Subjective tinnitus is the perception of sound in the absence of an external source. Tinnitus is often accompanied by hearing loss but not everyone with hearing loss experiences tinnitus. We examined neuroanatomical alterations associated with hearing loss and tinnitus in three groups of subjects: those with hearing loss with tinnitus, those with hearing loss without tinnitus and normal hearing controls without tinnitus. To examine changes in gray matter we used structural MRI scans and voxel-based morphometry (VBM) and to identify changes in white matter tract orientation we used diffusion tensor imaging (DTI). A major finding of our study was that there were both gray and white matter changes in the vicinity of the auditory cortex for subjects with hearing loss alone relative to those with tinnitus and those with normal hearing. We did not find significant changes in gray or white matter in subjects with tinnitus and hearing loss compared to normal hearing controls. VBM analysis revealed that individuals with hearing loss without tinnitus had gray matter decreases in anterior cingulate and superior and medial frontal gyri relative to those with hearing loss and tinnitus. Region-of-interest analysis revealed additional decreases in superior temporal gyrus for the hearing loss group compared to the tinnitus group. Investigating effects of hearing loss alone, we found gray matter decreases in superior and medial frontal gyri in participants with hearing loss compared to normal hearing controls. DTI analysis showed decreases in fractional anisotropy values in the right superior and inferior longitudinal fasciculi, corticospinal tract, inferior fronto-occipital tract, superior occipital fasciculus, and anterior thalamic radiation for the hearing loss group relative to normal hearing controls. In attempting to dissociate the effect of tinnitus from hearing loss, we observed that hearing loss rather than tinnitus had the greatest influence on gray and white matter alterations.