Studies applying structural neuroimaging to pedophiles are scarce and have shown conflicting results. Although first findings suggested reduced volume of the amygdala, pronounced gray matter decreases in frontal regions were observed in another group of pedophilic offenders. When compared to non-sexual offenders instead of community controls, pedophiles revealed deficiencies in white matter only. The present study sought to test the hypotheses of structurally compromised prefrontal and limbic networks and whether structural brain abnormalities are related to phenotypic characteristics in pedophiles. We compared gray matter volume of male pedophilic offenders and non-sexual offenders from high-security forensic hospitals using voxel-based morphometry in cross-sectional and correlational whole-brain analyses. The significance threshold was set to p < .05, corrected for multiple comparisons. Compared to controls, pedophiles exhibited a volume reduction of the right amygdala (small volume corrected). Within the pedophilic group, pedosexual interest and sexual recidivism were correlated with gray matter decrease in the left dorsolateral prefrontal cortex (r = -.64) and insular cortex (r = -.45). Lower age of victims was strongly associated with gray matter reductions in the orbitofrontal cortex (r = .98) and angular gyri bilaterally (r = .70 and r = .93). Our findings of specifically impaired neural networks being related to certain phenotypic characteristics might account for the heterogeneous results in previous neuroimaging studies of pedophilia. The neuroanatomical abnormalities in pedophilia seem to be of a dimensional rather than a categorical nature, supporting the notion of a multifaceted disorder.