Texture boundary segmentation is conventionally thought to be mediated by global differences in Fourier energy, i.e., loworder texture statistics. Here, we have examined the importance of higher order statistical structure of textures in a simple second-order segmentation task. We measured modulation depth thresholds for contrast boundaries imposed on texture samples extracted from natural scene photographs, using forced-choice judgments of boundary orientation (left vs. right oblique). We compared segmentation thresholds for contrast boundaries whose constituent textures were either intact or phase scrambled. In the intact condition, all the texture statistics were preserved, while in the phase-scrambled condition the higher order statistics of the same texture were randomized, but the lower order statistics were unchanged. We found that (1) contrast boundary segmentation is impaired by the presence of higher order statistics; (2) every texture shows impairment but some substantially more than others; and (3) our findings are not related to scrambling-induced changes in detectability. The magnitude of phase-scrambling effect for individual textures was uncorrelated with variations in their amplitude spectra, but instead we suggest that it might be related to differences in local edge structure or sparseness.