The three-dimensional (3D) organization of chromatin, on the length scale of a few genes, is crucial in determining the functional state—accessibility and amount of gene expression—of the chromatin. Recent advances in chromosome conformation capture experiments provide partial information on the chromatin organization in a cell population, namely the contact count between any segment pairs, but not on the interaction strength that leads to these contact counts. However, given the contact matrix, determining the complete 3D organization of the whole chromatin polymer is an inverse problem. In this work, a novel inverse Brownian dynamics method based on a coarse-grained bead-spring chain model has been proposed to compute the optimal interaction strengths between different segments of chromatin such that the experimentally measured contact count probability constraints are satisfied. Applying this method to the α-globin gene locus in two different cell types, we predict the 3D organizations corresponding to active and repressed states of chromatin at the locus. We show that the average distance between any two segments of the region has a broad distribution and cannot be computed as a simple inverse relation based on the contact probability alone. The results presented for multiple normalization methods suggest that all measurable quantities may crucially depend on the nature of normalization. We argue that by experimentally measuring predicted quantities, one may infer the appropriate form of normalization.