Vitamin D is critical to children’s skeletal development and health. Despite this, the factors which determine vitamin D concentrations during infancy remain incompletely understood. This article reviews the literature assessing the factors which can affect vitamin D status in infancy, including antenatal and postnatal vitamin D supplementation. Observational data supports that dietary intake of vitamin D, UV exposure, and geographic factors contribute significantly to infants’ vitamin D status, but the relationship is unclear regarding genetic variation, ethnicity, and maternal vitamin D status. Randomised controlled trials have compared higher versus lower doses of infant vitamin D supplementation, but no studies have compared infant vitamin D supplementation to placebo and eliminated external sources of vitamin D to fully quantify its effect on vitamin D status. Knowledge gaps remain regarding the factors associated with optimal vitamin D concentrations in infants—including key factors such as ethnicity and genetic variation—and further studies are needed.