Background: Exposure to high levels of pollen in infancy is a risk factor for allergic respiratory diseases in later childhood, but effects on lung function are not fully understood. We aim to examine associations between grass pollen exposure in the first months of life and lung function at 12 and 18 years, and explore potential modification. Methods: Using the Melbourne Atopy Cohort Study, a birth cohort of children with a family history of allergic diseases, we modeled the association between cumulative grass pollen exposure up to 3 months after birth, on FEV1, FVC, and FEV1/FVC ratio at 12 and 18 years. We also assessed modifying effects of residential greenness levels (derived from satellite imagery), asthma, and early life sensitization to ryegrass. Results: Grass pollen exposure in the first 7 days was associated with a reduction in FEV1(−15.5 mL; 95% CI: −27.6, −3.3 per doubling of pollen count) and FVC (−20.8 mL; −35.4, −6.1) at 12 years, but not at 18 years. Increase in cumulative grass pollen exposure up to 3 months was negatively associated with FVC at 12 and 18. Exposure to high residential greenness modified the association at 18 years. Conclusion: Early exposure to grass pollen was associated with decreased lung function in children and adolescents. Targeted interventions for pollen avoidance strategies that take into account local topography could be implemented alongside other clinical interventions such as immunotherapy.