This work shows the impact of microstructure and defect on the fatigue life of an AlSi10Mg manufactured by Additive Layer Manufacturing (ALM). Samples were manufactured via a laser powder-bed process: two configurations (0° and 90°) are considered in order to evaluate the impact of the building direction on fatigue properties. 3D X-Ray tomography was used to characterize the defect population. The microstructure was characterized by considering four parameters: melt-pools, crystallographic grains, dendritic structure and precipitates. The fatigue properties were determined by establishing S-N curves for machined samples, with and without T6 heat-treatment, at R = −1 under tensile loading. The size of the defect responsible for the fatigue failure was determined in each sample so as to establish a relationship between the fatigue limit and the defect size using Kitagawa-type diagrams. In order to study a broader range of defect size, artificial defects were introduced using electro-discharge machining. The following observations are made: (i) after heat-treatment, the boundaries of melt-pools and the dendritic structure are not visible. Si is organized into pure precipitates homogeneously distributed over space and intermetallic Fe based compounds are observed in the form of needles; (ii) The impact of building direction on fatigue life is seen only after T6 heat treatment; (iii) An improvement of the fatigue resistance is observed after T6, in spite of the presence of intermetallic needles; (iv) The fatigue limit is controlled by the defect size both before and after T6 heat treatment, and it seems that the influence of T6 decreases as the defect size increases.