Quantitative phase maps from Au and polystyrene nanospheres were obtained using the transport of intensity equation approach for phase retrieval in the electron microscope. These were compared to surface plasmon maps, as generated by electron energy loss spectroscopy. Large phase excursions were observed in the vacuum regions surrounding the Au nanospheres, yet not near adjacent polystyrene nanospheres. It is proposed that these phase excursions are associated with surface plasmon excitation. These results suggest that phase retrieval offers an alternative means for sensing and interpreting the optical response of nanostructured metals.