The elderly population is more susceptible to infections as a result of an altered immune response, commonly referred to as immunosenescence. Cytomegalovirus (CMV)-infection associated changes in blood lymphocytes are known to impact this process, but the interaction with gender remains unclear. Therefore, we analysed the effects and interaction of gender and CMV on the absolute numbers of a comprehensive set of naive and memory T- and B-cell subsets in people between 50 and 65 years of age. Enumeration and characterisation of lymphocyte subsets by flow cytometry was performed on fresh whole blood samples from 255 middle-aged persons. CMV-IgG serostatus was determined by ELISA. Gender was a major factor affecting immune cell numbers. CMV infection was mainly associated with an expansion of late-differentiated T-cell subsets. CMV+ males carried lower numbers of total CD4+, CD4+ central memory (CM) and follicular helper T-cells than females and CMV- males. Moreover, CMV+ males had significantly lower numbers of regulatory T (Treg)-cells and memory B-cells than CMV+ females. We here demonstrate an interaction between the effects of CMV infection and gender on T- and B-cells in middle-aged individuals. These differential effects on adaptive immunity between males and females may have implications for vaccination strategies at middle-age.