The contribution of the lateral prefrontal cortex (LPFC) to working memory is the topic of active debate. On the one hand, it has been argued that the persistent delay activity in LPFC recorded during some working memory tasks is a reflection of sensory storage, the notion supported by some lesion studies. On the other hand, there is emerging evidence that the LPFC plays a key role in the maintenance of sensory information not by storing relevant visual signals but by allocating visual attention to such stimuli. In this study, we addressed this question by examining the effects of unilateral LPFC lesions during a working memory task requiring monkeys to compare directions of two moving stimuli, separated by a delay. The lesions resulted in impaired thresholds for contralesional stimuli at longer delays, and these deficits were most dramatic when the task required rapid reallocation of spatial attention. In addition, these effects were equally pronounced when the remembered stimuli were at threshold or moved coherently. The contralesional nature of the deficits points to the importance of the interactions between the LPFC and the motion processing neurons residing in extrastriate area MT. Delay-specificity of the deficit supports LPFC involvement in the maintenance stage of the comparison task. However, because this deficit was independent of stimulus features giving rise to the remembered direction and was most pronounced during rapid shifts of attention, its role is more likely to be attending and accessing the preserved motion signals rather than their storage.