Sleep loss leads to both time-on-task slowing of responsiveness and increased frequency of transient response errors. The consequences of such errors during real-world visuomotor tasks, such as driving, are serious and life threatening. To investigate the neuronal underpinning of time-on-task and transient errors during a visuomotor tracking task following sleep restriction, we performed fMRI on 20 healthy individuals when well-rested and when sleep-restricted while they performed a 2-D pursuit-tracking task. Sleep restriction to 4-h time-in-bed was associated with significant time-on-task decline in tracking performance and an increased number of transient tracking errors. Sleep restriction was associated with time-on-task decreases in BOLD activity in task-related areas, including the lateral occipital cortex, intraparietal cortex, and primary motor cortex. In contrast, thalamic, anterior cingulate, and medial frontal cortex areas showed overall increases irrespective of time-on-task after sleep-restriction. Furthermore, transient errors after sleep-restriction were associated with distinct transient BOLD activations in areas not involved in tracking task per se, in the right superior parietal cortex, bilateral temporal cortex, and thalamus. These results highlight the distinct cerebral underpinnings of sustained and transient modulations in alertness during increased homeostatic drive to sleep. Ability to detect neuronal changes associated with both sustained and transient changes in performance in a single task allowed us to disentangle neuronal mechanisms underlying two important aspects of sustained task performance following sleep loss.