In recent years there has been increasing awareness of the limitations of conventional neuropsychological assessment in terms of predicting functional performance in everyday life. In response to the growing recognition of the need for ecological validity in rehabilitation settings, clinicians have begun to develop methods (e.g. rating scales and behaviour checklists) of assessing the impact of neuropsychological impairments on the everyday behaviour of the patient. However, there have been few attempts to investigate the validity or reliability of such scales. The aims of the present study were: (i) to design a rating scale of attentional behaviour, to be completed by therapists treating traumatically head-injured patients who were receiving a remedial intervention for their attentional deficits; (ii) to examine the scale’s correlation with neuropsychological measures of attention; (iii) to assess its internal structure and intra-rater reliability; and (iv) to examine the use of the scale by raters in different contexts and by individual raters over time. The scale developed showed some validity as a measure of attentional behaviour. It was quick and simple to administer. It showed modest, but statistically significant correlations with neuropsychological measures of attention, a high level of internal consistency, and excellent intra-rater reliability. The correlations between scores made by different raters in different contexts were much lower. Possible reasons for this finding are discussed, including the influence of context and the frame of reference of the rater. The importance of exploring and attempting to address potential sources of dissonance, both within and between raters, is emphasised.