Objective: Recognizing the prodrome of a first psychotic episode prospectively creates the opportunity of intervention, which could delay, ameliorate or even prevent onset. Valid criteria and a reliable methodology for identifying possible prodromes are needed. This paper describes an instrument, the Comprehensive Assessment of At-Risk Mental States (CAARMS), which has been designed for such a purpose. It has two functions: (i) to assess psychopathology thought to indicate imminent development of a first-episode psychotic disorder; and (ii) to determine if an individual meets criteria for being at ultra high risk (UHR) for onset of first psychotic disorder. This paper describes the pilot evaluation of the CAARMS. Method: Several methodologies were used to test the CAARMS. First, CAARMS scores in a group of UHR young people and the association between CAARMS scores and the risk of transition to psychotic disorder, were analysed. Second, CAARMS scores in a UHR group were compared to a control group. To assess concurrent validity, CAARMS-defined UHR criteria were compared to the existing criteria for identifying the UHR cohort. To assess predictive validity, the CAARMS-defined UHR criteria were applied to a sample of 150 non-psychotic help-seekers and rates of onset of psychotic disorder at 6-month follow-up determined for the CAARMS-positive (i.e. met UHR criteria) group and the CAARMS-negative (i.e. did not meet UHR criteria) group. The inter-rater reliability of the CAARMS was assessed by using pairs of raters. Results: High CAARMS score in the UHR group was significantly associated with onset of psychotic disorder. The control group had significantly lower CAARMS scores than the UHR group. The UHR criteria assessed by the CAARMS identified a similar group to the criteria measured by existing methodology. In the sample of non-psychotic help-seekers those who were CAARMS-positive were at significantly increased risk of onset of psychotic disorder compared to those who were CAARMS-negative (relative risk of 12.44 (95% CI = 1.5-103.41, p = 0.0025)). The CAARMS had good to excellent reliability. Conclusions: In these preliminary investigations, the CAARMS displayed good to excellent concurrent, discriminant and predictive validity and excellent inter-rater reliability. The CAARMS instrument provides a useful platform for monitoring subthreshold psychotic symptoms for worsening into full-threshold psychotic disorder.