P Drivers Project: Process Evaluation

BACKGROUND
OVERVIEW
The over-representation of young novice drivers continues to be one of road safety’s most intractable problems. In many countries, road crashes are one of the leading causes of death for those aged 17-25 years. Inexperience, poor judgment, underestimation of risk, and deliberate risk-taking behaviours have all been described in this group. Thus, young novice drivers remain a high-risk group, and require support to remain safe in their early years of solo driving, when crash risk is at its highest. Developing innovative and effective interventions (which can complement existing youth road safety countermeasures) is important in order to reduce crash risk.
The ‘P Drivers Project’ focuses on the development and delivery of a behavioural change program (‘the Program’) targeted at young drivers in their initial months of solo driving. The objectives of the Program are: to increase awareness among young novice drivers of the factors which contribute to their elevated crash risk; to improve safe driving behaviour among novice drivers; and ultimately, to reduce the number and severity of crashes involving young novice drivers.
A trial of the Program was implemented in Victoria and New South Wales, and this report focuses on the process evaluation of the trial, which aimed to evaluate processes related to Program delivery, and to identify barriers to potential future widespread implementation and delivery of the Program.
THE PROGRAM
A theoretical framework was used to guide the development of the Program curriculum, which was informed through a workshop with behaviour change experts and a review of relevant behaviour change literature. The focus was on target behaviours associated with the key crash types of young novice drivers (e.g., rear-end crashes, single-vehicle off-path crashes). The target behaviours of interest were: speed choice; hazard perception; car following; and gap selection. In addition, a number of key influences and barriers to behaviour change were considered in the Program. These included: peer influence; emotional control; distraction; other drivers and vehicles; and driving at high-risk times.
The Program comprised several components, which were completed in the following order:
1. Driver self-assessment survey (DSAS): An online survey about participant exposure to high-risk situations and behaviours, including immediate feedback.
2. Group session 1 (GS1): A 3-hour discussion session with around ten young drivers and a trained facilitator. Fostered links between behaviours and crash risk, and shared personal experiences.
3. On-road coaching session (ORC): Involved two to three participants taking it in turn to drive their own vehicle while being coached by a trained facilitator. When not driving their own vehicle, participants observed as passengers.
4. Group session 2 (GS2): A 3-hour discussion session with around ten young drivers and a trained facilitator. Involved reflection on previous components and the development of an individual action plan (Goal Plan).
5. Maintenance messages: To encourage maintenance of safe driving behaviours and to facilitate relapse prevention, some participants were sent regular messages via SMS and email over a 12-month period following completion of GS2.
A range of reimbursements were offered for participation in different components of the Program, including, for example, vouchers and prize draw entries.
THE PROGRAM TRIAL
The Program was evaluated using both process and outcome evaluations. The outcome evaluation was based only on participants in Victoria. The focus of process evaluation efforts were in NSW. A sub-set of Victorian participants who took part in the outcome evaluation also took part in the process evaluation. This enabled, in part, an assessment of whether the findings from NSW could be generalised. This report focuses only on the findings from the process evaluation.
METHODS
Process evaluations aim to assess the effectiveness of project delivery, and the current evaluation was structured using a range of widely accepted principles for robust process evaluation (context, reach, dose delivered, dose received, fidelity, recruitment and implementation – each is described in more detail in the results section).
A range of data collection methods were used, including:
• Quantitative: Metrics related to session information (location, time, number of participants); participant allocation, completions, drop-outs etc.; and quantitative survey responses.
• Qualitative: Interviews, focus groups and open-ended survey responses.
To ensure a balanced view, data were collected from both trial participants, and those involved with delivery of the Program (including project management staff, recruitment staff, and trainers/ facilitators). Data were also collected from those who were not interested in participating (where possible) to facilitate comparisons between those who chose to participate and those that did not.
The reference population for study was newly licensed P-plate drivers (P1) who resided in specified ‘hubs’ corresponding with the areas of NSW in which the Program was delivered. Hubs were chosen to represent a cross-section of novice drivers from different local environments and socio-economic backgrounds. In NSW, the hubs were located in Western Sydney, Dubbo, Tamworth and Lismore/Tweed heads. A total of 1,574 P1 drivers signed up to the Program in NSW with around 33% being from regional areas. Despite attempts to include indigenous and socio-economically diverse participants in the NSW trial, only around 3% were Indigenous Australians, 4% were classified as economically disadvantaged; and 3% were classified as having a low level of education.
Participants in the outcome evaluation in Victoria also took part in the process evaluation. In Victoria, 35,107 P1 drivers signed up to take part in the outcome evaluation, and were randomly allocated to either a treatment group (to complete the Program) or a control group (to complete surveys but not the Program). The full trial in Victoria was completed by 2,492 participants in the treatment group and 2,844 participants in the control group. Whether a given participant in the Victorian outcome evaluation also took part in the process evaluation was dependant on a number of factors including the data collection method, its purpose, and the timing of data collection.
RESULTS
Outcomes from the process evaluation have been summarised against the seven principles for process evaluations that formed the framework for the current evaluation: context, reach, dose delivered, dose received, fidelity, recruitment, and implementation.
CONTEXT
Context is concerned with aspects of the larger social, political and economic environment that may influence intervention implementation. The wider context of the Program included the reimbursement structure, and administrative elements.
• The reimbursements offered were generally well received by participants, who thought they were appropriate and fair
• While increasing the reimbursements increased initial participation, they may not have been as important for retention
• Participants were generally positive about the structure and contents of the Program
• A majority indicated that they would recommend the Program to their friends because of its value to driving safety, particularly for newly-licensed P1 drivers
REACH
Reach is concerned with the proportion of the intended target audience that participates in the intervention. This includes factors relating to participation, non-participation and drop-out.
• Reach represented a key challenge for the trial, including: Low recruitment rates from the eligible population; high rates of dropout; and lower than expected participation by disadvantaged groups (including Indigenous Australians and low socioeconomic status), and those living in regional areas
• The reimbursements offered were a major reason for participation, although an interest in improving driving safety was also prominent
• Key barriers to participation (and reasons for drop-out) included: not having the time available, the time demands of the Program, difficulty scheduling mutually convenient session times, and a lack of interest
DOSE DELIVERED
Dose delivered is concerned with the number or amount of intervention units delivered, and is usually a function of the efforts of the intervention providers.
• The overall average attendance rate was lowest for GS1 at around 60%, but reached nearly 90% for the ORC and GS2
• Average attendance rates following the introduction of additional reimbursements were somewhat higher than for the same period the previous year
• Between-stage attrition was highest between the participants consenting to participate in the trial and the scheduled time of first program task (DSAS), while early-stage attrition was higher in the NSW regional areas (compared with Sydney)
• Later stage attrition showed a less consistent pattern across study hubs
DOSE RECEIVED
Dose received is concerned with the extent to which participants recall and comprehend the delivered components, and feel that it has influenced them. This essentially measures engagement: the extent to which participants actively engage with, interact with, or use, materials provided.
• Dose received was generally positive
• There was a high level of engagement with the Program, and participants reported receiving the information well (including putting effort into changing behaviour as a result)
• A large proportion of participants believed they would use what they had learned from the Program to improve their driving, and many participants also reported they believed their driving had become safer as a result of participation
• Sydney participants and females reported being more influenced by the Program than their regional and male counterparts, respectively
• There was less evidence for a role of other personal characteristics
FIDELITY
Fidelity is concerned with the extent to which the intervention was delivered as planned, and represents the quality and integrity of the intervention as conceived by the intervention developers.
• Participants commended the Coaches and felt the delivery was of a high standard
• Participants responded favourably to Facilitators sharing their personal experiences
• Participants also liked that the Facilitators encouraged participant feedback
• A small number of interviewees in NSW suggested that the Facilitator seemed unfamiliar with the program
• Some Facilitators felt that they would have benefited from performance feedback and monitoring
• Recruitment staff felt that they required more information on the broader Program and its delivery to better perform in their role
RECRUITMENT
Recruitment is concerned with the procedures used to approach and attract participants. Recruitment often occurs at the individual and community/organisational level.
• In NSW, the most effective means of recruitment was the letter sent upon receipt of a P1 licence, whereas in Victoria, phone calls appeared to be the most common introduction to the Program
• Data from NSW and Victoria were relatively consistent, with key differences involving participant age (older in Victoria) and location-specific barriers to participation
IMPLEMENTATION
Implementation is concerned with the extent to which the intervention has been implemented and received by the intended audience.
• Overall, the Program was generally delivered as designed
• The main deviations from the planned delivery included: lengthy delays between stages; only one participant in the car for the ORC Session in NSW; and participants progressing through the program without completing pre-requisite stages
SUMMARY OF KEY FINDINGS
The overarching aims of the P Drivers Project process evaluation were to (1) evaluate the delivery processes of the Program (with a view to refining these processes as required), and (2) to gain a better understanding of relevant issues for possible future Program implementation and delivery (including assessing the willingness of young drivers to participate in this type of program). Key findings included:
Delivery processes were generally positively received
• The format and content were generally well received
• The adult education approach was appreciated, the small group discussion format was found to be engaging, and participants generally valued the opportunity for feedback on their driving.
• The Program appeared to be conveying its intended messages to increase awareness of hazards, risky driving, and consequences of crashing
• Participants reported adopting the safe driving strategies that they learned
• Feedback on participants’ own driving was highly valued, particularly during times (and in locations) that provided a sufficiently challenging driving environment
• The DSAS was seen as less valuable by participants (although this may relate to Facilitators failing to adequately review DSAS material in Group Session 1 activities)
• Having one rather than two participants in the ORC session was a problem in NSW - process changes implemented in Victoria part-way through Program delivery appeared to generally resolve this issue for Victorian participants
• Those participants that did have another participant in the ORC session tended to appreciate the opportunity to view the driving of the other participant less than getting feedback on their own driving
Reach represented a key challenge for the Project
• The willingness of young drivers to participate in such a program represented a key challenge, and would need to be addressed in any future implementation
• The target population reported being busy with many competing interests and demands (such as school, part-time work, study)
• Recruitment difficulties were exacerbated by participants finding the scheduling times and locations inconvenient
• Some P1 drivers did not have access to a car suitable for participation
• Significant reimbursement was required to achieve a relatively low recruitment rate
• Strategies to enhance participation might focus on key barriers to participation identified in this evaluation
Differences between population sub-groups were an important feature
• Those living in regional areas were less likely to participate, and more likely to drop-out, than their city counterparts
• Reimbursements were less effective in overcoming regional participants’ barriers to participation, which related more to inconvenient scheduling times and locations
• Potentially disadvantaged participants were under-represented in the study, although they appeared to be no more likely to drop-out once recruited
• The limited evidence available suggests that a primary deterrent was the group session format, while a primary reason for participation was the opportunity to learn about safe driving
• Targeted strategies to address barriers for population sub-groups are particularly important for enhancing participation
Willingness to pay for the Program
• Few people would be willing to pay more than $100 to participate
• Participants reported that it was likely that parents would meet any costs of participation (which is a further potential barrier for disadvantaged P1 drivers), which suggests that parents should be a focus of recruitment efforts
POTENTIAL REFINEMENTS TO THE PROGRAM
The evaluation identified a number of potential refinements to the Program. These include:
• Promotion: Highlight the value of the Program in promotional materials; recruit young drivers using electronic methods (particularly social media); target parents in order to encourage participation; improve materials first introducing the program to novice drivers and parents to make the Program more attractive.
• Administrative aspects: Make the website mobile responsive; provide maps and access instructions for locations; offer more sessions in more locations; provide more support and ongoing professional development for Facilitators, and ensure that facilitators are regularly given feedback/debriefings and kept up-to-date with any changes in the program.
• Program Content and Delivery:
o DSAS: Elicit information using more qualitative strategies; reduce repetition in the questions; ensure DSAS is completed before progression; ensure Facilitators fully understand the role and benefits of the DSAS.
o GS1: Incorporate more group activities, address session absenteeism in order to maximise effectiveness; replace the icebreaker activity ‘what it means to have a P1 licence’ with an alternative icebreaker; increase the pace of the session; shorten the session.
o ORC Session: Provide take-home sheet with personalised summary of session lessons; shorten the session; choose times/locations to maximise challenging driving environments so that learning outcomes are met.
o GS2: Include ice-breaker activities; refine the session by removing repetition where appropriate.
o Face-to-face sessions: Use strategies to ensure sufficient group sizes.
o Reminder messages: Explain the relevance of the Goal Plans and reminders, as well as refine their content and delivery.
o General content: Reduce repetition where appropriate; provide specific behavioural messages and strategies ‘to prevent returning to old, dangerous habits’.
CONCLUSIONS
Overall, the P Drivers Project was generally delivered as intended. The most successful elements related to the format and content of the Program. Participant engagement was high, with many participants reporting that they changed their behaviour as a result of what they learned. Reach represented a key challenge for the trial, largely due to the age group being targeted in combination with the time-requirements of the Program. The reimbursements offered were a major reason for participation, although an interest in improving driving safety was also prominent.