Time to take stock

Bryony Pawinska provides an interim evaluation of the new pre-registration period six months into the pilot scheme.

As the new scheme develops apace - 581 trainees, each with their own supervisor, and 49 assessors - New Year is a good time to start to take stock of where we are. The College is working to a very tight workplan for the pilot, and we are achieving our targets. But those targets alone do not tell us how it is for the participants - the trainees who are keeping their logbooks and experiencing the assessment visits, their supervisors, and the assessors themselves.

Why the change?
It may be helpful to reflect on why the College has proposed the introduction of such a radical new approach to assessing the fitness to practise and clinical skill of pre-registration trainees. Under the provisions of Section 13 of the Opticians Act 1989, the GOC approves any scheme for the registration of UK-trained optometrists. But approval is not a one-off process - the GOC's education committee visits the PQE Part II regularly and comments to the College on its findings. The visitors shared the aim of the College that the pre-reg period should encompass a more structured phase of post-degree education and training aimed at establishing fitness for practice.

First and foremost the change provides ongoing support from the College for trainees. The issue of having to pass all 10 sections of the examination first time has perhaps been overemphasised - while it is true that the first-time pass rate has been consistently low at around 30 per cent, the majority of trainees do pass at the second attempt. However, for many trainees the first formal assessment of their progress was when they took the PQEII. It seemed self-evident that, if there was more structured support, and ongoing assessment throughout the pre-reg period, not only could the final assessment be reduced in length and still retain its rigour, but the trainee and the supervisor would be able to assess more formally how well the trainee was developing in terms of clinical skill.

The end result for all parties must always be the assurance that the trainee is clinically competent and safe to practise. While the PQEII did a good enough job of providing that assurance, a competence-based approach to assessment takes account of far more than a good grasp of the theory. It ensures that it is the application of that theory to practice, ie during the patient episode, which is primarily assessed, rather than the theory itself.


Involving the wider optical community
From the outset, the College has been determined to involve all stakeholders in the development of the new scheme. Indeed, change, and especially such profound change, is only effective when key stakeholders are involved. The interests of employers, universities, the hospital eye service, the regulator and practitioners have been represented on a Steering Group (now transmuted into an Advisory Group) comprised of the College, the AOP, FODO, BUCO (British Universities Committee for Optometry) and the GOC. The Advisory Group receives the College's workplan, updated regularly, and is consulted, sometimes collectively and sometimes individually, on issues that have included data protection and quality assurance.

Ensuring rigour in the assessment process
The pilot introduces a very different structure from traditional pedagogical approaches and is more in line with modern educational thinking. The GOC Core Competencies (which were developed in partnership with the optical bodies) are the starting point. The College has produced, under the direction of Professor John Lawrenson, chairman of its Academic Committee, an assessment framework which shows not just the competency to be evidenced but the performance criteria for each competence (the 'outcome'), the range of patient episodes that should be experienced by the trainee in order to achieve each competence, and the acceptable evidence of competence.


Evidence of competence
Evidence can be primary (eg direct observation of the patient episode) or secondary (eg the trainee's logbook, which is reviewed monthly by the supervisor). The key to the success of this approach is the assessment visit, undertaken by College-appointed and trained assessors, who are at 'arm's length' from the trainee and supervisor. The assessor, using the assessment framework, observes the trainee examining patients, may use role play and/or ask probing questions, checks the logbook and matches it to patient records, and looks at other evidence such as copies of referral letters. At the end of the visit the assessor provides feedback to the trainee and the supervisor on those competencies the assessor considers to have been adequately evidenced, and those where more development is required. Development needs are translated into an action plan, signed by the trainee and the supervisor.

Funding the new Scheme
The College's core costs are low, as befits a charity. The old pre-registration period had only administrative cost implications whereas the PQEII was very expensive to run. Under the new scheme, there are significant cost implications in providing an assessor for each trainee, but the final assessment, being shorter, will have a lower cost.

Employers informed us that the average cost to them of the old scheme, when taking into account the cost of retakes, was around £1,900 per trainee, and the College secured the support of all stakeholders for an increase to £2,200. This includes the provision of ongoing support for trainees throughout the pre-registration period by College assessors and by a dedicated staff team, and one free set of retakes of the PQEII (retained during the pilot year).

The College has absorbed the one-off cost of retaining the PQEII this year and our financial projections show that we can easily cover the costs of continuing investment in the development of the new scheme.

In a previous article in OPTICIAN Professor Barry Winn asserts that the PQEII has been run on a shoestring. The College, being a small organisation, can be very tightly managed, ensuring very firm controls over all expenditure. Nothing the College does is on a 'shoestring' but everything it does is subject to the most rigorous financial control. 

In 2003 Professor Winn presented his proposal, including a funding model, to the College. The College considered it carefully - from our viewpoint it would have made very little difference financially if we had been funded directly by the universities to administer the pre-registration period rather than being a direct provider. Our decision to continue with the development of our own scheme was based on a number of concerns about the consortium approach proposed by Professor Winn. But more importantly, the College believed the experience for the trainee was likely to be better under its own proposed scheme. The trainee would be employed and paid, looked after by their employer, well supported by the College, and not expected to train off the job.

Close scrutiny of the figures presented to the College by Professor Winn last year showed that university overheads would account for 43 per cent of the cost of running the scheme, ie £2,415 per student, plus a further £369 per student that the university would have to pay to an organisation (the College?) to administer the pre-registration period. The Winn proposal included residential weekends at £200 per student, adding up to a total cost per student of £2,984.

Professor Winn proposed to seek HEFCE funding for a course with the status of the fourth year of a Masters degree, ie £4,491 per student, and additionally to charge each student £3,000 to cover the direct costs as outlined above. The College understands that HEFCE were not supportive, which is not surprising given that HEFCE was proposing at the time to reduce funding for the three-year optometry degree courses by changing the banding. HEFCE funding, if secured, would have yielded a surplus for participating universities to spend on research and development. However, it would be difficult to ensure that this was devoted to optometry.

Our discussions with employers indicated that they would prefer to have just one provider to deal with, and it was unclear whether employers would still be eligible for a Department of Health grant if the trainee had student status. For those employers who would wish to pay the trainee/student for the significant time that would still need to be spent in the workplace, this would be an issue. We were also concerned about the level of student debt that would accrue, with no demonstrable improvement in the quality of the experience.


How is it working so far?
The College is monitoring the pilot's development very closely. We encourage assessors and supervisors to feed back to us informally, and this month we are introducing a formal mechanism through focus groups of assessors, trainees and supervisors. Until we have the formal feedback, comments received to date are, to some extent, anecdotal. However, the main area of comment, other than administrative details, is on the time taken to complete the quarterly assessments. The College's original anticipation that assessors would complete two assessments per day has proved to be right, but those assessments are taking longer than we had envisaged. To some extent this is a feature of any new scheme and, as the assessors gain more experience, I do not doubt this will improve. However, it is important that the assessment is not rushed and the College and the GOC may need to consider, in any case, whether it is unreasonable to expect the trainee to complete the first quarter competencies as defined by the GOC within the first three to four months.

Quality Assurance
The GOC Handbook for providers of schemes for registration of UK-trained optometrists lists the criteria for approval, which relate to assuring quality. The College has mapped its provision against these criteria and is confident that it will be able to satisfy all of them.

Key aspects of quality assurance include: