This article is best viewed in a PDF Format.
|
|
|
|
Being confident that those entering the optometrists’ register are fit to practise unsupervised in a fast changing healthcare landscape is an aim that the regulator, the public and the profession all share. The question is how best to achieve it.
On behalf of the General Optical Council (GOC), the College of Optometrists determines whether pre-registration trainees have the necessary knowledge and skills to be eligible for registration. Until 2005, this was accomplished through a multi-part examination that had been in place since the beginning of the 1980s.
The education of healthcare professionals has been evolving over time, moving away from discipline-based teaching, through integrated curricula and problem-based learning to outcome or competency-based learning. Competency-based learning helps trainees to develop in combination the knowledge, skills and attitudes they need for effective performance and reflects the complexity of the tasks they will have to undertake when they qualify. In the early part of this decade, the GOC moved optometry in this direction by introducing a series of core competencies to be attained at undergraduate level (Stage 1) and a further series for the end of pre-registration training (Stage 2).
To reflect this style of learning, the College introduced the new Scheme for Registration in 2005 which fundamentally altered the structure of pre-registration training. Central to the new Scheme was an holistic approach to assessment integrating work-based assessment of competency using College-appointed assessors and a shortened final assessment under examination conditions. This radical new approach gave trainees a framework to support their learning experience by building their expertise through authentic tasks which grew in complexity and it gave them a clear indication of how they were progressing through feedback and action planning.
Supervisors and trainees have welcomed the introduction of work-based assessment. Both groups appreciate the structured format and have found the action plans that follow each assessment enormously helpful. And for the first time there was a focus on the quality of trainees’ experience through requiring them to see a broad range of specified clinical presentations rather than simply stipulating a minimum number of patients.
Reviewing the Scheme
Changing the way trainees were assessed was always going to be an iterative process and the College strives to respond quickly to feedback on the Scheme. We also want to ensure that our assessment methods comply with current thinking on good assessment practice.
In reviewing the Scheme the College has paid particular attention to two issues: assessors and examiners have indicated that trainees are not always keeping their knowledge and skills up to date once they have been assessed in particular areas and the structure of the Final Assessment does not lend itself to wide sampling.
Good assessment practice
There is a considerable body of research into assessing clinical competence. Choosing the best assessment methods is the first challenge and criteria have been described in the literature that can be used to help with this process, the most significant of which are:
-
Reliability (the candidates’ results would be reproducible if the same assessment was taken on different occasions)
-
Validity (the content of the assessments is drawn from across the competencies and the assessment discriminates appropriately between candidates of differing levels of ability and experience)
-
Educational impact (it is known that assessment drives learning and the assessment process must influence trainees to be proficient in the knowledge and skills needed for registration)
-
Cost-effectiveness (the assessments are funded by Scheme for Registration fees and these resources must be used as effectively as possible)
-
Acceptability (the assessments must be conducted properly and the methods must, therefore, be acceptable to those involved).
There is a balance to be achieved between these criteria, and our aim is to design an integrated assessment Scheme that both takes the criteria into account and reflects the way optometric practice is organised.
In 1990 GE Miller used a pyramid structure to show which type of assessment methods were appropriate for assessing competency.
Through this triangle, Miller suggests that written papers and vivas are the most appropriate methods for assessing knowledge (the recall of facts) and the application of knowledge in a clinical context. The GOC’s 82 Stage 2 competencies are a step higher, however, as they are primarily task-based. It goes without saying that trainees can only perform tasks successfully if they have the underpinning knowledge, but for pre-registration training, which is competence-based, the assessment process lies mainly in the two top tiers of the triangle where trainees show the assessor how they perform either in a simulated setting or in the authentic setting of the workplace.
Moving forward
The Scheme for Registration is an integrated system of assessment that allows trainees to develop their skills in a structured way. The framework it provides has proved an excellent tool for trainees and we wanted to build on this when deciding which assessment methods would ensure that the Scheme was valid, reliable, acceptable, cost-effective and had a positive impact on trainees’ learning.
Work-based assessment
The workplace is the obvious setting for assessing trainees’ competence. It is the natural place to observe trainees working with actual patients and most closely mirrors their day-to-day work. In addition, it allows assessors to give trainees constructive feedback and a plan for improving areas of weakness with the support of their supervisors.
We have begun to refine the current work-based assessment process and the framework within which the competencies are assessed, to improve the experience for trainees and make the system more robust.
Methods of assessment
Assessors have a number of different options for assessing trainees in the workplace. As well as direct observation, and the use of logbooks and patient records, assessors can use hypothetical case scenarios and questioning. This has allowed assessors to use a variety of methods to assess the same competency and has sometimes meant that the method chosen has not permitted trainees to demonstrate competence based on their own experience.
To ensure the work-based assessment is patient-centred, key skills must now be directly observed, and for other competencies structured case-based discussions centred on patient records are used. Hypothetical case scenarios are used only where the trainee has been unable to find the appropriate direct experience.
Specifying which assessment methods to use has made the process fairer to trainees as it means a more consistent approach. And it is a better measure of their competence, focusing on how they manage actual patients.
Impact on learning
It was of concern that trainees sometimes allowed their knowledge and skills to deteriorate once they had successfully completed the relevant competency. The structure of the assessment process is key to preventing this.
We have, therefore, restructured the assessment framework so that the order in which competencies are assessed reflects the order in which most trainees gain experience. And we have moved the broader-based competencies to the end of the work-based assessment to allow assessors to check maintenance of competence across the range of knowledge and skills required.
Building a robust system
For trainees beginning the Scheme from June 2009 the work-based assessment will become a two-stage process. Stage 1 will involve the assessment of all the competencies other than the broad-based competencies and Stage 2 will be a final work-based assessment visit.
During this final visit, the broader-based competencies will be assessed and we will move the assessment of Routine Examination and Contact Lens Fitting and Aftercare out of the Final Assessment into this final work-based assessment visit.
This move will mean that two key processes will be assessed in an authentic setting where trainees are familiar with the equipment and surroundings.
We will, however, use patients supplied by the College to ensure that patients from a central resource do not know the patients or their details in advance, and that the patient prescriptions are within a defined standardisation range.
A new assessor will undertake this final visit to bring a second professional judgement to the work-based assessment process for each trainee.
This pattern of assessment will help trainees to build up their knowledge and skills so that these are up-to-date when they reach the end of the pre-registration period.
Final Assessment
The need for a Final Assessment stems from two main points. The results of the Scheme for Registration must be robust as they determine whether or not a trainee is eligible for registration. This means that the GOC, employers and the public must be sure that a pass means that an optometrist is capable of practising unsupervised. The second point is that the structure of optometric practice means that, although they are assessed across a wide sample of tasks and patient conditions, it is impractical for trainees to be tested by multiple assessors in the workplace. This would be the ideal way of increasing reliability. We strive to improve consistency between assessors through training and through quality assurance methods such as observations and trainee interviews, and we are now more prescriptive about the assessment methods they use. Assessor variability cannot, however, be completely eliminated.
The purpose of the Final Assessment is, therefore, both to enhance the reliability of the Scheme for Registration and to act as a final check of key skills.
We have decided to use an assessment method which will concentrate on sampling across the competencies and across examiner judgments to increase the reliability of the assessment system. Miller’s triangle, described above, suggests another way of assessing competence. The Objective Structured Clinical Examination (OSCE) is a well-established method of examining clinical and communication skills and there is a considerable body of literature pointing to its success. Invented in Dundee in 1979 it is now used extensively in medicine, veterinary medicine, dentistry and other health and social care professions, both in the UK and worldwide.
The OSCE consists of a number of short clinical tasks, known as stations, which the candidates must undertake in sequence. In each station candidates interact with a standardised patient – an actor who has been trained to simulate the actions and reactions of a real patient – although on occasion the task can involve an anatomical model, video simulation or clinical image where this is more appropriate. The aim is to assess candidates in scenarios which simulate reality as closely as possible but from which the distractions from real life are removed. The examiner observes and does not intervene except in very limited circumstances. These key characteristics of the OSCE increase consistency – another factor in constructing a reliable examination.
The examination is considered ‘objective’ because it combines the judgements of a significant number of examiners. Professional judgement is by definition subjective, but combining the judgements reduces the impact of ‘hawks’ and ‘doves’ on any one candidate. The examination is ‘structured’ because the candidates all undertake the same broad range of tasks and it is ‘clinical’ because the tasks revolve around different patient-related scenarios.
We propose to introduce an OSCE that will assess candidates in the following common skills across a range of patient conditions:
-
History taking (including diagnosis)
-
Communication with patients and colleagues
-
Data interpretation
-
Clinical examination and practical skills.
Each OSCE station will focus on one particular area of competence, for example the ability to take a history from a patient with a headache, or the ability to undertake indirect ophthalmoscopy. The OSCE is a snapshot rather than an assessment of the whole consultation process, although stations do allow for some integration of skills. It is, however, the snapshot which allows for the wide sampling of content and examiner judgements that increases the reliability of the assessment process.
Summary
For trainees enrolling on the Scheme from June 2009, the Scheme for Registration will be an integrated assessment system in three stages.
Stage 1
Work-based assessment by an allocated assessor using direct observation and structured case-based discussions to allow detailed exploration of trainees’ competence in managing actual patients.
Stage 2
The assessment by a new assessor of the routine examination and contact lens fitting and aftercare processes in familiar surroundings on a patient supplied from a central resources by the College, and the assessment of the broader-based competencies to sample a variety of tasks to ensure that trainees are competent across the board.
Stage 3
An OSCE to allow wide sampling in terms of skills, cases and examiner judgements to enhance reliability.
This will ensure that the Scheme for Registration is fair to candidates, and will allow the GOC, employers and
the public to be assured that someone who has successfully completed the Scheme is capable of independent practice. l
-
Professor John Lawrenson is chairman of the Education Committee, and Josephine Mullin, director of education, at the College of Optometrists