What does it mean to make the grade?

Higher education institutions should be wary of relying too heavily on A-level examination grades when assessing applicants for places, because the system of translating exam marks into grades leads to results being less accurate.

In a study of one exam board's English A-level, Professor Gordon Rae of University of Ulster discovered that nearly two in every five candidates had been misclassified, getting either a better or a poorer grade than their marks deserved.

Professor Rae says that the number of grades employed at GCSE, the threshold mark that separates them, and the scores allocated to them, all influence accuracy. "Converting marks into grades is tantamount to replacing a relatively fine categorical scale with a coarser one," he says. "It results in a loss of accuracy because all the marks within a particular category are being treated as indistinguishable."

He points out that all exams are subject to some degree of error. A candidate may have been particularly fortunate in spotting which questions would come up, and do better than expected. A pneumatic drill outside the exam hall may reduce the performance of some candidates; others may arrive upset over an incident in their personal lives, or slightly unwell, and can under-perform. Such candidates can then get a classification that is below their potential - though, in the case of public exams, it will rarely be by more than one grade.

Added to this is the problem that there will be slight variations in the scores given by different markers.

"The majority of candidates with observed grade A have true grades of A or B; the great majority of candidates with observed grade B have true grades of A or B or C," he says.

In an investigation of one examination board's 1992 GCE A-level English literature exam, Professor Rae compared candidates' observed grades with their true grades - in other words, the grades they would have been awarded had the exam been perfectly reliable, with reasonable assumptions made about the accuracy and distribution of overall marks.

This showed that while 5 per cent of all candidates both deserved and attained a grade B, another 2.1 per cent deserved a B but received an A and 3.4 per cent who deserved a B got a C or D. Thus 61 per cent were seen to be correctly, and 39 per cent wrongly, graded.

"About 39 per cent of all candidates were wrongly classified because errors of measurement caused them to cross a boundary into a category they should not be occupying according to their true scores," Professor Rae says.

He believes the A-level system to be a well audited exam, and in the mathematical model he produced to establish the grade misclassification rate he made a generous assumption about the amount of error in the examination. "If university selectors are perturbed by these figures, they should realise the picture becomes rapidly worse if one were to assume the exam process to be even less reliable."

When, he points out, the seven points in the current exam scale are reduced to three, combining A with B, C with D and E, and N with U, the percentage of candidates correctly graded rises from 61 per cent to 86.4 per cent. "We thus have a situation in which reducing the number of grades decreases accuracy but increases the proportion of candidates correctly graded," Professor Rae explains. Fewer grades make for fewer, but more severe, misclassifications.

But, he stresses, the number of categories and the proportion of candidates allocated to these is not a simple matter, because they depend on wider issues - such as the need to motivate children, or to achieve comparability of standards from year to year.

The findings relating to the exam board surveyed could similarly apply to all other UK exam boards, though the exact percentages of misclassifications may vary from board to board and subject to subject, he adds. "All reputable exam boards try to achieve maximum reliability in the marks awarded. But in a real world exams cannot be made perfect, so that errors in grading will always occur. All that can be done is to make every effort to ensure the setting and marking of papers is as thorough as possible."