RGCL Anniversary Highlights, Day 9

Published on Aug, 18 2022 by RGCL.

Our long-standing collaboration with the US National Board of Medical Examiners (NBME)

It all began with a paper entitled “Computer-aided generation of multiple-choice tests” at the very first BEA (Building educational applications using Natural Language Processing) workshop in Edmonton, Canada, 2003 (19 years ago; this year, the workshop is in its 17th edition).

The paper describes a novel computer-aided procedure for generating multiple-choice tests from electronic instructional documents. The then-Director of the Center for Innovation at the National Board of Medical Examiners (NBME) read the paper, and arranged a meeting with RGCL team in Wolverhampton. They asked us to run the system on a medical text. They then got the results, and analysed them on their way back to Philadelphia. The rest, as they say, is history.

A fruitful collaboration was established. Various problems such as translation from American to British English, information extraction from test items, predicting response times and items difficulties, patient note scoring, and short answer scoring have been studied together, between RGCL and NBME. The collaboration officially started back in 2005 (17 years ago) and is still ongoing. An impact case study based on this collaboration was detailed and returned to the REF2021 exercise.

RGCL members Le An Ha and Victoria Yaneva at a conference organised by NBME. Victoria Yaneva is now a Senior Data Scientist for NBME

Through this collaboration, a few quotations of urban-legend status – such as an RGCL staff member being able to “walk on water”, or “Wolverhampton is well-known for two things: the industrial revolution, and its Research Group in Computational Linguistics” – have been born.

RGCL is indebted to Le An Ha for his continuous excellent work on the NBME project, and to Victoria Yaneva, whose work for NBME was evidently so impressive as to land her a job there as Senior Data Scientist!