RMIT student uses machine learning to decode doctors’ handwriting

Tiếng Việt

RMIT student uses machine learning to decode doctors’ handwriting

An RMIT University student has helped solve a big challenge standing in the way of digitising massive Vietnamese medical records: deciphering doctors’ notoriously difficult-to-read handwriting.

Working closely with the Hospital of Tropical Disease (HTD) and Oxford University Clinical Research Unit (OUCRU), Bachelor of Software Engineering student Phung Minh Tuan successfully developed an end-to-end pipeline to recognise text on scanned Vietnamese medical records, potentially accelerating the digitisation of medical health records that the Government initiated in 2019.

news-1-rmit-student-uses-machine-learning-to-decode-doctors-handwriting

Bachelor of Software Engineering student Phung Minh Tuan successfully developed an end-to-end pipeline to perform text recognition on scanned Vietnamese medical records.

Overcoming the complexities of Vietnamese handwriting

“There have been great advances in handwritten text recognition, however, most existing methods have been developed for English language and there is little-to-no methods dedicated for Vietnamese language,” shared Tuan, now in his final year with the School of Science & Technology at RMIT.

“Vietnamese handwritten text recognition is fundamentally more challenging than that of English due to the presence of more character classes, complex vocals and tonal symbols.”

It took Tuan more than three months of trial and error to figure out the most effective way to turn an image of a hard-copy medical document into an electronic version.

news-2-rmit-student-uses-machine-learning-to-decode-doctors-handwriting

An illustration of perspective warping operation

“We tackled those challenges by enhancing various tasks in the text recognition pipeline,” he said.

“We applied a denoising process, performed text segmentation at a word level, and applied the Bigram language model to improve the probabilities of the possible correction given the neighbouring words.

“Most importantly, we combined and implemented a deep learning architecture which consists of a ResNet for feature extraction, a BiLSTM network for text sequence modelling, [both artificial neural networks], and CTC for final transcription task.

“At that point, the final output sequence aligned to a lexicon to further boost the accuracy.”

news-3-rmit-student-uses-machine-learning-to-decode-doctors-handwriting

Phung Minh Tuan (pictured left) and RMIT School of Science & Technology lecturer and his supervisor Dr Dinh Ngoc Minh (pictured right).

A door has been opened to a better treatment process

RMIT School of Science & Technology lecturer and Tuan’s supervisor Dr Dinh Ngoc Minh emphasised the promising pipeline results, which can ultimately facilitate the digital transformation of medical centres and hospitals in Vietnam, to improve their readiness for adopting modern electronic health record management systems.

“The proposed work can accelerate the digitisation process of medical health record systems,” Dr Minh said.

“With the help of the machine in processing all the records, health facilities could gradually shift to an electronic system without any sudden change in protocol.

“Such systems would also allow remote medical centres or field health workers with limited computer access to continue with paper systems which could then be digitised easily.”

Dr Minh said that the patient records could be shared simply between departments, which could help reduce unnecessary tests and optimise treatment, to eventually improve health care quality.

“And most importantly, Tuan’s work can generate a digital medical-note dataset for a variety of potential medical machine-learning solutions,” Dr Minh said.

“In fact, our collaborators HTD and OUCRU are planning to use the generated data to develop expert diagnostic systems, improve the treatment process and minimalise errors in healthcare practices.

“Tuan has already proven himself as a professional software developer, and a quick-learner who has expanded his machine learning knowledge significantly during the six months of working on this project.”

Through this work, Tuan has been offered an internship at the OUCRU, and his work has been presented at the ACIS2020, an A-ranked international conference, the AHT conference, as well as the University’s recent student showcase.

Story: Hoang Ha

Campus and community

Science and technology

Media Release

08 December 2020