CardioBERT
Lior Tondovski, M.Sc. student at the department of Industrial Engineering
Advisors: Prof. Erez Shmueli & Dr. Noam Koenigstien
Via Zoom click here
Abstract:
THeart rate (HR) is an essential signal in many health and wellbeing scenarios and in some cases may serve as an important predictor. However, medical datasets that include continuous heart rate data are typically very small, and the high dimensionality of heart rate makes it very difficult to train a model that is likely to generalize well. Inspired by Google's BERT (2018), which has revolutionized the world of natural language processing (NLP) in recent years, we present a similar framework, tailored to the case of HR, to which we call CardioBERT. Specifically, the training of CardioBERT is divided into two main stages: 1) a pre-training stage, which utilizes the mask language model (MLM) training technique. This involves selectively masking certain segments of the heart rate sequence and prompting the model to predict the masked values, and 2) a finetuning stage in which we fine-tune the pre-trained model on small/medium datasets with a specific prediction task in mind. To evaluate our approach, we pre-trained CardioBert using data of thousands of participants in the PerMed study, and fine-tuned it for two different use cases (whose data also arrive from the PerMed study): 1) predicting the result of COVID-19 test results a day in advance, and 2) predicting the occurrence of a fever symptom a day in advance. In both considered use cases, the CardioBERT-based models outperformed all benchmark models that were trained directly on the target dataset, without a pre-training stage.
Bio:
Lior Tondovski is a M.Sc. student at the department of Industrial Engineering in Tel Aviv University, specializing in Data Science. Lior holds a B.SC. Degree in Industrial Engineering from Tel Aviv University. His research focuses on top-notch Deep Leaning Algorithms for Personalized Medicine. The research is supervised by Prof. Erez Shmueli and Dr. Noam Koenigstein.
E-Mail: lior@mail.tau.ac.il
