598 Reproducibility Project

Citation to the original paper:

Oleynik M, Kugic A, Kas ́a ˇc Z, Kreuzthaler M. Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text clas- sification. J Am Med Inform Assoc. 2019 Nov 1;26(11):1247-1254. doi: 10.1093/jamia/ocz149. PMID: 31512729; PMCID: PMC6798565.

Original paper's repo:

https://github.com/bst-mug/n2c2

Code Dependencies

• JDK8+
• python3 (to run official evaluation scripts)
• make (to compile fastText)
• gcc/clang (to compile fastText)

Steps for using this code repo:

• 1. You need to get the original/input dataset. One way to get is from https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/ by submitting data access request.
• 2. Once you have the input dataset, you need to put the data under the folder: /data/test , /data/train separatelly. 70% of the data is for training, and 30% of the data is for testing.
• 3. Run the program SentenceDumper.java to generate the sentences.txt file.
• 4. Run the program VocabularyDumper.java to generate the vocab.txt file.
• 5. Install fasttext program to your working machine. You can follow this link to install fasttext: https://fasttext.cc/docs/en/supervised-tutorial.html
• 6. Copy sentences.txt file to the folder scripts, then run the script train_embeddings.sh which will generate n2c2-fasttext model.
• 7. Copy vocab.txt file to the folder scripts. Download BioWordVec_PubMed_MIMICIII_d200.bin from https://github.com/ncbi-nlp/BioSentVec. Run the script print_pre_trained_vectors.sh to generate pre_trained embedding. Run the script print_self_trained_vectors.sh to generate self_trained embedding. Then, copy both the embeddings to the related folder as the generated class file by java.
• 8. To start the main program: navigate to ClassifierRunner.java program. Then, run the main method, and the program will start running. For my case, I disabled two Classifiers because the pre_trained embedding can not generated from my laptop due to my laptop has only 8 GB memory and failed to run the BioWordVec_PubMed_MIMICIII_d200.bin.
• 9. The program first loads all the classifiers, then it starts to parse the training/test dataset into list of patients class. For each classifier, it uses two validators: n2c2 official metrics and accuracy, fp/fn metrics to validate the model. Lastly, it writes the results under the stats folder, with -basic.csv and -official.csv as suffix.

Table of results:

I uploaded all outputs under original_output folder, including: Baseline model, RBC model, SVM model, LR model, LSTM model.

Overall F1 score per criterion on the test set, compared with the baseline, a majority classifier:

Overall accuracy per criterion on the test set, compared with the baseline, a majority classifier