Non-monotone transformation of biomarkers to improve diagnostic and screening accuracy in a DNA methylation study with trichotomous phenotypes

We propose a non-monotone transformation to biomarkers in order to improve the diagnostic and screening accuracy. The proposed quadratic transformation only involves modeling the distribution means and variances of the biomarkers and is therefore easy to implement in practice. Mathematical justification was rigorously established to support the validity of the proposed transformation. We conducted extensive simulation studies to assess the performance of the proposed method and compared the new method with the traditional methods. Case studies on real biomedical and epigenetics data were provided to illustrate the proposed transformation. In particular, the proposed method improved the AUC values for a large number of markers in a DNA methylation study and consequently led to the identification of greater number of important biomarkers and biologically meaningful genetic pathways.