GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification

We present GazeDiff, a novel architecture that leverages radiologists’ eye gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis. Code available here.