A new way to combine tissue images and gene data

HistoTx merges visual tissue structures with molecular activity through a shared transformer architecture.

Researchers developed HistoTx to combine H&E images and spatial transcriptomics. This method provides a multimodal view of tissue biology.

Spatial transcriptomics maps gene expression across tissue sections. Whole slide images capture morphological architecture.

The HistoTx architecture uses early fusion with a vision transformer. Transcript tokens are built at arbitrary spatial resolution and merged with image patch tokens.

The system uses a shared transformer stack for deep cross-modal interaction. It starts from a pretrained vision-only pathology foundation model.

When using only images, performance stays on par with the baseline. It improves on tasks like gene expression prediction.

Using both image and transcript inputs together outperforms using either modality alone.

Key facts

  • HistoTx uses an early-fusion vision transformer architecture.
  • The model performs better on gene expression prediction when using only images.
  • Jointly providing image and transcript inputs outperforms using one modality alone.

All sections

Edition