Arc Virtual Cell Challenge Primer

The Arc Virtual Cell Challenge is a competition to develop a model capable of predicting the effects of silencing a specific gene using CRISPR in a partially unseen cell type, a task known as context generalization. The ultimate goal is to create a "virtual cell" that can simulate cellular responses to parameter changes, potentially accelerating drug discovery by reducing the reliance on expensive and error-prone physical experiments.

Training Data and Modeling Constraints

The challenge utilizes a curated dataset of approximately 300,000 single-cell RNA sequencing profiles. The training set includes 220,000 cells, each represented by a transcriptome—a sparse row vector containing raw counts of RNA molecules for corresponding genes. Within this set, roughly 38,000 cells are unperturbed (control cells), which serve as the essential baseline for comparison.

The Observer Effect in Transcriptomics

Predicting cellular changes is complicated by the fact that reading a transcriptome destroys the cell. Because measurements cannot be taken from the same cell before and after a perturbation, researchers must use a population of basal (unperturbed) cells as a reference. This introduces biological heterogeneity and technical noise, which must be separated from the actual signal of the perturbation. The observed gene expression in perturbed cells is modeled as the sum of the true effect of the perturbation, the biological heterogeneity of the baseline population, and experiment-specific technical noise.

The STATE Baseline Model

Arc provided a baseline solution called STATE, which employs two transformer-based models: the State Embedding Model (SE) and the State Transition Model (ST).

State Transition Model (ST)

The State Transition Model acts as the cell simulator. It uses a Llama backbone and takes two inputs: a set of transcriptomes (or SE embeddings) for covariate-matched basal cells and one-hot vectors representing the gene perturbation. These inputs pass through independent 4-layer MLP encoders with GELU activations. If the model is producing a full transcriptome, the output passes through a learned decoder. ST is trained using Maximum Mean Discrepancy to minimize the difference between the probability distributions of predicted and actual outcomes.

State Embedding Model (SE)

The State Embedding Model is a BERT-like autoencoder designed to create semantic cell embeddings to improve cross-cell type generalization. The process involves several steps:

  1. Gene Embeddings: The model obtains amino acid sequences for all protein isoforms of a gene and processes them through ESM2 (a 15B parameter Protein Language Model). These are mean-pooled to create a transcript embedding, and then mean-pooled again to create a gene embedding.
  2. Projection: Gene embeddings are projected to the model dimension using a learned encoder with LayerNorm and SiLU activation.
  3. Cell Sentences: Each cell is represented by its top 2048 genes ranked by log fold expression level. These gene embeddings are constructed into a "cell sentence" including a [CLS] token (used as the final cell embedding) and a [DS] token (used to disentangle dataset-specific effects).
  4. Expression Modulation: To incorporate the magnitude of gene expression, a "soft binning" algorithm and two MLPs create expression encodings that are added to each gene embedding, similar to positional embeddings.

SE is trained by masking 1,280 genes per cell and tasking the model with predicting them.

Evaluation Metrics

Submissions are evaluated based on three primary metrics:

Perturbation Discrimination (PDisc)

This metric evaluates how well a model identifies relative differences between perturbations. It calculates the Manhattan distance between the predicted transcriptome and the ground truth, compared to the distance between the predicted transcriptome and all other perturbed transcriptomes in the test set. The result is normalized to a scale where 0 represents a perfect match, and the final score is normalized as PDiscNorm = 1 - 2 * PDisc.

Differential Expression (DE)

Differential Expression measures the fraction of truly affected genes that the model correctly identifies as significantly affected. The process involves:

  • Computing p-values via a Wilcoxon rank-sum test with tie correction for both predicted and ground truth distributions.
  • Applying the Benjamini-Hochberg procedure to modulate p-values and reduce false positives.
  • Calculating the intersection of the predicted differentially expressed genes and the ground truth set, normalized by the true number of differentially expressed genes.

Mean Average Error

This metric provides a straightforward measurement of the average error in the model's predictions.

Sources