Cheat sheet

gLM cheat sheet: HyenaDNA

One reference card for HyenaDNA: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

HyenaDNA

A million bases in one pass at single-base resolution, from a model small enough to be surprising

First public
arXiv, June 2023
Published
NeurIPS, December 2023
Lab
Stanford, Harvard, SynTensor, Mila
Architecture
Hyena, not a Transformer. Long convolutions that adapt to the input replace attention, so cost grows near linearly with length, not as its square.
Tokenization
Single nucleotide, one letter per token
Parameters
6.6M
Context
1,000,000 tokens, reached by ramping sequence length up in stages.
Training data
The human reference genome. One species.
Used for
Prediction and classification, including tasks that need long range

Headline result

State of the art on 12 of the 18 Nucleotide Transformer tasks with about 1.6M parameters, roughly 1,500x fewer than the 2.5B model it was compared against. On a separate suite, GenomicBenchmarks, up to +20 accuracy points on enhancer identification. 160x faster than a Transformer at 1M tokens.

What it does not do

Human only, and evaluated as a predictor. It trains by next-base prediction, but the paper does not test it as a sequence designer.

Commonly confused

The headline results come from a 1.6M parameter model, not the 6.6M largest checkpoint. And the 500x context claim is against standard attention only: models that skip or compress positions already reached about 100,000 bases.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On HyenaDNA specifically, two numbers need their conditions attached.

The parameter count on the card is 6.6M, the largest checkpoint, by the same convention the landscape figure uses. The headline result is not from that model. State of the art on 12 of 18 Nucleotide Transformer tasks came from a 1.6M parameter version, which is where the “1,500x fewer parameters” comparison against the 2.5B Nucleotide Transformer comes from. Both numbers are real and they belong to different checkpoints.

The 500x context claim is against dense attention models specifically, which the paper says plainly. Attention-based DNA models were capped near 512 to 4,000 tokens, under 0.001% of the human genome. Models using dilation or downsampling already reached about 100,000 bases, but they got there by averaging or skipping, which gives up single-base resolution. HyenaDNA’s actual claim is having both at once: a million bases of context with every base still visible.

The mechanism is the Hyena operator, a long convolution whose filter is produced by a small neural network, combined with input-controlled gating. Each layer still sees the whole sequence, like attention, but cost grows as O(L log L) rather than with the square of the length. At a million tokens that works out to 160x faster training than a comparable Transformer.

Numbers are from the NeurIPS paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.