Cheat sheet

gLM cheat sheet: DNABERT-2

One reference card for DNABERT-2: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

DNABERT-2

The efficient successor: tokens are frequent chunks, not fixed k-mers, plus a shared benchmark

First public
arXiv, June 2023
Published
ICLR, May 2024
Lab
Northwestern University, Stony Brook University
Architecture
BERT encoder with faster attention. Position comes from a distance penalty (ALiBi), not a learned table, so nothing caps the input length.
Tokenization
BPE, byte-pair encoding
Parameters
117M
Context
No fixed limit. Chunked tokens cut length about 5x.
Training data
32.5 billion bases from 135 species, roughly 12 times the human genome alone.
Used for
Prediction and classification after fine-tuning

Headline result

Average 66.80 on GUE, the paper's own benchmark of DNA classification tasks across four species, against 66.93 for the 2.5B Nucleotide Transformer, at 21x fewer parameters and about 92x less GPU time to pretrain. Beats the original DNABERT on 23 of 28 datasets, about 6 points higher on average.

What it does not do

Not a generator. It fills in hidden spans rather than writing sequence, and every reported result comes from full fine-tuning.

Commonly confused

The 21x comparison is against Nucleotide Transformer v1 (2.5B), the state of the art at the time, not the later 250M v2. And it is a near-match rather than a win: 66.80 against 66.93.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On DNABERT-2 specifically, the comparison is the part that gets repeated loosely. The 21x claim is against Nucleotide Transformer v1 at 2.5B parameters, which was the state of the art when the paper was written, not against the later 250M v2. And the result is a near-match rather than a win: 66.80 against 66.93 on GUE. The paper says “comparable performance”, and the card keeps that.

What is unambiguous is the cost. About 92x less GPU time to pretrain, 14 days on 8 RTX 2080Ti cards against 28 days on 128 A100s, and small enough to fine-tune on a consumer GPU. That is the actual argument the paper makes.

The tokenization change is the reason. Overlapping k-mers leak the answer during masked training, because a masked token can be read off its neighbours. Non-overlapping ones break the other way: shift the sequence by one base and the token sequence changes completely, so near-identical inputs look unrelated. Byte-pair encoding avoids both and cuts sequence length about 5x on the way.

GUE is the other half of the contribution and does not fit in ten fields. It is a multi-species benchmark of 36 datasets across 9 task types, with inputs from 70 to 10,000 bases, published so models could be compared on the same terms.

Numbers are from the ICLR paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.