Cheat sheet

gLM cheat sheet: DNABERT

One reference card for DNABERT: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

DNABERT

The model that brought BERT's pretrain-once, fine-tune-often recipe to DNA

First public
bioRxiv, September 2020
Published
Bioinformatics, February 2021
Lab
Northwestern University, Stony Brook University
Architecture
BERT. 12 Transformer layers, 768 hidden units, 12 attention heads. A bidirectional encoder trained by masking contiguous spans of k-mers and predicting them back.
Tokenization
k-mer, overlapping 6-mers
Parameters
~92M
Context
512 tokens, and pretraining sequences up to 510 bases. A separate DNABERT-XL was added for longer inputs.
Training data
The human reference genome. One species.
Used for
Prediction and classification, after task-specific fine-tuning

Headline result

On 690 datasets of measured transcription factor binding (ENCODE ChIP-seq), the only method with mean and median accuracy and F1 all above 0.9 (0.918 and 0.919). Pretrained on human only, it still beat models built from scratch when fine-tuned on mouse.

What it does not do

Human only, not a generator, and 512 tokens of context. Every downstream task needs its own fine-tuning run.

Commonly confused

Often quoted at 110M parameters. That is BERT-base with a 30,000-token text vocabulary. DNABERT's k-mer vocabulary is 4,101 tokens, so its embedding table is far smaller and the count works out near 92M.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On DNABERT specifically, the size is the field to be careful with. It is widely quoted at 110M parameters, which is BERT-base carrying a 30,000-token text vocabulary. DNABERT’s vocabulary is every possible 6-mer plus five special tokens, 4,101 in total, so its embedding table is much smaller and the count derived from the published config comes out near 92M. The paper itself states no parameter count, which is why the card carries a tilde.

The result worth reading closely is the transfer one. DNABERT was pretrained on the human genome alone, and human and mouse non-coding DNA is only about 50% alike, yet fine-tuning it on mouse data still beat models built from scratch. The same paper shows the flip side: without pretraining, the identical architecture either fails to train or performs poorly. That pairing is the argument for the whole pretrain-then-finetune paradigm, made in 2021.

One framing on the card is not the paper’s. “The first BERT for the genome” is accurate against the record, but the paper only says it adapted the idea of BERT to a genomic setting. It does not claim priority.

Numbers are from the Bioinformatics paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.