Human language comes pre-cut. Spaces and punctuation tell a model where one unit ends and the next begins. DNA gives you none of that: one long string, four letters, no boundaries.
So you have to choose where to cut, and the choice is not free. Cut too finely and the sequence gets long, which matters because attention cost grows with the square of the length. Cut too coarsely and you lose the single-base resolution that variant effect prediction depends on.
The video runs the same nine bases, ATGCCATGC, through all four schemes:
- Nucleotide level — every base is its own token. Nine bases, nine tokens. Full resolution, longest sequence. Used by Evo 2 and Nucleotide Transformer v3.
- k-mer, overlapping — a window of three slides one base at a time. Seven tokens, but neighbours share most of their bases, so a masked token leaks from the ones beside it. This is the flaw DNABERT-2 set out to fix.
- k-mer, non-overlapping — the window jumps by three. Three tokens from nine bases, which cuts the attention cost by roughly the square of that reduction. The Carbon family bet on this at 6-mers.
- Learned (BPE) — the vocabulary is built from the data by merging the most frequent pair, over and over. Two tokens here. Compact, but the same stretch can split differently depending on where it falls.
Nine bases become nine, seven, three, or two tokens. There is still no standard, and the trade-off between sequence length and base-level resolution is the reason why.