Oggetti

Descrizione

dta reg (aaaa-mm-gg)

Body

We're *excited* to introduce HyenaDNA, a genomic foundation model pretrained on sequences of up to 1 million tokens long at single nucleotide resolution.
We show that as one increases context length, we can reach better performance measured in <a href="https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#ppl">improved perplexity</a>.
HyenaDNA trains up to <a href="https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#runtime">160x faster than Transformers</a> with FlashAttention, uses a single-character tokenizer, and has global context at each layer.
We apply HyenaDNA on 28 genomic tasks (SotA on 23), using far fewer parameters than <a href="https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#previous">previous genomic models</a>, with examples that can fit on <a href="https://colab.research.google.com/drive/1wyVEQd4R3HYLTUOXEEQmp_I8aNC_aLhL#scrollTo=dgkZggOetkbR">colab</a>.
There has been amazing work applying foundation models (aka LLMs) to genomics [<a href="https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1">DNABERT</a>, <a href="https://www.biorxiv.org/content/10.1101/2023.01.11.523679v1">Nucleotide Transformer</a>, <a href="https://www.biorxiv.org/content/10.1101/2022.10.10.511571v1">GenSLMs</a>, <a href="https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1">GENA-LM</a>], modeling DNA as the &ldquo;language&rdquo; of life. Unfortunately, these works have been limited by the quadratic scaling of attention in Transformers, and thus far have typically used context lengths of between 512 - 4k tokens, depending on dense or sparse attention. That&rsquo;s less than 0.001% of the length of the human genome. (We also noticed that compared to protein models, which have fairly "short-range" sequences, genomics is far less established.)
&nbsp;

tipo ogg.

cat

subcat

proprietario

azienda

ute

selez1

op1

codart

codice

codclifor

motivo

url

tag

imgsrc

sunto_breve

sunto_medio

note

keyscrape

Lista tag dell'oggetto (oggettag)
cod_tag	value
dev	5
gen	5
genoma	5
llm	5
nn	5
tool	5
wpg	5

Lista delle righe dell'oggetto (oggettir)
tip_pro	cod_pro	descrizione	val_pro	pes_pro
WBK	LLM
	de		HyenaDNA: learning from DNA with 1 Million token context

---.