LOGISTAR — Scheda Oggetto

Dati oggetto

URL

Apri

Titolo / Descrizione

Sito / Azienda

Estratto / Body

&lt;p&gt;We&#039;re *excited* to introduce HyenaDNA, a genomic foundation model pretrained on sequences of up to &lt;em&gt;&lt;strong&gt;1 million tokens long&lt;/strong&gt;&lt;/em&gt; at &lt;em&gt;&lt;strong&gt;single nucleotide resolution&lt;/strong&gt;&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;We show that as one increases context length, we can reach better performance measured in &lt;a href=&quot;https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#ppl&quot;&gt;improved perplexity&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;HyenaDNA trains up to &lt;a href=&quot;https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#runtime&quot;&gt;160x faster than Transformers&lt;/a&gt; with FlashAttention, uses a single-character tokenizer, and has global context at each layer.&lt;/p&gt;
&lt;p&gt;We apply HyenaDNA on 28 genomic tasks (SotA on 23), using far fewer parameters than &lt;a href=&quot;https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna#previous&quot;&gt;previous genomic models&lt;/a&gt;, with examples that can fit on &lt;a href=&quot;https://colab.research.google.com/drive/1wyVEQd4R3HYLTUOXEEQmp_I8aNC_aLhL#scrollTo=dgkZggOetkbR&quot;&gt;colab&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;There has been amazing work applying foundation models (aka LLMs) to genomics [&lt;a href=&quot;https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1&quot;&gt;DNABERT&lt;/a&gt;, &lt;a href=&quot;https://www.biorxiv.org/content/10.1101/2023.01.11.523679v1&quot;&gt;Nucleotide Transformer&lt;/a&gt;, &lt;a href=&quot;https://www.biorxiv.org/content/10.1101/2022.10.10.511571v1&quot;&gt;GenSLMs&lt;/a&gt;, &lt;a href=&quot;https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1&quot;&gt;GENA-LM&lt;/a&gt;], modeling DNA as the &amp;ldquo;language&amp;rdquo; of life. Unfortunately, these works have been limited by the quadratic scaling of attention in Transformers, and thus far have typically used context lengths of between 512 - 4k tokens, depending on dense or sparse attention. That&amp;rsquo;s less than &lt;strong&gt;0.001%&lt;/strong&gt; of the length of the human genome. (We also noticed that compared to protein models, which have fairly &quot;short-range&quot; sequences, genomics is far less established.)&lt;/p&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;

Note

Voto oggetto (1-10)

Precisione (1-10)

Tag (7)

✎ Modifica tag 🤖 Suggerisci AI

           genoma,llm,dev,nn,tool         

dev 5 gen 5 genoma 5 llm 5 nn 5 tool 5 wpg 5

ID: 344 • Tipo: WPG • Reg: 2023-10-09 • Mod: 29/07/2026 20:16