LOGISTAR — Scheda Oggetto

Dati oggetto

URL

Apri

Titolo / Descrizione

Sito / Azienda

Estratto / Body

&lt;div id=&quot;content-center&quot; class=&quot;content-center&quot;&gt;
&lt;div class=&quot;main-article-author v2&quot;&gt;&lt;strong&gt;&amp;nbsp;Long Story Short about BERT&amp;zwj;&lt;/strong&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class=&quot;w-embed&quot;&gt;
&lt;p&gt;BERT stands for Bidirectional Encoder Representation from Transformers. The original &lt;a href=&quot;https://arxiv.org/pdf/1810.04805.pdf&quot;&gt;BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding&lt;/a&gt;, actually, explains everything you need to know about BERT.&lt;/p&gt;
&lt;p&gt;Honestly saying, there are much better articles on the Internet explaining what BERT is, for example, &lt;a href=&quot;https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270&quot;&gt;BERT Explained: State of the art language model for NLP&lt;/a&gt;. After reading this article, you may have some questions about attention mechanisms; this article &lt;a href=&quot;https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a&quot;&gt;Illustrated: Self-Attention&lt;/a&gt; explains attentions.&lt;/p&gt;
&lt;p&gt;In this paragraph I just want to run over the ideas of BERT and give more attention to the practical implementation.&lt;/p&gt;
&lt;p&gt;BERT solves two tasks simultaneously:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Next Sentence Prediction (NSP) ;&lt;/li&gt;
&lt;li&gt;Masked Language Model (MLM).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;h4&gt;Building BERT&lt;/h4&gt;
&lt;div class=&quot;w-embed&quot;&gt;
&lt;p&gt;To build BERT we need to work out three steps:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Prepare Dataset;&lt;/li&gt;
&lt;li&gt;Build a model;&lt;/li&gt;
&lt;li&gt;Build a trainer.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;BERT&lt;/h3&gt;
&lt;div class=&quot;w-embed&quot;&gt;
&lt;p&gt;BERT module is a container that combines all the modules together and returns the output.&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;prism  language-python&quot;&gt;&lt;span class=&quot;token keyword&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;token class-name&quot;&gt;BERT&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Module&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;  
  
    &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;self&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; vocab_size&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; dim_out&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; attention_heads&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;  
        &lt;span class=&quot;token builtin&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;BERT&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;__init__&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
  
        self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;embedding &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; JointEmbedding&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;vocab_size&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; dim_inp&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
        self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;encoder &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; Encoder&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; dim_out&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; attention_heads&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
  
        self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;token_prediction_layer &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Linear&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; vocab_size&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
        self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;softmax &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;LogSoftmax&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
        self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;classification_layer &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Linear&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
  
    &lt;span class=&quot;token keyword&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;token function&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;self&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; input_tensor&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; torch&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Tensor&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; attention_mask&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt; torch&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Tensor&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;  
        embedded &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;embedding&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;input_tensor&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
        encoded &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;encoder&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;embedded&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; attention_mask&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
  
        token_predictions &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;token_prediction_layer&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;encoded&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
  
        first_word &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; encoded&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;  
        &lt;span class=&quot;token keyword&quot;&gt;return&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;softmax&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;token_predictions&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;classification_layer&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;first_word&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We use linear layer (and softmax) with output that is equal to the vocabulary size for token prediction task.&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;prism  language-python&quot;&gt;self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;token_prediction_layer &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Linear&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; vocab_size&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;softmax &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;LogSoftmax&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim&lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token operator&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;token number&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And linear layer with output of 2 for next sentence prediction task&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;prism  language-python&quot;&gt;self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;classification_layer &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; nn&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;Linear&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;dim_inp&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt;  &lt;span class=&quot;token number&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The output of the network&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;argmax(NSP output) = [1, 0] is NOT next sentence
argmax(NSP output) = [0, 1] is next sentence
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Everything is simple in the &lt;code&gt;forward&lt;/code&gt;. At first we calculate embedding, then pass embeddings to our encoder.&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;prism  language-python&quot;&gt;embedded &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;embedding&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;input_tensor&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
encoded &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;encoder&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;embedded&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; attention_mask&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Secondly, we calculate the model output.&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;prism  language-python&quot;&gt;token_predictions &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;token_prediction_layer&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;encoded&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;  
 
first_word &lt;span class=&quot;token operator&quot;&gt;=&lt;/span&gt; encoded&lt;span class=&quot;token punctuation&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token number&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;token punctuation&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;token keyword&quot;&gt;return&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;softmax&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;token_predictions&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;token punctuation&quot;&gt;,&lt;/span&gt; self&lt;span class=&quot;token punctuation&quot;&gt;.&lt;/span&gt;classification_layer&lt;span class=&quot;token punctuation&quot;&gt;(&lt;/span&gt;first_word&lt;span class=&quot;token punctuation&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The full model graph is also available. To build the graph, run the script &lt;code&gt;graph.py&lt;/code&gt;. It saves the graph to &lt;code&gt;data/logs&lt;/code&gt; directory. Run tensorboard&lt;/p&gt;
&lt;pre class=&quot; language-bash&quot;&gt;&lt;code class=&quot;prism  language-bash&quot;&gt;tensorboard --logdir data/logs
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Open &lt;a href=&quot;http://localhost:6006&quot;&gt;http://localhost:6006&lt;/a&gt; in the browser, go to Graph tab. You should see the graph of our BERT model.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;h2&gt;Building BERT&lt;/h2&gt;
&lt;div class=&quot;w-embed&quot;&gt;
&lt;p&gt;To build BERT we need to work out three steps:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Prepare Dataset;&lt;/li&gt;
&lt;li&gt;Build a model;&lt;/li&gt;
&lt;li&gt;Build a trainer.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;/div&gt;

Note

Voto oggetto (1-10)

Precisione (1-10)

Tag (4)

✎ Modifica tag 🤖 Suggerisci AI

          ai,bert,pytorch,        

ai 5 bert 5 pytorch 5 wpg 5

ID: 114 • Tipo: WPG • Reg: 2022-07-03 • Mod: 29/07/2026 20:16