Oggetti

Descrizione

dta reg (aaaa-mm-gg)

Body

<div id="content-center" class="content-center">
<div class="main-article-author v2">&nbsp;Long Story Short about BERT&zwj;</div>
</div>
<div class="w-embed">
BERT stands for Bidirectional Encoder Representation from Transformers. The original <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>, actually, explains everything you need to know about BERT.
Honestly saying, there are much better articles on the Internet explaining what BERT is, for example, <a href="https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270">BERT Explained: State of the art language model for NLP</a>. After reading this article, you may have some questions about attention mechanisms; this article <a href="https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a">Illustrated: Self-Attention</a> explains attentions.
In this paragraph I just want to run over the ideas of BERT and give more attention to the practical implementation.
BERT solves two tasks simultaneously:
<ul>
<li>Next Sentence Prediction (NSP) ;</li>
<li>Masked Language Model (MLM).</li>
</ul>
&nbsp;
<h4>Building BERT</h4>
<div class="w-embed">
To build BERT we need to work out three steps:
<ul>
<li>Prepare Dataset;</li>
<li>Build a model;</li>
<li>Build a trainer.</li>
</ul>
<h3>BERT</h3>
<div class="w-embed">
BERT module is a container that combines all the modules together and returns the output.
<pre class=" language-python"><code class="prism language-python">class BERT(nn.Module): 
 
 def __init__(self, vocab_size, dim_inp, dim_out, attention_heads=4): 
 super(BERT, self).__init__() 
 
 self.embedding = JointEmbedding(vocab_size, dim_inp) 
 self.encoder = Encoder(dim_inp, dim_out, attention_heads) 
 
 self.token_prediction_layer = nn.Linear(dim_inp, vocab_size) 
 self.softmax = nn.LogSoftmax(dim=-1) 
 self.classification_layer = nn.Linear(dim_inp, 2) 
 
 def forward(self, input_tensor: torch.Tensor, attention_mask: torch.Tensor): 
 embedded = self.embedding(input_tensor) 
 encoded = self.encoder(embedded, attention_mask) 
 
 token_predictions = self.token_prediction_layer(encoded) 
 
 first_word = encoded[:, 0, :] 
 return self.softmax(token_predictions), self.classification_layer(first_word)
</code></pre>
We use linear layer (and softmax) with output that is equal to the vocabulary size for token prediction task.
<pre class=" language-python"><code class="prism language-python">self.token_prediction_layer = nn.Linear(dim_inp, vocab_size)
self.softmax = nn.LogSoftmax(dim=-1)
</code></pre>
And linear layer with output of 2 for next sentence prediction task
<pre class=" language-python"><code class="prism language-python">self.classification_layer = nn.Linear(dim_inp, 2)
</code></pre>
The output of the network
<pre><code>argmax(NSP output) = [1, 0] is NOT next sentence
argmax(NSP output) = [0, 1] is next sentence
</code></pre>
Everything is simple in the <code>forward</code>. At first we calculate embedding, then pass embeddings to our encoder.
<pre class=" language-python"><code class="prism language-python">embedded = self.embedding(input_tensor) 
encoded = self.encoder(embedded, attention_mask)
</code></pre>
Secondly, we calculate the model output.
<pre class=" language-python"><code class="prism language-python">token_predictions = self.token_prediction_layer(encoded) 
 
first_word = encoded[:, 0, :]
return self.softmax(token_predictions), self.classification_layer(first_word)
</code></pre>
The full model graph is also available. To build the graph, run the script <code>graph.py</code>. It saves the graph to <code>data/logs</code> directory. Run tensorboard
<pre class=" language-bash"><code class="prism language-bash">tensorboard --logdir data/logs
</code></pre>
Open <a href="http://localhost:6006">http://localhost:6006</a> in the browser, go to Graph tab. You should see the graph of our BERT model.
</div>
&nbsp;
&nbsp;
&nbsp;
</div>
<h2>Building BERT</h2>
<div class="w-embed">
To build BERT we need to work out three steps:
<ul>
<li>Prepare Dataset;</li>
<li>Build a model;</li>
<li>Build a trainer.</li>
</ul>
</div>
</div>

tipo ogg.

cat

subcat

proprietario

azienda

ute

selez1

op1

codart

codice

codclifor

motivo

url

tag

imgsrc

sunto_breve

sunto_medio

note

keyscrape

Lista tag dell'oggetto (oggettag)
cod_tag	value
ai	5
bert	5
pytorch	5
wpg	5

Lista delle righe dell'oggetto (oggettir)
tip_pro	cod_pro	descrizione	val_pro	pes_pro

---.