Transformer (machine learning model): Difference between revisions
WikiSysopIG (talk | contribs) |
WikiSysopIG (talk | contribs) |
||
| Line 249: | Line 249: | ||
\text{Tokens} | \text{Tokens} | ||
\rightarrow | \rightarrow | ||
\text{ | \text{Token IDs} | ||
\rightarrow | |||
\text{Embedding Vectors} | |||
\rightarrow | \rightarrow | ||
\text{Transformer} | \text{Transformer} | ||
</math> | |||
Each token is represented by a numerical vector, allowing the neural network to process linguistic information mathematically. | Each token is represented by a numerical vector, allowing the neural network to process linguistic information mathematically. | ||
| Line 269: | Line 271: | ||
</math> | </math> | ||
where <math>\mathcal{L}</math> represents the training loss. | where <math>\mathcal{L}(\theta)</math> represents the training loss. | ||
For language modeling, a common objective is to predict a token based on contextual information. | For language modeling, a common objective is to predict a token based on contextual information. | ||
| Line 281: | Line 283: | ||
P(x_t\mid x_1,\ldots,x_{t-1}) | P(x_t\mid x_1,\ldots,x_{t-1}) | ||
</math> | </math> | ||
where <math>x_t</math> represents the token at position <math>t</math>. | |||
The model is trained to assign high probability to appropriate next tokens. | The model is trained to assign high probability to appropriate next tokens. | ||