<img src='https://scrapbox.io/api/pages/nishio-en/GPT-4/icon' alt='GPT-4.icon' height="19.5"/>Attention is the psychological process by which we focus <a href="/en/focal%20point">focal point</a> on certain events or information and <a href="/en/ignore">ignore</a> others. In general, our <a href="/en/awareness">awareness</a> is <a href="/en/limited%20resources">limited resources</a> and we cannot pay attention to all information at the same time. Therefore, our brain deals with <a href="/en/information%20overload">information overload</a> situations by focusing on information that is deemed <a href="/en/important">important</a> in the moment.
<h2>machine learning context:</h2>
<ul>
<li><a href="/en/attention%20(heed)">attention (heed)</a> , <a href="/en/attention%20mechanism">attention mechanism</a>
<ul>
<li><a href="/en/addition%20caution">addition caution</a></li>
<li><a href="/en/internal%20volume%20caution">internal volume caution</a></li>
<li><a href="/en/Source%20Target%20Attention">Source Target Attention</a></li>
<li><a href="/en/self-caution">self-caution</a></li>
</ul>
</li>
<li>Note A is defined with query q, key k, and value v as follows
</li>
<li>$A(Q, K, V) = \mathrm{softmax}(QK^T)V$
</li>
<li>Additive and Intra-product Caution
<ul>
<li>Theoretically, the complexity is about the same, but the inner product attention can be computed with matrix products, so it is faster in practical use.</li>
<li>The claim that scaling with the dimension dk of the key gives better performance</li>
</ul>
</li>
<li>$A(Q, K, V) = \mathrm{softmax}({QK^T \over \sqrt{d_k}})V$
</li>
</ul>
<hr>
This page is auto-translated from <a href="https://scrapbox.io/nishio/%E3%82%A2%E3%83%86%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%B3">/nishio/アテンション</a> using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at <a href="https://twitter.com/nishio_en">@nishio_en</a>. I&#39;m very happy to spread my thought to non-Japanese readers.