20240506

BlitherBoom812 · May 6, 2024 · d4f9408 · d4f9408
1 parent 0ef2725
commit d4f9408
Show file tree

Hide file tree

Showing 19 changed files with 105 additions and 4 deletions.
diff --git a/scripts/katex-patch.js b/scripts/katex-patch.js
@@ -9,5 +9,7 @@ module.exports = function (data) {
     if (debug) {
         console.log("after replace: " + data.content.match(/<div>\$\$(.|\n)*?\$\$<\/div>/g));
     }
+    data.content = data.content.replace("\\{", "\\lbrace");
+    data.content = data.content.replace("\\}", "\\rbrace");
     return data;
 };
diff --git a/source/_posts/AI.md b/source/_posts/AI.md
@@ -1,6 +1,6 @@
 ---
 title: Introduction to Artificial Intelligence
-katex: false
+katex: true
 date: 2024-02-26 15:20:37
 tags:
 ---
@@ -295,3 +295,104 @@ Batch Normalization
 ![1714380763838](../images/AI/1714380763838.png)
 
 最后一层 Global Average Pooling：7\*7\*2048 -> 1\* 1 \* 2048
+
+### Recurrent Neural Network (RNN)
+
+#### Idea for Sequence Modeling
+
+Local Dependency
+
+![1714982108981](../images/AI/1714982108981.png)
+
+Parameter Sharing
+
+![1714982133935](../images/AI/1714982133935.png)
+
+### RNN
+
+![1714982188331](../images/AI/1714982188331.png)
+
+Go deeper
+
+![1714982215920](../images/AI/1714982215920.png)
+
+#### Standard Architectures
+
+![1714982075164](../images/AI/1714982075164.png)
+
+- RNNs can represent unbounded temporal dependencies
+- RNNs encode histories of words into a fixed size hidden vector 
+- Parameter size does not grow with the length of dependencies
+- RNNs are hard to learn long range dependencies present in data
+
+#### LSTM
+
+Multihead, shared bottom.
+
+![1714982405665](../images/AI/1714982405665.png)
+
+Gradient flow highway: remember history very well.
+
+NIPS 2015 Highway Network.
+
+#### Training Strategies
+
+Shift in Training & Inference
+
+![1714983390325](../images/AI/1714983390325.png)
+
+Use Scheduled Sampling to solve this
+
+![1714983452419](../images/AI/1714983452419.png)
+
+Problem: Gradient Explosion during continuously multiplication.
+
+Solution: Gradient Clipping
+
+![1714983641507](../images/AI/1714983641507.png)
+
+Variational Dropout
+
+![1714983788461](../images/AI/1714983788461.png)
+
+Layer Normalization
+
+![1714983988587](../images/AI/1714983988587.png)
+
+BN: Easy to compare between channels
+
+LN: Easy to compare between samples
+
+在图像任务上，我们一般认为 channel 之间的地位应该是相同的，因此常常采用 BN。
+
+### Transformer
+
+use attention to replace state space.
+
+![1714984460020](../images/AI/1714984460020.png)
+
+#### Attention
+
+![1714984778231](../images/AI/1714984778231.png)
+
+$$
+\text{Attention}(Q,K,V)=\text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
+$$
+
+Multi-Head Attention
+
+![1714985419724](../images/AI/1714985419724.png)
+
+Sparse?
+
+$W^o$ to maintain shape and jointly attend to information from different representation subspaces.
+
+#### FFN
+
+Position-wise FFN (Similar to multi  convolution kernels in CNN, shared parameters in every word.)
+
+![1714985971187](../images/AI/1714985971187.png)
+
+#### Positional Encoding
+
+![1714986050538](../images/AI/1714986050538.png)
diff --git a/source/_posts/SolidPhysics.md b/source/_posts/SolidPhysics.md
@@ -1,6 +1,6 @@
 ---
 title: 固体物理
-katex: false
+katex: true
 date: 2024-04-16 10:06:07
 tags:
 ---
@@ -26,7 +26,6 @@ $$
 
 ![1713239832006](../images/SolidPhysics/1713239832006.png)
 
-
 #### 一维近自由电子近似
 
 利用微扰求解薛定谔方程
@@ -90,7 +89,6 @@ $$
 
 $$
 \vec{\psi(r)}=e^{i\vec{k}\cdot\vec{r}}\vec{u(r)}
-
 $$
 
 被周期函数调幅的平面波。

diff --git a/source/images/AI/1714982075164.png b/source/images/AI/1714982075164.png
diff --git a/source/images/AI/1714982108981.png b/source/images/AI/1714982108981.png
diff --git a/source/images/AI/1714982133935.png b/source/images/AI/1714982133935.png
diff --git a/source/images/AI/1714982188331.png b/source/images/AI/1714982188331.png
diff --git a/source/images/AI/1714982215920.png b/source/images/AI/1714982215920.png
diff --git a/source/images/AI/1714982405665.png b/source/images/AI/1714982405665.png
diff --git a/source/images/AI/1714983390325.png b/source/images/AI/1714983390325.png
diff --git a/source/images/AI/1714983452419.png b/source/images/AI/1714983452419.png
diff --git a/source/images/AI/1714983641507.png b/source/images/AI/1714983641507.png
diff --git a/source/images/AI/1714983788461.png b/source/images/AI/1714983788461.png
diff --git a/source/images/AI/1714983988587.png b/source/images/AI/1714983988587.png
diff --git a/source/images/AI/1714984460020.png b/source/images/AI/1714984460020.png
diff --git a/source/images/AI/1714984778231.png b/source/images/AI/1714984778231.png
diff --git a/source/images/AI/1714985419724.png b/source/images/AI/1714985419724.png
diff --git a/source/images/AI/1714985971187.png b/source/images/AI/1714985971187.png
diff --git a/source/images/AI/1714986050538.png b/source/images/AI/1714986050538.png