Implementation of a self-made Encoder-Decoder Transformer in PyTorch (Multi-Head Attention is implemented too), inspired by "Attention is All You Need." Primarily designed for Neural Machine Translation (NMT), specifically for Chinese to Thai translation.
To install the required dependencies, run the following command:
pip install -r requirements.txtBefore using the model, make sure to download the necessary datasets and preprocess them accordingly. You can find an example of how to structure your data in the data.py file.
To train the model, execute:
python train.pyFor testing, you can use the example script:
python test_example.pyMake sure to customize the paths and configurations in the param.py and data.py file to match your setup as these codes were used for training Chinese to Thai translation model.
The project structure is organized as follows:
- models/: Contains the core components of the Transformer.
- layers/: Various layers used in the model.
- model/: Implementation of the Encoder, Decoder, and the complete Transformer architecture.
- train.py: Script for training the model.
- test_example.py: Script for testing the trained model on an example.
Adjust the hyperparameters and configurations in the param.py file before starting the training process. The trained model will be saved in the models/ directory.
Test the model using the testing script test_example.py. Modify the input examples to observe the model's outputs.
-
Attention is All You Need Paper
- Authors: Ashish Vaswani et al.
- Paper Link: arXiv:1706.03762
-
Transformer Implementation by Kevin Ko
- GitHub Repository: github.com/hyunwoongko/transformer
-
The Illustrated Transformer by Jay Alammar
- Article Link: jalammar.github.io/illustrated-transformer
-
TensorFlow Transformer Tutorial
- TensorFlow Text Tutorials: TensorFlow Transformer Tutorial