-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathEncoderDecoders
98 lines (80 loc) · 4.54 KB
/
EncoderDecoders
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
"""
https://huggingface.co/transformers/notebooks.html
https://github.com/google-research/bert
https://github.com/huggingface/transformers#model-architectures
https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences
https://github.com/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb
https://en.wikipedia.org/wiki/Cloze_test
http://nlp.seas.harvard.edu/2018/04/03/attention.html
https://cloud.google.com/tpu/
Tensor2Tensor Colab: https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb
Blogs/Exp:
https://openai.com/blog/better-language-models/
https://nlp.seas.harvard.edu/2018/04/03/attention.html#encoder-and-decoder-stacks
http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
http://jalammar.github.io/illustrated-bert/
https://jalammar.github.io/illustrated-transformer/
https://www.kaggle.com/ratan123/in-depth-guide-to-google-s-bert
https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
https://www.youtube.com/watch?v=xI0HHN5XKDo
https://www.youtube.com/watch?v=TQQlZhbC5ps
https://ruder.io/nlp-imagenet/
https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca
https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html
https://www.youtube.com/watch?v=rBCqOTEfxvg
http://mlexplained.com/2017/12/29/attention-is-all-you-need-explained/
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8#e18a-828e5fc317c7
https://medium.com/@jonathan_hui/nlp-bert-transformer-7f0ac397f524
https://www.tensorflow.org/tutorials/text/transformer
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
https://transformer.huggingface.co/doc/gpt2-large
https://nlp.seas.harvard.edu/2018/04/03/attention.html
http://www.site.uottawa.ca/~diana/csi5386/mikolov1,coling,2014,tutorial,fix,tomas,mikolov.pdf
https://stackoverflow.com/questions/37889914/what-is-a-projection-layer-in-the-context-of-neural-networks
https://challengeenthusiast.com/2019/10/15/pay-attention-and-you-shall-learn/
Related papers:
https://huggingface.co/transformers/bertology.html
All you need is attention: https://arxiv.org/abs/1706.03762
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding:https://arxiv.org/abs/1810.04805
Semi-supervised Sequence Learning: https://arxiv.org/abs/1511.01432
Deep contextualized word representations: https://arxiv.org/abs/1802.05365
OpenAI transformer: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
A large annotated corpus for learning natural language inference (https://nlp.stanford.edu/projects/snli/): http://nlp.stanford.edu/pubs/snli_paper.pdf
"""
NMT
https://github.com/tensorflow/nmt
https://arxiv.org/abs/1409.0473
https://arxiv.org/abs/1508.04025
https://arxiv.org/abs/1703.01619
https://www.youtube.com/watch?v=IxQtK2SjWWM
https://blog.floydhub.com/attention-mechanism/
https://towardsdatascience.com/light-on-math-ml-attention-with-keras-dc8dbc1fad39
https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec
NLP with Attention Models
DL: https://www.youtube.com/watch?v=W3_yaf3HvHU&list=PLbRMhDVUMngc7NM-gDwcBzIYZNFSK2N1a&index=1
https://www.youtube.com/watch?v=w8xbd8XI7U8
https://www.youtube.com/watch?v=4Yirbho6Kyo
https://www.jeremyjordan.me/autoencoders/
Sentiment
https://huggingface.co/datasets/amazon_us_reviews
Speech
https://techcrunch-com.cdn.ampproject.org/c/s/techcrunch.com/2020/12/03/mlcommons-debuts-first-public-database-for-ai-researchers-with-86000-hours-of-speech/amp/
https://mlcommons.org/en/peoples-speech/
BLEU Score:
#hassle-free computation of shareable, comparable, and reproducible BLEU scores. It also knows all the standard test sets and handles downloading, processing, and tokenization
https://github.com/mjpost/sacrebleu
https://cloud.google.com/translate/automl/docs/evaluate
Translated texts from the web(smaller): http://opus.nlpl.eu/
Larger: https://paracrawl.eu/
Both available through: https://www.tensorflow.org/datasets, https://www.tensorflow.org/datasets/catalog/overview
Books:
DIDL:
https://github.com/d2l-ai/d2l-en
https://www.amazon.science/latest-news/amazon-scientists-author-popular-deep-learning-book
Hinton:
https://arxiv.org/abs/2009.05673
JAX and Trax:
https://github.com/google/trax
Other resources
https://cs.uwaterloo.ca/~ppoupart/teaching/cs480-spring19/resources.html