Loomuliku keele töötlus 2018/19 kevad

Course schedule

This schedule is tentative and may be changed during the semester as necessary.

Textbooks:

• Yoav Goldberg, 2017. Neural Network Methods for Natural Language Processing

Week	Date	Topic
24	13.02.2019	Introduction to NLP [slides] [video]
25	20.02.2019	Language modeling [slides] [video]
26	27.02.2019	Word embeddings [slides] [video]
27	06.03.2019	Text classification [slides] [video]
28	13.03.2019	Sequence tagging [slides] [video]
29	20.03.2019	Morphology [slides] [video]
30	27.03.2019	Syntactic parsing [slides] [video]
31	03.04.2019	Lexical semantics [slides] [video]
32	10.04.2019	Attention mechanism [slides] [video]
33	17.04.2019	Question answering [slides] [video]
34	24.04.2019	Dialog systems [slides] [video]
35	01.05.2019	Spring Day - no lecture
36	08.05.2019	Machine translation - Mark Fišel [slides]
37	15.05.2019	Speech recognition - Tanel Alumäe (TUT) [slides] [video]
38	22.05.2019	Speech synthesis - Tambet Matiisen [slides] [video]
39	29.05.2019	NLP in industry - Risto Hinno (FeelingStream) [slides] [video]

Extra reading

• Lecture 1: Introduction
  • Natural language processing: a historical review
  • A Review of the Neural History of Natural Language Processing
  • Natural language processing: an introduction
• Lecture 2: Language modeling
  • Language Model: A Survey of the State-of-the-Art Technology
  • An Empirical Study of Smoothing Techniques for Language Modeling
  • A Neural Probabilistic Language Model
  • Recurrent Neural Network Based Language Model
  • Strategies for Training Large Vocabulary Neural Language Models
  • The Unreasonable Effectiveness of Recurrent Neural Networks
  • Character-Aware Neural Language Models
• Lecture 3: Word embeddings
  • Collobert et al., 2011.Natural Language Processing (Almost) from Scratch
  • Mikolov et al., 2013. Linguistic Regularities in Continuous Space Word Representations
  • Mikolov et al., 2013. Efficient Estimation of Word Representations in Vector Space
  • Goldberg and Levy, 2014. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method
  • Bojanowski et al., 2017. Enriching Word Vectors with Subword Information
  • Blog post: The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
  • Ling et al., 2015. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
  • Schnabel et al., 2015. Evaluation methods for unsupervised word embeddings
  • Pennington et al., 2014. GloVe: Global Vectors for Word Representation
  • Peters et al., 2018. Deep contextualized word representations
  • Devlin et al., 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Lecture 4: Text classification
  • Kim, 2014. Convolutional Neural Networks for Sentence Classification
  • Joulin et al., 2017. Bag of Tricks for Efficient Text Classification
  • Lai et al., 2015. Recurrent Convolutional Neural Networks for Text Classification
  • Zhou et al., 2015. A C-LSTM Neural Network for Text Classification
  • Zhang et al., 2015. Character-level Convolutional Networks for Text Classification
  • Aggarwal and Zhai, 2012. A Survey of Text Classification Algorithms
• Lecture 7: Syntactic parsing
  • Kipperwasser and Goldberg, 2016. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations
  • Dyer et al., 2015. Transition-Based Dependency Parsing with Stack Long Short-Term Memory
  • Dozat el al., 2017. Stanford’s Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task
• Lecture 8: Lexical semantics
  • Blog post: The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
  • Peters et al., 2018. Deep contextualized word representations
  • Devlin et al., 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Lecture 9: Attention
  • Bahdanau et al., 2015. Neural machine translation by jointly learning to align and translate
  • Luong et al., 2015. Effective approaches to attention-based neural machine translation
  • Vaswani et al., 2017. Attention is all you need
  • Incorporating Copying Mechanism in Sequence-to-Sequence Learning
  • Merity et al., 2016. Pointer sentinel mixture models
  • Huang et al., 2016. Attention-based multimodal machine translation
  • Yang et al., 2016. Hierarchical attention networks for document classification
  • Xu et al., 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
  • Cheng et al., 2016. Long Short-Term Memory-Networks for Machine Reading
  • Allamanis et al., 2016. A Convolutional Attention Network for Extreme Summarization of Source Code
• Lecture 10: Question answering
  • Jurafsky and Martin, 2018. Speech and language processing, 3rd ed draft, chapter 23
  • Chen et al., 2017. Reading Wikipedia to Answer Open-Domain Questions