Building Small Language Model: From Foundations to Bangla Financial Text Generation
Learn the core principles and techniques of language models while building a small Bangla language model that generates financial articles. Explore the five modules below, from language-model foundations to training and text generation.
Syllabus
Course syllabus
Instructor: Sabik Bin Sultan
- Introduction to Language Models
History of NLP · Transformer Architecture · Tokenization
- Data Preparation Pipeline
Text Preprocessing · Underfitting, Overfitting & Just-Right Fitting · Tokenization Fundamentals · Bangla Tokenization Challenges
- Transformer Architecture
Token and positional embeddings · Self-attention mechanism with causal masking · Multi-Head Attention · Cross-Attention
- Model Components
Feed-Forward Networks · Multi-Layer Perceptrons (MLPs) · Forward Pass · Gradient Explosion / Vanishing
- Training, Evaluation, Generation
AdamW Optimizer Configuration · Learning Rate Scheduling · Gradient Clipping · Training Loop Implementation · Autoregressive Decoding · Temperature Tuning
Module 1: Introduction to Language Models
History of NLP • Transformer Architecture • Tokenization
Module 1: Introduction to Language ModelsModule 2: Data Preparation Pipeline
Text Preprocessing • Underfitting, Overfitting & Just-Right Fitting • Tokenization Fundamentals • Bangla Tokenization Challenges
Module 2: Data Preparation PipelineModule 3: Transformer Architecture
Token and positional embeddings • Self-attention mechanism with causal masking • Multi-Head Attention • Cross-Attention
Module 3: Transformer ArchitectureModule 4: Model Components
Feed-Forward Networks • Multi-Layer Perceptrons (MLPs) • Forward Pass • Gradient Explosion / Vanishing
Module 4: Model ComponentsModule 5: Training, Evaluation, Generation
AdamW Optimizer Configuration • Learning Rate Scheduling • Gradient Clipping • Training Loop Implementation • Autoregressive Decoding • Temperature Tuning
Module 5: Training, Evaluation, Generation