LLM Knowledge and Interview Questions

Introduction

This repository collects concepts, study notes, and interview questions related to large language models. The material was compiled from public resources and the author's own notes. Contributions and corrections are welcome, and users who find the project useful are encouraged to star the repository.

For hands-on learning with limited hardware, the author also maintains several practical projects:

• tiny-llm-zh — build a small Chinese LLM from scratch and learn pretraining, fine-tuning, and reinforcement-learning techniques.
• tiny-rag — implement a simple RAG system with multi-path retrieval and reranking.
• tiny-mcp — implement MCP servers and clients with prompts and function calling, and use MCP to build agent projects.
• llama3-from-scratch-zh — implement Llama 3 from scratch, load official Meta weights, and debug locally on a machine with 16 GB of memory.

Another recommended resource is AI Interview Notes, which covers general deep learning, machine learning, recommendation systems, and search systems.

Online reading

Read the material online at LLMs Interview Note.

Notes

The answers are written and organized by the author. If you find errors or unreasonable explanations, corrections are welcome. The author also publishes occasional LLM content and interview experience through a WeChat public account.

<img src=https://github.com/wdndev/personal/blob/main/image/llmers_weixin.jpg width="427" height="156" alt="WeChat" />

Contents

The repository is organized around the following major topics:

1. Real interview questions — examples gathered from actual LLM-related interviews.
2. Large language model fundamentals — language models, tokenization, word vectors, part-of-speech tagging, parsing, Word2Vec, NLP feature extractors, deep-learning basics, and common LLM concepts.
3. Large language model architectures — Transformer attention, layer normalization, positional encoding, tokenization, model parameters, activation functions, MHA/MQA/GQA, decoding strategies, BERT, Llama, ChatGLM, and Mixture-of-Experts models.
4. Training datasets — dataset formats and model-parameter considerations.
5. Distributed training — data parallelism, pipeline parallelism, tensor parallelism, sequence parallelism, multidimensional hybrid parallelism, automatic parallelism, MoE parallelism, DeepSpeed, Megatron, training acceleration, and GPU-memory questions.
6. Supervised fine-tuning — core concepts, prompting, adapter tuning, LoRA, pretraining, Llama 2 fine-tuning, and ChatGLM3 fine-tuning.
7. Inference — inference frameworks such as vLLM, Text Generation Inference, FasterTransformer, and TensorRT-LLM, plus inference optimization and quantization.
8. Reinforcement learning — policy gradients, PPO, RLHF, DPO, and related interview questions.
9. Retrieval-Augmented Generation (RAG) — retrieval-augmented LLMs, RAG techniques, and large-model agent technologies.
10. Large language model evaluation — evaluation methods, hallucinations, their causes, and mitigation techniques.
11. LLM applications — chain-of-thought prompting, LangChain, and practical application topics.
12. Related courses and references — additional study material and source references.

The original repository contains detailed article links and chapter files for each topic. Use the online documentation or repository navigation to open the individual notes.