ChatGLM2-6B

Introduction

ChatGLM2-6B is the second-generation version of the open-source bilingual Chinese-English chat model ChatGLM-6B. It retains the smooth conversation experience and low deployment requirements of the first generation while providing stronger performance, longer context, and more efficient inference.

Key Improvements

1. Stronger Performance — ChatGLM2-6B uses the GLM hybrid objective and was pre-trained on 1.4 trillion bilingual tokens, followed by human-preference alignment training. Compared with the first-generation model, it significantly improves performance across benchmarks including MMLU, C-Eval, GSM8K, and BBH.

2. Longer Context — Using FlashAttention, the base model's context length was extended from 2K to 32K. Dialogue alignment was trained with an 8K context length, allowing the model to support longer and more extensive conversations.

3. More Efficient Inference — ChatGLM2-6B uses Multi-Query Attention to increase inference speed and reduce GPU memory consumption. Under the official implementation, inference speed is approximately 42% faster than the first-generation model. With INT4 quantization, a 6 GB GPU can support substantially longer conversations.

4. Open Model — ChatGLM2-6B is available for academic research, with commercial use subject to the conditions of its model license.

Getting Started

Environment Setup

Clone the repository and install the required dependencies:

git clone https://github.com/THUDM/ChatGLM2-6B
cd ChatGLM2-6B
pip install -r requirements.txt

PyTorch 2.0 or later is recommended for improved inference performance.

Usage

ChatGLM2-6B can be loaded through Transformers:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained(
    "THUDM/chatglm2-6b",
    trust_remote_code=True
)

model = AutoModel.from_pretrained(
    "THUDM/chatglm2-6b",
    trust_remote_code=True,
    device="cuda"
).eval()

response, history = model.chat(
    tokenizer,
    "Hello!",
    history=[]
)

print(response)

Conversation history can be passed back to the model to support multi-turn conversations.

Inference Efficiency

ChatGLM2-6B employs Multi-Query Attention to improve inference speed and reduce the GPU memory required by the KV cache. It also uses a causal mask during dialogue training, allowing KV cache data from previous conversation rounds to be reused.

Typical GPU memory requirements depend on quantization. FP16/BF16 inference requires roughly 13 GB, INT8 requires around 8 GB, and INT4 can operate with approximately 5–6 GB depending on the workload.

Quantization

If GPU memory is limited, the model can be loaded using 8-bit or 4-bit quantization:

model = AutoModel.from_pretrained(
    "THUDM/chatglm2-6b",
    trust_remote_code=True
).quantize(8).cuda()

Quantization reduces memory requirements at the cost of some model performance, while still allowing ChatGLM2-6B to generate natural conversational responses.

CPU Deployment

ChatGLM2-6B can also run without a GPU:

model = AutoModel.from_pretrained(
    "THUDM/chatglm2-6b",
    trust_remote_code=True
).float()

CPU inference is considerably slower and requires substantially more system memory.

macOS Deployment

On Apple Silicon Macs, ChatGLM2-6B can use the MPS backend provided by PyTorch. The model can be loaded locally and moved to MPS:

model = AutoModel.from_pretrained(
    "your local path",
    trust_remote_code=True
).to("mps")

Memory requirements depend on model precision and quantization.

API Deployment

Install FastAPI and Uvicorn:

pip install fastapi uvicorn

Then start the included API server:

python api.py

By default, the API runs locally on port 8000 and can be used to integrate ChatGLM2-6B with other applications.

License

The repository code is licensed under Apache-2.0. Use of the ChatGLM2-6B model weights is governed separately by the project's Model License. Consult the repository's current license terms before using the model commercially.

About ChatGLM2-6B

ChatGLM2-6B was developed as an open-source bilingual conversational language model intended to support research and development around large language models. Its combination of Chinese-English capabilities, relatively low hardware requirements, quantization support, longer context, and efficient inference makes it suitable for experimentation, local deployment, conversational applications, and downstream development.