GPTQ¶

GPTQ is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with transformers and also how to quantize your own model with AutoGPTQ.

Usage of GPTQ Models with Transformers¶

Now, Transformers has officially supported AutoGPTQ, which means that you can directly use the quantized model with Transformers. The following is a very simple code snippet showing how to run Qwen2-7B-Instruct-GPTQ-Int4 (note that for each size of Qwen2, we provide both Int4 and Int8 quantized models) with the quantized model:

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2-7B-Instruct-GPTQ-Int4", # the quantized model
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B-Instruct-GPTQ-Int4")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512,
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Usage of GPTQ Quantized Models with vLLM¶

Attention

vllm does not support GPTQ quantized Qwen2 MoE models at the moment (version 0.5.2).

vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with AutoGPTQ with vLLM. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and Qwen2-7B-Instruct-GPTQ-Int4:

python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2-7B-Instruct-GPTQ-Int4

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen2-7B-Instruct-GPTQ-Int4",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me something about large language models."}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "repetition_penalty": 1.05,
  "max_tokens": 512
}'

or you can use python client with openai python package as shown below:

from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="Qwen/Qwen2-7B-Instruct-GPTQ-Int4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me something about large language models."},
    ],
    temperature=0.7,
    top_p=0.8,
    max_tokens=512,
)
print("Chat response:", chat_response)

Quantize Your Own Model with AutoGPTQ¶

Important

AutoGPTQ does not officially support quantizing Qwen2 MoE models at the moment (version 0.7.1). Consider using this fork.

If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code:

git clone https://github.com/AutoGPTQ/AutoGPTQ
cd AutoGPTQ
pip install -e .

Suppose you have finetuned a model based on Qwen2-7B, which is named Qwen2-7B-finetuned, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run:

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

# Specify paths and hyperparameters for quantization
model_path = "your_model_path"
quant_path = "your_quantized_model_path"
quantize_config = BaseQuantizeConfig(
    bits=8, # 4 or 8
    group_size=128,
    damp_percent=0.01,
    desc_act=False,  # set to False can significantly speed up inference but the perplexity may slightly bad
    static_groups=False,
    sym=True,
    true_sequential=True,
    model_name_or_path=None,
    model_file_base_name="model"
)
max_len = 8192

# Load your tokenizer and model with AutoGPTQ
# To learn about loading model to multiple GPUs,
# visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoGPTQForCausalLM.from_pretrained(model_path, quantize_config)

However, if you would like to load the model on multiple GPUs, you need to use max_memory instead of device_map. Here is an example:

model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    quantize_config,
    max_memory={i: "20GB" for i in range(4)}
)

Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example:

import torch

data = []
for msg in dataset:
    text = tokenizer.apply_chat_template(msg, tokenize=False, add_generation_prompt=False)
    model_inputs = tokenizer([text])
    input_ids = torch.tensor(model_inputs.input_ids[:max_len], dtype=torch.int)
    data.append(dict(input_ids=input_ids, attention_mask=input_ids.ne(tokenizer.pad_token_id)))

where each msg is a typical chat message as shown below:

[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me who you are."},
    {"role": "assistant", "content": "I am a large language model named Qwen..."}
]

Then just run the calibration process by one line of code:

import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S"
)
model.quantize(data, cache_examples_on_gpu=False)

Finally, save the quantized model:

model.save_quantized(quant_path, use_safetensors=True)
tokenizer.save_pretrained(quant_path)

It is unfortunate that the save_quantized method does not support sharding. For sharding, you need to load the model and use save_pretrained from transformers to save and shard the model. Except for this, everything is so simple. Enjoy!

Troubleshooting¶

Issue: With transformers and auto_gptq, the logs suggest CUDA extension not installed. and the inference is slow.

auto_gptq fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its installation guide to install a pre-built wheel or try installing auto_gptq from source.

Issue: Qwen2-7B-Instruct-GPTQ-Int8 and Qwen2-1.5B-Instruct-GPTQ-Int8 inferencing with transformers and auto_gptq, RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 is raised or endless of !!!!... is generated, depending on the PyTorch version.

The fused CUDA kernels for 8-bit quantized models in auto_gptq that are also accessible to transformers is the one called cuda_old. It is not numerically stable for Qwen2 models. There are two workarounds:

Use vllm:

vllm uses a custom kernel for 8-bit GPTQ quantized models based on exllama_v2.
Use the triton kernel if auto_gptq must be used:

The triton kernel in auto_gptq is not accessible to transformers. Follow these steps:
1. Copy the content of quantization_config in config.json to quantize_config.json in the model files;
2. Use AutoGPTQForCausalLM.from_quantized from auto_gptq instead of AutoModelForCausalLM.from_pretrained from transformers to load the model;
3. Pass use_triton to from_quantized (and make sure you have triton and nvcc installed).

Issue: Self-quantized Qwen2-72B-Instruct-GPTQ with vllm, ValueError: ... must be divisible by ... is raised. The intermediate size of the self-quantized model is different from the official Qwen2-72B-Instruct-GPTQ models.

After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2-72B is 29568. Unfortunately, \(29568 \div 128 = 231\). Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with tensor_parallel_size=1, i.e., one GPU card.

A workaround is to make the intermediate size divisible by \(128 \times 8 = 1024\). To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality.

Try the following:

import torch
from torch.nn import functional as F

from transformers import AutoModelForCausalLM

# must use AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-72B-Instruct", torch_dtype="auto")

# this size is Qwen2-72B only
pad_size = 128

sd = model.state_dict()

for i, k in enumerate(sd):
    v = sd[k]
    print(k, i)
    # interleaving the padded zeros
    if ('mlp.up_proj.weight' in k) or ('mlp.gate_proj.weight' in k):
        prev_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568*2, -1)[:pad_size*2]
        new_v = torch.cat([prev_v, v[pad_size:]], dim=0)
        sd[k] = new_v
    elif 'mlp.down_proj.weight' in k:
        prev_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568*2)[:, :pad_size*2]
        new_v = torch.cat([prev_v, v[:, pad_size:]], dim=1)
        sd[k] = new_v

# this is a very large file; make sure your RAM is enough to load the model
torch.save(sd, '/path/to/padded_model/pytorch_model.bin')

This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the intermediate_size in config.json to 29696. Finally, you can quantize the saved model checkpoint.