Quantization

Quantization is a technique that stores a neural network's numbers in lower precision, for example 8-bit or 4-bit integers instead of 16-bit or 32-bit floating point. The model gets smaller and usually faster, at the cost of some accuracy.

Why it matters

Memory is often the limiting factor for running large language models. Halving the bits roughly halves the memory for the weights, which can be the difference between needing a data-centre GPU and running on a laptop or a single consumer card. Lower precision also reduces memory bandwidth needs, which speeds up inference. Quality loss is often small at 8-bit and grows at 4-bit and below, depending on the method and model.

Example

7B parameters at 16-bit: 7e9 x 2 bytes = ~14 GB
7B parameters at 8-bit:  7e9 x 1 byte  = ~7 GB
7B parameters at 4-bit:  7e9 x 0.5 byte = ~3.5 GB

These figures cover weights only; extra memory is needed for the context cache.

Common mistakes

• Assuming quantization is free. Evaluate on your own task before and after.
• Quantizing a model below the point where reasoning or code quality visibly drops.
• Confusing it with distillation, which trains a different, smaller model.

Related

• Model parameters
• Model distillation
• Open-weight model
• Inference

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.