KVQuant enables 70B parameter LLMs to run on 8GB RAM

A 4-bit KV cache quantization technique reduces memory requirements for large models.