Hugging Face's logo

TheHouseOfTheDude
/
M2411-123B-Animus-V12.0_Compressed-Tensors

Text Generation
English
vllm
conversational
roleplay
rp
compressed-tensors
awq
w4a16
quantized
Model card Files
xet
Community
M2411-123B-Animus-V12.0_Compressed-Tensors / README.md
phaedawg's picture
phaedawg
Adding W8A16
6fa8aae verified
preview code
|
raw
history blame contribute delete
6.61 kB
metadata
language:
  - en
library_name: vllm
pipeline_tag: text-generation
tags:
  - text-generation
  - conversational
  - roleplay
  - rp
  - compressed-tensors
  - awq
  - w4a16
  - quantized
base_model: Darkhn/M2411-123B-Animus-V12.0
base_model_relation: quantized
quantized_by: TheHouseOfTheDude
license: other

M2411-123B-Animus-V12.0 — Quantized (compressed-tensors for vLLM)

This repository provides quantized runtime builds of
Darkhn/M2411-123B-Animus-V12.0, repackaged for vLLM using the compressed-tensors format.

TL;DR

  • Quantized with W4A16 (INT4 weights / A16 activations) or W8A16 (INT8 weights / A16 activations) for vLLM via --quantization compressed-tensors.
  • Same calibration recipe as our recent cards: 512 chat samples, 2048 max sequence length, dataset neuralmagic/LLM_compression_calibration (messages rendered with the model’s chat template).
  • Weight-only AWQ; group size 128; lm_head kept in higher precision; exported with save_compressed=True.

Revisions & Branches

The main branch is a landing page (model card + links). Runnable artifacts live in per-quant branches.

  • main — placeholder / landing page
  • W4A16 — 4-bit weights / 16-bit activations (compressed-tensors)
  • W8A16 — 8-bit weights / 16-bit activations (compressed-tensors)

Quick links

What’s inside (per revision)

  • Sharded quantized weights (*.safetensors) + index (model.safetensors.index.json)
  • config.json with compressed-tensors metadata (weight_format, quantization, quantization_config, etc.)
  • Tokenizer artifacts (tokenizer.json, tokenizer.model, merges/vocab if applicable)
  • Optional: chat_template.jinja (inherits the finetune’s chat style)

Exact file lists may differ between branches — see Files and versions for each revision.

Quantization & calibration details (same script/recipe as prior card)

Method / flow

  • llmcompressor oneshot pipeline with an AWQModifier (weight-only quantization).

Targets / exclusions

  • Quantize Linear layers; ignore lm_head (kept in higher precision).

Weights / grouping

  • INT4 (num_bits=4, type="int", symmetric=True)
  • Strategy: "group" with group_size=128 (Marlin-friendly layout)
  • Activations are not quantized (runtime A16: BF16/FP16)

Calibration dataset & preprocessing

  • Dataset: neuralmagic/LLM_compression_calibration, split train
  • NUM_CALIBRATION_SAMPLES = 512 (random subset with fixed seed)
  • MAX_SEQUENCE_LENGTH = 2048
  • Each example’s messages is rendered via tokenizer.apply_chat_template(..., tokenize=False), then tokenized with:
    • max_length=2048, truncation=True, padding=False, add_special_tokens=False

Compression call

  • oneshot(..., max_seq_length=2048, num_calibration_samples=512, tokenizer=tokenizer) on the preprocessed dataset

Export for vLLM

  • Saved with save_compressed=True so vLLM loads the compressed-tensors runtime layout directly

This mirrors the exact group size, sequence length, sample count, and dataset parameters used in the previous quantization script you provided.

Context length

  • Calibration context: up to 2048 tokens per sample (as above).
  • Model context window: inherited from Darkhn/M2411-123B-Animus-V12.0; quantization does not change rope/position encodings, only the numeric representation of the weights.

Quickstart — vLLM (compressed-tensors)

Install vLLM (recent version recommended):

pip install vllm

Serve (adjust to your hardware):

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve TheHouseOfTheDude/M2411-123B-Animus-V12.0_Compressed-Tensors \
  --quantization compressed-tensors \
  --tensor-parallel-size 8 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.70 \
  --dtype bfloat16

Example Chat Completions:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TheHouseOfTheDude/M2411-123B-Animus-V12.0_Compressed-Tensors",
    "messages": [
      {"role":"system","content":"You are Animus — helpful, precise, and safe."},
      {"role":"user","content":"Draft a scene outline with shifting perspectives."}
    ],
    "max_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.95
  }'

Note: compressed-tensors is a vLLM runtime format. Loading directly with vanilla 🤗 Transformers is not supported.
For Transformers, use a compatible quant (e.g., GPTQ/AWQ export) or the full-precision finetune.

Prompting / chat template

This package follows the finetuned parent’s chat conventions. If a chat_template.jinja is present, libraries that support apply_chat_template will automatically format messages.

Guidelines:

  • Keep the system message concise (behavior, tone, safety constraints).
  • Provide clear user instructions; for multi-step tasks, list steps explicitly.

Intended use & safety

The finetuned parent model is marked “not for all audiences” and may produce mature or intense themes. This quantization:

  • Does not change underlying behavior or content tendencies.
  • Only changes weight storage for efficient inference.

Apply appropriate content filters / policies for your deployment context.

Lineage

Hardware tips

  • 123B-class models benefit from multi-GPU tensor parallel for throughput.
  • Long contexts are KV-cache heavy — tune --max-model-len and batch size.
  • Prefer BF16 on GPUs with native support; otherwise FP16.
  • Enable P2P/NVLink when available; consider CUDA Graphs if stable.

Changelog

  • V12.0 (current) — Initial compressed-tensors W4A16 quantization with 512-sample / 2048-token AWQ calibration; vLLM-ready packaging.