FP4-quant

This repository organizes three FP4-related experiment pipelines (with reproducible scripts) for Llama-3.1-8B-Instruct:

• FlatQuant + MXFP4
• QuaRot + MXFP4
• MR-GPTQ (MXFP format)

Layout

• methods/flatquant_mxfp4
• methods/quarot_mxfp4
• methods/mr_gptq_mxfp4
• scripts/
• docs/structure.md

Reproducible Scripts

• scripts/run_flatquant_mxfp4_llama3_1_8b_instruct_w4a4kv16_rtn.sh
• scripts/run_quarot_mxfp4_llama3_1_8b_instruct_w4a4kv16_gptq.sh
• scripts/run_quarot_mxfp4_llama3_1_8b_instruct_w4a4kv16_rtn.sh
• scripts/run_mr_gptq_mxfp4_llama3_1_8b_instruct_eval.sh

Notes

• Model path defaults to: /home/yaobc/model/Llama-3.1-8B-Instruct
• Conda env defaults to: flatquant_mxfp4
• GPU id defaults to: CUDA_VISIBLE_DEVICES=0 (adjust as needed)

Run With nohup

Example (FlatQuant + MXFP4):

nohup bash scripts/run_flatquant_mxfp4_llama3_1_8b_instruct_w4a4kv16_rtn.sh \
  > flatquant_mxfp4_llama3_1_8b_instruct.nohup.log 2>&1 &

Example (QuaRot + MXFP4 GPTQ):

nohup bash scripts/run_quarot_mxfp4_llama3_1_8b_instruct_w4a4kv16_gptq.sh \
  > quarot_mxfp4_llama3_1_8b_instruct_gptq.nohup.log 2>&1 &

Example (MR-GPTQ eval):

nohup bash scripts/run_mr_gptq_mxfp4_llama3_1_8b_instruct_eval.sh \
  > mr_gptq_mxfp4_llama3_1_8b_instruct.nohup.log 2>&1 &