Skip to content

Update ipex backend to be deprecated. #1599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
XuehaoSun merged 7 commits into from
Mar 24, 2026
Merged

Update ipex backend to be deprecated. #1599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
65b0716
Update ipex backend to be deprecated.
xin3he Mar 23, 2026
cbf5757
Update docs/step_by_step.md
xin3he Mar 24, 2026
027e57a
Update docs/step_by_step.md
xin3he Mar 24, 2026
2d30693
Update docs/step_by_step.md
xin3he Mar 24, 2026
24f5e8f
Update docs/step_by_step_CN.md
xin3he Mar 24, 2026
526b711
Update docs/step_by_step_CN.md
xin3he Mar 24, 2026
998490d
Update step_by_step_CN.md
xin3he Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions auto_round/inference/backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -509,7 +509,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand All @@ -524,7 +524,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand All @@ -539,7 +539,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand All @@ -554,7 +554,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand All @@ -569,7 +569,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand All @@ -584,7 +584,7 @@ def fp8_static_scheme_checker(
priority=6,
checkers=[ark_feature_checker],
alias=["ark"],
compute_dtype=["float32", "float16"],
compute_dtype=["float32", "float16", "bfloat16"],
data_type=["int"],
act_bits=WOQ_DEFAULT_ACT_BITS,
requirements=["torch>=2.8.0", "auto-round-lib"],
Expand Down
5 changes: 5 additions & 0 deletions auto_round/inference/convert_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,11 @@ def post_init(model: torch.nn.Module, used_backends: list[str]) -> None:
need_gptqmodel_init = True
elif backend.startswith(("ipex", "auto_round_kernel")):
need_ipex_init = True
if backend.startswith("ipex"):
logger.warning_once(
f"Backend '{backend}' is deprecated and will be removed in a future release. "
"Please use the 'ark' backend instead (requires auto-round-lib and torch>=2.8.0)."
)

# AutoGPTQ post-init
if need_autogptq_init:
Expand Down
16 changes: 10 additions & 6 deletions docs/step_by_step.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -629,7 +629,7 @@ AutoRound automatically selects the best available backend based on the installe

### CPU

Supports 2, 4, and 8 bits. We recommend using intel-extension-for-pytorch (IPEX) for 4 bits inference.
Supports 2, 4, and 8 bits. We recommend using auto-round-lib (ark) for inference. When using the ark backend, ensure that your PyTorch version is >= 2.8.0.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
Expand All @@ -645,7 +645,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal

### Intel GPU

Supports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for inference.
Supports 4,8 bits. When using the ark backend, PyTorch (torch) >= 2.8.0 is required. We recommend using auto-round-lib (ark) for inference.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
Expand Down Expand Up @@ -698,13 +698,13 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
AutoRound automatically selects the backend for each layer based on compatibility. In general, the priority order is Marlin > ExLLaMAV2 > Triton, but the final choice depends on factors such as group size, bit width, packing format, hardware device, and other implementation details.

The backend may not always be the most suitable for certain devices.
You can specify your preferred backend such as "ipex" for CPU and Intel GPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.
You can specify your preferred backend such as "ark" for CPU and Intel GPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig

model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
quantization_config = AutoRoundConfig(backend="ipex")
quantization_config = AutoRoundConfig(backend="ark")
model = AutoModelForCausalLM.from_pretrained(
model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto"
)
Expand All @@ -715,13 +715,17 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
```
| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
|----------------------------------------------|--------------|---------|-----------|----------|-----------------|-----------------------------------|
| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
| ark | cpu | 2,4,8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib<br/>torch>=2.8.0 |
| ark | cpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib<br/>torch>=2.8.0 |
| ark | xpu | 4,8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib<br/>torch>=2.8.0 |
| ark | xpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib<br/>torch>=2.8.0 |
| ipex(To be deprecated) | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq/gptq_zp+-1 | gptqmodel |
| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 3 | gptq_zp+-1 | auto-gptq<br/>transformers<5.0.0 |
| gptq:cuda | cuda | 2,3,4,8 | FP16 | 1 | gptq_zp+-1 | auto-gptq<br/>transformers<5.0.0 |
| triton | xpu/cuda | 2,4,8 | BF16/FP16 | 2 | gptq/gptq_zp+-1 | auto-round |
| awq | cuda | 4 | FP16 | 5 | awq | auto-awq<br/>transformers<4.57.0 |
| awq | cuda | 4 | FP16 | 5 | awq | auto-awq<br/>transformers<4.57.0 |
| gptqmodel:awq or<br/>gptqmodel:awq_exllamav2 | cuda | 4 | BF16/FP16 | 6 | awq | gptqmodel |
| gptqmodel:awq_marlin | cuda | 4,8 | FP16 | 5 | awq | gptqmodel |
| gptqmodel:awq_gemm | cuda | 4 | FP16 | 3 | awq | gptqmodel |
Expand Down
14 changes: 9 additions & 5 deletions docs/step_by_step_CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -590,7 +590,7 @@ AutoRound 支持十余种推理后端，并会根据已安装的库自动选择
​**请勿在推理过程中手动将量化后的模型迁移到其他设备**​（例如执行 `model.to('cpu')`），否则可能导致意外错误。

### CPU
支持 2、4、8 bits 模型，其中 **4-bits 推理推荐搭配 intel-extension-for-pytorch（IPEX）** 使用，示例：
支持 2、4、8 bits 模型，推荐搭配 **auto-round-lib（ark）**（需 PyTorch>=2.8.0）进行推理，示例：
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

Expand All @@ -604,7 +604,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
```

### 英特尔 GPU
**仅支持 4-bits 模型**，推荐搭配 **IPEX** 使用，示例：
支持 4、8 bits 模型，推荐搭配 **auto-round-lib（ark）**（需 PyTorch>=2.8.0）进行推理，示例：
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

Expand Down Expand Up @@ -652,7 +652,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
AutoRound 会根据兼容性为每个层自动选择推理后端，默认优先级大致为：Marlin > ExLLaMAV2 > Triton。最终的选择会受到 **group size、bit width、packing format、hardware device** 等多种因素的影响。

默认选择的后端并非在所有设备上都是最优的，你可根据需求或硬件兼容性手动指定后端：
- CPU/英特尔 GPU：推荐`ipex`
- CPU/英特尔 GPU：推荐`ark`（需要 `auto-round-lib` 和 `torch>=2.8.0`）
- CUDA：可指定`marlin`/`exllamav2`/`triton`

**注意**：手动指定后端的话，可能要安装相关依赖。
Expand All @@ -662,7 +662,7 @@ AutoRound 会根据兼容性为每个层自动选择推理后端，默认优先
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig

model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
quantization_config = AutoRoundConfig(backend="ipex")
quantization_config = AutoRoundConfig(backend="ark")
model = AutoModelForCausalLM.from_pretrained(
model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto"
)
Expand All @@ -676,7 +676,11 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
#### 各推理后端的详细支持说明
| 后端名称 | 支持设备 | 支持位宽 | 支持数据类型 | 优先级 | 打包格式 | 依赖库要求 |
|-------------------------|----------------|----------------|--------------|--------|-----------------|--------------------------------|
| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
| ark | cpu | 2、4、8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib<br/>torch>=2.8.0 |
| ark | cpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib<br/>torch>=2.8.0 |
| ark | xpu | 4、8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib<br/>torch>=2.8.0 |
| ark | xpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib<br/>torch>=2.8.0 |
| ipex（即将废弃） | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
| marlin | cuda | 4、8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
| exllamav2/<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
| exllamav2/<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq<br/>transformers<5.0.0 |
Expand Down
Loading