Add comprehensive model card for Mixture of Horizons
Browse filesThis PR adds a comprehensive model card for the Mixture of Horizons in Action Chunking model.
It includes:
- The paper link: https://huggingface.co/papers/2511.19433
- The project page: https://timsty1.github.io/moh/
- The code repository: https://github.com/Timsty1/MixtureOfHorizons/tree/main
- Relevant metadata (`license: apache-2.0`, `pipeline_tag: robotics`, `library_name: transformers`).
- A summary of the model and its method.
- Detailed installation instructions, including the necessary modification to the `transformers` library.
- A sample Python code snippet for inference, directly from the GitHub README.
- The BibTeX citation.
This ensures better discoverability and usability of the model on the Hugging Face Hub.
README.md
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
pipeline_tag: robotics
|
| 4 |
+
library_name: transformers
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
# Mixture of Horizons in Action Chunking
|
| 8 |
+
|
| 9 |
+
This repository hosts the official models and code for the paper:
|
| 10 |
+
[**Mixture of Horizons in Action Chunking**](https://huggingface.co/papers/2511.19433)
|
| 11 |
+
|
| 12 |
+
Project Page: https://timsty1.github.io/moh/
|
| 13 |
+
Code Repository: https://github.com/Timsty1/MixtureOfHorizons/tree/main
|
| 14 |
+
|
| 15 |
+
## Introduction
|
| 16 |
+
Vision-language-action (VLA) models have shown remarkable capabilities in robotic manipulation, but their performance is sensitive to the **action chunk length** used during training, termed **horizon**. This paper proposes a **mixture of horizons (MoH)** strategy to mitigate the inherent trade-off between long-term foresight and short-term precision observed with fixed horizons. MoH rearranges action chunks into segments with different horizons, processes them in parallel with a shared action transformer, and fuses outputs. This approach allows MoH to exploit both long-term foresight and short-term precision jointly within a single model, improving performance and generalizability with minimal overhead. MoH also enables dynamic inference with adaptive horizons, achieving higher throughput while preserving superior performance.
|
| 17 |
+
|
| 18 |
+
<div align="center">
|
| 19 |
+
<table border="0" cellspacing="0" cellpadding="0">
|
| 20 |
+
<tr>
|
| 21 |
+
<td align="center" width="50%">
|
| 22 |
+
<img src="https://huggingface.co/Timsty/mixture_of_horizons/resolve/main/figure/study_of_horizons_pi0.png" alt="Trade-off Effect" width="100%">
|
| 23 |
+
</td>
|
| 24 |
+
<td align="center" width="50%">
|
| 25 |
+
<img src="https://huggingface.co/Timsty/mixture_of_horizons/resolve/main/figure/intro_motivation_v2.png" alt="Mixture of Horizons" width="100%">
|
| 26 |
+
</td>
|
| 27 |
+
</tr>
|
| 28 |
+
<tr>
|
| 29 |
+
<td align="center" valign="top">
|
| 30 |
+
Figure 1: Trade-off between long-term foresight and short-term precision induced by single horizon
|
| 31 |
+
</td>
|
| 32 |
+
<td align="center" valign="top">
|
| 33 |
+
Figure 2: Overview of the proposed mixture-of-horizons strategy
|
| 34 |
+
</td>
|
| 35 |
+
</tr>
|
| 36 |
+
</table>
|
| 37 |
+
</div>
|
| 38 |
+
|
| 39 |
+
## Quick Start
|
| 40 |
+
|
| 41 |
+
### 1. Environment Setup
|
| 42 |
+
|
| 43 |
+
Clone the repository and set up the conda environment:
|
| 44 |
+
|
| 45 |
+
```bash
|
| 46 |
+
git clone git@github.com:Timsty1/MixtureOfHorizons.git
|
| 47 |
+
conda create -n moh -y python=3.10
|
| 48 |
+
conda activate moh
|
| 49 |
+
pip install uv
|
| 50 |
+
cd MixtureOfHorizons
|
| 51 |
+
uv pip install -r requirements.txt
|
| 52 |
+
pip install packages/libero
|
| 53 |
+
pip install packages/openpi-client
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
### 2. Modify Transformers Library
|
| 57 |
+
|
| 58 |
+
This implementation requires modifying the `transformers` library to support PyTorch-type $\pi$ series models, which rely on *gemma*, *paligemma*, and *siglip*.
|
| 59 |
+
|
| 60 |
+
First, locate your conda environment path:
|
| 61 |
+
```bash
|
| 62 |
+
conda info --base
|
| 63 |
+
```
|
| 64 |
+
Then, copy the provided files to the transformers library directory (replace `YOUR_CONDA_DIR` with the path found above):
|
| 65 |
+
```bash
|
| 66 |
+
cp -r ./src/openpi/models_pytorch/transformers_replace/* YOUR_CONDA_DIR/envs/moh/lib/python3.10/site-packages/transformers/
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
### 3. Inference with Code
|
| 70 |
+
You can use our provided "eagenerate" for speedup generation just like using 'generate' from Hugging Face. Here is an example.
|
| 71 |
+
|
| 72 |
+
```python
|
| 73 |
+
import torch
|
| 74 |
+
from eagle.model.ea_model import EaModel
|
| 75 |
+
from fastchat.model import get_conversation_template
|
| 76 |
+
|
| 77 |
+
# Replace with paths to your base model and EAGLE model checkpoints
|
| 78 |
+
# Example: base_model_path = "lmsys/vicuna-13b-v1.3", EAGLE_model_path = "Timsty/mixture_of_horizons"
|
| 79 |
+
base_model_path = "path/to/your/base_model"
|
| 80 |
+
EAGLE_model_path = "path/to/your/eagle_model"
|
| 81 |
+
|
| 82 |
+
model = EaModel.from_pretrained(
|
| 83 |
+
base_model_path=base_model_path,
|
| 84 |
+
ea_model_path=EAGLE_model_path,
|
| 85 |
+
torch_dtype=torch.float16,
|
| 86 |
+
low_cpu_mem_usage=True,
|
| 87 |
+
device_map="auto",
|
| 88 |
+
total_token=-1
|
| 89 |
+
)
|
| 90 |
+
model.eval()
|
| 91 |
+
your_message="Hello"
|
| 92 |
+
conv = get_conversation_template("vicuna") # Use the correct template for your base model
|
| 93 |
+
conv.append_message(conv.roles[0], your_message)
|
| 94 |
+
conv.append_message(conv.roles[1], None)
|
| 95 |
+
prompt = conv.get_prompt()
|
| 96 |
+
input_ids=model.tokenizer([prompt]).input_ids
|
| 97 |
+
input_ids = torch.as_tensor(input_ids).cuda()
|
| 98 |
+
output_ids=model.eagenerate(input_ids,temperature=0.5,max_new_tokens=512)
|
| 99 |
+
output=model.tokenizer.decode(output_ids[0])
|
| 100 |
+
print(output)
|
| 101 |
+
```
|
| 102 |
+
**Note:** Vicuna, LLaMA2-Chat, and LLaMA3-Instruct are both chat models. You need to use the correct chat template, otherwise it will cause abnormal output from the model and affect the performance of EAGLE.
|
| 103 |
+
|
| 104 |
+
## ❤️ Acknowledgment
|
| 105 |
+
|
| 106 |
+
We express our gratitude to [OpenPi](https://github.com/Physical-Intelligence/openpi/tree/main), [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO), and [RoboTwin](https://robotwin-platform.github.io/) for their open-source contributions.
|
| 107 |
+
|
| 108 |
+
## 📝 Citation
|
| 109 |
+
If you feel that this paper, models, or codes are helpful, please cite our paper, thanks for your support!
|
| 110 |
+
|
| 111 |
+
```bibtex
|
| 112 |
+
@article{jing2025mixture_of_horizons,
|
| 113 |
+
title={Mixture of Horizons in Action Chunking},
|
| 114 |
+
author={Jing, Dong and Wang, Gang and Liu, Jiaqi and Tang, Weiliang and Sun, Zelong and Yao, Yunchao and Wei, Zhenyu and Liu, Yunhui and Lu, Zhiwu and Ding, Mingyu},
|
| 115 |
+
journal={arXiv preprint arXiv:2511.19433},
|
| 116 |
+
year={2025}
|
| 117 |
+
}
|
| 118 |
+
```
|