nielsr HF Staff commited on
Commit
eccb524
·
verified ·
1 Parent(s): 77b7492

Add comprehensive model card for Mixture of Horizons

Browse files

This PR adds a comprehensive model card for the Mixture of Horizons in Action Chunking model.

It includes:
- The paper link: https://huggingface.co/papers/2511.19433
- The project page: https://timsty1.github.io/moh/
- The code repository: https://github.com/Timsty1/MixtureOfHorizons/tree/main
- Relevant metadata (`license: apache-2.0`, `pipeline_tag: robotics`, `library_name: transformers`).
- A summary of the model and its method.
- Detailed installation instructions, including the necessary modification to the `transformers` library.
- A sample Python code snippet for inference, directly from the GitHub README.
- The BibTeX citation.

This ensures better discoverability and usability of the model on the Hugging Face Hub.

Files changed (1) hide show
  1. README.md +118 -0
README.md ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ pipeline_tag: robotics
4
+ library_name: transformers
5
+ ---
6
+
7
+ # Mixture of Horizons in Action Chunking
8
+
9
+ This repository hosts the official models and code for the paper:
10
+ [**Mixture of Horizons in Action Chunking**](https://huggingface.co/papers/2511.19433)
11
+
12
+ Project Page: https://timsty1.github.io/moh/
13
+ Code Repository: https://github.com/Timsty1/MixtureOfHorizons/tree/main
14
+
15
+ ## Introduction
16
+ Vision-language-action (VLA) models have shown remarkable capabilities in robotic manipulation, but their performance is sensitive to the **action chunk length** used during training, termed **horizon**. This paper proposes a **mixture of horizons (MoH)** strategy to mitigate the inherent trade-off between long-term foresight and short-term precision observed with fixed horizons. MoH rearranges action chunks into segments with different horizons, processes them in parallel with a shared action transformer, and fuses outputs. This approach allows MoH to exploit both long-term foresight and short-term precision jointly within a single model, improving performance and generalizability with minimal overhead. MoH also enables dynamic inference with adaptive horizons, achieving higher throughput while preserving superior performance.
17
+
18
+ <div align="center">
19
+ <table border="0" cellspacing="0" cellpadding="0">
20
+ <tr>
21
+ <td align="center" width="50%">
22
+ <img src="https://huggingface.co/Timsty/mixture_of_horizons/resolve/main/figure/study_of_horizons_pi0.png" alt="Trade-off Effect" width="100%">
23
+ </td>
24
+ <td align="center" width="50%">
25
+ <img src="https://huggingface.co/Timsty/mixture_of_horizons/resolve/main/figure/intro_motivation_v2.png" alt="Mixture of Horizons" width="100%">
26
+ </td>
27
+ </tr>
28
+ <tr>
29
+ <td align="center" valign="top">
30
+ Figure 1: Trade-off between long-term foresight and short-term precision induced by single horizon
31
+ </td>
32
+ <td align="center" valign="top">
33
+ Figure 2: Overview of the proposed mixture-of-horizons strategy
34
+ </td>
35
+ </tr>
36
+ </table>
37
+ </div>
38
+
39
+ ## Quick Start
40
+
41
+ ### 1. Environment Setup
42
+
43
+ Clone the repository and set up the conda environment:
44
+
45
+ ```bash
46
+ git clone git@github.com:Timsty1/MixtureOfHorizons.git
47
+ conda create -n moh -y python=3.10
48
+ conda activate moh
49
+ pip install uv
50
+ cd MixtureOfHorizons
51
+ uv pip install -r requirements.txt
52
+ pip install packages/libero
53
+ pip install packages/openpi-client
54
+ ```
55
+
56
+ ### 2. Modify Transformers Library
57
+
58
+ This implementation requires modifying the `transformers` library to support PyTorch-type $\pi$ series models, which rely on *gemma*, *paligemma*, and *siglip*.
59
+
60
+ First, locate your conda environment path:
61
+ ```bash
62
+ conda info --base
63
+ ```
64
+ Then, copy the provided files to the transformers library directory (replace `YOUR_CONDA_DIR` with the path found above):
65
+ ```bash
66
+ cp -r ./src/openpi/models_pytorch/transformers_replace/* YOUR_CONDA_DIR/envs/moh/lib/python3.10/site-packages/transformers/
67
+ ```
68
+
69
+ ### 3. Inference with Code
70
+ You can use our provided "eagenerate" for speedup generation just like using 'generate' from Hugging Face. Here is an example.
71
+
72
+ ```python
73
+ import torch
74
+ from eagle.model.ea_model import EaModel
75
+ from fastchat.model import get_conversation_template
76
+
77
+ # Replace with paths to your base model and EAGLE model checkpoints
78
+ # Example: base_model_path = "lmsys/vicuna-13b-v1.3", EAGLE_model_path = "Timsty/mixture_of_horizons"
79
+ base_model_path = "path/to/your/base_model"
80
+ EAGLE_model_path = "path/to/your/eagle_model"
81
+
82
+ model = EaModel.from_pretrained(
83
+ base_model_path=base_model_path,
84
+ ea_model_path=EAGLE_model_path,
85
+ torch_dtype=torch.float16,
86
+ low_cpu_mem_usage=True,
87
+ device_map="auto",
88
+ total_token=-1
89
+ )
90
+ model.eval()
91
+ your_message="Hello"
92
+ conv = get_conversation_template("vicuna") # Use the correct template for your base model
93
+ conv.append_message(conv.roles[0], your_message)
94
+ conv.append_message(conv.roles[1], None)
95
+ prompt = conv.get_prompt()
96
+ input_ids=model.tokenizer([prompt]).input_ids
97
+ input_ids = torch.as_tensor(input_ids).cuda()
98
+ output_ids=model.eagenerate(input_ids,temperature=0.5,max_new_tokens=512)
99
+ output=model.tokenizer.decode(output_ids[0])
100
+ print(output)
101
+ ```
102
+ **Note:** Vicuna, LLaMA2-Chat, and LLaMA3-Instruct are both chat models. You need to use the correct chat template, otherwise it will cause abnormal output from the model and affect the performance of EAGLE.
103
+
104
+ ## ❤️ Acknowledgment
105
+
106
+ We express our gratitude to [OpenPi](https://github.com/Physical-Intelligence/openpi/tree/main), [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO), and [RoboTwin](https://robotwin-platform.github.io/) for their open-source contributions.
107
+
108
+ ## 📝 Citation
109
+ If you feel that this paper, models, or codes are helpful, please cite our paper, thanks for your support!
110
+
111
+ ```bibtex
112
+ @article{jing2025mixture_of_horizons,
113
+ title={Mixture of Horizons in Action Chunking},
114
+ author={Jing, Dong and Wang, Gang and Liu, Jiaqi and Tang, Weiliang and Sun, Zelong and Yao, Yunchao and Wei, Zhenyu and Liu, Yunhui and Lu, Zhiwu and Ding, Mingyu},
115
+ journal={arXiv preprint arXiv:2511.19433},
116
+ year={2025}
117
+ }
118
+ ```