meituan-longcat/UNO-Scorer-Qwen3-14B

UNO-Scorer: A Unified General Scoring Model for UNO-Bench

📖 Introduction

UNO-Scorer is a lightweight yet high-precision LLM-based evaluation model designed to efficiently automate the evaluation of Large Multimodal Models (LMMs) with minimal computational overhead.

Core Functionality:

• Input: Question + Reference Answer + Model Response
• Processing: Analyzes correctness by comparing each sub-question against the reference answer
• Output: Numerical score + Detailed evaluation reasoning for each sub-question

Built upon the powerful Qwen3-14B backbone, UNO-Scorer is fine-tuned on 13K high-quality in-house data. It overcomes the limitations of traditional Overall Reward Models (ORMs) by supporting 6 distinct question types, with particular excellence in Multi-Step Open-Ended Questions (MO).

📊 Performance

UNO-Scorer demonstrates superior performance in automated evaluation, particularly in handling complex Multi-Step Open-Ended Questions. We compared the accuracy of our scorer against other advanced evaluators on our test set:

Model	Accuracy
Seed-1.5-VL	0.9118
GPT-4.1	0.9457
UNO-Scorer (Ours)	0.9505

Experiments show that UNO-Scorer surpasses even proprietary frontier models like GPT-4.1 in this specific evaluation domain with lower cost.

💻 Usage

⚡ Quick Start (HuggingFace Transformers)

Get started with UNO-Scorer in just a few lines of code:

pip install -U transformers
python3 test_scorer_hf.py --model-name /path/to/UNO-Scorer

Minimal Example:

⚠️ Critical: The prompt template below is simplified for illustration. Only the complete prompt template in test_scorer_hf.py will properly activate the model's fine-tuned scoring capabilities. Custom or simplified prompts will not achieve optimal results.

Click to expand complete prompt template

def process_score_prompt(question, reference, response):
    promt_template = """请先通读问题信息，然后基于参考答案对模型回复的结果进行正确性打分。每道题可能包含多个小问，每个小问都已给出了相应的参考答案和分值，请逐小问校验模型回复是否正确，正确得对应分值，错误或漏答得0分，累计计分，有如下要求。

---

### 要求1：信息梳理

- 梳理出如下信息
  - 问题内容
  - 参考答案（可适度完善表达，但不改变核心内容）
  - 模型回复（需要将模型回复中的指代关系与参考答案对齐）
  - 分值

### 要求2：判断题型

- 明确该小问属于以下哪种题型之一，并基于该类型的打分标准进行打分，需要给出详细的比对过程。
  - **数值型**，要求模型回复与标准答案的数值完全相同，不允许有误差。例，`问题：北京奥运会是哪一年？参考答案：2008，模型回复：2004，打分结果：错误。`
  - **枚举型**，要求模型回复列举出参考答案的全部对象，缺一不可、错一不可，允许同义词等语义相近的表达，题中有顺序要求则必须按顺序枚举。例，`图中出现了哪些动物？参考答案：大熊猫、河马、长颈鹿，模型回复：河马、小熊猫、长颈鹿，打分结果：错误。 `注：“/”表示“或”，如，XXA/XXB，表示回答出任意一项即可。
  - **选择题**，要求模型回复与参考答案相同的选项或选项内容。例，`问题：李白是哪个朝代的诗人？A. 唐朝 B. 宋朝 C. 元朝，模型回复：李白是唐朝诗人，打分结果：正确。`
  - **判断题**，要求模型回复与参考答案的判断一致。例，`问题：图中鼠标是否放在了笔记本电脑左侧？参考答案：是，模型回复：图中鼠标在笔记本电脑的左侧。打分结果：正确。`
  - **简答题**，要求模型回复包括与参考答案语义一致的短语或表达，允许表达方式不同。例，`问题：视频中最后放入锅中的食材是什么？参考答案：洋葱，模型回复：胡萝卜。打分结果：错误。`
  - **论述题**，要求模型回复包含参考答案的核心观点。例，`问题：请简要论述为什么要保护生物多样性。参考答案：维持生态平衡，模型回复：保护生物多样性能够让生态系统保持稳定，促进人类社会的可持续发展。打分结果：正确。`

### 要求3：打分标准

- **完全正确**：得满分。
- **错误或漏答**：得0分。
- 如模型回复与参考答案大意相同但细节略有差别，且非核心内容，视为正确，具体参考参考答案的详细要求。
- 若模型回复未直接给出答案，需主动归纳总结结论，只关注结论是否一致。
- 每小问独立打分，前序错误不影响后续小问的结果。

### 要求4：输出格式

- 逐小问列出得分说明。
- 所有小问得分相加，在<score></score>中给出总分，例如：<score>5</score>

---

## 问题信息
{{question}}
## 参考答案
{{reference}}
## 模型回复
{{response}}
## 逐小问打分"""
    
    prompt = promt_template.replace("{{question}}", remove_thought_block(question.strip()))
    prompt = prompt.replace("{{reference}}", reference)
    prompt = prompt.replace("{{response}}", response)
    return prompt

from transformers import AutoModelForCausalLM, AutoTokenizer
import re

def extract_score(text):
    matches = re.findall(r'<score>([\d.]+)</score>', text)
    return float(matches[-1]) if matches else 0.0

tokenizer = AutoTokenizer.from_pretrained("meituan-longcat/UNO-Scorer-Qwen3-14B")
model = AutoModelForCausalLM.from_pretrained(
    "meituan-longcat/UNO-Scorer-Qwen3-14B",
    torch_dtype="auto",
    device_map="auto"
)

# Prepare scoring prompt
question = "Which animal appears in the image?"
reference = "Sub-question 1: Elephant, total score 10 points"
response = "I see an elephant in the image."

# This prompt template is simplified for illustration.
prompt = f"""Please score the model's response based on the reference answer.

Question: {question}
Reference Answer: {reference}
Model Response: {response}

Provide a step-by-step analysis and output the total score in <score></score> tags."""

# Generate score
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=4096,
    do_sample=False
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 
result = tokenizer.decode(output_ids, skip_special_tokens=True)
print("Score response:\n", result)
score = extract_score(result)
print(f"Score: {score}")

🔄 How It Works

UNO-Scorer evaluates model responses through a structured process:

1. Information Organization: Extracts question content, reference answer, model response, and scoring criteria
2. Question Type Classification: Identifies the question type (multiple-choice, numerical, enumeration, yes/no, short-answer, or essay)
3. Detailed Comparison: Compares model response against reference answer using type-specific criteria
4. Score Extraction: Outputs final score in <score>X</score> format (where X is 0-10)

📥 Input Format Requirements

The model expects three key inputs:

Component	Description	Example
Question	The original question posed to the model	"Which animals appear in the image?"
Reference Answer	Ground truth answer with point allocation (sum to 10)	Sub-question 1: Elephant, total score 10 points
Model Response	The response from the model being evaluated	"I see an elephant in the image."

Reference Answer Formatting (Critical!)

Since the model is trained primarily on Chinese corpora, formatting reference answers in Chinese yields significantly better results. However, English formatting is also supported.

For Single-Answer Questions:

1. {Answer}, total score 10 points, focus only on final answer correctness
1. {答案}，总分10分，无需关注推理过程，最终答案正确即可

For Multi-Part Questions:

1. {Sub-Answer A} ({X} points); 2. {Sub-Answer B} ({Y} points)
1. {子答案A}（{X}分）; 2. {子答案B}（{Y}分）

With Custom Scoring Criteria:

1. {Answer}, total score 10 points, scoring criteria: {detailed criteria}
1. {答案}，总分10分，评分标准：{详细标准}

📤 Output Format

The model returns:

• Detailed Evaluation: Step-by-step analysis for each sub-question
• Score Tag: <score>X</score> where X ranges from 0 to 10

Example output:

Sub-question 1:

Question Content: How many apples are in the image?
Reference Answer: 2
Model Response: There are two appels.
Points: 10 point
Question Type: Numerical
Comparison Process: The reference answer is "2" and the model response is "two". The numerical values are completely identical, with only the expression format differing. This meets the scoring standard for numerical questions.

Scoring Explanation: Completely correct, awarded 10 point.

<score>10</score>

📋 Complete Evaluation Example

See test_scorer_hf.py for a full working example with multiple question types:

• Multiple-choice questions
• Yes/No questions
• Open-ended questions
• Multi-part questions

Run the example:

python3 test_scorer_hf.py --model-name /path/to/UNO-Scorer

🚀 Optimized Inference with vLLM (Recommended for Production)

For large-scale evaluation tasks, we strongly recommend using vLLM for significant performance improvements:

# 1. Clone the repository
git clone https://github.com/meituan-longcat/UNO-Bench.git
cd UNO-Bench/uno-eval

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run vLLM-based inference
bash examples/test_scorer_vllm.sh

Why vLLM?

• 10-20x faster inference compared to standard HuggingFace
• Better batching support for multiple evaluation tasks
• Lower memory footprint
• Optimized for production deployments

⚠️ Important Notes

1. Language: Chinese formatting in reference answers produces significantly better results due to the model's training data composition
2. Point Allocation: Reference answers must have total points equal to 10
3. Score Extraction: Always look for <score>X</score> in the output
4. Batch Processing: Use vLLM for evaluating multiple responses efficiently
5. Question Type Awareness: Ensure reference answers clearly specify the question type for optimal scoring

🎯 Supported Question Types

UNO-Scorer supports evaluation across 6 distinct question types:

Question Type	Description	Scoring Rule
Multiple-Choice	Select correct option from given choices	Response must match the correct option exactly
Numerical	Provide specific numerical values	No tolerance for numerical errors
Enumeration	List all required items	Must include all items, no omissions or errors
Yes/No	Binary judgment questions	Response judgment must match reference answer
Short-Answer	Brief factual answers	Semantic equivalence acceptable, expression flexibility allowed
Essay	Longer analytical responses	Must contain core viewpoints from reference answer

📜 Citation

If you find this model or the UNO-Bench useful for your research, please cite our paper:

@misc{chen2025unobench,
      title={UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models},
      author={Chen Chen and ZeYang Hu and Fengjiao Chen and Liya Ma and Jiaxing Liu and Xiaoyu Li and Ziwen Wang and Xuezhi Cao and Xunliang Cai},
      year={2025},
      eprint={2510.18915},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.18915},
}

⚖️ License & Disclaimer

This model is released under the Apache 2.0 License. It is based on Qwen3-14B. Please strictly follow the license and usage policy of the original Qwen model series.

Disclaimer: This model is designed for research and evaluation purposes. Users are responsible for ensuring their use complies with applicable laws and regulations.

🤝 Contributing

We welcome contributions and feedback! Please feel free to:

• Report issues or bugs
• Suggest improvements
• Share your evaluation results
• Contribute enhancements

For more information, visit our GitHub repository.