HiTZ/whisper-large-v2-ca

Whisper Large-V2 Catalan

Model summary

Whisper Large-V2 Catalan is an automatic speech recognition (ASR) model for Catalan (ca) speech. It is fine-tuned from [openai/whisper-large-v2] on the Catalan subset of Mozilla Common Voice 13.0, achieving a Word Error Rate (WER) of 4.6716% on the evaluation split.

This variant provides improved transcription accuracy and model efficiency compared to the original large model, leveraging enhancements from the V2 architecture.

---

Model description

• Architecture: Transformer-based encoder–decoder (Whisper V2)
• Base model: openai/whisper-large-v2
• Language: Catalan (ca)
• Task: Automatic Speech Recognition (ASR)
• Output: Text transcription in Catalan
• Decoding: Autoregressive sequence-to-sequence decoding

Fine-tuned to optimize transcription quality on Catalan audio.

---

Intended use

Primary use cases

• High-accuracy transcription of Catalan audio
• Research and development in Catalan ASR
• Media, educational, or accessibility applications

Out-of-scope use

• Real-time transcription without optimization
• Speech translation
• Safety-critical applications without further validation

---

Limitations and known issues

• Performance may degrade on:
  • Noisy or low-quality recordings
  • Conversational or spontaneous speech
  • Regional dialects underrepresented in Common Voice
• Occasional transcription errors on challenging audio

---

Training and evaluation data

• Dataset: Mozilla Common Voice 13.0 (Catalan subset)

• Data type: Crowd-sourced, read speech

• Preprocessing:

  • Audio resampled to 16 kHz
  • Text normalized using Whisper tokenizer
  • Filtering of invalid or problematic samples

• Evaluation metric: Word Error Rate (WER) on held-out evaluation set

---

Evaluation results

Metric	Value
WER (eval)	4.6716%

---

Training procedure

Training hyperparameters

• Learning rate: 1e-5
• Optimizer: Adam (β1=0.9, β2=0.999, ε=1e-8)
• LR scheduler: Linear
• Warmup steps: 500
• Training steps: 20,000
• Train batch size: 32
• Eval batch size: 16
• Gradient accumulation steps: 2
• Seed: 42

Training results (summary)

Training Loss	Epoch	Step	Validation Loss	WER
0.1072	1.02	1000	0.1637	7.0329
0.0239	3.02	2000	0.1784	7.0277
0.0507	5.02	3000	0.1754	6.5773
0.0571	7.02	4000	0.1620	6.5047
0.0193	9.02	5000	0.1821	6.4887
0.0625	11.02	6000	0.1443	6.7585
0.0752	13.02	7000	0.1653	5.9097
0.0359	15.02	8000	0.1406	5.8760
0.0565	17.01	9000	0.1496	5.9680
0.0196	19.01	10000	0.1788	5.2746
0.0215	21.01	11000	0.1539	5.3895
0.0178	23.01	12000	0.1800	5.3764
0.0114	25.01	13000	0.1709	5.2078
0.0123	27.01	14000	0.1827	5.2003
0.0337	29.01	15000	0.1553	5.3655
0.0108	31.01	16000	0.1476	4.9151
0.0194	33.01	17000	0.1396	4.8477
0.0472	35.0	18000	0.1202	4.8717
0.0401	37.0	19000	0.1494	4.6716
0.0127	39.0	20000	0.1187	4.7276

---

Framework versions

• Transformers 4.33.0.dev0
• PyTorch 2.0.1+cu117
• Datasets 2.14.4
• Tokenizers 0.13.3

---

How to use

from transformers import pipeline

hf_model = "HiTZ/whisper-large-v2-ca"  # replace with actual repo ID
device = 0  # set to -1 for CPU

pipe = pipeline(
    task="automatic-speech-recognition",
    model=hf_model,
    device=device
)

result = pipe("audio.wav")
print(result["text"])

---

Ethical considerations and risks

• This model transcribes speech and may process personal data.
• Users should ensure compliance with applicable data protection laws (e.g., GDPR).
• The model should not be used for surveillance or non-consensual audio processing.

---

Citation

If you use this model in your research, please cite:

@misc{dezuazo2025whisperlmimprovingasrmodels,
  title={Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages},
  author={Xabier de Zuazo and Eva Navas and Ibon Saratxaga and Inma Hernáez Rioja},
  year={2025},
  eprint={2503.23542},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Please, check the related paper preprint in arXiv:2503.23542 for more details.

---

License

This model is available under the Apache-2.0 License. You are free to use, modify, and distribute this model as long as you credit the original creators.

---

Contact and attribution

• Fine-tuning and evaluation: HiTZ/Aholab - Basque Center for Language Technology
• Base model: OpenAI Whisper
• Dataset: Mozilla Common Voice

For questions or issues, please open an issue in the model repository.