DE-T5-Sci-Transfer-15k

Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: cross_lingual_transfer/logs/train/.../step-step=015000.ckpt.

Model Details

• Base: EN-T5-Sci weights, German tokenizer
• Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup 1.5k, grad clip 1.0
• Effective batch: 48 (per-GPU 48, grad accumulation 1)
• Objective: Span corruption (15 % noise, mean span 3)

Training Data

German subset of the Unpaywall-derived scientific corpus (sliding windows 512 tokens, 50 % overlap). Same cleaning pipeline as the English run.

Evaluation (Global-MMLU, zero-shot)

Metric	EN	DE
Overall accuracy	0.2738	0.2700
Humanities	0.2559	0.2536
STEM	0.2867	0.2851
Social Sciences	0.3058	0.3055
Other	0.2562	0.2443

This is the best-performing German checkpoint across both languages in the final evaluation (evaluation_results/scientific_crosslingual_transfer_eval_full_15k).

Intended Use

Zero-shot QA in German/English scientific domains, or as a strong starting point for German task-specific fine-tuning (NER, relation extraction, etc.).

Limitations

• Still inherits T5-base context length/parameter budget.
• Evaluated only on Global-MMLU; downstream fine-tuning recommended for specialized tasks.
• Training corpus is domain-specific (scientific); may underperform on casual text.

Citation

Please cite the thesis (Nikolas Rauscher, 2025) and the WECHSEL paper (Minixhofer et al. 2022).