mjbommar/magic-bert-50m-mlm

Magic-BERT 50M MLM

A BERT-style transformer model trained for binary file understanding using masked language modeling (MLM). This model learns byte-level patterns in binary files, including magic bytes, headers, and structural patterns across 106 file types.

Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches signatures at fixed offsets. Magic-BERT learns structural patterns throughout the file, enabling use cases where you don't have clean file boundaries:

• Network streams: Classifying packet payloads mid-connection, before headers arrive
• Disk forensics: Identifying file types during carving, when scanning raw disk images without filesystem metadata
• Fragment analysis: Working with partial files, slack space, or corrupted data
• Adversarial contexts: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

Model Description

Magic-BERT uses a custom BERT architecture with absolute position embeddings, trained on binary file data using a byte-level BPE tokenizer. The MLM objective teaches the model to predict masked bytes given surrounding context, which implicitly learns file format structure.

Property	Value
Parameters	59M
Hidden Size	512
Layers	8
Attention Heads	8
Max Sequence Length	512 tokens
Vocabulary Size	32,768 (byte-level BPE)
Position Encoding	Absolute (learned embeddings)

Tokenizer

The tokenizer uses the Binary BPE methodology introduced in Bommarito (2025). The original Binary BPE tokenizers (available at mjbommar/binary-tokenizer-001-64k) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types including documents, images, audio/video, archives, and source code.

Intended Uses

Primary use cases:

• Fill-mask: Predicting missing bytes in binary files
• Magic byte and file signature recognition
• Feature extraction for downstream classification
• Research on binary file structure

Example tasks:

• Completing partial file headers
• Identifying file type from structure
• Anomaly detection in binary data

Detailed Use Cases

Network Traffic Analysis

When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Magic-BERT's structural understanding can identify file types from interior content.

Disk Forensics & File Carving

During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. Magic-BERT can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

Incident Response

Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

Embedded Content Detection

Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings from Magic-BERT enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.

Training

Data

Trained on a diverse corpus of binary files spanning 106 MIME types, including:

• Documents (PDF, Office formats, OpenDocument)
• Images (PNG, JPEG, GIF, WebP, TIFF)
• Audio/Video (MP3, MP4, WebM, FLAC)
• Archives (ZIP, GZIP, 7z, TAR)
• Executables (ELF, PE, Mach-O)
• And 90+ additional formats

Procedure

Phase	Steps	Learning Rate	Batch Size	Objective
MLM Pre-training	100,000	1e-4	240	Masked LM (15% masking)

Data augmentation: 50% of samples use random byte offset to reduce position bias.

Evaluation Results

Perplexity by Region

Region	Perplexity
Magic Bytes (0-9)	1.07
Header (10-49)	1.06
Body (50+)	1.05
Overall	1.05

Fill-Mask Accuracy

Metric	Value
Top-1 Accuracy	58.9%
Top-5 Accuracy	73.5%
Mean Reciprocal Rank	0.67

Representation Quality

Metric	Value
Linear Probe Accuracy	87.0%
Silhouette Score	0.39
Separation Ratio	2.78

Architecture: Absolute vs Rotary Position Embeddings

This model uses absolute position embeddings, where each position (0-511) has a learned embedding vector added to the token embedding. This is the original BERT approach.

An alternative is Rotary Position Embeddings (RoPE), used by the RoFormer variant. RoPE encodes relative position through rotation matrices applied to query and key vectors in attention, rather than learning absolute position vectors.

Key finding from our experiments: Both approaches show similar position bias (~47-48% accuracy drop at offset 1000). Position bias is primarily a data distribution issue (files naturally start at offset 0) rather than an architecture limitation.

Aspect	Magic-BERT (this)	RoFormer
Position Encoding	Absolute (learned)	RoPE (rotary)
Parameters	59M	42.3M
Perplexity	1.05	1.13
Fill-mask Top-1	58.9%	61.8%
Probing Accuracy	87.0%	85.0%

Magic-BERT achieves slightly better perplexity and probing accuracy, while RoFormer achieves better fill-mask accuracy with fewer parameters.

MLM vs Classification: Two-Phase Training

This is the Phase 1 (MLM) model. The training pipeline has two phases:

Phase	Model	Task	Purpose
Phase 1	This model	Masked Language Modeling	Learn byte-level patterns and file structure
Phase 2	magic-bert-50m-classification	Contrastive Learning	Optimize embeddings for file type discrimination

When to use each:

• Use this model (MLM) for: fill-mask tasks, research, or as a base for custom fine-tuning
• Use classification model for: file type detection, similarity search, production classification

How to Use

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-mlm")

model.eval()

# Read a binary file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")

# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1  # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id

# Predict masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0, mask_pos].topk(5)

print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))

Getting Embeddings

from transformers import AutoModel

# Load base model for embeddings
model = AutoModel.from_pretrained(
    "mjbommar/magic-bert-50m-mlm",
    trust_remote_code=True
)

with torch.no_grad():
    outputs = model(**inputs)
    # CLS token embedding
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [batch_size, 512]
    # Mean pooling
    mean_embedding = outputs.last_hidden_state.mean(dim=1)  # [batch_size, 512]

Limitations

1. Position bias: The model performs best when file content starts at position 0. Accuracy drops ~48% when content starts at offset 1000. This reflects training data distribution, not architectural limitations.

2. Sequence length: Limited to 512 tokens. Longer files require truncation or chunking.

3. Text files: Lower performance on high-entropy or highly variable content (e.g., encrypted data, random bytes).

4. Domain specificity: Trained on common file formats; may not generalize to rare or proprietary formats.

Model Selection Guide

Use Case	Recommended Model	Reason
Fill-mask / byte prediction	This model	Best perplexity (1.05)
Research baseline	This model	Established BERT architecture
Classification + fill-mask	magic-bert-50m-classification	Retains 41.8% fill-mask capability
Production classification	magic-bert-50m-roformer-classification	Highest accuracy (93.7%), efficient (42M params)

Related Models

• magic-bert-50m-classification: Same architecture fine-tuned for classification (89.7% accuracy)
• magic-bert-50m-roformer-mlm: RoFormer variant with rotary position embeddings
• magic-bert-50m-roformer-classification: RoFormer variant fine-tuned for classification (93.7% accuracy, recommended for production)

Related Work

This model builds on the Binary BPE tokenization approach:

• Binary BPE Paper: Bommarito (2025) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
• Binary BPE Tokenizers: Pre-trained tokenizers for executables are available at mjbommar/binary-tokenizer-001-64k.

Key difference: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}