AdithyaSK commited on
Commit
56d5d23
·
verified ·
1 Parent(s): 90497a2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -58
README.md CHANGED
@@ -9,63 +9,7 @@ tags:
9
  - retrieval
10
  - colbert
11
  - late-interaction
12
- pipeline_tag: image-text-to-text
13
  ---
14
 
15
- # Merged ColGemma3 Model
16
-
17
- This model is a merged version of multiple ColGemma3 models using the **linear** merging technique.
18
-
19
- ## Source Models
20
-
21
- 1. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert)
22
- 2. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert)
23
-
24
-
25
- ## Merge Method: LINEAR
26
-
27
- Linear interpolation: Weighted average of model parameters.
28
-
29
- ## Model Architecture
30
-
31
- ColGemma3 is a vision-language model for late interaction retrieval:
32
- - **Base**: Gemma3 vision-language model
33
- - **Vision Encoder**: Processes images into patch embeddings
34
- - **Custom Projection**: Projects embeddings to 128 dimensions
35
- - **Retrieval**: Uses MaxSim scoring for multi-vector retrieval
36
-
37
- ## Usage
38
-
39
- ```python
40
- from colpali_engine.models.gemma3.colgemma3 import ColGemma3, ColGemmaProcessor3
41
- from PIL import Image
42
- import torch
43
-
44
- # Load model and processor
45
- model = ColGemma3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1", torch_dtype=torch.bfloat16, device_map="auto")
46
- processor = ColGemmaProcessor3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1")
47
-
48
- # Process images
49
- images = [Image.open("document.png")]
50
- batch_images = processor.process_images(images).to(model.device)
51
-
52
- # Process queries
53
- queries = ["What is this document about?"]
54
- batch_queries = processor.process_queries(queries).to(model.device)
55
-
56
- # Generate embeddings
57
- with torch.no_grad():
58
- img_embeddings = model(**batch_images)
59
- query_embeddings = model(**batch_queries)
60
-
61
- # Compute similarity scores
62
- scores = processor.score([query_embeddings[0]], [img_embeddings[0]])
63
- ```
64
-
65
- ## Citation
66
-
67
- If you use this model, please cite the original ColGemma3 work and the source models.
68
-
69
- ---
70
-
71
- *This model was automatically merged using [Modal](https://modal.com) infrastructure.*
 
9
  - retrieval
10
  - colbert
11
  - late-interaction
12
+ pipeline_tag: visual-document-retrieval
13
  ---
14
 
15
+ # ColNetraEmbed