---
library_name: transformers
datasets:
- laicsiifes/flickr30k-pt-br
language:
- pt
metrics:
- bleu
- rouge
- meteor
- bertscore
base_model:
- google/paligemma-3b-pt-224
pipeline_tag: image-text-to-text
model-index:
- name: PaliGemma-Flickr30K-Translated
  results:
  - task:
      name: Image Captioning
      type: image-text-to-text
    dataset:
      name: Flickr30K
      type: laicsiifes/flickr30k-pt-br
      split: test
    metrics:
    - name: CIDEr-D
      type: cider
      value: 47.25
    - name: BLEU@4
      type: bleu
      value: 18.63
    - name: ROUGE-L
      type: rouge
      value: 39.69
    - name: METEOR
      type: meteor
      value: 48.68
    - name: BERTScore
      type: bertscore
      value: 69.87
    - name: CLiP-Score
      type: clipscore
      value: 60.13
license: mit
---

# 🎉 PaliGemma fine-tuned in Flickr30K Translated for Brazilian Portuguese Image Captioning

PaliGemma (google/paligemma-3b-pt-224) model fine-tuned for image captioning on [Flickr30K Portuguese](https://huggingface.co/datasets/laicsiifes/flickr30k-pt-br) (translated version using Google Translator API).


## 🤖 Model Description


## 🧑‍💻 How to Get Started with the Model

Use the code below to get started with the model.

- **Install libraries:**

```bash
pip install transformers==4.45.2 bitsandbytes==0.45.2 peft==0.13.2
```

- **Python code:**

```python
import requests
import torch
from PIL import Image

from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig
from huggingface_hub import login

# Use your HuggingFace API key, since Paligemma is available through user form submission
login('hf_...')

# load a fine-tuned image captioning model, and corresponding tokenizer and image processor
model = PaliGemmaForConditionalGeneration.from_pretrained(
    'google/paligemma-3b-pt-224',
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type='nf4',
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map={'':0}
)
model.load_adapter('laicsiifes/paligemma-flickr30k_pt')
processor = PaliGemmaProcessor.from_pretrained('laicsiifes/paligemma-flickr30k_pt', trust_remote_code=True)

# preprocess an image
image = Image.open(requests.get("http://images.cocodataset.org/val2014/COCO_val2014_000000458153.jpg", stream=True).raw)
inputs = processor(
    text='caption pt\n',
    images=image,
    return_tensors='pt'
).to('cuda:0')

# generate caption
generated_ids = model.generate(**inputs, max_new_tokens=25)
prediction = generated_ids[:, inputs['input_ids'].shape[1]:].tolist()
generated_text = processor.batch_decode(prediction, skip_special_tokens=True)[0]
```

```python
import matplotlib.pyplot as plt

# plot image with caption
plt.imshow(image)
plt.axis("off")
plt.title(generated_text)
plt.show()
```

![image/png](example.png)

## 📈 Results

The evaluation metrics: CIDEr-D, BLEU@4, ROUGE-L, METEOR, BERTScore
(using [BERTimbau](https://huggingface.co/neuralmind/bert-base-portuguese-cased)), and CLIP-Score (using [CAPIVARA](https://huggingface.co/hiaac-nlp/CAPIVARA)).

| Model | #Params | CIDEr | BLEU-4 | ROUGE-L | METEOR | BERTScore | CLIP-Score |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **ViTucano 1B** | 1.53B | 78.78 | 26.29 | 46.67 | 50.25 | 73.44 | 55.48 |
| **ViTucano 2B** | 2.88B | **80.03** | **27.32** | **47.31** | **50.90** | **73.69** | 56.16 |
| **PaliGemma** | 2.92B | 47.25 | 18.63 | 39.69 | 48.68 | 69.87 | **60.13** |
| **Phi-3 V** | 4.15B | 75.47 | 27.26 | 47.24 | 48.24 | 73.20 | 55.44 |
| **LLaMa 3.2 V** | 11.70B | 70.94 | 23.97 | 45.05 | 47.13 | 72.71 | 55.65 |

## 📋 BibTeX entry and citation info

Coming soon. For now, please reference the model adapter using its Hugging Face link.