Automated VLM Annotations with Gemma4
Offering a prediction API that returns detected objects and bounding boxes I now want to extend this service by adding a more detailed image description from a Vision-Language model like nanoVLM or SmolVLM. The first step is to use my custom dataset with object detection labels to generate image descriptions using a big model like Gemma4. This dataset with extended annotation can then be used to train a small model that can be run on an embedded device or as an easy drop in to a Cloud service that doesn´t kill the throughput.
Locally Hosted Annotation Model
The easiest way to get the image description to be used to train the small model is to host a local VLM with oLLaMa:
curl -fsSL https://ollama.com/install.sh | sh
Once installed, I will start with trying a smaller Gemma4 model:
ollama pull gemma4:e4b
Annotation Pipeline
Data Input/Output
My input annotation are in the YOLO format:
0 0.755238 0.563542 0.205773 0.872917
12 0.819812 0.298639 0.076305 0.160000
Where the float numbers are nomralized coordinates of the bounding box around the detected object and the first integer number can be mapped to a list of class names:
[ 'person','car','motor','train','truck','dog','cat','van','bird','deer','bus','plate','face','bike','plane','boat','helicopter','horse','cow' ]
I can add this additional information to my Gemma4 prompt to make sure that the VLM focuses on those objects and does not get lost describing the background of the image in too much details. But instead of the raw box coordinates I will translate the YOLO coordinates (x_center, y_center) into spatial regions, dividing the image into 4 quadrants:
x < 0.5 and y < 0.5 -> upper leftx >= 0.5 and y < 0.5 -> upper rightx < 0.5 and y >= 0.5 -> lower leftx >= 0.5 and y >= 0.5 -> lower right
For the two annotation give above this translates to:
- Object 1 :
- Annotation line: 0 0.755238 0.563542 0.205773 0.872917
- What it is: Person (Index 0).
- Where it is: Lower right quadrant.
- Why: The center coordinates are $x = 0.755$ (well past the $0.5$ middle mark to the right) and $y = 0.563$ (just below the $0.5$ middle mark downwards).
- Size: It is quite a large detection vertically, taking up roughly 87% of the image's total height (h = 0.872).
- Object 2:
- Annotation line: 1 0.819812 0.298639 0.076305 0.160000
- What it is: Face (Index 12).
- Where it is: Upper right quadrant.
- Why: The center coordinates are x = 0.819 (far to the right) and y = 0.298 (high up in the top half of the frame).
- Size: This is a much smaller object, spanning only about 7.6% of the width and 16% of the image height.
When parsed by the logic script, this exact file results in the following appended text statement for Gemma4:
"Focus on the key objects the image contains: Car in the upper right, Person in the lower right."
To train a smaller VLM like nanoVLM or smolVLM I need to output a standard VLM instruction-tuning format (like LLaVA or Hugging Face's multi-modal datasets):
{
"image": "path/to/image.jpg",
"conversations": [
{ "from": "human", "value": "<image>\nTranslate the visual scene into detailed English and German descriptions, accompanied by core keywords. Focus on the key objects the image contains: Car in the upper right, Person in the lower right." },
{ "from": "gpt", "value": "{\"description_en\": \"...\", \"description_de\": \"...\", \"keywords_en\": [...] }", \"keywords_de\": [...] }" }
]
}
Python Code
import os
import json
from pydantic import BaseModel, Field
import ollama
# CONFIGURATION
IMAGES_DIR = "./images" # Folder containing image files
LABELS_DIR = "./labels" # Folder containing corresponding .txt files
OUTPUT_FILE = "vlm_training_dataset.jsonl"
MODEL_NAME = "gemma4:e4b" # Change to your local model tag
# Map YOLO class index to your string labels
CLASS_LIST = ['person','car','motor','train','truck','dog','cat','van','bird','deer','bus','plate','face','bike','plane','boat','helicopter','horse','cow']
# PYDANTIC SCHEMA
class ImageAnnotationSchema(BaseModel):
description_en: str = Field(description="A concise description of the scene in English.")
description_de: str = Field(description="A concise description of the scene in German.")
keywords_en: list[str] = Field(description="A list of 5-10 relevant keywords capturing the core scene in English.")
keywords_de: list[str] = Field(description="A list of 5-10 relevant keywords capturing the core scene in German.")
def get_spatial_description(txt_path, classes):
"""
Parses a YOLO file, maps classes, determines their general spatial
quadrant, and returns a string of unique object-location descriptions.
"""
if not os.path.exists(txt_path):
return "" # Background image or missing label file
unique_spatial_objects = set()
with open(txt_path, 'r') as f:
for line in f:
parts = line.strip().split()
if len(parts) != 5:
continue
class_id = int(parts[0])
x_center = float(parts[1])
y_center = float(parts[2])
# Safely get class label
class_name = classes[class_id].capitalize() if class_id < len(classes) else "Object"
# Map normalized coordinates to quadrants
horizontal = "left" if x_center < 0.5 else "right"
vertical = "upper" if y_center < 0.5 else "lower"
quadrant = f"{vertical} {horizontal}"
unique_spatial_objects.add(f"{class_name} in the {quadrant}")
if not unique_spatial_objects:
return ""
# Format as: "Person in the lower left, Car in the lower right"
return ", ".join(sorted(list(unique_spatial_objects)))
def process_dataset():
# Find all supported images
supported_extensions = ('.jpg', '.jpeg', '.png')
image_files = [f for f in os.listdir(IMAGES_DIR) if f.lower().endswith(supported_extensions)]
print(f" INFO :: Found {len(image_files)} images to process...")
for img_name in image_files:
img_path = os.path.join(IMAGES_DIR, img_name)
base_name = os.path.splitext(img_name)[0]
txt_path = os.path.join(LABELS_DIR, f"{base_name}.txt")
# Parse spatial objects from labels directory
spatial_objects = get_spatial_description(txt_path, CLASS_LIST)
# Build conditional instructions
base_instruction = "Translate the visual scene into detailed English and German descriptions, accompanied by core keywords in both respective languages."
if spatial_objects:
user_instruction = f"{base_instruction} Focus on the key objects the image contains: {spatial_objects}. Especially point out identifying features like colors, destinctive clothing or vehicle model types."
else:
user_instruction = base_instruction
system_prompt = (
"You are an expert security data annotation agent who has to point out anomalies, suspicious activities and identifying features of observed persons, animals or vehicles. Generate a concise, highly accurate description of the scene, paying specific attention to the actions, context, and positions of the listed objects. Do not invent objects not listed or visible."
)
print(f"Processing image: {img_name} (Has detections: {bool(spatial_objects)})")
try:
# Call Ollama utilizing structured JSON mode
response = ollama.chat(
model=MODEL_NAME,
messages=[
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": user_instruction,
"images": [img_path]
}
],
format=ImageAnnotationSchema.model_json_schema()
)
# Parse output to ensure structural compliance
structured_data = ImageAnnotationSchema.model_validate_json(response.message.content)
# Format exactly for fine-tuning our small VLM later
training_sample = {
"image": img_path,
"conversations": [
{
"from": "human",
"value": f"<image>\n{user_instruction}"
},
{
"from": "gpt",
"value": structured_data.model_dump_json()
}
]
}
# OPEN AND CLOSE THE FILE INSTANTLY FOR EVERY IMAGE (Safe & unbuffered)
with open(OUTPUT_FILE, "a", encoding="utf-8") as out_f:
out_f.write(json.dumps(training_sample, ensure_ascii=False) + "\n")
print(f"INFO :: {img_name} processed and annotations saved")
except Exception as e:
print(f"ERROR :: Failed processing {img_name}: {e}")
if __name__ == "__main__":
process_dataset()
Results
An example output:
{"image": "./images/1008_Helicopter.jpg", "conversations": [{"from": "human", "value": "<image>\nTranslate the visual scene into detailed English and German descriptions, accompanied by core keywords in both respective languages. Focus on the key objects the image contains: Helicopter in the upper right, Person in the upper right. Especially point out identifying features like colors, destinctive clothing or vehicle model types."}, {"from": "gpt", "value": "{\"description_en\":\"A large, dark-colored military utility helicopter, identified by 'SIKORSKY' markings, is shown flying or hovering at a very low altitude over a stationary, light-colored rectangular building. The aircraft dominates the upper frame and features complex rotor systems. The structure below has a clean, functional design with visible square windows along its facade. **Observation:** The helicopter maintains proximity to the facility, suggesting active operation, surveillance, or tactical transport. No identifiable individuals are visible in the immediate vicinity or upper right quadrant.\",\"description_de\":\"Ein großes, dunkel gefärbtes Militär-Universalhelikopter, erkennbar an den 'SIKORSKY'-Markierungen, ist sehr tief über ein stationäres, hellfarbiges rechteckiges Gebäude zu fliegen oder schweben. Das Fluggerät dominiert das obere Bilddrittel und verfügt über komplexe Rotorsysteme. Die Struktur darunter hat ein sauberes, funktionales Design mit sichtbaren quadratischen Fenstern entlang ihrer Fassade. **Beobachtung:** Der Helikopter hält eine Nähe zum Gelände, was auf aktive Operationen, Überwachung oder taktischen Transport hindeutet. Es sind keine identifizierbaren Personen in der unmittelbaren Umgebung oder im oberen rechten Quadranten sichtbar.\",\"keywords_en\":[\"Military Helicopter\",\"Sikorsky\",\"Low Altitude Flight\",\"Utility Aircraft\",\"Rectangular Building\",\"Operational Scene\"],\"keywords_de\":[\"Militärhelikopter\",\"Sikorsky\",\"Niedrige Höhe\",\"Universalflugzeug\",\"Rechteckiges Gebäude\",\"Betriebsszene\"]}"}]}