Training a small VLM
- Training
- Testing the Torch model
- Export to OpenVino
- Testing the OpenVino models
- Deployment
- Python Code
I found the VLM tutorial by @mrdbourke - VLM Fine Tune Tutorial that explains how to fine tune SmolVLM2-500M-Video-Instruct using a custom dataset.
I started by generating VLM annotations using Gemma4 for my custom dataset with the goal of deploying the code in OpenVino format to an Intel Core i9 server.
But then I ran into a bunch of issues trying to use Daniel Bourkes code on an acient double-1080GTX system :( The modified code can be found in my Github repository:
- Github
main.py: Create annotation for custom YOLO dataset using Gemma4. I am using a YOLO object detector to create bboxes and then use those annotation to point out points-of-interest in my dataset.training/main.py: Use the created Gemma annotation to train the SmolVLM model.
If you have more recent hardware ~ then there is nothing to see here... follow the original code instead!
Training
SmolVLM2-500M-Observer
{'loss': 0.2905, 'grad_norm': 1.0703125, 'learning_rate': 0.0002, 'epoch': 5.99}
{'loss': 0.3097, 'grad_norm': 1.171875, 'learning_rate': 0.0002, 'epoch': 6.0}
{'loss': 0.294, 'grad_norm': 0.90625, 'learning_rate': 0.0002, 'epoch': 6.0}
{'loss': 0.29, 'grad_norm': 1.09375, 'learning_rate': 0.0002, 'epoch': 6.0}
{'eval_loss': 0.4115491211414337, 'eval_runtime': 25300.6761, 'eval_samples_per_second': 0.314, 'eval_steps_per_second': 0.157, 'epoch': 6.0}
6%|████████████ | 95190/1586500 [308:44:49<4893:50:53, 11.81s/it
SmolVLM-256M-Observer
{'loss': 0.423, 'grad_norm': 1.171875, 'learning_rate': 0.0002, 'epoch': 2.99}
{'loss': 0.4026, 'grad_norm': 1.1796875, 'learning_rate': 0.0002, 'epoch': 3.0}
{'loss': 0.4048, 'grad_norm': 1.109375, 'learning_rate': 0.0002, 'epoch': 3.0}
{'loss': 0.4219, 'grad_norm': 1.078125, 'learning_rate': 0.0002, 'epoch': 3.0}
{'loss': 0.404, 'grad_norm': 1.0, 'learning_rate': 0.0002, 'epoch': 3.0}
{'eval_loss': 0.46206653118133545, 'eval_runtime': 21894.3322, 'eval_samples_per_second': 0.362, 'eval_steps_per_second': 0.181, 'epoch': 3.0}
3%|██▋ | 47595/1586500 [116:25:59<3097:17:33, 7.25s/it
SmolVLM2-256M-Observer
{'loss': 0.6415, 'grad_norm': 1.328125, 'learning_rate': 0.0002, 'epoch': 0.98}
{'loss': 0.6239, 'grad_norm': 1.578125, 'learning_rate': 0.0002, 'epoch': 0.98}
{'loss': 0.6161, 'grad_norm': 1.453125, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.653, 'grad_norm': 1.5390625, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.585, 'grad_norm': 1.3125, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.6205, 'grad_norm': 1.4453125, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.6124, 'grad_norm': 1.5859375, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.6171, 'grad_norm': 1.5, 'learning_rate': 0.0002, 'epoch': 0.99}
{'loss': 0.6098, 'grad_norm': 1.7421875, 'learning_rate': 0.0002, 'epoch': 1.0}
{'loss': 0.6316, 'grad_norm': 1.46875, 'learning_rate': 0.0002, 'epoch': 1.0}
{'loss': 0.6125, 'grad_norm': 1.3046875, 'learning_rate': 0.0002, 'epoch': 1.0}
{'eval_loss': 0.6205337047576904, 'eval_runtime': 20795.0222, 'eval_samples_per_second': 0.381, 'eval_steps_per_second': 0.191, 'epoch': 1.0}
1%|▉ | 15864/1586400 [33:58:00<2149:50:37, 4.93s/it
{'loss': 0.6645, 'grad_norm': 1.6328125, 'learning_rate': 0.0002, 'epoch': 1.0}
{'loss': 0.5624, 'grad_norm': 1.3515625, 'learning_rate': 0.0002, 'epoch': 1.0}
{'loss': 0.5539, 'grad_norm': 1.4453125, 'learning_rate': 0.0002, 'epoch': 1.0}
Testing the Torch model
> python training/testing_trained_torch_model.py
INFO :: Trying to load fine tuned model from ../training/SmolVLM2-500M-Observer/checkpoint-47595
processor_config.json: 100%|████████████████████████████████████████████| 67.0/67.0 [00:00<00:00, 594kB/s]
chat_template.json: 100%|████████████████████████████████████████████████| 430/430 [00:00<00:00, 2.39MB/s]
preprocessor_config.json: 100%|██████████████████████████████████████████| 599/599 [00:00<00:00, 4.87MB/s]
tokenizer_config.json: 28.6kB [00:00, 9.22MB/s]
vocab.json: 801kB [00:00, 1.67MB/s]
merges.txt: 466kB [00:00, 7.22MB/s]
tokenizer.json: 3.55MB [00:00, 17.4MB/s]
added_tokens.json: 4.74kB [00:00, 1.92MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████| 868/868 [00:00<00:00, 5.92MB/s]
Device set to use cpu
INFO :: Trying to get random sample data from motion_vlm_training_dataset.jsonl
INFO :: Starting to work with random sample: {'messages': [{'role': 'system', 'content': [{'type': 'text', 'text': 'You are an expert security data annotation agent who has to point out anomalies, suspicious activities and identifying features of observed persons, animals or vehicles. Generate a concise, highly accurate description of the scene, paying specific attention to the actions, context, and positions of the listed objects. Do not invent objects not listed or visible.'}]}, {'role': 'user', 'content': [{'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=500x333 at 0x7FD46A62E510>}, {'type': 'text', 'text': 'Translate the visual scene into detailed English and German descriptions, accompanied by core keywords in both respective languages. Focus on the key objects the image contains: Cat in the lower left. Especially point out identifying features like colors, destinctive clothing or vehicle model types.\nOnly return valid JSON in he following form:\n\n```json\n{\n "description_en": "", # type string - the image description in english\n "description_de": "", # type string - the image description in german\n "keywords_en": [], # type list of strings - single english keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database\n "keywords_de": [], # type list of strings - single german keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database\n}\n```\n'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': '{"description_en":"A highly detailed annotation of a single domestic cat positioned prominently in the lower-left and center of the frame. The feline is characterized by distinct brown and black tabby striping patterns across its entire coat. It exhibits bright, striking greenish-yellow eyes with an alert, direct gaze directed towards the viewer. The posture suggests vigilance and attention. The background environment indicates an indoor setting near a window with vertical slat blinds or curtains visible directly behind the animal. Natural light enters from the right side, illuminating the cat\'s fur and creating highlights. No human activity or suspicious objects are present.","description_de":"Eine hochdetaillierte Annotation einer einzelnen Hauskatze, die prominent im linken unteren und mittleren Bereich des Bildes positioniert ist. Das Tier zeichnet sich durch deutliche braune und schwarze Tabby-Streifenmuster über seinem gesamten Fell aus. Es weist auffallend helle, grün-gelbe Augen mit einem wachsamen, direkten Blick auf den Betrachter auf. Die Haltung lässt Wachsamkeit und Aufmerksamkeit vermuten. Die Hintergrundumgebung deutet auf einen Innenraum in der Nähe eines Fensters hin, hinter dem vertikale Jalousien oder Vorhänge sichtbar sind. Natürliches Licht fällt von der rechten Seite ein und beleuchtet das Fell der Katze sowie die Umgebung. Es sind keine menschlichen Aktivitäten oder verdächtigen Objekte vorhanden.","keywords_en":["Cat","Feline","Tabby Cat","Brown and Black Stripes","Greenish-Yellow Eyes","Alert Stance","Indoor Setting","Blinds/Curtains"],"keywords_de":["Katze","Felin","Tabbykatze","Braun und Schwarz Streifen","Grünlich-gelbe Augen","Wache Haltung","Innenraum","Jalousien/Vorhänge"]}'}]}]}
INFO :: Ideal output for selected sample:
{"description_en":"A highly detailed annotation of a single domestic cat positioned prominently in the lower-left and center of the frame. The feline is characterized by distinct brown and black tabby striping patterns across its entire coat. It exhibits bright, striking greenish-yellow eyes with an alert, direct gaze directed towards the viewer. The posture suggests vigilance and attention. The background environment indicates an indoor setting near a window with vertical slat blinds or curtains visible directly behind the animal. Natural light enters from the right side, illuminating the cat's fur and creating highlights. No human activity or suspicious objects are present.","description_de":"Eine hochdetaillierte Annotation einer einzelnen Hauskatze, die prominent im linken unteren und mittleren Bereich des Bildes positioniert ist. Das Tier zeichnet sich durch deutliche braune und schwarze Tabby-Streifenmuster über seinem gesamten Fell aus. Es weist auffallend helle, grün-gelbe Augen mit einem wachsamen, direkten Blick auf den Betrachter auf. Die Haltung lässt Wachsamkeit und Aufmerksamkeit vermuten. Die Hintergrundumgebung deutet auf einen Innenraum in der Nähe eines Fensters hin, hinter dem vertikale Jalousien oder Vorhänge sichtbar sind. Natürliches Licht fällt von der rechten Seite ein und beleuchtet das Fell der Katze sowie die Umgebung. Es sind keine menschlichen Aktivitäten oder verdächtigen Objekte vorhanden.","keywords_en":["Cat","Feline","Tabby Cat","Brown and Black Stripes","Greenish-Yellow Eyes","Alert Stance","Indoor Setting","Blinds/Curtains"],"keywords_de":["Katze","Felin","Tabbykatze","Braun und Schwarz Streifen","Grünlich-gelbe Augen","Wache Haltung","Innenraum","Jalousien/Vorhänge"]}
`generation_config` default values have been modified to match model-specific defaults: {'do_sample': True}. If this is not desired, please set these values explicitly.
INFO :: Generated output:
{"description_en":"The scene is an indoor, well-lit environment featuring a large, alert cat positioned prominently in the lower-left foreground. The animal displays a distinct brown and black tabby coat pattern (resembling a European Shorthair). It has vivid bright eyes and is oriented directly toward the viewer. The background indicates the presence of structural blinds or sheer curtains on the left side, suggesting an interior setting outside or near the natural light source. No anomalies or suspicious activities are observed.","description_de":"Die Szene ist eine gut beleuchtete Innenraumumgebung mit einem großen, wachsamen Katzen, der prominent im vorderen linken Bereich positioniert ist. Das Tier weist ein deutliches braun-schwarzes Tabby-Muster (ähnlich einem europäischen Joch-/Grauhaar) auf. Es hat leuchtendichte Augen und ist direkt auf den Betrachter ausgerichtet. Der Hintergrund deutet auf die Präsenz struktureller Jalousien oder einem entschränkten Vorhang auf der linken Seite hin und lässt auf einen Innenbereich außerhalb oder in der Nähe der natürlichen Lichtquelle schließen. Es sind keine Anomalien oder verdächtigen Aktivitäten zu beobachten.","keywords_en":["Cat","Tabby Pattern","Brown and Black Coat","Alert Position","Indoor Environment","Blinds/Curtains"],"keywords_de":["Katze","Tabby-Muster","Braun und Schwarzer Fell","Wache Position","Innenraum","Jalousien/Vorhang"]}
Export to OpenVino
pip install --upgrade optimum-intel openvino openvino-tokenizers nncf num2words optimum[openvino] optimum[onnxruntime]
optimum-cli export openvino \
--model ./SmolVLM-256M-Observer/checkpoint-31730 \
--task image-text-to-text \
--trust-remote-code \
openvino_smolvlm_f32_e5
optimum-cli export openvino \
--model ./SmolVLM2-500M-Observer/checkpoint-95190 \
--task image-text-to-text \
--weight-format int8 \
--trust-remote-code \
openvino_smolvlm2_int8_e5
optimum-cli export openvino \
--model ./SmolVLM2-500M-Observer/checkpoint-95190 \
--task image-text-to-text \
--weight-format int4 \
--group-size 64 \
--trust-remote-code \
openvino_smolvlm2_int4_e5
# SmolVLM-256M-Observer-8bit
## 255.9 MiB (268,309,044)
# SmolVLM2-500M-Observer-8bit
## 495.8 MiB (519,925,439)
Testing the OpenVino models
> python training/testing_trained_openvino_model_initial.py
INFO :: Getting random sample data from motion_vlm_training_dataset.jsonl
INFO :: Working with random sample...
INFO :: Ideal output:
{"description_en":"A natural scene captured in a vertical frame, featuring two main elements: a bird and flowering plants. \n\nBird: Located prominently in the lower left quadrant. The subject is a small passerine bird, observed either flying or mid-perch, with wings partially extended. Identifying features include mottled brown plumage on the back and wings, contrasting sharply with lighter tan underparts and a visible dark eye/head area. The posture suggests active movement.\n\nFlowers: Dominating the upper right quadrant is a dense cluster of vibrant flowers. These blooms are characterized by rich shades of red and orange, growing in cascading stalks. They belong to an ornamental or natural flora. \n\nContext: The background is uniformly soft-focus (bokeh), providing no contextual details other than suggesting an outdoor, vegetated environment.","description_de":"Eine natürliche Szene, aufgenommen in einem Hochformatbild, die zwei Hauptelemente zeigt: einen Vogel und blühende Pflanzen. \n\nVogel: Positioniert im linken unteren Quadranten. Das Motiv ist ein kleiner Singvogel, der entweder fliegt oder kurz vor dem Landen festgehalten wurde und dessen Flügel teilweise ausgefahren sind. Auffällige Merkmale umfassen gesprenkeltes braunes Gefieder am Rücken und an den Flügeln, das sich deutlich von den helleren Unterpartien und einem sichtbaren dunklen Augen-/Kopfbereich abhebt. Die Haltung lässt auf aktive Bewegung schließen.\n\nBlüten: Der obere rechte Quadrant wird von einem dichten Cluster lebendiger Blumen dominiert. Diese Blüten zeichnen sich durch satte Rot- bis Orangetöne aus und wachsen in kaskadierenden Stielen. Es handelt sich um eine Zierpflanze oder natürliche Flora. \n\nKontext: Der Hintergrund ist gleichmäßig unscharf (Bokeh), was keinerlei Kontextdetails außer einer vermutlichen, bewaldeten Umgebung liefert.","keywords_en":["Wildlife","Passerine bird","In flight/Perching","Brown plumage","Red and orange flowers","Natural flora","Upper right quadrant","Lower left quadrant"],"keywords_de":["Wildtiere","Singvogelart","Im Flug/Landen","Braunes Gefieder","Rot- und Orangenblüten","Naturflora","Oberes rechtes Quadranten","Linkes unteres Quadranten"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_f32_e5
INFO :: OpenVINO Generated Output:
{"description_en":"A single bird is prominently featured in the lower left quadrant of the frame. The bird exhibits mottled brown and tan plumage with distinct streaking on its chest and belly. It possesses a relatively short, pointed beak. The animal is observed in mid-flight, positioned low over the dense floral arrangement. The background consists of soft-focus, deep green foliage, suggesting an outdoor natural habitat.","description_de":"Ein einzelner Vogel ist prominent im unteren linken Quadranten des Bildes zu sehen. Der Vogel weist gemustertes braunes und hellbraunes Gefieder mit deutlicher Streifung auf Brust und Bauch auf. Er besitzt einen relativ kurzen, spitzer Schnabel. Das Tier wird in der Flugposition dargestellt, niedrig über die dichte Blumenarrangement positioniert. Der Hintergrund besteht aus unscharfem, tiefgrünem Laubwerk, was auf einen natürlichen Außenlebensraum hindeutet.","keywords_en":["Bird","Mottled brown plumage","Short beak","Mid-flight","Floral arrangement","Lower left quadrant"],"keywords_de":["Vogel","Gemustertes braunes Gefieder","Kurzer Schnabel","In der Flugposition","Blumenarrangement","Unteres linkes Viertel"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_int8_e5
INFO :: OpenVINO Generated Output:
{"description_en":"A single small bird is observed in mid-flight, positioned across the lower-left quadrant of the frame. The bird exhibits mottled brown and tan plumage with distinct streaking visible on its chest and belly. It possesses a relatively short, pointed beak. The background consists of dense, saturated green foliage, suggesting an outdoor natural habitat.","description_de":"Ein einzelner kleiner Vogel wird im Flug beobachtet und befindet sich im unteren linken Quadranten des Bildes. Der Vogel weist gemustertes braun-beige Gefieder mit deutlicher Streifung auf Brust und Bauch auf. Er besitzt einen relativ kurzen, spitzen Schnabel. Der Hintergrund besteht aus dichtem, gesättigtem grünem Laubwerk, was auf einen natürlichen Außenlebensraum hindeutet.","keywords_en":["Bird","In flight","Brown plumage","Lower left quadrant","Natural habitat"],"keywords_de":["Vogel","Im Flug","Braunes Gefieder","Unteres linkes Viertel","Natürlicher Lebensraum"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_int4_e5
INFO :: OpenVINO Generated Output:
{"description_en":"The scene captures a natural outdoor environment dominated by dense foliage and flowering plants. The primary visible object is a bright, vivid red flower structure (likely a fuchsia or similar flowering ornamental) positioned centrally and extending upwards toward the upper frame area. This flower structure acts as a focal point for the composition. In the lower left quadrant, a small, highly detailed, brownish-tan bird specimen is observed in motion, suggesting active flight or rapid movement across the foreground relative to its position relative to the main subject (lower center/left). This action suggests rapid passage through the frame’s depth of field or rapid movement through the frame’s depth of field. The background is heavily blurred (shallow depth of field), indicating dense foliage or flowering plants that dominate the mid-ground and background, creating high contrast and depth of field separation. The overall context suggests an outdoor, natural, and highly floral environment.","description_de":"Die Szene fängt eine Außenumgebung mit dichtem Laubwerk und Blütenpflanzen dominiert von einer überfähigen Begegnung der Flora und von Blütenwerkwerkstücken. Das primäre sichtbare Objekt ist die kontrastreiche und leuchtend rote Blume-/Blätterwerkstruktur (vermutlich eine Fuchsia- oder Blumengebrüschenkonstruktion), die im Zentrum und im oberen Bereich des Rahmens positioniert ist und sich durch den oberem Bildrand hineingebunden wird. Im linken unteren Quadranten wird ein kleines, detaillierter, bräunlich-beigefarbener Vogelspezimenortgebildung bei Bewegung beobachtet, was auf eine aktive Flughöhe oder Bewegung durch den Bereich des Hintergrunds hinweist (Richtung Mitte/Links). Die Bewegung erfolgend durch den Bereichsdunkel-/Überführungshintergrund und bewegt sich durch den stark verschwommenen Tiefenschwellenbildung des Bildes (Depth of Field / Low Light
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_f32_e1
INFO :: OpenVINO Generated Output:
{"description_en":"A small passerine bird is prominently featured in the lower left quadrant, actively flying or hovering near a cluster of bright red flowers. The bird exhibits mottled brown plumage on its head and breast, contrasting with lighter streaking visible on its back and wings. It is positioned low to the ground, facing slightly toward the right side of the frame. The surrounding context consists of dense, vibrant red foliage and blossoms, indicating a natural outdoor setting. No suspicious activity or anomalies are observed; the bird appears to be engaged in routine wildlife movement.","description_de":"Ein kleiner Singvogel ist prominent im unteren linken Quadranten zu sehen und fliegt oder schwebt in der Nähe eines Cluster leuchtend roten Blumen. Der Vogel weist gemustertes braunes Gefieder auf dem Kopf und Brustbereich auf, das einen Kontrast zu hellerem Streifmuster an Rücken und Flügeln bildet. Er ist niedrig am Boden positioniert und blickt leicht nach rechts aus dem Bild heraus. Der umgebende Kontext besteht aus dichtem, leuchtend roten Laubwerk und Blüten, was auf eine natürliche Außenumgebung hindeutet. Es sind keine verdächtigen Aktivitäten oder Anomalien festzustellen; der Vogel scheint ein routinemäßiger Wildtierbewegung zu sein.","keywords_en":["Bird","Passerine","Mottled brown plumage","Streaked wings","Flowers","Natural setting","Lower left quadrant"],"keywords_de":["Vogel","Singvogel","Gefiederbraunes Gefieder","Gestreifte Flügel","Blumen","Natürliche Umgebung","Unteres linkes Viertel"]}
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_int8_e1
INFO :: OpenVINO Generated Output:
{"description_en":"A small bird is prominently featured in the lower left quadrant, actively flying or hovering near a cluster of bright red and orange blossoms. The bird exhibits mottled brown plumage with distinct white markings on its chest and underbelly. Its beak is visible, and it appears to be moving from left to right. The surrounding context is an outdoor environment dominated by dense, vibrant red and orange blossoms, which are partially obscured by foliage and branches. No suspicious activity or anomalies are observed; the bird is simply transiting through the flora.","description_de":"Ein kleiner Vogel ist prominent im linken unteren Quadranten zu sehen und fliegt oder schwebt in der Nähe eines Cluster leuchtend roten und orangefarbenen Blüten. Der Vogel weist gemustertes braunes Gefieder mit deutlichen weißen Markierungen an Brust und Unterbauch auf. Sein Schnabel ist sichtbar, und er scheint von links nach rechts zu bewegen. Der umgebende Kontext ist eine Außenumgebung, die von dichtem, leuchtend roten und orangefarbenen Blüten dominiert wird, die teilweise von Laubwerk und Ästen verdeckt sind. Es sind keine verdächtigen Aktivitäten oder Anomalien festzustellen; der Vogel bewegt sich lediglich durch die Flora.","keywords_en":["Bird","Mottled brown plumage","Hovering/Flying","Blooms","Red and orange blossoms","Outdoor environment"],"keywords_de":["Vogel","Gefieder braun","Schwebend/Flyt","Blüten","Rot und orange Blüten","Außenumgebung"]}
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_int4_e1
INFO :: OpenVINO Generated Output:
{"description_en":"A medium-sized bird is prominently located in the lower left quadrant, actively flying or hovering near a cluster of bright red flowers. The bird exhibits streaky brown plumage across its back and wings, contrasting with mottled tan and brown feathers on its chest and belly area. Its head is held high, indicating high energy level. Multiple bright red petals surround the area where the bird is positioned. These petals are densely clustered around the main cluster of flowers, with multiple petals forming a dense cluster structure around the bird's body area. The background consists of soft-focus, dark green/dark brown foliage, while the foreground is dominated by out-of-focus (shades of deep red) bright red flowers and clusters of bright red petals. No other distinct birds or animals are visible in the upper left corner; only the foreground and background elements are present.","description_de":"Ein mittelgroßer Vogel befindet sich prominent im linken unteren Quadranten, der aktiv zu fliegen oder zu schwegen steht/gehen. Der Vogel weist einen gestreiften braunen Gefieder am Rücken und den Flügeln auf, kontrastierend mit ein gesprenkeltes, hellbraunes Fellfarben am Brust- und Bauchbereich. Seine Kopf ist erhörend gehalten, was auf einen hohes Energiebene hindeutet. Die mehrschneites (rot) bei der Vogel positioniert sind vorwiegend dicht zu seinem Haufen. Zusammen mit leuchtend roten Blüten im Bereich des Vogels, der am Vogel sichtbar sind, werden dargestellten (mit leuchtendem (rot) Rosé-Blumen) und Ansammlung von leuchtendem (rotes) Rotblumen sowie Ansammlung von leuchtendem (rot) Blumen. Es sind kein weitere einzelne Artvogel oder ein Hund zu sehen; es sind kein weitere Artvogel oder einen Gegenündungselementen im oberen linken Bereich vorhanden.","keywords_en":["Bird
> python training/inference_openvino_model.py
INFO :: Getting random sample data from motion_vlm_training_dataset.jsonl
INFO :: Working with random sample...
INFO :: Ideal output:
{"description_en":"A single adult male is captured in a three-quarter profile, positioned centrally in the frame. He has dark hair and appears middle-aged. He is dressed in formal attire consisting of a dark tailored suit jacket and a light-colored collared shirt. His posture is stationary, and he is looking directly forward with a neutral expression. The background consists of blurred greenery, providing no contextual identifiers or activity data.","description_de":"Ein einziger erwachsener Mann wird in einem Dreiviertelprofil zentral im Bild festgehalten. Er hat dunkles Haar und wirkt mittleren Alters. Er ist in formelle Kleidung gekleidet, bestehend aus einem dunkel gehaltenen maßgeschneiderten Sakko und einem hellfarbigen Kragenhemd. Seine Haltung ist stationär, und er blickt direkt nach vorne mit neutralem Ausdruck. Der Hintergrund besteht aus unscharfem Grün und liefert keine Kontextidentifikatoren oder Aktivitätsdaten.","keywords_en":["Male subject","Dark suit jacket","Formal attire","Three-quarter profile","Stationary","Neutral expression"],"keywords_de":["Männliches Subjekt","Dunkles Sakkosakko","Formelle Kleidung","Dreiviertelprofil","Stationär","Neutraler Ausdruck"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_f32_e5
INFO :: OpenVINO Generated Output:
{"description":"A single adult male is visible, occupying the center frame. He has dark hair and appears to be middle-aged. The man is wearing a dark suit jacket (likely black or navy) over a light-colored shirt and a patterned tie. His expression is serious and focused, suggesting he is speaking or delivering a presentation. No suspicious activities or anomalies are observed; the individual is stationary and engaged in public speaking.","keywords":["Adult Male","Dark Suit","Presentation","Speaking","Lower Left Face","Lower Right Person"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_int8_e5
INFO :: OpenVINO Generated Output:
{"description":"A single male subject is visible, occupying the center frame. The individual has dark hair and is looking slightly off-camera to his right. He is wearing a dark suit jacket over a light-colored collared shirt and a patterned tie. His posture suggests he is standing or speaking in an indoor setting. No suspicious activities or anomalies are observed; the person appears to be engaged in conversation or presentation.","keywords":["Male subject","Dark suit","Collared shirt","Patterned tie","Portrait","Lower left/Right focus"]}
INFO :: Loading OpenVINO model from SmolVLM2-500M-Observer/openvino_smolvlm2_int4_e5
INFO :: OpenVINO Generated Output:
{"description":"The scene captures a partial view of at least two individuals in close proximity, suggesting a public event or event setup. The Person in the lower left/Center-Left (Face/Person) is partially visible and occupies the foreground/mid-left area. This individual appears to be positioned behind or slightly above the main subject, facing generally toward the center-right but angled slightly downward or to the side. Due to cropping and limited visibility, specific identifying features such as clothing colors, facial hair details, or clear facial structure cannot be confirmed for this portion of the face/person described. The Person in the lower right/Lower-Left (Face/Person) is visible only as a partial profile view of another individual, largely cropped out of view, suggesting only a partial profile of an individual whose face is visible only partially relative to the main subject's face profile. Due to cropping and limited visibility, specific identifying features like facial hair details or distinct facial structure details cannot be confirmed for this portion of the face/person described.","description":"The scene captures a partial view of at least two individuals in close proximity, suggesting a public event or event setup. The Person in the lower left/Center-Left (Face/Person) is partially visible and occupies the foreground/mid-left area. Due to cropping and limited visibility, specific identifying features like clothing colors, facial hair details, or clear facial structure details cannot be confirmed for this portion of the face/person described. The Person in the lower right/Lower-Left (Face/Person) is visible only as a partial profile view of another individual, largely cropped out of view. Due to cropping and limited visibility, specific identifying features like facial hair details or distinct facial structure details cannot be confirmed for this portion of the face/person described.","description":"The scene captures a partial view of at least two individuals in close proximity, suggesting a public event or event setup. The Person in the lower left/Center-Left (Face/Person) is partially visible and occupies the foreground/mid-left area. Due to cropping and limited visibility, specific identifying features like clothing colors, facial hair details, or clear facial structure details cannot be confirmed for this portion of the face/person described. The Person in the lower right/Lower-Left (Face/Person) is visible only as a partial profile view of another individual, largely cropped out of view. Due to cropping and limited visibility, specific identifying features like facial hair details or distinct facial structure details cannot be confirmed for this portion of
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_f32_e2
INFO :: OpenVINO Generated Output:
{"description":"A close-up portrait of an adult male subject, positioned centrally and occupying the lower half of the frame. The man has dark, receding hair and is looking directly at the camera with a serious or engaged expression. He is wearing formal attire consisting of a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly dark and indistinct, suggesting an outdoor setting or heavily shadowed environment. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up portrait of an adult male subject, positioned centrally and occupying the lower half of the frame. The man has dark, receding hair and is looking directly at the camera with an attentive or serious expression. He is formally dressed in a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly dark and indistinct, suggesting an outdoor setting or heavily shadowed environment. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up portrait of an adult male subject, positioned centrally and occupying the lower half of the frame. The man has dark, receding hair and is looking directly at the camera with an engaged or serious expression. He is formally dressed in a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly dark and indistinct, suggesting an outdoor setting or heavily shadowed environment. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up portrait of an adult male subject, positioned centrally and occupying the lower half of the frame. The man has dark, receding hair and is looking directly at the camera with an attentive or serious expression. He is formally dressed in a dark suit jacket (appears black or deep navy) over a light colored collared shirt. The background is uniformly dark and indistinct, suggesting an outdoor setting or heavily shadowed environment. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up portrait of an adult male subject, positioned centrally and occupying the lower half of the frame. The man has dark, receding hair and is looking directly at the camera with an attentive or serious expression. He is formally dressed in a dark suit jacket (appears black or deep navy) over a light colored collared shirt
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_int8_e2
INFO :: OpenVINO Generated Output:
{"description":"A close-up, head-and-shoulders portrait of an adult male subject. The man has dark, receding hair and is looking directly at the camera with a serious or engaged expression. He is wearing formal attire: a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly blurred (bokeh effect), suggesting an outdoor setting or large crowd. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up, head-and-shoulders portrait of an adult male subject. The man has dark, receding hair and is looking directly at the camera with an engaged or serious expression. He is dressed in formal attire: a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly blurred (bokeh effect), suggesting an outdoor setting or large crowd. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"A close-up portrait of a single adult male subject. The man has dark, receding hair and is looking directly at the camera with an engaged or serious expression. He is dressed in formal attire: a dark suit jacket (appears black or deep navy) over a light-colored collared shirt. The background is uniformly blurred (bokeh effect), suggesting an outdoor setting or large crowd. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"An outdoor portrait featuring a single adult male subject. The man has dark, receding hair and is looking directly at the camera with an engaged or serious expression. He is dressed in formal attire: a dark suit jacket (appears black or deep navy) over a light colored collared shirt. The background is uniformly blurred (bokeh effect), suggesting an outdoor setting or large crowd. No suspicious activities or anomalies are observed; the subject appears to be posing for a photograph.","description":"An outdoor portrait featuring a single adult male subject. The man has dark, receding hair and is looking directly at the camera with an engaged or serious expression. He is dressed in formal attire: a dark suit jacket (appears black or deep navy) over a light colored collared shirt. The background is uniformly blurred (bokeh effect), suggesting an outdoor setting or large crowd. No suspicious activities or anomalies are observed; the subject appears to be posing for
INFO :: Loading OpenVINO model from SmolVLM-256M-Observer/openvino_smolvlm_int4_e2
INFO :: OpenVINO Generated Output:
{"description":"The image is a close-up portrait featuring one adult male subject positioned centrally in the frame. The primary focus is on the face and upper body of the person. The individual has dark, receding hair and appears to be middle-aged to elderly. He is wearing formal attire, consisting of a dark (appears black or deep navy) suit jacket over a light blue collared shirt and a dark (appears black or deep charcoal) suit jacket over a lighter (white/cream) collared shirt. His expression is serious and focused, suggesting he may be speaking or addressing an audience. He is positioned facing slightly right relative to the camera's left axis, looking slightly off-camera to his right. No other persons or objects are visible besides the main subject.","description":""The scene is static and focused on one primary individual: the primary subject. The primary focus is on the face and upper body of the person, who is positioned centrally in the lower left and right quadrants. This individual has dark, receding, receding-haired hair and appears to be middle-aged to elderly. He is formally dressed in a dark (appears black or deep navy) suit jacket over a light blue collared shirt and a dark (appears black or deep charcoal) dark suit jacket over a lighter (white/creme) collared shirt. His facial expression is serious and focused, suggesting he may be speaking or addressing an audience relative to his left and right. No other persons or objects are visible besides the main subject; only the main subject is present and centered in the frame. No anomalies or suspicious activities are observed; the focus remains solely on the primary subject's appearance and physical attributes.","keywords: Adult male, receding hair, dark suit, light blue collared shirt, dark jacket, light blue collared shirt, receding hair, receding hair, formal attire, professional portrait, East/West focus, Lower left, Lower right focus, Light-colored shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared shirt, Light-colored collared
INFO :: Loading OpenVINO model from SmolVLM2-256M-Observer/openvino_smolvlm2_f32_e1
INFO :: OpenVINO Generated Output:
{'description': 'Ein Nahaufnahme-Porträt eines erwachsenen männlichen Subjekts, das zentral im Bild positioniert ist. Der Mann hat kurzes, dunkles Haar und trägt formelle Kleidung, bestehend aus einem dunklen Anzugsakko (wirkt schwarz oder tief marineblau) über einem hellfarbigen Hemd. Er blickt direkt in die Kamera. Der Hintergrund ist einheitlich und unscharf (Bokeh-Effekt), was auf eine Aufnahme in einer Außenumgebung hindeutet. Es sind keine verdächtigen Aktivitäten oder Anomalien festzustellen; der Subjekt wirkt stationär.'}
INFO :: Loading OpenVINO model from SmolVLM2-256M-Observer/openvino_smolvlm2_int8_e1
INFO :: OpenVINO Generated Output:
{'description': 'Ein Nahaufnahme-Porträt eines erwachsenen männlichen Subjekts, das zentral im Bild positioniert ist. Der Mann hat kurzes, ordentlich geschnittenes, grau-braunes Haar und trägt dunkle Kleidung (wahrscheinlich schwarze oder marineblaue Blazer/Anzug). Er blickt direkt in die Kamera mit einem neutralen Ausdruck. Sein Gesichtsausdruck ist neutral und aufmerksam. Der Hintergrund ist einheitlich und unscharf (Bokeh-Effekt), was auf eine Aufnahme in einer Außenumgebung hindeutet.'}
INFO :: Loading OpenVINO model from SmolVLM2-256M-Observer/openvino_smolvlm2_int4_e1
INFO :: OpenVINO Generated Output:
{'description': 'Die Szene ist eine Aufnahme mit einem einzelnen männlichen Subjekt, die im unteren rechten Quadranten positioniert ist. Der Mann hat kurz geschnittes, ordentlich gestyltes Haar und kurz geschnittes Haar. Er trägt formelle dunkle (schwarzes oder marineblaues) Anzüge (sichtbare Kombinationen in weiß/hellbraun) über einem hellblaues Hemd. Sein Gesichter ist leicht nach links ausgerichtet. Die Person befindet sich im unteren rechten Quadranten, lehnt sich vom Betrachter und leicht nach links vom Betrachter. Sein Gesichter ist leicht nach links ausgerichtet. Es sind keine anderen Objekte oder Objekte sichtbar; der Fokus ist rein an der Person und dem Subjekt.'}
Deployment
FROM python:3.10-slim
# Install system dependencies
RUN apt-get update && apt-get install -y \
libgl1 \
libglib2.0-0 \
&& rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app
# Install Python requirements
RUN pip install --no-cache-dir openvino optimum-intel transformers datasets pillow numpy
# Copy application script
COPY inference.py /app/inference.py
# Copy the pre-exported OpenVINO model directory directly into the image
COPY openvino_smolvlm2_int8 /app/openvino_smolvlm2_int8
# Set environment variable to run completely offline (prevents huggingface hub lookup attempts)
ENV HF_HUB_OFFLINE=1
CMD ["python", "inference.py"]
Python Code
Training
import os
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoProcessor, AutoModelForVision2Seq, TrainingArguments
from trl import SFTTrainer
# ==========================================
# CONFIGURATION
# ==========================================
MODEL_ID = "HuggingFaceTB/SmolVLM-256M-Instruct"
JSONL_DATASET_PATH = "vlm_training_dataset.jsonl"
OUTPUT_DIR = "./smolvlm_finetuned"
BATCH_SIZE = 2 # Keep low for consumer hardware; scale up if VRAM allows
GRAD_ACCUM_STEPS = 4 # Simulates a larger batch size (2 * 4 = 8 effective batch size)
NUM_EPOCHS = 3
# ==========================================
# 1. LOAD AND PREPARE DATASET
# ==========================================
# Hugging Face's datasets handles parsing the local JSONL file automatically
dataset = load_dataset("json", data_files=JSONL_DATASET_PATH, split="train")
# We must map our customized JSONL format to the exact dict structure Hugging Face expects
def format_conversations(example):
formatted_messages = []
# Translate our ShareGPT style ("from": "human"/"gpt", "value": "...")
# to Hugging Face standard conversation dictionaries
for message in example["conversations"]:
role = "user" if message["from"] == "human" else "assistant"
formatted_messages.append({
"role": role,
"content": [{"type": "text", "text": message["value"]}]
})
return {"messages": formatted_messages, "images": [example["image"]]}
# Process dataset
formatted_dataset = dataset.map(format_conversations)
# Split into train and validation sets
split_dataset = formatted_dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
# ==========================================
# 2. INITIALIZE PROCESSOR & MODEL
# ==========================================
processor = AutoProcessor.from_pretrained(MODEL_ID)
# Configure LoRA to freeze base weights and train only small adapter layers (saves massive VRAM)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"], # Targets the visual and text attention layers
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
# Load base model in 8-bit precision (or use device_map="auto" for raw float16/bfloat16)
model = AutoModelForVision2Seq.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
device_map="auto"
)
# ==========================================
# 3. CONFIGURE TRAINING RUN
# ==========================================
training_args = TrainingArguments(
output_dir=OUTPUT_DIR,
num_train_epochs=NUM_EPOCHS,
per_device_train_batch_size=BATCH_SIZE,
gradient_accumulation_steps=GRAD_ACCUM_STEPS,
learning_rate=2e-4,
weight_decay=0.01,
logging_steps=10,
evaluation_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
fp16=not torch.cuda.is_bf16_supported(),
bf16=torch.cuda.is_bf16_supported(),
remove_unused_columns=False, # Crucial: Let TRL handle custom multimodal elements
report_to="none" # Change to "wandb" if you want live visualization
)
# ==========================================
# 4. EXECUTE TRAINING
# ==========================================
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
peft_config=lora_config,
processor=processor, # SFTTrainer natively knows how to coordinate image/text tokens
)
print("Starting SmolVLM Fine-tuning...")
trainer.train()
# Save final adapter weights and processing configs
trainer.save_model(OUTPUT_DIR)
processor.save_pretrained(OUTPUT_DIR)
print(f"Training Complete! Finetuned weights successfully saved to: {OUTPUT_DIR}")
Testing
import random
import torch
from datasets import Dataset, Image
from transformers import AutoProcessor, pipeline
MODEL_ID = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
FINE_TUNED_MODEL_DIR = "../training/SmolVLM2-500M-Observer/checkpoint-47595"
TRAINING_JSON = "motion_vlm_training_dataset.jsonl"
SYSTEM_MESSAGE = "You are an expert security data annotation agent who has to point out anomalies, suspicious activities and identifying features of observed persons, animals or vehicles. Generate a concise, highly accurate description of the scene, paying specific attention to the actions, context, and positions of the listed objects. Do not invent objects not listed or visible."
OUTPUT_FORMAT = """
Only return valid JSON in he following form:
{
"description_en": "", # type string - the image description in english
"description_de": "", # type string - the image description in german
"keywords_en": [], # type list of strings - single english keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database
"keywords_de": [], # type list of strings - single german keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database
}
"""
OUTPUT_TOKENS = 512
DEVICE = "cpu" # run on cpu / cuda
def format_annotation_data(batch: dict) -> dict:
formatted_messages = []
for img, conversations in zip(batch["image"], batch["conversations"]):
try:
# Ensure image is RGB
rgb_image = img.convert("RGB")
# Format into your VLM message schema
messages = [
{
"role": "system",
"content": [{"type": "text", "text": SYSTEM_MESSAGE}],
},
{
"role": "user",
"content": [
{"type": "image", "image": rgb_image},
{
"type": "text",
"text": conversations[0]["value"] + OUTPUT_FORMAT,
},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": conversations[1]["value"]}],
},
]
formatted_messages.append(messages)
except Exception as e:
print(f"WARNING :: Skipping corrupt image: {e}")
formatted_messages.append(None)
return {"messages": formatted_messages}
def train_test_split(dataset_json):
# Load dataset
dataset = Dataset.from_json(dataset_json)
# Let HF handle PIL conversion
dataset = dataset.cast_column("image", Image())
# Apply dynamic transform
dataset.set_transform(format_annotation_data)
# shuffle and 80:20 split
split_dataset = dataset.train_test_split(test_size=0.2, seed=44)
train_dataset = split_dataset["train"]
test_dataset = split_dataset["test"]
return train_dataset, test_dataset
print(f"INFO :: Trying to load fine tuned model from {FINE_TUNED_MODEL_DIR}")
processor = AutoProcessor.from_pretrained(MODEL_ID)
fine_tuned_pipeline = pipeline(
"image-text-to-text",
model=FINE_TUNED_MODEL_DIR,
dtype=torch.bfloat16,
processor=processor,
device=DEVICE,
)
fine_tuned_pipeline.model.generation_config.do_sample = (
True # False for deterministic outputs
)
fine_tuned_pipeline.model.generation_config.pad_token_id = (
processor.tokenizer.eos_token_id
)
print(f"INFO :: Trying to get random sample data from {TRAINING_JSON}")
train_dataset, test_dataset = train_test_split(TRAINING_JSON)
random_sample = random.choice(test_dataset)
print(f"INFO :: Starting to work with random sample: {random_sample}")
random_sample_input = random_sample["messages"][1]
random_sample_image = random_sample_input["content"][0]["image"]
random_sample_output = random_sample["messages"][-1]
print(
f"INFO :: Ideal output for selected sample:\n{random_sample_output['content'][0]['text']}"
)
fine_tuned_pipeline_output = fine_tuned_pipeline(
text=[random_sample_input], max_new_tokens=OUTPUT_TOKENS
)
fine_tuned_pipeline_output_text = fine_tuned_pipeline_output[0]["generated_text"][-1][
"content"
]
print(f"INFO :: Generated output:\n{fine_tuned_pipeline_output_text}")
import random
import numpy as np
import openvino as ov
from datasets import Dataset, Image
from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor
OPENVINO_MODEL_DIR = [
"SmolVLM2-500M-Observer/openvino_smolvlm2_f32_e3",
"SmolVLM2-500M-Observer/openvino_smolvlm2_int8_e3",
"SmolVLM2-500M-Observer/openvino_smolvlm2_int4_e3",
]
TRAINING_JSON = "motion_vlm_training_dataset.jsonl"
SYSTEM_MESSAGE = "You are an expert security data annotation agent who has to point out anomalies, suspicious activities and identifying features of observed persons, animals or vehicles. Generate a concise, highly accurate description of the scene, paying specific attention to the actions, context, and positions of the listed objects. Do not invent objects not listed or visible."
OUTPUT_FORMAT = """
Only return valid JSON in he following form:
{
"description_en": "", # type string - the image description in english
"description_de": "", # type string - the image description in german
"keywords_en": [], # type list of strings - single english keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database
"keywords_de": [], # type list of strings - single german keywords that describe the depicted scene best and allow searching for similar scenes in a SQL database
}
"""
OUTPUT_TOKENS = 512
DEVICE = "cpu"
def format_annotation_data(batch: dict) -> dict:
formatted_messages = []
for img, conversations in zip(batch["image"], batch["conversations"]):
try:
# Ensure image is RGB
rgb_image = img.convert("RGB")
# Format into your VLM message schema
messages = [
{
"role": "system",
"content": [{"type": "text", "text": SYSTEM_MESSAGE}],
},
{
"role": "user",
"content": [
{"type": "image", "image": rgb_image},
{
"type": "text",
"text": conversations[0]["value"] + OUTPUT_FORMAT,
},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": conversations[1]["value"]}],
},
]
formatted_messages.append(messages)
except Exception as e:
print(f"WARNING :: Skipping corrupt image: {e}")
formatted_messages.append(None)
return {"messages": formatted_messages}
def train_test_split(dataset_json):
# Load dataset
dataset = Dataset.from_json(dataset_json)
# Let HF handle PIL conversion
dataset = dataset.cast_column("image", Image())
# Apply dynamic transform
dataset.set_transform(format_annotation_data)
# shuffle and 80:20 split
split_dataset = dataset.train_test_split(test_size=0.2, seed=44)
train_dataset = split_dataset["train"]
test_dataset = split_dataset["test"]
return train_dataset, test_dataset
if __name__ == "__main__":
print(f"INFO :: Getting random sample data from {TRAINING_JSON}")
train_dataset, test_dataset = train_test_split(TRAINING_JSON)
random_sample = random.choice(test_dataset)
print(f"INFO :: Working with random sample...")
# Extract raw user messages (containing system, user prompt, and PIL image)
sample_messages = random_sample["messages"]
# Keep only system and user role for text generation inference
inference_messages = [sample_messages[0], sample_messages[1]]
random_sample_image = sample_messages[1]["content"][0]["image"]
random_sample_output = sample_messages[-1]
print(f"INFO :: Ideal output:\n{random_sample_output['content'][0]['text']}")
for model_directory in OPENVINO_MODEL_DIR:
print(f"INFO :: Loading OpenVINO model from {model_directory}")
# Load processor and OpenVINO model
processor = AutoProcessor.from_pretrained(
model_directory, trust_remote_code=True
)
ov_model = OVModelForVisualCausalLM.from_pretrained(
model_directory, device=DEVICE, trust_remote_code=True
)
# Apply the model's chat template
prompt_text = processor.apply_chat_template(
inference_messages, tokenize=False, add_generation_prompt=True
)
# Process text and image together via processor
inputs = processor(
text=[prompt_text], images=[random_sample_image], return_tensors="pt"
)
# Generate tokens using the OpenVINO inference engine
generated_ids = ov_model.generate(
**inputs,
max_new_tokens=OUTPUT_TOKENS,
do_sample=True,
temperature=0.2,
pad_token_id=processor.tokenizer.eos_token_id,
)
# Strip prompt tokens from output sequence and decode text
input_token_len = inputs["input_ids"].shape[1]
generated_tokens = generated_ids[:, input_token_len:]
ov_output_text = processor.batch_decode(
generated_tokens, skip_special_tokens=True
)[0]
print(f"INFO :: OpenVINO Generated Output:\n{ov_output_text}")