Project information

• Category: MLOps
• Client: INSTAR Deutschland GmbH (Waletech) Shenzhen, China
• Project date: 01 Feb, 2026
• Project URL: INSTAR Cloud

The problem: a chain of individually broken AI features — motion, face, plate, scene — each with its own model, its own crash modes, none stable under real load.

What I did: I built the one serving stack that ties them together — a distributed AI inference platform on a CPU-only server. Three resident model servers keep object detection (YOLOv10 / LiteRT-TFLite), plate OCR (Fast-Plate-OCR / LiteRT), and a fine-tuned SmolVLM (OpenVINO) scene observer loaded once in memory behind a Redis job queue: sorted-set task queues for batching, thread-safe claim-and-process worker loops, and UUID-keyed request/result slots so every prediction call resolves without blocking the others. A watchdog validates each incoming alarm job (JSON schema), then orchestrates a pool of Ray actors (up to 18 concurrent, each 1 CPU capped) with round-robin dispatching, result collection, and fleet-wide structured logging. A connector layer then writes each detection into the customer-facing MySQL database — alarms, plate hits, face matches — triggers email notifications with per-tenant rules, and cross-server load balancing routes each job to the least-loaded node.

Six months of model optimization (OpenVINO / LiteRT conversion), memory budgeting (partitioning the Ray heap from the shared-memory store), and fault-tolerance work drove the failure modes out: no more memory leaks, random crashes, or multi-minute predictions at peak. It ships as a documented HashiCorp Nomad job — canary-first rollout with automatic rollback on a failing health check, a bounded retry policy, and strict per-task CPU/memory limits — so a broken model can't take down the fleet, just that one allocation. Deployed on Nomad behind a dedicated Zabbix dashboard for the cloud team.

The payoff: the system that makes every AI project above stop being a demo and become a service.