Skip to main content

Guangzhou, China

Triton Inference Server

Triton is optimized to provide the best inferencing performance by using GPUs, but it can also work on CPU-only systems. In both cases you can use the same Triton Docker image.

docker pull nvcr.io/nvidia/tritonserver:24.06-py3

Its primary use cases are:

  • Serving multiple models from a single server instance.
  • Dynamic model loading and unloading without server restart.
  • Ensemble inference, allowing multiple models to be used together to achieve results.
  • Model versioning for A/B testing and rolling updates.

Prerequisites​

pip install tritonclient\[all\]
import contextlib
from pathlib import Path
import subprocess
import time

from tritonclient.http import InferenceServerClient
from ultralytics import YOLO

Exporting YOLO Models into the ONNX Format​

# Load the PyTorch model
model_8n = YOLO("weights/yolov8n.pt")
# Export the ONNX model
onnx_file = model_8n.export(format="onnx", dynamic=True) # export to weights/yolov8n.onnx

Setting Up Triton Model Repository​

# Define paths
model_name = "yolo8n"
triton_repo_path = Path("deploy") / "triton"
triton_model_path = triton_repo_path / model_name
# Create directories
(triton_model_path / "1").mkdir(parents=True, exist_ok=True)
# Copy ONNX model to Triton path
Path(onnx_file).rename(triton_model_path / "1" / "model.onnx")
# Create config file
(triton_model_path / "config.pbtxt").touch()

Running Triton Inference Server (CPU)​

docker run --rm -v ./deploy/triton:/models -p 8000:8000 --name tritonserver nvcr.io/nvidia/tritonserver:24.06-py3 tritonserver --model-repository=/models
=============================
== Triton Inference Server ==
=============================

NVIDIA Release 24.06 (build 98458813)
Triton Server Version 2.47.0

I0630 10:50:50.713271 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo8n (version 1)"
I0630 10:50:50.713306 1 onnxruntime.cc:3010] 
W0630 10:50:50.811660 1 onnxruntime.cc:1104] "autofilled max_batch_size to 4 for model 'yolo8n' since batching is supporrted but no max_batch_size is specified in model configuration. Must specify max_batch_size to utilize autofill with a larger max batch size"
I0630 10:50:50.814366 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo8n_0 (CPU device 0)"
I0630 10:50:50.814429 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo8n_1 (CPU device 0)"
I0630 10:50:50.944107 1 onnxruntime.cc:3075] 
I0630 10:50:51.123288 1 model_lifecycle.cc:838] "successfully loaded 'yolo8n'"
I0630 10:50:51.382747 1 model_lifecycle.cc:838] "successfully loaded 'yolo9s'"

+--------+---------+--------+
| Model  | Version | Status |
+--------+---------+--------+
| yolo8n | 1       | READY  |
+--------+---------+--------+
# Wait for the Triton server to start
triton_client = InferenceServerClient(url="localhost:8000", verbose=False, ssl=False)

# Wait until model is ready
for _ in range(10):
    with contextlib.suppress(Exception):
        assert triton_client.is_model_ready(model_name)
        break
    time.sleep(1)
!curl -v 192.168.2.210:8000/v2/health/ready

    * Trying 192.168.2.210:8000...
    * Connected to 192.168.2.210 (192.168.2.210) port 8000 (#0)
    > GET /v2/health/ready HTTP/1.1
    > Host: 192.168.2.210:8000
    > User-Agent: curl/7.81.0
    > Accept: */*
    > 
    * Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Content-Length: 0
    < Content-Type: text/plain
    < 
    * Connection #0 to host 192.168.2.210 left intact
curl http://192.168.2.210:8000/v2/models/yolo8n/config

{"name":"yolo8n","platform":"onnxruntime_onnx","backend":"onnxruntime","runtime":"","version_policy":{"latest":{"num_versions":1}},"max_batch_size":0,"input":[{"name":"images","data_type":"TYPE_FP32","format":"FORMAT_NONE","dims":[1,3,640,640],"is_shape_tensor":false,"allow_ragged_batch":false,"optional":false}],"output":[{"name":"output0","data_type":"TYPE_FP32","dims":[1,21,8400],"label_filename":"","is_shape_tensor":false}],"batch_input":[],"batch_output":[],"optimization":{"priority":"PRIORITY_DEFAULT","input_pinned_memory":{"enable":true},"output_pinned_memory":{"enable":true},"gather_kernel_buffer_threshold":0,"eager_batching":false},"instance_group":[{"name":"yolo8n","kind":"KIND_GPU","count":1,"gpus":[0],"secondary_devices":[],"profile":[],"passive":false,"host_policy":""}],"default_model_filename":"model.onnx","cc_model_filenames":{},"metric_tags":{},"parameters":{},"model_warmup":[]}
# Load the Triton Server model
triton_model = YOLO("http://192.168.2.210:8000/yolo8n", task="detect")
# Run inference on the server
results = triton_model("datasets/test/face2.jpg")
    
    image 1/1 /opt/app/datasets/test/face2.jpg: 640x640 3 class0s, 52.0ms
    Speed: 2.4ms preprocess, 52.0ms inference, 3.3ms postprocess per image at shape (1, 3, 640, 640)

Exporting YOLO Models for Triton Inference​

# Load the PyTorch model
model_9s = YOLO("runs/detect/yolov9s/weights/best.pt")
# Export the TensorRT model
onnx_file9s = model_9s.export(format="onnx", device=0) # export to runs/detect/yolov9s/weights/best.onnx

Setting Up Triton Model Repository​

# Define paths
model9s_name = "yolo9s"
triton9s_repo_path = Path("deploy") / "triton"
triton9s_model_path = triton9s_repo_path / model9s_name
# Create directories
(triton9s_model_path / "1").mkdir(parents=True, exist_ok=True)
# Copy ONNX model to Triton path
Path(onnx_file9s).rename(triton9s_model_path / "1" / "model.onnx")
# Create config file
(triton9s_model_path / "config.pbtxt").touch()

Running Triton Inference Server (GPU)​

Use the following command to run Triton with the example model repository you just created. The NVIDIA Container Toolkit must be installed for Docker to recognize the GPU(s). The --gpus=1 flag indicates that 1 system GPU should be made available to Triton for inferencing.

docker run --gpus=all --rm -v ./deploy/triton:/models \
 -p 8000:8000 -p 8001:8001 -p 8002:8002 --name tritonserver \
 nvcr.io/nvidia/tritonserver:24.06-py3 tritonserver --model-repository=/models  --model-control-mode=poll --repository-poll-secs 30
=============================
== Triton Inference Server ==
=============================

NVIDIA Release 24.06 (build 98458813)
Triton Server Version 2.47.0

I0702 08:51:29.912004 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo8n (version 1)"
I0702 08:51:29.912052 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo8m (version 1)"
I0702 08:51:29.912073 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo9s (version 1)"
I0702 08:51:29.912091 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo9m (version 1)"
I0702 08:51:31.036723 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo9m_0 (GPU device 0)"
I0702 08:51:31.764582 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo8m_0 (GPU device 0)"
I0702 08:51:32.050735 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo9s_0 (GPU device 0)"
I0702 08:51:32.181237 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo8n_0 (GPU device 0)"
I0702 08:51:32.302135 1 model_lifecycle.cc:838] "successfully loaded 'yolo9m'"
I0702 08:51:32.302465 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo9t (version 1)"
I0702 08:51:32.410140 1 model_lifecycle.cc:838] "successfully loaded 'yolo8m'"
I0702 08:51:32.410426 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolov10n (version 1)"
I0702 08:51:32.518006 1 model_lifecycle.cc:838] "successfully loaded 'yolo9s'"
I0702 08:51:32.518282 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolov10s (version 1)"
I0702 08:51:32.556986 1 model_lifecycle.cc:838] "successfully loaded 'yolo8n'"
I0702 08:51:32.557358 1 onnxruntime.cc:3010] "TRITONBACKEND_ModelInitialize: yolo8s (version 1)"
I0702 08:51:32.712628 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo9t_0 (GPU device 0)"
I0702 08:51:32.830182 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolov10n_0 (GPU device 0)"
I0702 08:51:33.154025 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolov10s_0 (GPU device 0)"
I0702 08:51:33.530147 1 onnxruntime.cc:3075] "TRITONBACKEND_ModelInstanceInitialize: yolo8s_0 (GPU device 0)"
I0702 08:51:33.617135 1 model_lifecycle.cc:838] "successfully loaded 'yolo9t'"
I0702 08:51:33.677292 1 model_lifecycle.cc:838] "successfully loaded 'yolov10n'"
I0702 08:51:33.772530 1 model_lifecycle.cc:838] "successfully loaded 'yolov10s'"
I0702 08:51:33.822564 1 model_lifecycle.cc:838] "successfully loaded 'yolo8s'"
I0702 08:51:33.823015 1 server.cc:604] 

+----------+---------+--------+
| Model    | Version | Status |
+----------+---------+--------+
| yolo8m   | 1       | READY  |
| yolo8n   | 1       | READY  |
| yolo8s   | 1       | READY  |
| yolo9m   | 1       | READY  |
| yolo9s   | 1       | READY  |
| yolo9t   | 1       | READY  |
| yolov10n | 1       | READY  |
| yolov10s | 1       | READY  |
+----------+---------+--------+

I0702 08:51:33.839930 1 grpc_server.cc:2463] "Started GRPCInferenceService at 0.0.0.0:8001"
I0702 08:51:33.840086 1 http_server.cc:4692] "Started HTTPService at 0.0.0.0:8000"
I0702 08:51:33.882137 1 http_server.cc:362] "Started Metrics Service at 0.0.0.0:8002"
!curl -v 192.168.2.210:8000/v2/health/ready

    * Trying 192.168.2.210:8000...
    * Connected to 192.168.2.210 (192.168.2.210) port 8000 (#0)
    > GET /v2/health/ready HTTP/1.1
    > Host: 192.168.2.210:8000
    > User-Agent: curl/7.81.0
    > Accept: */*
    > 
    * Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Content-Length: 0
    < Content-Type: text/plain
    < 
    * Connection #0 to host 192.168.2.210 left intact
curl http://192.168.2.210:8000/v2/models/yolo8n/config

{"name":"yolo8n","platform":"onnxruntime_onnx","backend":"onnxruntime","runtime":"","version_policy":{"latest":{"num_versions":1}},"max_batch_size":0,"input":[{"name":"images","data_type":"TYPE_FP32","format":"FORMAT_NONE","dims":[1,3,640,640],"is_shape_tensor":false,"allow_ragged_batch":false,"optional":false}],"output":[{"name":"output0","data_type":"TYPE_FP32","dims":[1,21,8400],"label_filename":"","is_shape_tensor":false}],"batch_input":[],"batch_output":[],"optimization":{"priority":"PRIORITY_DEFAULT","input_pinned_memory":{"enable":true},"output_pinned_memory":{"enable":true},"gather_kernel_buffer_threshold":0,"eager_batching":false},"instance_group":[{"name":"yolo8n","kind":"KIND_GPU","count":1,"gpus":[0],"secondary_devices":[],"profile":[],"passive":false,"host_policy":""}],"default_model_filename":"model.onnx","cc_model_filenames":{},"metric_tags":{},"parameters":{},"model_warmup":[]}