TensorRT-LLM

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

TensorRT-LLM AI

Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P]

I’ve been working on a CUDA-first inference runtime for small-batch / realtime ML workloads. The core idea is simple: instead of treating PyTorch / TensorRT / generic graph runtimes as the main execution path, I rewrite the model inference path directly with C++/CUDA kernels. This started from robotics / VLA workloads, but the problem is more general. In small-batch inference, the bottleneck is often not just a single slow GEMM. A lot of latency comes from the runtime glue around the math: fragmented small kernels norm / residual / activation boundaries quantize / dequantize overhead layout transitions Python / runtime scheduling graph compiler fusion failures precision conversion around FP8 / FP4 regions For cloud LLM serving, batching can hide a lot of this. For robotics, VLA, world models, and other realtime workloads, batch size is usually 1. There is nowhere to hide. Every launch, sync, and format boundary shows up directly in latency. Some current results from my implementation: Model / workload Hardware FlashRT latency Pi0.5 Jetson Thor ~44 ms Pi0 Jetson Thor ~46 ms GROOT N1.6 Jetson Thor ~41–45 ms Pi0.5 RTX 5090 ~17.6 ms GROOT N1.6 RTX 5090 ~12.5–13.1 ms Pi0-FAST RTX 5090 ~2.39 ms/token Qwen3.6 27B RTX 5090 ~129 tok/s with NVFP4 Motus / Wan-style world model RTX 5090 ~1.3s baseline → targeting ~100ms E2E The Motus / world-model case is especially interesting. The baseline path is around 1.3s end-to-end. The target is ~100ms E2E, but the hard part is not simply “use a faster GEMM”. The bottlenecks are VAE, joint attention, launch fragmentation, and a large amount of glue around the actual math. One lesson from this work: lower precision is not automatically a win. FP8 has been consistently useful. FP4 / NVFP4 is more mixed. It can help memory footprint and some large GEMM regions, but if the FP4 region is small, discontinuous, or surrounded by conversion / scaling overhead, the end-to-end speedup can be tiny. For example, in some VLA / world-model paths, FP4 over FP8 only gives a few percent latency improvement unless the region is large and deeply fused. This changed how I think about inference optimization. For large-batch cloud serving, generic runtimes and batching are often enough. For realtime small-batch inference, the runtime overhead becomes the workload. Curious if others have seen similar behavior with torch.compile, TensorRT, XLA, Triton, or custom CUDA kernels. At what point do you stop trying to make a generic compiler optimize the model, and just rewrite the inference path directly? Implementation: https://github.com/LiangSu8899/FlashRT submitted by /u/Diligent-End-2711 [link] [comments]

TurboOCR: 270–1200 img/s OCR with Paddle + TensorRT (C++/CUDA, FP16) [P]

I had about 940,000 PDFs to process. Running VLMs over a million pages is slow and expensive, and that gap is only getting worse as OCR moves toward transformer and VLM-based approaches. They’re great for complex understanding, but throughput and cost can become a bottleneck at scale. PaddleOCR (the non VL version), in my opinion the best non-VLM open source OCR, only handled ~15 img/s on my RTX 5090, which was still too slow. PaddleOCR-VL was crawling at 2 img/s with vLLM. PaddleOCR runs single-threaded Python with FP32 inference and no kernel fusion. Turbo-OCR replaces that with C++/CUDA, FP16 TensorRT, fused kernels, batched recognition, and multi-stream pipeline pooling. It takes images and PDFs via HTTP/gRPC and returns bounding boxes, text, and layout regions (PP-DocLayoutV3, 25 classes). Layout is toggleable per request and only adds ~20% to inference time. Results: 270 img/s on text-heavy pages without layout, 1,200+ on sparse ones. Works well for real-time RAG where you need a document indexed instantly, or for bulk processing large collections cheaply. Trade-offs: complex table extraction and structured output (invoice → JSON) still need VLM-based OCR like PaddleOCR-VL. I'm working on bringing structured extraction, markdown output, table parsing, and more languages to Turbo-OCR while sacrificing as little speed as possible.. Tested on Linux, RTX 50-series, CUDA 13.2. https://github.com/aiptimizer/TurboOCR submitted by /u/Civil-Image5411 [link] [comments]