AI Tech Inspire

One Backend to Rule the Edge: Vulkan + ncnn for Vendor-Agnostic Inference

What if your edge app could offload neural nets to whatever GPU the user has—without adding a heavyweight driver install or picking a vendor side? At AI Tech Inspire, we spotted a pragmatic approach from the PostSlate team that does exactly that by pairing Vulkan with ncnn. The result: vendor-agnostic GPU inference that runs on NVIDIA, AMD, Intel integrated graphics, and even Apple Silicon devices, while delivering substantial speed-ups and smaller model footprints.

Quick facts from the report

  • Use case: PostSlate, a video editing tool, runs on-device ML for face detection and embeddings.
  • Hardware diversity: Targets NVIDIA discrete GPUs, AMD, Intel integrated, and Apple Silicon.
  • Constraint: CUDA is ruled out due to vendor lock-in; the team wanted a single backend that runs everywhere.
  • Choice: Adopted ncnn with its Vulkan backend.
  • Reported performance on an RTX 4070 (fp16): ArcFace R50 from 30 ms (ONNX CPU) → 3 ms (ncnn Vulkan); SCRFD from 25 ms → 2.5 ms.
  • Model size change (ArcFace): 174 MB (ONNX fp32) → 87 MB (ncnn fp16 weight storage).
  • Primary benefit cited: Vulkan drivers are present on the machines they ship to; no vendor-specific runtime installs required.
  • Full write-up: getpostslate.com/blog/faster-local-inference.

Key takeaway: Vendor-agnostic GPU inference is practical today with Vulkan + ncnn. Speed is a bonus—portability is the headline.

Why this matters for edge ML

Edge deployments live in the real world, where you can’t assume a CUDA-capable card, a specific driver version, or a user comfortable installing 1 GB of dependencies. If your app does real-time video analysis, privacy-preserving on-device inference, or anything latency-sensitive, your success hinges on tapping whatever compute is available. That means a portable GPU API and a runtime that doesn’t box you into one vendor.

Vulkan checks the portability box: it’s an industry-standard low-level graphics and compute API, widely shipped across Windows and Linux, with portability layers available for platforms like Apple’s Metal via MoltenVK. And ncnn brings a lightweight, mobile-and-desktop-friendly inference engine that speaks Vulkan out of the box. Together, they form a credible “works almost everywhere” path that’s appealing if you’ve been juggling CUDA, Metal, DirectML, and friends just to reach your users.

What PostSlate actually did

The team runs common computer vision tasks—face detection and face embedding—entirely on-device to support their video editing workflow. They needed a single GPU backend that would run on:

  • NVIDIA discrete GPUs
  • AMD GPUs
  • Intel integrated graphics
  • Apple Silicon Macs

They selected ncnn’s Vulkan backend and reported strong wins. On an RTX 4070 with fp16 compute, ArcFace R50 dropped from 30 ms (ONNX CPU) to 3 ms, and SCRFD fell from 25 ms to 2.5 ms. ArcFace model storage also halved from 174 MB (ONNX fp32) to 87 MB (ncnn fp16). Even if the exact numbers vary across devices, the overall pattern is clear: offloading to GPU via Vulkan can be a 10× swing in latency for these tasks—and the footprint savings help with shipping and cold-start performance.

Notably, the team says the deciding factor wasn’t raw speed; it was the ability to depend on existing Vulkan drivers rather than ship or require vendor-specific runtimes.

Why Vulkan + ncnn hits a sweet spot

  • Portability first: Vulkan runs across vendors and OSes, and where it’s not native, portability layers (like MoltenVK) are commonly used. That means fewer code paths than juggling CUDA, Metal, and DirectML separately.
  • Lightweight engine: ncnn is tuned for mobile and desktop deployment with minimal dependencies. It’s not trying to be everything to everyone—it’s trying to be fast and small.
  • Reasonable model pipeline: Starting from ONNX models, you can convert with tools like onnx2ncnn and enable fp16 to cut size and memory bandwidth.
  • Privacy and UX: On-device inference avoids roundtrips to the cloud and can yield smoother user experiences in video tooling and beyond.

There’s a broader industry signal here: while ecosystems like PyTorch and TensorFlow dominate training, inference at the edge often benefits from runtimes that prioritize footprint, ops coverage for common models, and cross-vendor acceleration. Vulkan-backed inference fits that bill.

How this compares to other routes

  • CUDA/TensorRT: Excellent performance on NVIDIA hardware; not a fit if your users aren’t guaranteed to have an NVIDIA GPU or the right drivers.
  • Metal: Great on Apple Silicon; not portable elsewhere.
  • DirectML: Windows-centric, integrates nicely with system drivers, but doesn’t address macOS/Linux in one stroke.
  • OpenCL: A cross-vendor compute API with broad history, but momentum and tooling in the ML space have shifted toward Vulkan and vendor-specific stacks.
  • WebGPU: Emerging as a portable compute target in browsers and beyond; promising, but still maturing for heavy native inference pipelines.

In short, if you want one backend for “real computers in the wild,” Vulkan is a pragmatic pick today—especially when paired with an engine like ncnn that avoids heavy dependencies.

Practical steps to try this approach

If you’re shipping an app that needs real-time CV or audio on edge devices, here’s a starter mental model:

  • Start with ONNX: Convert your PyTorch or TensorFlow model to ONNX. Validate numerics on CPU first.
  • Convert to ncnn: Use onnx2ncnn, then consider ncnnoptimize. Enable fp16 where numerically safe to halve model storage and reduce bandwidth.
  • Wire up Vulkan: In your runtime, initialize Vulkan once (watch for portability extensions like VK_KHR_portability_subset where relevant) and share the context across your ncnn pipelines.
  • Benchmark realistically: Test on a representative spread: NVIDIA + AMD + Intel iGPU + Apple Silicon. Measure warm vs. cold runs, batch sizes, and typical input resolutions.
  • Fall back gracefully: Keep a CPU path for systems with incompatible drivers or conservative power states. A simple if (gpu_available) ... can save support tickets.
  • Ship fewer dependencies: The pitch here is “no extra vendor runtime.” Verify your installers rely on what’s already present, or bundle only what’s necessary.

For developers used to server-side accelerators, the upside is surprising: when your app is GPU-portable and self-contained, onboarding friction drops. Users don’t have to hunt for drivers or SDKs, and your support matrix gets simpler.

Caveats to keep in mind

  • Numbers vary by device: The 10×+ speed-ups cited are on a specific GPU (RTX 4070). Integrated GPUs or older cards will show different profiles.
  • fp16 accuracy: Most vision models handle fp16 well, but validate critical thresholds (e.g., face matching distances) before switching everything to half precision.
  • Driver quirks: Vulkan coverage is strong, but watch for platform-specific edges and portability layers. Thorough QA across OS + driver versions is essential.
  • Ops coverage: Ensure ncnn supports the operators your model uses—or be ready to re-export or swap layers.

Why this angle is worth trying

Many teams default to “NVIDIA-first, sort out the rest later.” The PostSlate report flips that script: they prioritized shipping everywhere with one GPU backend and got speed as a byproduct. If your product depends on real-time inference and your users have a mix of GPUs, their results suggest a low-drama path forward.

At AI Tech Inspire, we like this approach because it reframes the edge inference question from “which vendor do we bet on?” to “how do we bet on the user’s device?” It’s a subtle shift, but it changes architectural decisions—from installer flows to benchmark targets.

One more thought experiment: if your roadmap includes multimodal features (e.g., face + voice + gesture), a vendor-agnostic GPU stack can reduce surface area as you add models. You may not hit absolute peak perf on every single device, but you might ship faster, support more users, and still deliver real-time UX.

Resources and next steps

If you test this route, consider starting with detection/embedding baselines like ArcFace (InsightFace) and SCRFD. They’re widely used, easy to validate, and clearly demonstrate latency wins when GPU-accelerated.

Bottom line: If your edge app needs to “just work” across GPUs without extra installs, Vulkan + ncnn is a compelling, developer-friendly playbook to try next.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.