Skip to main content

NPU Acceleration on Qualcomm Boards

Introduction​

This document provides instructions for verifying and testing the Neural Processing Unit (NPU) on Qualcomm-based boards for developers.

On Qualcomm SoCs the AI accelerator is the Hexagon NPU, exposed on modern parts as the Hexagon Tensor Processor (HTP). It is part of the wider Qualcomm AI Engine (CPU + Adreno GPU + Hexagon NPU). The NPU runs on the compute DSP (cDSP), which the application processor reaches through FastRPC.

The software stack is the Qualcomm AI Engine Direct SDK (QNN), distributed as part of the Qualcomm AI Runtime (QAIRT) SDK. QNN exposes a single API with one backend library per accelerator:

Backend libraryTargetModels
libQnnCpu.soCPU (reference)FP32
libQnnGpu.soAdreno GPUFP32 / FP16
libQnnHtp.soHexagon NPU (HTP)quantized only (INT8/INT16)

The HTP backend loads a skel library on the DSP whose version must match the SoC's Hexagon architecture (libQnnHtpV<NN>Skel.so):

Board / SoCHexagon versionHTP skel library
QCS6490 / QCM6490, RB3 Gen2V68libQnnHtpV68Skel.so
QCS8550 / QCM8550V73libQnnHtpV73Skel.so
QRB5165 (RB5)V66libQnnHtpV66Skel.so

The authoritative SoC-to-Hexagon-version table is in Qualcomm's HTP backend documentation. Confirm the version for your exact SoC before selecting a skel library.

Verifying NPU Functionality​

After booting the image on the board, you can check that the cDSP (which hosts the NPU) is running and reachable.

Check the FastRPC device nodes​

The NPU is reached through FastRPC. The cDSP node is the relevant one:

ls -l /dev/fastrpc*

You should see a /dev/fastrpc-cdsp node (and usually /dev/fastrpc-cdsp-secure).

Check the remoteproc state​

The cDSP firmware is loaded by the kernel remoteproc subsystem. The index is not fixed, so match by name:

for d in /sys/class/remoteproc/remoteproc*; do
  echo "$d $(cat $d/name) $(cat $d/state)"
done

The entry whose name is cdsp should report state running.

Check kernel messages​

dmesg | grep -iE 'cdsp|remoteproc|fastrpc'

A healthy boot shows the cDSP firmware loading and coming up, for example:

remoteproc remoteproc1: Booting fw image qcom/qcs6490/.../cdsp.mbn, size ...
remoteproc remoteproc1: remote processor cdsp is now up

The cDSP firmware (cdsp.mbn) lives under /lib/firmware/qcom/<soc>/. The HTP skel libraries are loaded by the DSP from a path on the ADSP_LIBRARY_PATH (commonly /usr/lib/rfsa/adsp/):

ls /usr/lib/rfsa/adsp/libQnnHtpV*Skel.so

Running Inference with QNN​

Before running, set up the runtime environment so the loader and the DSP can find the QNN libraries and the HTP skel:

export LD_LIBRARY_PATH=/opt/:$LD_LIBRARY_PATH
export PATH=/opt:$PATH
export ADSP_LIBRARY_PATH="/opt/;/usr/lib/rfsa/adsp;/dsp"

Running a model with qnn-net-run​

qnn-net-run runs a compiled model against one of the QNN backends. The model is a compiled .so, the backend is selected with --backend, and inputs are listed in a text file:

# NPU (HTP) — requires a quantized model
qnn-net-run \
  --model libinception_v3_quantized.so \
  --backend libQnnHtp.so \
  --input_list input_list.txt \
  --output_dir output_htp

# CPU reference (float)
qnn-net-run \
  --model libinception_v3.so \
  --backend libQnnCpu.so \
  --input_list input_list.txt \
  --output_dir output_cpu

Running the same model on the CPU backend is the easiest way to confirm the NPU result is correct. Useful flags:

  • --retrieve_context <file>: run a pre-serialized context binary instead of --model.
  • --use_native_input_files / --use_native_output_files: use native (non-float) data types for I/O.

A throughput-oriented variant, qnn-throughput-net-run, is also available for benchmarking.

Running Inference with ONNX Runtime​

ONNX Runtime can target the NPU through the QNN Execution Provider, selecting the HTP backend with backend_path:

import onnxruntime as ort

session = ort.InferenceSession(
    "model.qdq.onnx",                       # quantized (QDQ) model required for HTP
    providers=["QNNExecutionProvider"],
    provider_options=[{"backend_path": "libQnnHtp.so"}],
)

Common provider options include htp_performance_mode (burst, balanced, high_performance, sustained_high_performance, …), profiling_level (off, basic, detailed), vtcm_mb, and enable_htp_fp16_precision. The HTP backend requires a quantized (QDQ) ONNX model.

Preparing Models​

The HTP/NPU backend runs quantized models only (typically INT8 weights with INT8/INT16 activations); libQnnCpu.so and libQnnGpu.so run float models.

There are two ways to obtain a model that runs on the NPU.

Qualcomm AI Hub compiles and profiles PyTorch / ONNX / TensorFlow models in the cloud and emits precompiled, device-specific binaries. For the NPU, choose the qnn_context_binary target runtime (SoC-specific, NPU-only) and select your chipset. A library of pre-optimized models is available at github.com/qualcomm/ai-hub-models and huggingface.co/qualcomm.

Building from a framework model with the QNN toolchain​

# 1. Convert the framework model to QNN sources (.cpp + .bin).
#    Passing --input_list (calibration data) triggers INT8 quantization.
qnn-onnx-converter \
  --input_network model.onnx \
  --input_dim input 1,3,240,320 \
  --out_node output \
  --input_list calib_list.txt \
  --output_path model_quantized.cpp

# 2. Compile the sources into a loadable .so for the target.
qnn-model-lib-generator \
  -c model_quantized.cpp \
  -b model_quantized.bin \
  -o model_libs

# 3. (Optional) Pre-finalize into a serialized context binary
#    for fast load and use with --retrieve_context.
qnn-context-binary-generator \
  --backend libQnnHtp.so \
  --model model_libs/.../libmodel_quantized.so \
  --binary_file model_ctx

Equivalent converters exist for other frameworks: qnn-tensorflow-converter, qnn-pytorch-converter, qnn-tflite-converter.

Legacy SNPE: older deployments may use the SNPE tools (snpe-onnx-to-dlc, snpe-net-run) and the .dlc model format. SNPE is superseded by QNN — new work should target QNN.

References​