Skip to main content

NPU Acceleration on MediaTek Genio Boards

Introduction​

This document provides instructions for enabling, verifying, and testing the Neural Processing Unit (NPU) on MediaTek Genio boards for developers.

On MediaTek Genio platforms the AI accelerator is exposed through the NeuroPilot software stack. The hardware is referred to as the NPU (the older term APU, AI Processing Unit, still appears in driver names and logs) and is built from two kinds of compute engines:

  • MDLA (MediaTek Deep Learning Accelerator): a CNN accelerator supporting FP16, INT16 and INT8.
  • VPU (Vision Processing Unit, a Cadence Tensilica DSP): a general AI/vision DSP supporting FP32, FP16, INT16 and INT8.

Which engines are available depends on the SoC:

BoardSoCAI engines
Genio 350-EVKMT8365VPU only (no MDLA)
Genio 510-EVKMT8390MDLA 3.0 + VPU
Genio 700-EVKMT8395MDLA 3.0 + VPU
Genio 1200-EVKMT8395APU 3.0 (2× MDLA 2.0 + 2× VPU)

Enabling NPU Support​

To enable Machine Learning support in your build, activate the Machine Learning Support in the distribution configuration.

  1. Enter the menuconfig utility: 
    . ./seco-setup.sh -m
  2. Navigate to the following path: Distro Features -> Enable support for Machine Learning
  3. Select and enable Machine Learning Support.
  4. Save your configuration and build the image.
Genio 1200 note:

The APU is brought up through a device-tree overlay (apusys.dtbo). Make sure the overlay is applied for your board, otherwise the NPU will not be available at runtime.

Verifying NPU Functionality​

After booting the image on the board, you can check that the NPU is recognized by the kernel. The kernel driver is named apusys.

Check kernel messages​

dmesg | grep -i apusys

Check for the NPU device node​

The exact device node name depends on the board image. Look for an APU/MDLA/VPU related node:

ls /dev | grep -iE 'apu|mdla|vpu'

Check the APUSYS debugfs interface​

When the APU is active, a debugfs directory is present:

ls /sys/kernel/debug/apusys/

You can also use it to force the NPU to its maximum frequency for benchmarking:

echo dvfs_debug 0 > /sys/kernel/debug/apusys/power

When inference actually runs on the NPU, you will see apusys log lines similar to:

[ apusys ][ info ] apusysSession: Session (0x...) : thd (runtime_api_sam) version (3) ...
[ apusys ][ info ] run: Cmd v2 (0x...) : run
[ apusys ][ info ] run: Cmd v2 (0x...) : run done (0)

Running Inference with TensorFlow Lite​

The sample models and label files are located under /usr/share/label_image/ and /usr/share/benchmark_dla/.

There are two ways to run a TensorFlow Lite model on the NPU:

  • Online (Neuron Stable Delegate): the .tflite model is compiled on the fly by the TFLite delegate. Available on Genio 510/700/1200 (MDLA boards).
  • Offline (Neuron SDK): the .tflite model is compiled ahead of time into a .dla archive and executed with the neuronrt runtime. See Running Inference with the Neuron SDK.

MDLA boards — Neuron Stable Delegate (Genio 510/700/1200)​

On boards with an MDLA, the NPU is targeted through the Neuron Stable Delegate (/usr/lib/libneuron_stable_delegate.so). Unlike a classic external delegate, it is loaded through a settings file that points delegate_path at the delegate library.

Performance Benchmark (benchmark_model)

benchmark_model \
  --stable_delegate_settings_file=/usr/share/label_image/stable_delegate_settings.json \
  --use_nnapi=false --use_xnnpack=false --use_gpu=false \
  --min_secs=20 \
  --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite

The stable_delegate_settings.json file must set delegate_path to /usr/lib/libneuron_stable_delegate.so.

Genio 350 — NNAPI Delegate (VPU)​

The Genio 350 has no MDLA and runs inference on the VPU through the NNAPI external delegate (/usr/lib/nnapi_external_delegate.so). This board uses the classic --external_delegate_path form:

benchmark_model \
  --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite \
  --external_delegate_path=/usr/lib/nnapi_external_delegate.so \
  --num_runs=10

Image Classification (label_image.py)​

A Python image-classification example is provided in /usr/share/label_image/.

cd /usr/share/label_image

# CPU (reference)
python3 label_image.py \
  --label_file labels_mobilenet_quant_v1_224.txt \
  --image grace_hopper.jpg \
  --model_file mobilenet_v1_1.0_224_quant.tflite

# Genio 350 — NNAPI delegate (VPU)
python3 label_image.py \
  --label_file labels_mobilenet_quant_v1_224.txt \
  --image grace_hopper.jpg \
  --model_file mobilenet_v1_1.0_224_quant.tflite \
  -e /usr/lib/nnapi_external_delegate.so

Other delegates (GPU / Arm NN)​

For comparison, the image also ships GPU and Arm NN delegates. These run on the Mali GPU / Cortex-A CPU, not on the NPU:

# GPU delegate
benchmark_model \
  --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite \
  --use_gpu=1 --allow_fp16=0 --gpu_precision_loss_allowed=0 --num_runs=10

# Arm NN delegate (GPU + CPU)
benchmark_model \
  --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite \
  --external_delegate_path=/usr/lib/libarmnnDelegate.so.29 \
  --external_delegate_options="backends:GpuAcc,CpuAcc" --num_runs=10

Running Inference with the Neuron SDK​

The Neuron SDK is the offline path: a TensorFlow Lite model is compiled into a .dla (Deep Learning Archive) with the ncc-tflite compiler and executed on the NPU with the neuronrt runtime.

Compile a model (ncc-tflite)​

Select the target engine with --arch (mdla3.0, mdla2.0, mdla1.5, vpu, or a combination such as mdla2.0,vpu):

# Quantized (INT8) model for MDLA 3.0
ncc-tflite --arch=mdla3.0 yolov5s_int8.tflite -o yolov5s_int8.dla

# Heterogeneous: MDLA 2.0 + VPU
ncc-tflite --arch mdla2.0,vpu ssd_mobilenet_v1_coco_quantized.tflite -o ssd_mobilenet_v1_coco_quantized.dla

The NPU is designed for quantized INT8/UINT8 models. Useful flags:

  • --relax-fp32: run an FP32 model as FP16 (FP32 is not natively executed on MDLA).
  • --int8-to-uint8: convert signed INT8 to UINT8; required to run asymmetric signed 8-bit models on MDLA 1.0 / 1.5, which lack native INT8.
  • --req-per-ch-conv: re-quantize per-channel convolutions whose output scales are unsupported (may reduce accuracy).
  • --show-exec-plan: show how the graph is partitioned across the --arch targets.
  • --check-target-only: check operator support per device without compiling.

Run a compiled model (neuronrt)​

# Show the input/output tensor IDs of a .dla
neuronrt -d -a yolov5s_int8.dla

# Run on the NPU hardware, 10 iterations
neuronrt -m hw -a yolov5s_int8.dla -i input.bin -o output.bin -c 10

Key flags: -m hw (run on hardware), -a (DLA file), -i/-o (input/output binaries), -c (run count), -d (dump tensor I/O info), -b (boost frequency).

Demo scripts​

The image ships end-to-end demos that compile and run a model on the NPU:

# Compile + run a .dla on the NPU
cd /usr/share/demo_dla && python3 label_image.py

# Benchmark across engines; results in benchmark.log
cd /usr/share/benchmark_dla && python3 benchmark.py --auto && cat benchmark.log

The benchmark.log reports per-engine inference time (ms), for example:

mobilenet_v2_1.0_224_quant.tflite, mdla3.0, : 2.48
mobilenet_v2_1.0_224_quant.tflite, vpu,     : 18.58

References​