Skip to main content

NPU Acceleration on NXP Boards

Introduction​

This document provides instructions for enabling, verifying, and testing the Neural Processing Unit (NPU) on NXP-based boards for developers.

Enabling NPU Support​

To enable NPU support in your build, you need to activate the Machine Learning Support in the distribution configuration.

  1. Enter the menuconfig utility: 
    . ./seco-setup.sh -m
  2. Navigate to the following path: Distro Features -> Enable support for Machine Learning
  3. Select and enable Machine Learning Support.
  4. Save your configuration and build the image.

Verifying NPU Functionality​

After booting the image on the board, you can check if the NPU is enabled and recognized by the kernel.

Check for the NPU device node​

To verify that the NPU is available, check for the presence of the NPU device node:

ls /dev/mxc_npu

The presence of /dev/mxc_npu indicates that the NPU driver is loaded and the hardware is recognized.

Check kernel messages​

You can also check the kernel log for messages related to the NPU driver initialization.

dmesg | grep galcore

You should see output indicating the Galcore driver has been initialized, which is part of the NPU driver stack. The output should look something like this:

[    1.234567] galcore: new galcore device registered.

This confirms that the kernel has loaded the necessary modules for the NPU.

Running Inference with TensorFlow Lite​

The NXP eIQ Machine Learning Software Development Environment uses TensorFlow Lite to leverage the NPU for hardware acceleration. This is achieved through different "delegates" depending on the i.MX platform.

The examples and models are located at /usr/bin/tensorflow-lite-2.18.0/examples.

i.MX 8 Platforms (VX Delegate)​

For i.MX 8 series, the VX Delegate is used to execute the computational graph on the NPU or GPU. To target the NPU, set the USE_GPU_INFERENCE environment variable to 0.

Image Classification (label_image)

USE_GPU_INFERENCE=0 ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libvx_delegate.so

Performance Benchmark (benchmark_model)

./benchmark_model --graph=mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libvx_delegate.so

i.MX 93 Platforms (Ethos-U Delegate)​

For i.MX 93, the Ethos-U Delegate accelerates inference on the Ethos-U NPU.

Image Classification (label_image)

./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libethosu_delegate.so

Performance Benchmark (benchmark_model)

./benchmark_model --graph=mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libethosu_delegate.so

i.MX 9 Platforms with Neutron-S NPU (Neutron Delegate)​

For i.MX 9 series with a Neutron-S NPU, the Neutron Delegate is used.

Image Classification (label_image)

./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libneutron_delegate.so

Performance Benchmark (benchmark_model)

./benchmark_model --graph=mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libneutron_delegate.so

Profiling​

To get detailed profiling information and identify if any operations are falling back to the CPU, use the --enable_op_profiling=true flag with the benchmark_model tool.

./benchmark_model --graph=mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=<path_to_delegate> --enable_op_profiling=true

Running Inference with ONNX Runtime​

The ONNX Runtime engine can be used to run ONNX models on the NPU. This is achieved by using the VsiNpuExecutionProvider.

Note on ONNX Models: The ONNX Model Zoo is largely deprecated. While some links may still work, it is recommended to search for and download models from Hugging Face.

Warning on Quantized Models: The VsiNpuExecutionProvider has limited support for quantized INT8 models. While FP32 models may run, INT8 models can fail or fall back to CPU execution. Always verify that your model is running on the NPU.

Testing with onnx_test_runner​

A common tool for testing ONNX models is onnx_test_runner. You can use it to run a model with the VSI NPU execution provider.

  1. Get a model: Download an ONNX model. For this example, we'll use the MobileNetV2 model. The following command downloads a model from the (deprecated) ONNX Model Zoo.

    wget https://github.com/onnx/models/raw/refs/heads/main/Computer_Vision/mobilenetv2_140_Opset18_timm/mobilenetv2_140_Opset18.onnx

  2. Run the test: Use onnx_test_runner to run the model. The -e vsi_npu flag tells ONNX Runtime to use the VSI NPU execution provider.

    onnx_test_runner -e vsi_npu mobilenetv2_140_Opset18.onnx

Legacy Test with onnxruntime_test​

The following example uses the onnxruntime_test utility.

  1. Download the Model

    First, you need a pre-trained ONNX model. You can download the MobileNetV2 model from the ONNX Model Zoo.

    wget https://github.com/onnx/models/raw/refs/heads/main/Computer_Vision/mobilenetv2_140_Opset18_timm/mobilenetv2_140_Opset18.onnx

  2. Run the ONNX Runtime Test

    Use the onnxruntime_test utility to run the model on the NPU.

    onnxruntime_test mobilenetv2_140_Opset18.onnx

    Note: The command to specify the execution provider may vary depending on your version of ONNX Runtime. If the above command does not use the NPU, you may need to specify the execution provider. For example, with a different version of the tool, the command might be onnxruntime_perf_test mobilenetv2_140_Opset18.onnx -e vsi_npu. Please consult the documentation for your specific version of ONNX Runtime.

  3. Expected Output on i.MX 8M Plus

    When you run the test on an i.MX 8M Plus board with the NPU enabled, you should see output similar to the following, indicating that the model is being executed on the NPU:

    [I:onnxruntime:, vsinpu_execution_provider.cc:58 Get] Start to create VsiNpuExecutionProvider
    [I:onnxruntime:, vsinpu_execution_provider.cc:68 Get] Finish to create VsiNpuExecutionProvider
    ...
    Running model with VSI-NPU EP
    ...
    Test passed

    The key messages to look for are the creation of the VsiNpuExecutionProvider and the "Test passed" message. This confirms that the ONNX Runtime is correctly using the NPU for inference.