native-onnx

Android-first React Native ONNX Runtime bindings built with Nitro Modules.

native-onnx gives React Native apps a small, typed API for creating ONNX Runtime sessions, selecting Android execution backends, running tensor inference, benchmarking sessions, and experimenting with mobile diffusion model pipelines.

Status

This package is early and Android-focused.

Current support:

• Android native module via Nitro Modules
• ONNX Runtime session creation
• CPU, XNNPACK, NNAPI, Qualcomm QNN HTP, and Qualcomm QNN GPU backend selection
• async tensor inference
• model input/output introspection
• common tensor helpers
• benchmark helpers
• experimental Qualcomm Stable Diffusion v2.1 component pipeline helpers

Not supported yet:

• iOS
• full DiffusionPipeline.generate({ prompt }) prompt-to-image generation
• tokenizer and scheduler orchestration inside the library
• arbitrary dynamic-shape diffusion pipelines

Installation

npm install native-onnx react-native-nitro-modules

or:

bun add native-onnx react-native-nitro-modules

Then rebuild the Android app:

cd android
./gradlew assembleDebug

The package currently depends on:

implementation "com.microsoft.onnxruntime:onnxruntime-android-qnn:1.27.0"

Quick Start

import { NativeOnnx, TensorTools } from 'native-onnx'

const session = await NativeOnnx.createSession('/data/local/tmp/model.onnx', {
  backend: 'XNNPACK',
  threads: 4,
})

const input = TensorTools.float32(
  'pixel_values',
  [1, 3, 224, 224],
  new Float32Array(1 * 3 * 224 * 224)
)

const outputs = await session.run([input])
const logits = TensorTools.toFloat32Array(outputs[0])

session.close()

Backend Selection

const backends = NativeOnnx.getSupportedBackends()
const hasNpu = NativeOnnx.isBackendSupported('QNN_HTP')

Supported backend option strings:

CPU
XNNPACK
NNAPI
QNN_HTP
QNN_GPU

Backend meaning:

• CPU: ONNX Runtime default CPU execution.
• XNNPACK: optimized CPU execution.
• NNAPI: Android Neural Networks API compatibility path.
• QNN_HTP: Qualcomm QNN HTP backend, targeting Snapdragon NPU.
• QNN_GPU: Qualcomm QNN GPU backend, targeting Snapdragon/Adreno GPU.

For strict NPU/GPU validation, disable CPU fallback:

const session = await NativeOnnx.createSession(modelPath, {
  backend: 'QNN_HTP',
  threads: 4,
  requireFullBackend: true,
  qnnPerformanceMode: 'burst',
  qnnGraphOptimizationMode: '3',
})

requireFullBackend: true sets session.disable_cpu_ep_fallback=1. Session creation should fail if ONNX Runtime cannot place the full graph on the selected backend.

getSupportedBackends() only reports which execution providers are present in the runtime package/device environment. It does not prove a specific model fully runs on NPU. For user-facing NPU readiness, create the session with requireFullBackend: true and run a real model-specific smoke test.

Session Options

type CreateSessionOptions = {
  backend: string
  threads: number
  requireFullBackend?: boolean
  qnnPerformanceMode?: string
  qnnGraphOptimizationMode?: string
  qnnProfilingLevel?: string
  qnnProfilingFilePath?: string
  ortProfilingFilePath?: string
}

QNN option mapping:

• qnnPerformanceMode maps to htp_performance_mode.
• qnnGraphOptimizationMode maps to htp_graph_finalization_optimization_mode.
• qnnProfilingLevel maps to QNN profiling_level.
• qnnProfilingFilePath maps to QNN profiling_file_path.
• ortProfilingFilePath enables ONNX Runtime session profiling.

Tensor Inference

session.run(...) accepts named tensors and returns named tensors:

type OnnxTensorValue = {
  name: string
  dtype: string
  shape: number[]
  data: ArrayBuffer
}

Supported tensor dtypes:

FLOAT
DOUBLE
INT8
INT16
INT32
INT64
BOOL
UINT8
UINT16
FLOAT16
BFLOAT16

Aliases accepted for input tensors:

• FLOAT32 -> FLOAT
• FLOAT64 -> DOUBLE
• BOOLEAN -> BOOL

The ArrayBuffer byte length must exactly match dtype * shape.

If a typed-array view has a byte offset, pass a sliced buffer:

const data = view.buffer.slice(view.byteOffset, view.byteOffset + view.byteLength)

TensorTools

import { TensorTools } from 'native-onnx'

const input = TensorTools.float32('input', [1, 3, 224, 224], pixels)
const emptyMask = TensorTools.createEmptyTensor('mask', 'UINT8', [1, 1, 224, 224])

const outputs = await session.run([input, emptyMask])
const result = TensorTools.toFloat32Array(outputs[0])

Useful helpers:

• createTensor(name, dtype, shape, data)
• createEmptyTensor(name, dtype, shape)
• float32, float64, int8, int16, int32, int64
• uint8, uint16, bool, float16, bfloat16
• toTypedArray, toFloat32Array, toFloat64Array
• toInt32Array, toUint8Array, toUint16Array
• assertDType, assertShape
• byteLengthForShape, elementCount, validateShape

uint16, float16, and bfloat16 expect raw packed Uint16Array values. float16 and bfloat16 do not convert normal JavaScript number values into half-precision formats yet.

BenchmarkTools

import { BenchmarkTools } from 'native-onnx'

const stats = await BenchmarkTools.benchmarkSession(session, [input], {
  warmupRuns: 2,
  runs: 10,
})

console.log(stats.meanMs, stats.p95Ms)

Use the same model and input tensors when comparing CPU, XNNPACK, NNAPI, QNN_HTP, and QNN_GPU.

Diffusion Pipeline

The diffusion API is experimental. It is meant to move reusable image-generation pipeline work into this package instead of forcing each app to wire ONNX sessions manually.

import { DiffusionPipeline } from 'native-onnx'

const pipeline = await DiffusionPipeline.load({
  modelDir,
  backend: 'QNN_HTP',
  threads: 4,
  metadataJson,
  sessionOptions: {
    requireFullBackend: true,
    qnnPerformanceMode: 'burst',
    qnnGraphOptimizationMode: '3',
  },
})

const metadata = pipeline.getMetadata()

pipeline.close()

By default, DiffusionPipeline.load(...) expects a Qualcomm Stable Diffusion v2.1 precompiled QNN ONNX extraction directory containing:

text_encoder.onnx
unet.onnx
vae.onnx
text_encoder_qairt_context.bin
unet_qairt_context.bin
vae_qairt_context.bin

If your app extracts files under different names or nested directories, pass explicit component paths:

const pipeline = await DiffusionPipeline.load({
  modelDir,
  backend: 'QNN_HTP',
  modelFiles: {
    textEncoder: 'stable_diffusion_v2_1/text_encoder.onnx',
    unet: 'stable_diffusion_v2_1/unet.onnx',
    vae: 'stable_diffusion_v2_1/vae.onnx',
  },
})

Recommended Model Preparation Flow

Apps should own download, extraction, storage, selection, deletion, and UI state. The library should own the ONNX pipeline smoke test.

import { DiffusionPipeline } from 'native-onnx'

async function prepareImageModel(archivePath: string, modelDir: string) {
  await extractArchiveIfNeeded(archivePath, modelDir)

  await assertFilesExist(modelDir, [
    'text_encoder.onnx',
    'unet.onnx',
    'vae.onnx',
    'text_encoder_qairt_context.bin',
    'unet_qairt_context.bin',
    'vae_qairt_context.bin',
    'metadata.json',
  ])

  const metadataJson = await readText(`${modelDir}/metadata.json`)
  const pipeline = await DiffusionPipeline.load({
    modelDir,
    backend: 'QNN_HTP',
    threads: 4,
    metadataJson,
    sessionOptions: {
      requireFullBackend: true,
      qnnPerformanceMode: 'burst',
      qnnGraphOptimizationMode: '3',
    },
  })

  try {
    const metadata = pipeline.getMetadata()

    if (
      metadata.textEncoder.inputs.length === 0 ||
      metadata.unet.inputs.length === 0 ||
      metadata.vae.inputs.length === 0
    ) {
      throw new Error('Diffusion model sessions loaded without expected inputs')
    }
  } finally {
    pipeline.close()
  }
}

Only mark a downloaded model as ready after this smoke test passes.

Component-level calls:

const textEmbedding = await pipeline.encodeTokens(tokens)

const noisePrediction = await pipeline.runUnet({
  latent,
  timestep,
  textEmbedding,
})

const image = await pipeline.decodeLatent(finalLatent)

DiffusionTensorTools includes the model-specific tensor helpers used by those calls, including Qualcomm Stable Diffusion v2.1 metadata, seeded latent noise, UNet/VAE UINT16 quantization, and VAE image tensor to RGBA conversion.

generate(...) is part of the public API shape but intentionally throws for now. The package currently owns ONNX session loading and component execution. Remaining work before prompt-to-image generation:

• tokenizer
• scheduler loop
• classifier-free guidance
• image file encoding/writing

Qualcomm Stable Diffusion Target

The first image-generation target is Qualcomm's optimized Stable Diffusion v2.1:

• Model repo: https://huggingface.co/qualcomm/Stable-Diffusion-v2.1
• AI Hub model page: https://aihub.qualcomm.com/models/stable_diffusion_v2_1
• Export recipe: https://github.com/qualcomm/ai-hub-models/tree/main/src/qai_hub_models/models/stable_diffusion_v2_1
• QNN EP docs: https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html

Recommended asset type for Android NPU testing:

Stable-Diffusion-v2.1
Runtime: PRECOMPILED_QNN_ONNX or QNN_CONTEXT_BINARY
Precision: w8a16
Backend: QNN_HTP

Download with Qualcomm's CLI:

pip install qai_hub_models_cli
qai-hub-models info Stable-Diffusion-v2.1
qai-hub-models perf Stable-Diffusion-v2.1
qai-hub-models fetch Stable-Diffusion-v2.1 --runtime qnn_context_binary --precision w8a16

Or export for a specific device family:

pip install "qai-hub-models[stable-diffusion-v2-1]"
qai-hub configure --api_token API_TOKEN
qai-hub-models export stable_diffusion_v2_1 \
  --target-runtime qnn_context_binary \
  --precision w8a16 \
  --device "Samsung Galaxy S25 (Family)"

Qualcomm's published release assets list QAIRT 2.45 and ONNX Runtime 1.25.0. This package currently builds against onnxruntime-android-qnn:1.27.0, so verify asset/runtime compatibility on the target device before treating results as final.

Qualcomm's precompiled Stable Diffusion components use UINT16 activation tensors. ONNX Runtime Java can describe UINT16, but does not expose a public OnnxJavaType.UINT16 input carrier. This package isolates the Android workaround inside the native bridge.

Model Notes

QNN HTP/NPU is not a generic "run anything on NPU" switch. Expect these constraints:

• models should have fixed/static input shapes
• QNN HTP commonly requires quantized models
• operator support varies by QNN backend
• unsupported graph parts may fall back or fail depending on session config

For image generation, prefer ONNX models that are:

• already exported to ONNX
• mobile-friendly or distilled
• split into smaller components when possible
• available in FP16 or quantized form
• compatible with static input shapes

Troubleshooting

Unsupported backend

Check available backend strings on the device:

console.log(NativeOnnx.getSupportedBackends())

Session works but may not be on NPU

Use requireFullBackend: true and run a model-specific smoke test. Provider availability alone does not prove full-graph NPU execution.

Tensor byte-size mismatch

Check the model's expected input metadata:

console.log(session.getInputs())

Then make sure your tensor dtype, shape, and ArrayBuffer.byteLength match exactly.

Development

bun install
bun run codegen
bun run typecheck
bun run build

Run bun run codegen after changing any src/specs/*.nitro.ts file.

Before publishing:

bun run build
npm pack --dry-run

License

MIT