React Native

Dimensioning (React Native)

This guide documents the React Native surface for the Vision SDK's 3D box dimensioning feature. It covers integration, usage, gating, live guidance, custom overlays, telemetry, troubleshooting, and known limitations.

Contents

• Quick overview
• Installation & permissions
• Device capability gating
• VisionDimensioning helpers
• DimensioningView props, events and commands
• Live guidance and custom overlays
• Telemetry
• Cloud segmentation
• Troubleshooting
• Known limitations
• Source references

# Quick overview

• Dimensioning provides approximate length × width × height measurements for detected boxes using ARKit + LiDAR and optional cloud segmentation refinement (YOLO + SAM2).
• The React Native package exposes:
  • DimensioningView - native view component emitting onCapture, onError, onMeasurementUpdate, onOverlayUpdate and onTelemetry (iOS only), plus stop() / start() through a ref.
  • VisionDimensioning.deviceCapabilities() - inspect device support.
  • VisionDimensioning.prefetchModels() - warm the bundled CoreML models.

Always gate UI with deviceCapabilities() to avoid mounting the native view on unsupported devices.

# Installation

Dimensioning ships with react-native-vision-sdk.

        npm install react-native-vision-sdk
# or
yarn add react-native-vision-sdk

      

No additional RN packages are required, but ensure your iOS build has the Vision SDK dimensioning subspec available (see native integration notes in the SDK docs).

# Info.plist

Add camera and LiDAR usage descriptions to Info.plist:

        <key>NSCameraUsageDescription</key>
<string>Camera access is required to measure package dimensions.</string>
<key>Privacy - LiDAR Usage Description</key>
<string>LiDAR access is required for accurate 3D measurement.</string>

      

No extra ARKit permission keys are required beyond these usage descriptions.

# Device capability gating

Dimensioning only runs usefully on LiDAR-equipped iOS devices. Use the capability helper to gate rendering:

        import { VisionDimensioning } from 'react-native-vision-sdk';

async function canUseDimensioning() {
  const caps = await VisionDimensioning.deviceCapabilities();
  return caps.lidar && caps.arWorldTracking;
}

      

On simulator and on Android the helper returns all-false flags; the native view either emits LidarUnavailable or renders a placeholder.

# VisionDimensioning helpers

• deviceCapabilities(): Promise<DimensioningCapabilities> - returns { lidar, arWorldTracking, sceneReconstruction }.
• prefetchModels(): Promise<void> - idempotent warm-up for the bundled CoreML models.

Usage tips:

• prefetchModels() is optional. The models ship unencrypted and need no network, so it only saves the first capture's CoreML compile cost.
• deviceCapabilities() is safe to call cross-platform (stubbed on Android).

# DimensioningView component

# Props

# Events

# Commands

DimensioningView forwards a ref exposing camera control. ARKit and AVCaptureSession cannot share the rear camera, so call stop() before mounting <VisionCamera>:

        const dimRef = useRef<DimensioningViewHandle>(null);

<DimensioningView ref={dimRef} style={{ flex: 1 }} />

dimRef.current?.stop();   // tears the AR view down, releases the camera
dimRef.current?.start();  // re-creates it

      

# Example - basic usage

        import React, { useEffect, useState } from 'react';
import { Text } from 'react-native';
import { DimensioningView, VisionDimensioning } from 'react-native-vision-sdk';

export default function RNDimensioningScreen() {
  const [supported, setSupported] = useState(false);

  useEffect(() => {
    VisionDimensioning.deviceCapabilities().then((c) => setSupported(c.lidar));
  }, []);

  if (!supported) return <Text>Dimensioning not supported on this device</Text>;

  return (
    <DimensioningView
      mode="offline"
      measurementUnit="centimeters"
      maximumTrackCount={3}
      onCapture={(m) => console.log('measurement', m)}
      onError={(e) => console.warn('dimensioning error', e)}
      style={{ flex: 1 }}
    />
  );
}

      

# Capture payload

onCapture receives a DimensioningMeasurement:

        type DimensioningMeasurement = {
  id: string;
  trackId: string;            // stable id of the physical box
  timestamp: number;          // Unix seconds
  length: number;             // value in `lengthUnit`
  lengthUnit: string;
  width: number;
  widthUnit: string;
  height: number;
  heightUnit: string;
  distanceFromCamera: number;
  distanceFromCameraUnit: string;
  confidence: number;         // 0...1
  usedCloudSAM: boolean;      // true when the cloud path ran
  volume: number;             // cubic meters
  imagePixelSize: { width: number; height: number };
  boxVertices2D: Array<{ x: number; y: number }>;
};

      

boxVertices2D holds the 8 projected box corners in the captured frame's pixel space - indices 0–3 the base face, 4–7 the top face, corner k under corner k+4. With imagePixelSize that's enough to draw the measured box over a photo. Both are empty/zero when the SDK kept no frame.

The raw JPEG is deliberately not forwarded - shipping an image across the bridge on every capture would dwarf the rest of the payload.

# Live guidance - onMeasurementUpdate

Fires continuously with a coarse tracking state (searching → groundFound → boxDetected → stable) and the in-progress dimensions of every tracked box. Use it for a "hold steady… ready" HUD:

        <DimensioningView
  onMeasurementUpdate={(u) => {
    setStatus(u.trackingState);
    setBoxCount(u.tracks.length);
    // u.tracks[i].measurement is null until the pipeline has a reading
    // u.primaryTrackId marks the box the SDK considers primary
  }}
/>

      

# Custom overlays - onOverlayUpdate

Set overlayMode="callback" to suppress the SDK's own graphics and receive every overlay primitive each frame. All geometry arrives in view-space points, so it maps 1:1 onto the view:

        <DimensioningView
  overlayMode="callback"
  onOverlayUpdate={(f) => {
    f.boxes;   // boxVertices2D, contour2D, boundingBox, isStable, isSelected
    f.planes;  // boundary2D, center2D - detected support planes
    f.hud;     // statusText, guidanceText, isCapturing, groundPlanePrompt
  }}
/>

      

This fires on every frame. Keep the handler cheap and avoid setting state you don't render.

# Telemetry

Opt in with enableTelemetry and read onTelemetry:

        <DimensioningView
  enableTelemetry
  onTelemetry={(e) => {
    if (e.type === 'measurementAborted') {
      // 'timeout' | 'user_cancel' | 'no_dimensions'
      console.log('capture failed:', e.reason);
    } else {
      console.log(e.lengthCm, e.confidence, e.durationMs, e.consensusLevel);
    }
  }}
/>

      

Events go only to your handler. The SDK has no analytics backend of its own for dimensioning and makes no telemetry network calls. 'measurementAborted' is the only signal for a capture that gave up mid-session.

# Cloud segmentation

mode="offline" runs entirely on-device with no network and no key. mode="online" adds a cloud segmentation step that can improve accuracy on harder surfaces, at the cost of a per-capture HTTP request; usedCloudSAM is true on results where it landed.

Pass credentials explicitly:

        <DimensioningView
  mode="online"
  cloudUrl="https://…"
  cloudApiKey="…"
  cloudSdkId="…"
/>

      

Leave them unset to fall back to the host app's VSDKConstants - the behaviour before these props existed. With neither set, onError fires with MissingCredentials (0).

# Bundled CoreML models

The YOLO + SAM2 models ship unencrypted inside the framework as of VisionSDK 2.7.0. There is no decryption-key fetch, no first-launch network round-trip, and no Apple Developer Team requirement - dimensioning works offline from first launch for any signing team.

# Troubleshooting

• LidarUnavailable on-device: verify you are on a LiDAR-capable device (iPhone 12 Pro or later, iPad Pro 2020 or later) and that camera permission is granted.
• Simulator: dimensioning does not function there - deviceCapabilities() returns all-false and mounting DimensioningView emits an error.
• Android: the module is stubbed. Props and commands are accepted as no-ops and no events fire.
• No boxes detected: the whole box must be in frame with its top face visible, resting on a flat horizontal surface, roughly 40–90 cm away, held steady for 1–2 seconds. Cuboids only.
• A capture never completes: enable telemetry and read 'measurementAborted' - its reason distinguishes a timeout from a no-dimensions failure.
• Camera conflict with <VisionCamera>: ARKit and AVCaptureSession cannot share the rear camera. Call stop() on the dimensioning ref before mounting the scanner, and start() when you come back.

# Known limitations

• iOS-only; Android shows a placeholder view.
• onError covers pre-flight failures only (see the callout above).
• No imperative capture from JS. capture(), captureAll(), shutdown(), pause(), resume() and clearCaptured() exist on the native DimensioningSession, but that type has no camera preview and the preview view exposes no session handle - so they cannot be driven from the RN component. stop() / start() cover the camera-release case.
• The captured frame's raw JPEG is not forwarded across the bridge.
• Measurements are approximate - plan around roughly ±3–5 cm per dimension, and offer a re-capture.

# Source references

• React Native SDK: react-native-vision-sdk (DimensioningView, types, helpers)
• iOS native module: packagexlabs/vision-sdk-ios (VisionSDKDimensioning)
• CocoaPods: pod 'VisionSDK/Dimensioning' subspec in the iOS SDK distribution