Inferences

Shipping Label Inference

You can have the inference via the API or the vision-SDK while the vision-SDK will do the network request for you, the responses from this document will remain the same.

# New Inference

To create a new inference for shipping-labels, you'll need to pass the image_url as a base64 encoded data URL or public web URL. There's a soft cap of about 2.5MB per image for the base64 URL, and the whole request body must stay under 10MB.

You can also send several images of the same package in one inference - for example a top and a bottom photo when you do not know which side carries the label. PackageX runs the models over every image, picks the one that carries the label, and returns one inference. See Multiple Images Per Inference.

You're also able to specify what type of image the parser is looking at, so it can better extract data accurately along with other helpful information listed below:

image_url string | Array.<String> (required)
A base64 encoded data URL or public web URL, or an array of up to 4 of them when you are sending several images of the same package. A web URL must be https and name a public host rather than an IP address. See Multiple Images Per Inference for the full rules.

image string | Array.<String>
An alias for image_url, accepting the same single value or array. When both are supplied, image wins. One of image or image_url is required, and null on either key counts as "not set" so the other is used - but an empty string or empty array on image does not fall back to image_url, it returns 400 inference.image. Send only the key you mean rather than blanking one out.

barcode_values Array.<String> | Array.<Array.<String>>
Any existing values you've already extracted from the barcode. These are merged with the values read from the image rather than replacing them, and the result is returned as barcode_list.

With more than one image, send an array of lists so each entry lines up with the image at the same position in image / image_url - barcode_values: [["1Z..."], []] gives the first barcode to the first image and none to the second. Each image is extracted using only its own barcodes. A single flat list sent alongside several images is taken as belonging to the first.

Because they are attached to a specific image, the values you send follow the same rule as the barcodes we read off the images: barcode_list carries the selected image's, or every image's when options.images.barcodes_from_all is set. With one image it always carries yours.

location_id String
The ID of the location you want to attribute to this scan.

layout_id String
The ID of the layout to attribute to this scan. The layout must be associated with the specified location_id.

child_shipments Array.<String>
An array of shipment IDs to attach as child shipments to the parent shipment created from this inference. This enables the parent-child shipment hierarchy where a parent shipment (e.g., a pallet or consolidated box) contains multiple child shipments. Can also be an object with add, remove, or set arrays for updating an existing inference. Only applies when the tracker type is "outbound". Child shipments must be at the same location and have a status of created, delivered, or return_to_sender.

options Object
Option contains different other optional parameters you can provide along with the image. Those parameters are described below

        enum TrackingStatusOutstanding {
  provider_at_pickup
  package_accepted
  out_for_delivery
  provider_at_dropoff
  scheduled
  driver_dispatched
  package_at_waypoint
  in_transit
  pickup_available
  recipient_at_pickup
  added_to_container
  removed_from_container
  added_to_vehicle
  removed_from_vehicle
  delivered_to_provider
}

      

        enum TrackerCloseShipmentsOption {
  on_provider_updates
  on_all_updates
  off
}

      

        enum TrackerCloseShipmentsStatus {
  delivered
  picked_up
}

      

        const data = {
  image_url: "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAASA...", //truncated
  barcode_values: [], //If you have already extracted any barcode values from a shipping label
  location_id: null, //You an optionally pass the location ID to later filter scans by location
  layout_id: null, //You can optionally pass the layout ID. The layout must be associated with the specified location_id
  child_shipments: ["ship_abc123", "ship_def456"], //Optional array of child shipment IDs to attach to the parent shipment (outbound only)
  options: {
    tracker: {
      status: "pickup_available",
      create_automatically: false,
      type: "inbound",
      parcel: "always_create",
      chained_status: "in_transit",
      auto_close_shipments_option: "on_all_updates",
      auto_close_shipments_status: "picked_up",
      create_destination_shipment: false,
      destination_shipment_location_id: null,
      create_synchronously: false, //create the tracker inside this request; requires create_automatically
    },
    images: {
      //only used when you send several images - see "Multiple Images Per Inference"
      classification: "from_extraction_only",
      on_ambiguity: "use_best",
      barcodes_from_all: false, //true collects barcodes from every image, not just the winner
    },
    postprocess: {
      parse_addresses: ["sender", "recipient"],
    },
    match: {
      location: false,
      search: ["recipient"],
    },
  },
};

const res = await fetch("https://api.packagex.io/v1/inferences/images/shipping-labels", {
  method: "POST",
  headers: {
    "PX-API-KEY": process.env.PX_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(data),
}).then((res) => res.json());

const scan = response.data;

      

# Multiple Images Per Inference

image and image_url each accept a single value or an array of up to 4 values, so you can send several photos of the same package in one request - for example a top and a bottom frame when you do not know which side carries the label. PackageX runs the models over every image, picks the one that carries the shipping label, and returns one inference built from that image.

        const data = {
  image: [
    "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ...", //truncated
    "data:image/jpeg;base64,/9j/4AAQSkZJRgACBQ...", //truncated
  ],
  metadata: { camera_0: "top", camera_1: "bottom" }, //your own labels, with the image index in the key name - keep metadata flat
  options: {
    images: { classification: "from_extraction_only", on_ambiguity: "error" },
  },
};

      

Rules:

• Maximum 4 images per inference. More returns 400 with the code inference.too_many_images.
• The whole request body must stay under 10MB, shared across all the images. For large photos use image_url rather than base64. Extraction runs at the resolution you send, so higher resolution is not wasted; the image_url values returned on the inference point at a copy downscaled to a 1024px bounding box.
• An empty array returns 400 inference.image. An entry that is neither a URL nor a base64 string is rejected in validation instead, as 400 validation.failed.
• Entries may mix the two forms - one base64 data URL alongside two web URLs in the same array is fine.
• If both image and image_url carry values, image is used and image_url is ignored. Send your images through one of the two, not both.
• A web URL must be 255 characters or shorter, query string included, or the request fails validation with 400 validation.failed. Worth checking before you switch to image_url: pre-signed storage links routinely exceed that. Base64 data URLs are not subject to the limit.
• A web URL must be https and name a host, not an address. A bare IPv4 or IPv6 literal, localhost, and the cloud metadata host are rejected with 400 (inference.url, inference.hostname), and a non-https scheme with 400 inference.protocol. If your images live somewhere that is not reachable over public HTTPS, send them inline as base64 data URLs instead. Data URLs are exempt from all of this.
• An image we cannot fetch, decode or read fails the whole request, and nothing is persisted - no inference, no inference.created event, so there is nothing to look up afterwards. That covers an unreachable or non-image URL, an undecodable base64 payload, and an OCR failure, on any one of the images. A fetch that stalls is abandoned rather than held open, and comes back as 408 file.timeout_error; an implausibly large file is refused as 400 file.too_large; and if the host serving the URL answers with a 5xx you get a 502 file.download, which is worth retrying, unlike the others. All of these are a different failure from the per-image extraction errors below, which are recorded on the inference and leave the other images to compete.
• Multiple images are supported only when creating an inference. The update endpoint accepts no image / image_url at all, so anything you send there is ignored rather than rejected: an update never re-runs inference, so there is nothing for an image to act on.
• Sending more than one image alongside the vision-SDK's on_device_extraction returns 400 inference.image: one already-extracted result cannot be attributed to one of several images. A single image is fine either way, whether you pass it bare or as a one-element array.

The inference itself is built from the winning image: image_url, raw_text and every extracted field come from it. media_urls lists every stored image regardless of which one won, and barcode_list can be widened to include barcodes read from the losing images too with options.images.barcodes_from_all, since a barcode on the other side of the package is still real data - by default it carries the winning image's barcodes only.

# Which fields hold what, with one image versus several

Where each of these fields gets its value depends on how many images you sent:

On the missing_label and ambiguous_label paths the inference still ends in status: "error", but the extraction is kept, not discarded. Everything the images yielded is returned: raw_text, tracking_number, the carrier and party fields, the parcel dimensions, the merged barcode_list, your metadata, and an entry in images for every image carrying its own scores, document_classification and error. On ambiguous_label the top-ranked candidate is adopted into image_url and the extracted fields, because it is the answer we would have given; on missing_label no candidate was eligible, so those stay null. One field is deliberately withheld: hash is null on both paths, so re-sending the same image is not rejected as a duplicate of an inference that produced nothing. What does not happen on either path is the rest of the pipeline: no contact matching, no tracker, no media handling and no logs. errors carries that single code and nothing else, and the request fields that are normally applied after extraction are not applied either - the sender / recipient details you passed, the provider and reference-number overrides, media, a log entry, child_shipments, and the postprocess.provider_selection step that would otherwise pick between two extracted carriers (so provider reports what was read off the label, unfiltered). What you sent that does survive is metadata, the tags you set or added, notes, location_id, layout_id, weight and dimensions. The barcode_values you sent follow the same per-image rule as the barcodes we read off the images, so with several images they reach barcode_list only from the selected image - which means on missing_label, where nothing was selected, they are not carried onto the row. Set options.images.barcodes_from_all if you need them retained regardless of the outcome. With one image they are always kept.

Gate on status and errors, and on nothing else. Neither image_url nor images[].selected nor a populated tracking_number means the inference succeeded - all three can be set on a refused ambiguous_label result, where they describe the answer we would have given rather than one you should act on. This is deliberate: a refused inference that tells you what it saw is worth far more to whoever investigates it than an empty row.

# Image classification modes

options.images.classification controls what role the document classifier plays over your images.

Under "classifier_feeds_extraction", an image classified as something other than a shipping label is not extracted; it is still returned in images, carrying the class in document_classification, with scores: null (there was no extraction to score) and error: null (nothing failed - it was simply never sent to the label model). If classification rules out every image, all of them are extracted anyway rather than failing the inference, and an image whose classification call itself errored is recorded as classification_failed and extracted too - a classifier outage is not evidence about the image, so it neither drops the image nor costs it any score.

# How the winning image is chosen

Each image is scored on the hard signals in its extraction - a tracking number, a tracking number read from a barcode, a recognized provider (the catch-all other earns nothing, since it is no evidence of a label), a barcode being present, a complete recipient or sender address, a service level - plus the document class when a classifier ran. The two halves are reported separately as scores.extraction and scores.classification, and scores.total is their sum floored at 0. The classification half is graded by how confidently the classifier said "not a shipping label": shipping_label adds; a positive identification of a different document type is strong evidence against the image and costs the most; unknown means the classifier looked and could not place it, which is real but weak negative evidence and costs a partial penalty; and classification_failed, or no classification at all, costs nothing, because docking every image during a classifier outage would be a verdict about our infrastructure rather than about the image. The highest scores.total wins, and ties are broken by the amount of text read, then by submission order. The exact weights are internal and may be tuned; do not build logic on specific scores values, only on selected and on relative order.

The document class alone can never win an image. An image must carry evidence from its own extraction - a non-zero scores.extraction - to be considered at all; the class may reorder images that already have such evidence, but it cannot promote one that has none. Eligibility is judged on scores.extraction and not on scores.total, so an image with real extracted evidence stays in the running even when a confident non-label class floors its scores.total to 0. This follows from the classifier returning a bare class with no confidence score. A classification penalty therefore never removes an image from consideration - it only affects the ranking, and whether the image still clears the plausibility bar that feeds the ambiguity check. That is worth knowing, because it explains an otherwise puzzling outcome: a weakly-extracted image classified unknown can still be the one that gets selected, while no longer counting as a competing label for the purpose of declaring the batch ambiguous. So two blank package sides that the classifier happens to call shipping_label give missing_label, not an ambiguous pair.

If no image is eligible, what you get back depends on whose fault it was. A per-image 4xx is a real answer about that image - 418 image.no_text for a blank side, 400 image.invalid - so the inference ends in missing_label with status: "error" and a successful HTTP status. But if any of the images failed with a 5xx instead, or with no recognizable status at all, that upstream error becomes the request's error: the call fails with the upstream status rather than returning a 200 carrying missing_label. The inference row is still recorded, with status: "error" and each image's own error, so you can see what happened. Laundering an outage into missing_label would have you divert every parcel while the real fault went unnoticed.

A single image is always used. Scoring exists only to choose between images, so it can never reject the only image you sent. missing_label and ambiguous_label are therefore only possible when you send 2 or more images.

# Ambiguity

Two frames of the same package showing the same label is the normal case and is not ambiguous - the higher-scoring one simply wins. An inference is ambiguous when two or more images both look like shipping labels and they cannot be reconciled:

• their tracking numbers differ (so this may be two packages, or a misread), or
• none of them produced a tracking number (so there is nothing to tell them apart).

options.images.on_ambiguity decides what happens:

• "use_best" (default) uses the highest-scoring image and proceeds. Nothing in the response says the batch was ambiguous: no field records it, and because only the winner's extraction is returned you cannot compare the losing tracking numbers yourself. The other candidates are listed in images with their scores, so a close second is a hint that something was there to choose between - but it is only a hint. If your process needs to know for certain, use "error".
• "error" refuses to guess: status is "error" and errors contains ambiguous_label. The extraction is still returned in full - the top-ranked candidate's fields are on the inference and every image's scores are in images - but the pipeline stops there: no matching, no tracker. Treat the populated fields as a report, not a result, and gate on errors.

Use "error" when acting on the wrong package is worse than handling an exception - an automated conveyor, for example.

# New response fields

images Array.<Object> | null
One entry per image you sent, in the order you sent them - present only when you sent two or more. null when you sent a single image, since there was nothing to choose between and the inference's own fields already describe it, and null on inferences created before multiple images were supported.

A two-image images array, under the default "from_extraction_only", where the second frame was a blank package side:

        "images": [
  {
    "index": 0,
    "image_url": "https://storage.packagex.io/scans/org_.../0.jpg",
    "document_classification": null,
    "scores": { "extraction": 70, "classification": 0, "total": 70 },
    // "timings": {
    //   "total_ms": 1840,
    //   "total_stages": 4,
    //   "stages": [
    //     { "index": 0, "stage": "prepare", "time_ms": 190 },
    //     { "index": 1, "stage": "upload", "time_ms": 160 },
    //     { "index": 2, "stage": "ocr", "time_ms": 620 },
    //     { "index": 3, "stage": "extract", "time_ms": 1030 }
    //   ]
    // },
    "selected": true,
    "error": null
  },
  {
    "index": 1,
    "image_url": "https://storage.packagex.io/scans/org_.../1.jpg",
    "document_classification": null,
    "scores": null,
    // "timings": {
    //   "total_ms": 970,
    //   "total_stages": 4,
    //   "stages": [
    //     { "index": 0, "stage": "prepare", "time_ms": 180 },
    //     { "index": 1, "stage": "upload", "time_ms": 150 },
    //     { "index": 2, "stage": "ocr", "time_ms": 410 },
    //     { "index": 3, "stage": "extract", "time_ms": 330 }
    //   ]
    // },
    "selected": false,
    "error": { "code": "image.no_text", "message": "The provided image did not have any text to extract", "status": 418 }
  }
]

      

Those seven keys are the whole entry. The score numbers are illustrative - they are relative, and the weights may be tuned.

Only the winning image's extraction is returned: what the winner extracted is what you read off the inference itself (raw_text, provider, tracking_number, and the rest), and the other images' extractions are not part of the response - a losing image is described by its scores, document_classification and error alone. The raw OCR payloads, image hashes and image metadata are recorded internally and are not returned either.

# New error codes

Both appear in the inference's errors array with status: "error", and both are returned with a successful HTTP status - the failure is in the body so a single response tells you everything. When either is set it is the only entry in errors, since the inference stops before the steps that raise the others. Neither can occur with a single image.

Two request-level codes belong to the same feature, and both are real HTTP 400s rather than body errors: inference.too_many_images for more than 4 images, and inference.image for an image that is missing, blank, or sent as more than one image alongside the vision-SDK's on_device_extraction. Malformed option values (a misspelled classification, a non-boolean barcodes_from_all, a web URL over 255 characters) fail validation as 400 validation.failed.

# Synchronous Tracker Creation

By default, when options.tracker.create_automatically is true the tracker is created shortly after the inference response is returned: the create response comes back with status: "transforming" and tracker still null, and you learn about the tracker from the inference.transformed webhook or by polling.

Setting options.tracker.create_synchronously to true creates the tracker inside the request, so the response already carries status: "completed" and tracker.id. Nothing to poll for. This is intended for callers on a hard deadline - an automated conveyor that must know, before the package reaches the next checkpoint, whether it was accepted or must be diverted.

        const data = {
  image: ["data:image/jpeg;base64,...", "data:image/jpeg;base64,..."],
  options: {
    images: { on_ambiguity: "error" },
    tracker: {
      create_automatically: true, //required
      create_synchronously: true,
      type: "inbound", //recommended - see below
    },
    match: { search: [] }, //matching is the slowest stage; omit it if you need the fastest answer
  },
};

const res = await fetch("https://api.packagex.io/v1/inferences/images/shipping-labels", {
  method: "POST",
  headers: {
    "PX-API-KEY": process.env.PX_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(data),
}).then((res) => res.json());

if (res.data.status === "completed" && res.data.tracker?.id) {
  // the package exists
} else {
  // divert - res.data.errors says why (e.g. "missing_label", "ambiguous_label", "no_matches")
}

      

Things to know before enabling it:

• It only works alongside create_automatically: true. On its own it does nothing; it is forced to false when create_automatically is not set.
• A key you leave out falls back to your organization setting, including create_automatically. Sending options: { tracker: { create_synchronously: true } } on its own leaves auto-creation to settings.inferences.shipping_labels.tracker.create_automatically, so it creates a tracker synchronously if that setting is on, and creates nothing if it is off. Send create_automatically: true explicitly when you want it regardless of the organization setting. The example above does that, and names a type, so it reads the same whatever the organization is configured to do.
• It applies to later writes on the same inference too, not only to the create. The flag is stored on the inference, so if a follow-up update is what finally moves it to transforming - you supplied the contact that was missing, for example - that update creates the tracker inside its own request and returns it, the same way the create would have.
• It applies only when the inference reached transforming. An inference that ends in status: "error" - no confident contact match, an ambiguous label, a missing required property - does not get a tracker, synchronously or otherwise.
• Failures are reported in the body, not as an HTTP error. If tracker creation fails, the response still succeeds and the inference comes back with status: "error" and the reason in errors. This is deliberate, so an automated caller can branch on one response instead of handling a body in one path and an HTTP error in another. Never treat a successful HTTP status alone as "the package exists" - check status and tracker.id.
• Webhook order and count are unchanged. You get the same two events in the same order either way: inference.created carrying status: "transforming", then inference.transformed. The only difference is when they arrive - both are emitted during the request, so inference.transformed can reach your endpoint while you are still reading the HTTP response. Note that inference.transformed is emitted on failure as well as success, so read status on the payload rather than treating the event itself as confirmation that a tracker exists. See Events.
• "outbound" trackers are slower. An outbound tracker registers the package with the carrier inside your request, adding a third-party network call to your critical path. An "inbound" tracker does not. Prefer "inbound" for latency-sensitive integrations, and budget for the extra hop if you need "outbound".

# Inbound Rerouting (Multi-Hop)

When inbound multi-hop rerouting is enabled at the organization or location level, the system can automatically detect when a package does not have a recipient match at the scanned location and attempt to find the correct recipient across all locations in the organization.

If a match is found at a different location, the inference response will include a destination_location object indicating where the package should be rerouted to.

If the system matches a recipient but cannot determine the destination location, the inference will include a destination_location_unknown error. You can resolve this by updating the inference with a destination_location_id:

        const data = {
  destination_location_id: "loc_..."  // Must be one of the matched recipient's associated locations
};

const res = await fetch("https://api.packagex.io/v1/inferences/images/shipping-labels/sli-id", {
  method: "POST",
  headers: {
    "PX-API-KEY": process.env.PX_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(data),
}).then((res) => res.json());

const scan = res.data;
// scan.destination_location => { id: "loc_..." }
// scan.errors => [] (destination_location_unknown resolved)

      

# Inference Exceptions

The Exceptions feature allows us to flag specific shipping label inferences that contain anomalies or irregularities detected during processing. When an inference is in exception state, it can not be transformed however it can be reviewd and resolved out of the exception state and then it can be transformed.

This capability is useful in scenarios where the OCR model successfully processes the label image but encounters issues that affect the quality or reliability of the extracted data. Examples include:

unknown_contact - Contact information cannot be reliably identified.

addressed_generically - The label is addressed to a generic title (e.g., "Customer" or "Resident").

damaged_label - The physical label appears partially destroyed or unreadable.

suspicious - The label contains patterns that may indicate fraud or tampering.

arrived_opened - The package is detected to have been opened prior to delivery.

We can update an existing inference record and set it to exception, along with appropriate exceptions type.

        const data = {
  exceptions: {
    unknown_contact: true,
    suspicious: true
  }
};

const res = await fetch("https://api.packagex.io/v1/inferences/images/shipping-labels/sli-id", {
  method: "POST",
  headers: {
    "PX-API-KEY": process.env.PX_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(data),
}).then((res) => res.json());

const scan = response.data;