Skip to content

API Reference

This document covers the public API exported from src/index.ts. All examples assume: 

import {
  loadSlp,
  saveSlp,
  saveSlpToBytes,
  loadVideo,
  Video,
  Mp4BoxVideoBackend,
  Labels,
  LabeledFrame,
  Instance,
  PredictedInstance,
  Skeleton,
  Track,
  LabelsSet,
  SuggestionFrame,
  ROI,
  UserROI,
  PredictedROI,
  SegmentationMask,
  UserSegmentationMask,
  PredictedSegmentationMask,
  BoundingBox,
  UserBoundingBox,
  PredictedBoundingBox,
  Centroid,
  UserCentroid,
  PredictedCentroid,
  getCentroidSkeleton,
  CENTROID_SKELETON,
  LabelImage,
  UserLabelImage,
  PredictedLabelImage,
  normalizeLabelIds,
  readTrackMateCsv,
  loadTrackMate,
  isTrackMateFile,
  loadUltralytics,
  saveUltralytics,
  readLabels as readUltralytics,
  writeLabels as writeUltralytics,
  readLabelsSet as readUltralyticsSet,
  parseLabelFile,
  detectLineFormat,
  loadJabs,
  JABS_DEFAULT_SKELETON,
  makeJabsDefaultSkeleton,
  predictionToInstance,
  staticObjectToRoi,
  makeSimpleSkeleton,
  Identity,
  Instance3D,
  PredictedInstance3D,
  Camera,
  CameraGroup,
  RecordingSession,
  toDict,
  fromDict,
  toNumpy,
  fromNumpy,
  roisToGeoJSON,
  roisFromGeoJSON,
  writeGeoJSON,
  readGeoJSON,
} from "@talmolab/sleap-io.js";

Core I/O

loadSlp(source, options)

Read .slp from a path, URL, File, FileSystemFileHandle, or byte buffer.

const labels = await loadSlp("/data/session.slp", {
  openVideos: false,
  h5: { stream: "auto", filenameHint: "session.slp" },
});
  • source: string path/URL, File, FileSystemFileHandle, Uint8Array, or ArrayBuffer.
  • options.openVideos (default true): set false to skip opening video backends.
  • options.h5.stream: "auto" | "range" | "download" (browser URL streaming).
  • options.h5.filenameHint: helps name temporary files.
  • options.lazy (default false): use lazy loading for on-demand frame materialization.

saveSlp(labels, filename, options)

Write .slp (Node/Electron).

await saveSlp(labels, "/tmp/roundtrip.slp", {
  embed: false,
  restoreOriginalVideos: true,
});
  • options.embed: which frame images to embed, as Python sleap-io's embed:
  • true / "all": every labeled frame and every suggestion ("all+suggestions" is an alias)
  • "user": user-labeled frames (user instances, negative frames, user centroids/boxes/masks/ROIs)
  • "suggestions" / "user+suggestions"
  • an explicit list of [video, frameIdx] pairs: exactly those frames
  • "source": no images; restore source video paths before writing
  • false (default): no embedding (already-embedded images are still kept)
  • options.embedAllVideos (default true, as Python's embed_all_videos): when embedding, a video with no frames to embed becomes an empty embedded video referencing the package (keeping its shape and source video), so the package is portable. false leaves such videos as external references.
  • options.onProgress(current, total, message): called as frames are read ("Embedding frames") and written ("Writing frames"). Return false to cancel: the save throws ExportCancelled.
  • options.restoreOriginalVideos (default true): keep original video paths.

saveSlpToBytes(labels, options)

Serialize Labels to SLP format and return the bytes. Works in both Node.js and browser environments.

const bytes: Uint8Array = await saveSlpToBytes(labels, { embed: false });
  • labels: Labels object to serialize.
  • options: the same embed, embedAllVideos, onProgress and restoreOriginalVideos options as saveSlp.
  • Returns Promise<Uint8Array>.

loadVideo(source, options)

Open a Video with an appropriate backend.

// From file path (Node.js) or URL (browser)
const video = await loadVideo("/data/movie.mp4", { openBackend: true });
const frame0 = await video.getFrame(0);
video.close();

// From File object (browser)
const video = await loadVideo(fileInput.files[0]);
  • source: string (path or URL) or File (browser).
  • options.dataset: dataset path for embedded HDF5 sources.
  • options.openBackend (default true): set false for deferred open.
  • options.backend: explicit backend type ("mp4box", "mediabunny", "media", "hdf5").

Video Backends

Video

Video wraps a backend and exposes shape, fps, getFrame, getFrameTimes, and close.

const video = await loadVideo("/data/movie.mp4");
const timestamps = await video.getFrameTimes(); // number[] | null

// shape and fps have setters
video.shape = [100, 480, 640, 3];
video.fps = 30;
  • shape and fps have both getters and setters.
  • getFrame(index, signal?) accepts an optional AbortSignal to cancel in-flight decodes.
  • getFrameTimes() returns null if the backend does not implement it.

Virtual crops

Video.crop returns a virtual, on-read cropped view (no pixels are copied or re-encoded). See Virtual Cropping for the full guide.

const view = full.crop([320, 200, 576, 456]); // [x1, y1, x2, y2], x2/y2 exclusive
view.shape; // [frames, 256, 256, channels]
view.cropRect; // [320, 200, 576, 456]
view.isCropped; // true
view.cropFill; // 0
view.sourceVideo === full; // true

// Region specs (exactly one): explicit rect, bbox, roi, or center+size.
full.crop(null, { bbox: [x1, y1, x2, y2] });
full.crop(null, { roi: myRoi, margin: 8 });
full.crop(null, { center: [cx, cy], size: [128, 128], fill: 0 });

// Coordinate helpers (identity passthrough on an uncropped video):
const local = view.toCropCoords(sourcePts); // subtract (x1, y1)
const back = view.toSourceCoords(local); // add (x1, y1)
  • Video.crop(crop?, opts?) — one region spec across crop / opts.bbox / opts.roi / (opts.center + opts.size); plus opts.margin, opts.fill (out-of-bounds pad value, default 0), and opts.shareDecode (default true, reuse the source decoder for mosaics).
  • Video.fromCrop(video, crop?, opts?) — static wrapper around video.crop(...) (video must be a Video; path strings are unsupported in the JS port).
  • Getters: isCropped, cropRect ([x1,y1,x2,y2] | null), cropFill.
  • toCropCoords(pts) / toSourceCoords(pts) — NaN-preserving copies; accept [x,y] pairs or a flat interleaved buffer.
  • Standalone primitives: cropPoints, uncropPoints, cropFrame, and the CropVideoBackend class are all exported (browser-safe).

Mp4BoxVideoBackend

Browser-only backend for .mp4 with WebCodecs + mp4box. Supports range requests when the server honors Range and falls back to full download.

// From URL
const backend = new Mp4BoxVideoBackend("/data/movie.mp4");

// From File or Blob (browser)
const backend = new Mp4BoxVideoBackend(file);

const video = new Video({ filename: "/data/movie.mp4", backend });
const t = await video.getFrameTimes();

Constructor: new Mp4BoxVideoBackend(source: string | File | Blob, options?)

Notes: - Requires WebCodecs (VideoDecoder, EncodedVideoChunk). - Uses a Range: bytes=0-0 probe to detect range support (no longer requires HEAD). - Provides precise frame timestamps via getFrameTimes(). - getFrame() uses an async queue internally to prevent race conditions, and accepts an optional AbortSignal.

Hdf5VideoBackend

Used for .slp, .h5, or .hdf5 sources (embedded frames).

  • Selected automatically by loadVideo() for HDF5 inputs.
  • Decodes embedded PNG/JPEG frames in browsers; raw frames require shape + channelOrder.

MediaVideoBackend

Browser fallback when WebCodecs/mp4box is unavailable.

  • Uses HTMLVideoElement + canvas.
  • No getFrameTimes() support.

Labels & Model Classes

Key types:

  • Labels, LabeledFrame, Instance, PredictedInstance
  • Skeleton, Track
  • Video, SuggestionFrame, LabelsSet
  • Camera, CameraGroup, RecordingSession, InstanceGroup (camera/3D utilities)
  • Identity (ground-truth animal identity across sessions)
  • Instance3D, PredictedInstance3D (3D keypoint data)
  • LazyDataStore, LazyFrameList (lazy loading)
  • ROI, UserROI, PredictedROI (spatial annotations)
  • SegmentationMask, UserSegmentationMask, PredictedSegmentationMask (binary masks)
  • BoundingBox, UserBoundingBox, PredictedBoundingBox (detection/tracking)
  • Centroid, UserCentroid, PredictedCentroid (point detections/tracking)
  • LabelImage, UserLabelImage, PredictedLabelImage (dense instance segmentation)
  • Point has fields { xy, visible, complete, score? } where score is an optional confidence value.
  • LabeledFrame.isNegative (boolean, default false): marks negative-annotated frames.
  • SuggestionFrame.group (string, default "default"): the suggestion group name.
const skeleton = new Skeleton(["nose", "tail"]);
const inst = Instance.fromArray([[10, 20], [30, 40]], skeleton);
const video = new Video({ filename: "/data/movie.mp4" });
const frame = new LabeledFrame({ video, frameIdx: 0, instances: [inst] });
const labels = new Labels({ labeledFrames: [frame], skeletons: [skeleton], videos: [video] });

// Add a video with deduplication
labels.addVideo(video); // no-op if already present

Frame and track index lookups

Labels builds lazy Map indices for O(1) frame and track lookups, replacing linear scans through labeledFrames. The frame-aware find() / getCentroids() / getBboxes() / getMasks() / getLabelImages() / getRois() methods use these indices internally when both video and frameIdx are passed.

// O(1) frame lookup
const lf = labels.getFrame(video, 42);  // LabeledFrame | null

// O(1) track annotation lookup, sorted by frameIdx
const annotations = labels.getTrackAnnotations(video, track);

// Force index rebuild after external mutations
labels.reindex();

Indices are built lazily on first access and auto-invalidated when frames are added or videos replaced. See the Lazy Loading section below for the caveat that applies when the Labels is lazy.

Deep copy and video remapping

Labels.copy() produces a fully independent copy of a Labels object with consistent internal references — frame videos point to the copy's video objects, instance skeletons point to the copy's skeletons, etc. Both eager and lazy Labels are supported.

Labels.replaceVideos() rewrites video references across every collection in one call: labeledFrames, suggestions, sessions, staticRois, temporalRois, masks, labelImages, and centroids.

// Deep copy (eager or lazy — matches the original)
const copy = labels.copy();

// Suppress backend auto-opening on the new copy (faster, safer for bulk ops)
const detached = labels.copy({ openVideos: false });

// Replace video references — specify both old and new
labels.replaceVideos({ oldVideos: [oldVid], newVideos: [newVid] });

// Or infer old from current
labels.replaceVideos({ newVideos: [newVid] });

// Or pass a Map
labels.replaceVideos({ videoMap: new Map([[oldVid, newVid]]) });

LazyDataStore.copy() duplicates the underlying HDF5 column arrays so that two Labels wrapping the same source file can diverge safely.

Frame merging

Merge annotations from one LabeledFrame into another with strategy-aware handling. LabeledFrame.mergeAnnotations(other, strategy?, threshold?) supports six strategies, applied across all annotation modalities (centroids, bboxes, masks, label images, ROIs). To populate a single LabeledFrame in the first place, see Adding annotations to frames below.

Strategy Behavior
"keep_original" Keep self only, discard everything from other
"keep_new" Replace self's annotations with (copies of) other's
"keep_both" (default) Deduplicate by object identity, union both sets (items from other are shallow-copied before insertion)
"replace_predictions" Keep user annotations from self; drop self's predicted and take all predicted from other
"auto" Spatially match by centroid distance; user beats predicted; add unmatched
"update_tracks" Spatially match and cascade track assignments only

The auto and update_tracks strategies compare centroid distances against threshold (default 5.0 px). Empty masks and ROIs whose centroid is null are skipped by the matcher.

Under "auto", mask merging honors the fromPredicted provenance link before spatial matching: if a user mask in either frame links to a predicted mask in the other (by object identity), that pair is matched regardless of distance, and only falls back to centroid proximity when no link exists. Other annotation modalities have no fromPredicted link, so they are unaffected.

The strategy argument is typed as MergeStrategy, re-exported from the package root for type-safe call sites:

import type { MergeStrategy } from "@talmolab/sleap-io.js";

const strategy: MergeStrategy = "auto";
lfA.mergeAnnotations(lfB, strategy, 3.0);

// Default behavior (keep_both): dedupe by identity, union everything
lfA.mergeAnnotations(lfB);

// Spatial auto merge with a 3 px threshold
lfA.mergeAnnotations(lfB, "auto", 3.0);

// Keep only predicted updates from other, leave user annotations alone
lfA.mergeAnnotations(lfB, "replace_predictions");

Lazy Loading

Load SLP files with on-demand frame materialization for better performance on large datasets.

// Load lazily - only metadata is parsed initially
const labels = await loadSlp("large_dataset.slp", { lazy: true });
console.log(labels.isLazy);  // true

// Access individual frames on demand
const frame = labels.labeledFrames.at(0);

// Materialize all frames when needed
labels.materialize();
console.log(labels.isLazy);  // false

Key classes:

  • LazyDataStore: Holds raw HDF5 column data; supports materializeFrame(idx), materializeAll(), copy(), and toNumpy() fast path.
  • LazyFrameList: Array-like container with at(idx), length, toArray(), [Symbol.iterator](), and materializedCount.

Note: labels.getFrame() and labels.getTrackAnnotations() throw on lazy Labels. These O(1) lookups rely on fully materialized frame/track indices and would otherwise silently return stale or partial results. You have two options, both of which trigger full materialization under the hood:

  1. Call labels.materialize() explicitly, then use the O(1) lookups.
  2. Call labels.find({ video, frameIdx }), which materializes internally on first call and then uses the same O(1) index. It returns LabeledFrame[] — take the first element.

Both paths do the same amount of work on the first call; there's no cheap "lazy-preserving" variant of frame lookup by (video, frameIdx).

// Option 1: materialize first, then O(1) lookup
labels.materialize();
const lf = labels.getFrame(video, 42);

// Option 2: find() — equivalent, materializes on first call
const [frame] = labels.find({ video, frameIdx: 42 });   // LabeledFrame | undefined

ROI & Segmentation Masks

Spatial annotations stored alongside pose data (SLP format 1.5+).

ROI

Region of interest with GeoJSON-like geometry. ROI is abstract; use UserROI or PredictedROI for construction, or use the static factory methods which return UserROI.

// Create from polygon vertices
const roi = ROI.fromPolygon([[0,0], [100,0], [100,100], [0,100]], {
  category: "region",
});

// Create from multi-polygon
const roi = ROI.fromMultiPolygon([
  [[[0,0], [10,0], [10,10], [0,10], [0,0]]],
  [[[20,20], [30,20], [30,30], [20,30], [20,20]]],
]);

// Predicted ROI with confidence score
const pred = new PredictedROI({
  geometry: { type: "Polygon", coordinates: [[[0,0],[10,0],[10,10],[0,10],[0,0]]] },
  score: 0.95,
});

// Properties
roi.bounds;       // { minX, minY, maxX, maxY }
roi.area;         // polygon area
roi.centroidXy;   // [x, y]
roi.isBbox;       // true if axis-aligned rectangle
roi.isPredicted;  // false for UserROI, true for PredictedROI

// Explode multi-geometries into individual ROIs (preserves subclass)
const parts = roi.explode();  // ROI[]

// Convert to GeoJSON Feature
const feature = roi.toGeoJSON();

// Convert to mask
const mask = roi.toMask(480, 640);

Note: ROI.fromBbox() and ROI.fromXyxy() are deprecated. Use BoundingBox.fromXywh() or BoundingBox.fromXyxy() instead.

Note: ROI.video is retained for static ROIs (arena boundaries, chamber outlines stored on labels.staticRois); ROI.frameIdx was removed in v0.3.0. Frame-bound ROIs derive their frame from the parent LabeledFrame — attach them with lf.append(roi).

SegmentationMask

RLE-encoded binary mask. SegmentationMask is abstract; use UserSegmentationMask or PredictedSegmentationMask for direct construction. The fromArray() factory returns UserSegmentationMask.

// Create from 2D boolean array
const mask = SegmentationMask.fromArray(boolArray, 480, 640, {
  category: "segmentation",
});

// Create from array with stride (downsampled prediction)
const mask = SegmentationMask.fromArray(data, 120, 160, {
  stride: 4,  // sets scale to [0.25, 0.25]
});

// Predicted mask with score and optional score map
const pred = new PredictedSegmentationMask({
  rleCounts, height: 480, width: 640,
  score: 0.92,
  scoreMap: new Float32Array(480 * 640),  // optional per-pixel confidence
});

// Properties
mask.data;                // Uint8Array (decoded)
mask.area;                // pixel count
mask.bbox;                // { x, y, width, height }
mask.isPredicted;         // false for User, true for Predicted
mask.scale;               // [number, number] spatial scale (default [1, 1])
mask.offset;              // [number, number] spatial offset (default [0, 0])
mask.hasSpatialTransform; // true if scale != [1,1] or offset != [0,0]
mask.imageExtent;         // { height, width } in image coordinates

// Resample to target size (removes spatial transform)
const fullRes = mask.resampled(480, 640);

// Trace the actual mask boundary as closed polygon rings in image space
// (honors scale/offset). Disjoint blobs and holes each get their own ring;
// an empty mask returns []. Pure data — works in the browser.
const rings = mask.contours();   // number[][][]

// Convert to a polygon ROI tracing the real outline: a Polygon (one blob,
// holes nested as interior rings) or MultiPolygon (disjoint blobs). A
// predicted mask yields a PredictedROI carrying its score.
const roi = mask.toPolygon();

// Convert to BoundingBox object (with metadata)
const bb = mask.toBbox();

toPolygon() traces the boundary (as of v0.5.0) rather than returning the bounding-box rectangle. For the axis-aligned box use mask.bbox or mask.toBbox(). The traced outline is exact and axis-aligned ("staircase"); consumers wanting smooth curves can post-process the rings (e.g. Chaikin subdivision — the demo does this). get data() memoizes the decoded raster, so repeated data/area/bbox/contours() reads decode the RLE once (treat the returned buffer as read-only).

Adopting predictions (human-in-the-loop). PredictedSegmentationMask.toUser() mirrors Instance.fromPredicted: it returns a new UserSegmentationMask that copies the RLE raster and carries over metadata (name, category, source, track, trackingScore, instance, scale, offset) while dropping prediction-only fields (score, scoreMap). It sets fromPredicted on the returned mask as a provenance link.

const userMask = pred.toUser();      // userMask.fromPredicted === pred
const detached = pred.toUser(false); // detached.fromPredicted === null

Note: like Instance.fromPredicted, the mask's fromPredicted link IS persisted to the SLP format. It is stored as an index into the saved mask list and re-linked on read, so it survives a save/load round-trip as long as the source prediction is also saved (in the same or another frame). If the source prediction is not saved, the link loads as null. Recording at least one link bumps the SLP format_id to 2.4; the column is always written and reads are presence-gated, so files written before this column existed load the link as null.

LabeledFrame.unusedPredictedMasks returns the predicted masks in a frame that no user mask has adopted — the mask analogue of LabeledFrame.unusedPredictions. A prediction is adopted when a user mask in the same frame links to it via fromPredicted (checked first, by identity) or, lacking a link, spatially overlaps it (bbox-centroid within 5 px). The remaining unclaimed predictions are returned, e.g. for surfacing unreviewed detections in a proofreading UI.

Rendering masks in the browser

drawMasks and drawLabelImage are browser-safe raster compositors (pure ImageData blending — no skia-canvas, no Node built-ins). They are exported from the browser entry, so a UI can composite masks onto a <canvas> client-side without the Node renderImage path:

import { drawMasks } from "@talmolab/sleap-io.js"; // browser build

const ctx = canvas.getContext("2d");
ctx.drawImage(videoFrame, 0, 0);
const img = ctx.getImageData(0, 0, canvas.width, canvas.height);
drawMasks(img, frame.masks, {
  colors: frame.masks.map(maskColorRgb),  // e.g. by track identity
  alpha: 0.45,
});
ctx.putImageData(img, 0, 0);

// Crisp vector outlines on top, from the data model:
for (const mask of frame.masks) {
  for (const ring of mask.contours()) {
    ctx.beginPath();
    ctx.moveTo(ring[0][0], ring[0][1]);
    for (const [x, y] of ring) ctx.lineTo(x, y);
    ctx.closePath();
    ctx.stroke();
  }
}

The vector overlays (drawBboxes, drawRois, drawCentroids, applyOverlay) and the full renderImage/renderVideo pipeline draw through skia-canvas and remain Node-only (main entry). See demo/ for a complete browser example that fills masks with drawMasks and outlines them with smoothed contours().

renderImage/renderVideo auto-draw a frame's centroids (filled circle markers, colored by track via the pose palette, scoped to the rendered video), so a centroid-only frame renders without an explicit overlay. drawCentroids is also exported for standalone use:

import { drawCentroids } from "@talmolab/sleap-io.js"; // Node (skia-canvas)

drawCentroids(imageData, frame.centroids, {
  colors: frame.centroids.map(centroidColorRgb), // e.g. by track identity
  markerSize: 5,
  alpha: 1,
});

BoundingBox

Axis-aligned or rotated bounding box for detection/tracking workflows. BoundingBox is abstract; use UserBoundingBox or PredictedBoundingBox for direct construction.

Bounding boxes use corner coordinates (x1, y1, x2, y2) as their primary storage. Center-based properties (xCenter, yCenter, width, height) are available as computed getters.

Construct a user-annotated bbox and attach it to a frame:

const bbox = new UserBoundingBox({
  x1: 0, y1: 20, x2: 100, y2: 100,
  category: "animal",
});

// The parent LabeledFrame provides video + frameIdx context
let lf = labels.getFrame(labels.videos[0], 3);
if (!lf) {
  lf = new LabeledFrame({ video: labels.videos[0], frameIdx: 3 });
  labels.append(lf);
}
lf.append(bbox);

Construct a predicted bbox with confidence score:

const predBbox = new PredictedBoundingBox({
  x1: 0, y1: 20, x2: 100, y2: 100,
  score: 0.95,
});

Factory methods (return UserBoundingBox):

const fromCorners = BoundingBox.fromXyxy(10, 20, 110, 100);  // corner coords
const fromTopLeft = BoundingBox.fromXywh(10, 20, 100, 80);   // top-left + size

Properties and conversions:

// Stored properties
bbox.x1; bbox.y1; bbox.x2; bbox.y2;  // corner coordinates
bbox.angle;      // rotation in radians (default 0)

// Computed properties
bbox.xCenter;    // (x1 + x2) / 2
bbox.yCenter;    // (y1 + y2) / 2
bbox.width;      // abs(x2 - x1)
bbox.height;     // abs(y2 - y1)
bbox.xyxy;       // [x1, y1, x2, y2] (axis-aligned bounding box)
bbox.xywh;       // { x, y, width, height }
bbox.corners;    // number[][] (4 corner points, respects rotation)
bbox.bounds;     // { minX, minY, maxX, maxY }
bbox.area;       // width * height
bbox.centroidXy; // [x, y]
bbox.isPredicted; // true for PredictedBoundingBox
bbox.isRotated;  // true if angle != 0

// Conversion
const roi = bbox.toRoi();              // Polygon ROI
const mask = bbox.toMask(480, 640);    // SegmentationMask

Centroid

Point representing the center of an object. Centroid is abstract; use UserCentroid or PredictedCentroid.

// Create user centroid (attach to a frame to provide video + frameIdx context)
const c = new UserCentroid({ x: 100, y: 200, track });
let lf = labels.getFrame(video, 0);
if (!lf) {
  lf = new LabeledFrame({ video, frameIdx: 0 });
  labels.append(lf);
}
lf.append(c);

// Create predicted centroid with confidence
const pc = new PredictedCentroid({
  x: 50, y: 60, z: 3.5,
  score: 0.95,
  trackingScore: 0.8,
  track,
  name: "spot1", source: "trackmate",
});
let lf1 = labels.getFrame(video, 1);
if (!lf1) {
  lf1 = new LabeledFrame({ video, frameIdx: 1 });
  labels.append(lf1);
}
lf1.append(pc);

// Properties
c.xy;           // [x, y]
c.yx;           // [y, x] (row, col)
c.xyz;          // [x, y, z | null]
c.isPredicted;  // false for User, true for Predicted

// Convert to single-node Instance
const inst = pc.toInstance();  // PredictedInstance with centroid skeleton

// Create from Instance (center of mass, bbox center, or anchor node)
const centroid = Centroid.fromInstance(instance);
const centroid = Centroid.fromInstance(instance, { method: "bboxCenter" });
const centroid = Centroid.fromInstance(instance, { method: "anchor", node: "head" });

// Instance convenience method (requires centroid.ts import)
const centroid = instance.toCentroid();
const centroid = instance.toCentroid("anchor", "head");

// Instance centroid shortcut (no Centroid object, just coordinates)
instance.centroidXy;  // [x, y] | null (mean of visible points)

Centroid Skeleton

// Shared single-node skeleton for centroid conversions
const skel = getCentroidSkeleton();  // Skeleton(["centroid"])
CENTROID_SKELETON === getCentroidSkeleton();  // true (singleton)

Labels Annotation Access

Annotations (Centroid, BoundingBox, SegmentationMask, LabelImage, frame-bound ROI) live on individual LabeledFrame instances, not on Labels. The Labels getters return flattened read-only views across all frames; mutation always goes through the owning frame.

// Read-only property getters (flat views across all frames)
labels.rois;           // ROI[]
labels.masks;          // SegmentationMask[]
labels.bboxes;         // BoundingBox[]
labels.centroids;      // Centroid[]
labels.labelImages;    // LabelImage[]
labels.identities;     // Identity[]
labels.staticRois;     // Static (video-level) ROIs, e.g. arena boundaries
labels.temporalRois;   // Frame-bound ROIs (flat view across frames)

// Filtered queries (all support `predicted` filter; frame-aware filters use O(1) indices)
labels.getRois({ video, frameIdx: 0, category: "arena", predicted: false });
labels.getMasks({ video, category: "segmentation", predicted: true });
labels.getBboxes({ video, frameIdx: 0, predicted: true });
labels.getCentroids({ video, frameIdx: 0, track, predicted: true });
labels.getLabelImages({ video, frameIdx: 0, track, category: "cell" });

// Note: getRois({ video }) returns only frame-bound ROIs for that video.
// For static ROIs, use labels.staticRois directly.

// Add a video (deduplicates automatically)
labels.addVideo(video);

Adding annotations to frames

Use LabeledFrame.append(annotation) to add any annotation type. It routes by runtime type to the appropriate per-frame list (lf.centroids, .bboxes, .masks, .labelImages, .rois) and handles Instance and PredictedInstance as well. If you know the target list at the call site, you can also push directly.

Note: Labels and LabeledFrame each have an append() method. labels.append(lf) adds a LabeledFrame to a Labels (and also registers its video and tracks). lf.append(annotation) adds an annotation to a single LabeledFrame. They're different operations at different levels of the hierarchy.

import {
  Labels,
  LabeledFrame,
  UserCentroid,
  UserBoundingBox,
  UserSegmentationMask,
} from "@talmolab/sleap-io.js";

// Build a frame with mixed annotations
const lf = new LabeledFrame({ video, frameIdx: 5 });
lf.append(new UserCentroid({ x: 100, y: 200, track }));
lf.append(new UserBoundingBox({ x1: 50, y1: 50, x2: 150, y2: 150 }));
lf.append(UserSegmentationMask.fromArray(maskData, 480, 640));

// Or push directly if you know the type
lf.centroids.push(new UserCentroid({ x: 10, y: 20 }));

// Pass frames as labeledFrames when constructing Labels
const labels = new Labels({ labeledFrames: [lf], videos: [video], skeletons: [skeleton] });

// To mutate an existing Labels, look up the target frame first
let existing = labels.getFrame(video, 5);
if (!existing) {
  existing = new LabeledFrame({ video, frameIdx: 5 });
  labels.append(existing);
}
existing.append(anotherBbox);

Static ROIs (video-level, e.g. arena boundaries) live on labels.staticRois. Use labels.addStaticRoi(roi) to add one — the method also registers the ROI's track on labels.tracks if present:

labels.addStaticRoi(arenaRoi);

SLP format versions

The SLP format version (format_id) is set automatically based on content:

Version Trigger
1.5 ROIs or masks present
1.6 ROIs with instance associations
1.7 Bounding boxes
1.8 Label images
1.9 Identities or predicted annotations
2.0 BBox corner-based format
2.1 Spatial metadata (scale/offset)
2.2 Chunked label image storage
2.3 Virtual video crops present
2.4 Persisted mask fromPredicted provenance link
Virtual Crops (Format 2.3+)

A virtually-cropped video stores its crop rect in a dedicated top-level /video_crops dataset (a single JSON string), written only when at least one video is cropped. Its videos_json entry describes the uncropped source, and its source_video is the uncropped original — so a reader that does not understand /video_crops simply loads the full-frame source (a graceful, lossy degrade, never an error).

[
  { "video": 2, "crop": [128, 96, 384, 352], "fill": 0 },
  { "video": 5, "crop": [0, 0, 256, 256], "fill": 0 }
]
  • video — integer index into the videos_json video list.
  • crop — [x1, y1, x2, y2] in source pixel coordinates, x2/y2 exclusive.
  • fill — out-of-bounds fill value (regions outside the source are padded, not clamped).

The presence of any crop bumps format_id to 2.3. Files with no crops are byte-identical to earlier versions (no /video_crops dataset, no version bump).

Label Images (Dense Segmentation)

Dense integer label images for instance segmentation workflows (Cellpose, StarDist, SAM, Mask R-CNN output). Each pixel stores an object ID (0 = background, positive integers = objects) with per-object metadata (track, category, name).

LabelImage is abstract; use UserLabelImage or PredictedLabelImage, or use the static factory methods which return UserLabelImage.

Creating Label Images

// From a 2D integer array. Tracks are NOT created by default — pass them
// explicitly, or set `createTracks: true` to auto-create one per label ID.
const li = LabelImage.fromArray(
  new Int32Array([0, 0, 1, 1, 0, 2, 2, 0, 0]),
  3, 3,  // height, width
  { tracks: [trackA, trackB] },
);

// Pure segmentation (no tracking): omit tracks (objects carry track = null).
const segLi = LabelImage.fromArray(maskData, 480, 640);
// Opt in to auto-created tracks named by label ID:
const trackedLi = LabelImage.fromArray(maskData, 480, 640, { createTracks: true });

// From segmentation masks (composites multiple binary masks)
const li2 = LabelImage.fromMasks(masks);

// From per-object binary masks (SAM / Mask R-CNN output)
const li3 = LabelImage.fromBinaryMasks(
  [mask1_2d, mask2_2d, mask3_2d],
  {
    tracks: [trackA, trackB, trackC],
    categories: ["cell", "cell", "debris"],
    scores: [0.95, 0.88, 0.72],  // creates PredictedLabelImage if provided
  },
);

// Batch creation from a stack of 2D arrays (one per frame)
const labelImages = LabelImage.fromStack({
  data: [frame0_2d, frame1_2d, frame2_2d],
  tracks: new Map([[1, trackA], [2, trackB]]),
  createTracks: true,  // auto-create Track objects from label IDs
});

// Attach each label image to its frame
labelImages.forEach((li, i) => {
  let lf = labels.getFrame(video, i);
  if (!lf) {
    lf = new LabeledFrame({ video, frameIdx: i });
    labels.append(lf);
  }
  lf.append(li);
});

Frame and video context for label images comes from the parent LabeledFrame. Factory methods return plain UserLabelImage objects; use lf.append(li) to attach them to a frame.

Properties and Methods

// Data access
li.data;       // Int32Array (flat, row-major)
li.height;     // number
li.width;      // number
li.objects;    // Map<number, LabelImageObjectInfo>
li.nObjects;   // number of objects
li.labelIds;   // sorted unique non-zero label IDs in data
li.tracks;     // Track[] from object metadata
li.categories; // Set<string> from object metadata

// Spatial transforms (for downsampled predictions)
li.scale;               // [number, number] (default [1, 1])
li.offset;              // [number, number] (default [0, 0])
li.hasSpatialTransform; // boolean
li.imageExtent;         // { height, width } in image coordinates

// Mask extraction
const mask = li.getObjectMask(1);         // Uint8Array for label ID 1
const mask = li.getTrackMask(trackA);     // Uint8Array for all objects with trackA
const mask = li.getCategoryMask("cell");  // Uint8Array for all "cell" objects

// Iteration over objects
for (const [track, category, binaryMask] of li.items()) {
  console.log(track?.name, category, binaryMask.length);
}

// Conversion
const masks = li.toMasks();                       // SegmentationMask[]
const bboxes = li.toBboxes();                     // BoundingBox[]
const fullRes = li.resampled(480, 640);           // new LabelImage at full resolution

// Predicted label image
li.isPredicted;  // false for UserLabelImage, true for PredictedLabelImage
// PredictedLabelImage also has: score, scoreMap, scoreMapScale, scoreMapOffset

LabelImageObjectInfo

Per-object metadata stored in LabelImage.objects:

interface LabelImageObjectInfo {
  track: Track | null;
  category: string;
  name: string;
  instance: Instance | null;
  score?: number | null;           // per-object confidence (PredictedLabelImage)
  trackingScore?: number | null;   // per-link tracking cost
}

normalizeLabelIds()

Normalize label IDs across multiple frames so each Track or category gets a globally consistent ID. Mutates label images in place.

// Normalize by track (default) — each Track gets a unique, consistent ID
const trackMap: Map<Track, number> = normalizeLabelIds(labelImages);

// Normalize by category — same category string gets the same ID
const catMap: Map<string, number> = normalizeLabelIds(labelImages, { by: "category" });

Identity & 3D Data Structures

Identity

Ground-truth animal identity that persists across videos, sessions, and experiments. Unlike Track (which represents a temporal trajectory within a single video), Identity represents the actual animal.

const id = new Identity({ name: "mouse_A", color: "#ff0000" });

id.name;      // "mouse_A"
id.color;     // "#ff0000"
id.metadata;  // Record<string, unknown>

Instance3D and PredictedInstance3D

3D keypoint data from multi-camera triangulation.

// User-annotated 3D instance
const inst3d = new Instance3D({
  points: [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
  skeleton,
  score: 0.95,
});

inst3d.nVisible;  // number of non-NaN keypoints
inst3d.isEmpty;   // true if all keypoints are NaN/null

// Predicted 3D instance (with per-keypoint scores)
const pred3d = new PredictedInstance3D({
  points: [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
  skeleton,
  score: 0.95,
  pointScores: [0.99, 0.87],
});

3D Fields on Existing Types

// Labels.identities — all ground-truth identities
labels.identities;  // Identity[]

// Camera.size — image dimensions
camera.size;  // [width, height] | undefined

// InstanceGroup — links identity and 3D data to multi-camera groups
group.identity;    // Identity | undefined
group.instance3d;  // Instance3D | undefined
group.points;      // delegates to instance3d.points if available

Lossless, lazy-native session grouping

Cross-camera grouping (FrameGroup / InstanceGroup) round-trips losslessly through read → object model → write without materializing frames, on all three read paths (eager, lazy, streaming). Instead of resolving the grouping against a materialized frame table at read time, the model stores the as-read index refs (camcorder_to_lf_and_inst_idx_map and labeled_frame_by_camera) and resolves them lazily:

// Typed access — same public API as before, now lazy under the hood.
const fg = labels.sessions[0].frameGroups.get(0)!;
fg.instanceGroups[0].instanceByCamera; // Map<Camera, Instance> (resolved on demand)
fg.labeledFrameByCamera;               // Map<Camera, LabeledFrame> (resolved on demand)

Under the lazy reader, resolving one cross-camera ref materializes only that one frame (via Labels.frameAt(i)); an untouched session serializes straight from the stored refs with zero frame materialization. Mutating a group (accessing its map, then editing/reordering instances) re-derives indices from the live objects on write, so edits are reflected correctly.

On write, the sessions_json shape matches Python sleap-io's session_to_dict byte-for-byte: calibration is keyed cam_0, cam_1, … (with each camera's name kept as a field inside its dict), while camcorder_to_video_idx_map, camcorder_to_lf_and_inst_idx_map, and labeled_frame_by_camera are keyed by the bare integer camera index ("0", "1", …). These are two different key spaces on purpose — both Python and sleap-io.js resolve cameras by their order in calibration / positional index, so the calibration key string is cosmetic. This is a convergence to Python's existing format, not a new on-disk feature, so it does not bump format_id (that namespace is shared with Python, which already defines 2.5+). Reading stays tolerant of legacy name-keyed calibration, so loading an older file and re-saving upgrades the sessions to the canonical shape in place.

Raw sessions_json access (rawJson / rawSessionsJson) — deprecated

Deprecated / transitional, opt-in only. The typed session model above is now a faithful projection of sessions_json, so consumers should read typed objects rather than this raw dict. rawJson is a temporary bridge that will be removed once LUCID migrates. It is undefined by default — pass { rawSessions: true } to a read entrypoint to capture it. Capturing it deep-clones the whole session payload (doubling session memory in Labels.copy()), which is why it is off by default.

// OPT-IN: capture the verbatim parsed sessions_json dict per session.
const labels = await readSlp(path, { rawSessions: true });
labels.sessions[0].rawJson;
//   -> { calibration, camcorder_to_video_idx_map,
//        camcorder_to_lf_and_inst_idx_map, frame_group_dicts, metadata, ... }

// Index-aligned convenience getter (derived from sessions, no stored field)
labels.rawSessionsJson;         // Array<Record<string, unknown> | undefined>
labels.rawSessionsJson[0];      // === labels.sessions[0].rawJson

// Default (no option): rawJson is undefined, no clone, no copy() bloat.
const lean = await readSlp(path);
lean.sessions[0].rawJson;       // undefined

rawSessions is accepted by readSlp, readSlpLazy, and readSlpStreaming. When captured, rawJson exposes every key, including ones sleap-io.js does not itself model.

Caveats (when captured):

  • Read-time snapshot. rawJson is captured at read time and is NOT updated when you mutate the object model afterwards.
  • Never re-written to disk. Writes are rebuilt from the object model (the single source of truth); rawJson is a pure in-memory read surface over the already-existing sessions_json dataset. It is not stored in provenance or the /metadata blob, so saving does not bloat those.
  • Index-aligned. rawSessionsJson[i] corresponds to sessions[i]; entries are undefined for sessions constructed in-memory or read without rawSessions.

Metadata round-trip (Option 1)

Opaque metadata dicts round-trip losslessly through read → object model → write at every level — RecordingSession.metadata (including nested metadata.lucid), FrameGroup.metadata, InstanceGroup.metadata, and CameraGroup.metadata. Arbitrary unknown keys survive by JSON-equality.

Note: CameraGroup.metadata previously read back as {} (silently dropped on read); it is now populated from calibration.metadata.

Object-model write limitations (Option 3)

The write path (serializeSession) rebuilds sessions_json from a fixed field set, so it is lossless for everything sleap-io.js structurally models — per-camera intrinsics/extrinsics (name, rotation, translation, matrix, distortions, size), cam_N-keyed calibration with name kept as a field, integer-index-keyed camcorder_to_video_idx_map / camcorder_to_lf_and_inst_idx_map / labeled_frame_by_camera, and all metadata dicts. camcorder_to_lf_and_inst_idx_map now survives losslessly — it is derived from live objects when the group was mutated, and otherwise falls back to the as-read index refs (so it is preserved even in lazy mode). The following remain recoverable only via rawSessions:

  1. Any unmodeled top-level key NOT under a metadata sub-dict (at session / per-camera / frame-group / instance-group level) is dropped.
  2. Frame groups with zero instance groups are skipped entirely on write.
  3. Frame groups are keyed by frame_idx; groups sharing (or missing) a frame_idx collapse to one entry on read (a pre-existing fidelity gap).

Legacy note: older sleap-io.js builds keyed calibration by camera name when a camera had a distinct name, diverging from Python's cam_N convention. Reading tolerates both (cameras resolve by order); re-saving canonicalizes to cam_N.

SLEAP Analysis CSV export

Export Labels to the SLEAP Analysis CSV — one row per instance per frame with columns track, frame_idx, instance.score, {node}.score, {node}.x, {node}.y, … (node columns sorted alphabetically; missing/invisible points and user-instance scores are empty cells). labelsToCsv is browser-safe (returns the string); saveLabelsCsv writes to disk (Node).

import { labelsToCsv, saveLabelsCsv } from "@talmolab/sleap-io.js";

const csv = labelsToCsv(labels);               // string (browser + Node)
await saveLabelsCsv(labels, "out.csv");        // Node file write

// Span the whole video: emit NaN rows for unlabeled frames, padded to the video
// length when known (mirrors the Analysis-HDF5 export; sleap-io PR #480).
await saveLabelsCsv(labels, "out.csv", { includeEmpty: true });

// Other options: video (Video | index filter), includeScore (default true),
// startFrame / endFrame (inclusive / exclusive).

GeoJSON I/O

Convert ROIs to/from GeoJSON format.

// Convert ROIs to GeoJSON FeatureCollection
const geojson = roisToGeoJSON(labels.rois);

// Serialize to JSON string
const json = writeGeoJSON(labels.rois);

// Parse GeoJSON back to ROIs
const rois = roisFromGeoJSON(geojson);
const rois = readGeoJSON(jsonString);

TrackMate CSV I/O

Import TrackMate (ImageJ/Fiji) CSV tracking exports as PredictedCentroid objects. Node.js only.

import { readTrackMateCsv, loadTrackMate, isTrackMateFile } from "@talmolab/sleap-io.js";

// Check if a file is a TrackMate spots CSV
isTrackMateFile("data_spots.csv");  // true/false

// Load spots CSV (auto-detects sibling _edges.csv and .tif video)
const labels = readTrackMateCsv("data_spots.csv");
labels.centroids;  // PredictedCentroid[]
labels.tracks;     // Track[] (from TRACK_ID column)

// With explicit options
const labels = readTrackMateCsv("data_spots.csv", {
  edgesPath: "data_edges.csv",  // optional, auto-detected from _spots suffix
  video: "data.tif",            // optional, auto-detected from _spots suffix
});

// Public API alias
const labels = loadTrackMate("data_spots.csv");

TrackMate CSV format: - *_spots.csv — spot detections with POSITION_X, POSITION_Y, POSITION_Z, FRAME, QUALITY, TRACK_ID - *_edges.csv — frame-to-frame linkages; LINK_COST is stored as trackingScore on target centroids

Ultralytics YOLO I/O

Read and write Ultralytics YOLO datasets — pose, detection, and segmentation. A dataset is a directory of data.yaml + per-split images/ and labels/. The format of each label line is auto-detected by value count. Node.js only (directory-based I/O).

import { loadUltralytics, saveUltralytics } from "@talmolab/sleap-io.js";

// Load a split (auto-detects pose/detection/segmentation per line)
const labels = loadUltralytics("path/to/yolo_dataset", { split: "train" });
labels.labeledFrames;  // pose instances (5 + 3k value lines)
labels.bboxes;         // UserBoundingBox / PredictedBoundingBox (5 / 6 value lines)
labels.rois;           // UserROI from segmentation polygons

// Pointing at data.yaml directly also works
loadUltralytics("path/to/yolo_dataset/data.yaml");

// Write a dataset (default task "pose"; also "detect" / "segment")
await saveUltralytics(labels, "out_dir", { splitRatios: { train: 0.8, val: 0.2 } });
await saveUltralytics(labels, "out_dir", { task: "detect" });   // writes labels.bboxes
await saveUltralytics(labels, "out_dir", { task: "segment" });  // writes labels.rois

Lower-level helpers are also exported: parseLabelFile(path, skeleton, [h, w], { classNames }) returns { instances, rois, bboxes }; detectLineFormat(parts) returns "detection" | "detection_conf" | "segmentation" | "pose"; plus writeLabelFile, writeBboxLabelFile, writeRoiLabelFile, readLabelsSet, parseDataYaml, createDataYaml, and createSkeletonFromConfig.

Label-line formats (coordinates normalized to [0, 1]): - Detection — class_id x_center y_center width height [confidence] → UserBoundingBox (5 values) or PredictedBoundingBox (6 values). - Segmentation — class_id x1 y1 x2 y2 … xn yn → UserROI polygon (multi-polygons are exploded on write; interior holes are dropped with a warning). - Pose — class_id x_center y_center width height x1 y1 v1 … xn yn vn → Instance (visibility 0 = invisible, 1/2 = visible).

Image I/O note: image dimensions are read from the file header (PNG/JPEG/GIF/BMP/TIFF) rather than by decoding pixels. On write, frames backed by an on-disk image file are copied verbatim; frames yielding raw ImageData pixels are encoded to PNG; otherwise the frame is skipped with a warning. The imageFormat/imageQuality options apply only to the raw-pixel PNG path.

JABS Pose I/O

Read JABS (Jackson Lab Animal Behavior System) pose files. JABS files are HDF5 on disk, so this reader is Node.js only. Matches Python sleap-io v0.7.0 (PR #371): instances are PredictedInstance objects with per-point confidence, and static objects are UserROIs rather than synthetic skeletons.

import { loadJabs } from "@talmolab/sleap-io.js";

// Read a JABS pose file (pose versions 2–6). Async (HDF5 via h5wasm).
const labels = await loadJabs("recording_pose_est_v5.h5");

labels.labeledFrames;   // frames of PredictedInstance poses
labels.tracks;          // one Track per JABS identity (v2 → a single track "1")
labels.staticRois;      // static objects as UserROI (also in labels.rois)
labels.videos[0];       // Video named after the pose file (…_pose_est_vN.h5 → ….avi)

// Optional: override the skeleton (must have one node per keypoint column)
const labels2 = await loadJabs("recording_pose_est_v2.h5", { skeleton: myMouseSkeleton });

Static objects become UserROIs with source: "jabs" and category: "arena" for corners (or "anchor" otherwise); a single-point object (e.g. lixit) is a Point geometry, multi-point (e.g. corners) is a MultiPoint. The default JABS_DEFAULT_SKELETON is the 12-node "Mouse" skeleton; makeJabsDefaultSkeleton(), predictionToInstance(), staticObjectToRoi(), and makeSimpleSkeleton() are also exported.

JABS coordinates are stored (y, x) and flipped to (x, y) on read. Static-object coordinates are kept as-stored. The writer (saveJabs) is not yet implemented — the common path is a one-time JABS → SLP conversion.

Skeleton Codecs

JSON (jsonpickle format)

import { readSkeletonJson, isSkeletonJson } from "@talmolab/sleap-io.js";

const skeleton = readSkeletonJson(jsonString);

Training Config

import { readTrainingConfigSkeletons, isTrainingConfig } from "@talmolab/sleap-io.js";

const skeletons = readTrainingConfigSkeletons(jsonString);

Codecs & Numpy Helpers

Dictionary codec

const dict = toDict(labels);
const restored = fromDict(dict);

Numpy codec

const array = toNumpy(labels, { returnConfidence: true });
const rebuilt = fromNumpy(array, { video, skeleton, returnConfidence: true });

Also available: - Labels.numpy({ video, returnConfidence }) - Labels.fromNumpy(data, { video, skeleton, trackNames, firstFrame })

Streaming Options

SLP streaming is handled by HDF5 open helpers:

const labels = await loadSlp("https://example.com/session.slp", {
  openVideos: false,
  h5: { stream: "range", filenameHint: "session.slp" },
});
  • stream: "auto" uses range streaming when available.
  • stream: "download" forces full file download.
  • Node only supports string paths or byte buffers for SLP inputs.
  • Browser supports URL, File, FileSystemFileHandle, or byte buffers.

Error Handling Notes

Common runtime errors to anticipate:

  • SLP parsing:
  • Missing /metadata group in invalid .slp.
  • Unsupported source type in browser or Node.
  • Failed fetch for remote SLP (bad URL/status).

  • h5wasm FS unavailable in unsupported environments.

  • MP4/WebCodecs:

  • WebCodecs not supported or non-browser environment.
  • No video tracks or unsupported codec.

  • Failed fetch or no video source available.

  • Media backend:

  • Browser-only usage.
  • Video load or seek failures.