This should eventually be replaced with a more comprehensive approach to documentation (e.g., via readthedocs.org), but this markdown file will do for now.
ctrl+zorcmd+z: Undoctrl+shift+zorcmd+shift+z: Redoscroll: Zoom -- up for in, down for outctrl+scrollorshift+scrollorcmd+scroll: Change frame -- down for next, up for previousscrollclick+drag: Pan- Hold
shiftwhen closing a polygon to continue annotating a new region or hole. - Hold
shiftwhen moving the cursor inside a polygon to begin annotating a new region or hole. - Press
Escapeorcrtl+zto cancel the start of a new region or hole. - Press
Escapeto exit brush/erase mode. - Press
Tabto set the zoom to focus on the next annotation - Press
Shift+Tabto set the zoom to focus on the previous annotation
When the ulabel.js file is included, it attaches its class definition to the window object. Therefore, within the document, you may create a new annotation session with
let ulabel = new ULabel(...);Note that in order to begin the session, you must thereafter call
ulabel.init(() => {/* behavior on ready */})ULabel is the only name that ulabel.js will add to the global namespace.
The constructor is used to specify the configuration for an "annotation session". It has the following interface
class ULabel({
// Required arguments
container_id: string,
image_data: string | string[],
username: string,
submit_buttons: function | ULabelSubmitButton[],
subtasks: object,
// Optional arguments
task_meta: object,
annotation_meta: object,
px_per_px: number,
initial_crop: InitialCrop,
initial_line_size: number,
instructions_url: string,
toolbox_order: AllowedToolboxItem[],
distance_filter_toolbox_item: FilterDistanceConfig,
image_filters_toolbox_item: ImageFiltersConfig,
reset_zoom_keybind: string,
show_full_image_keybind: string,
create_point_annotation_keybind: string,
delete_annotation_keybind: string,
delete_vertex_keybind: string,
keypoint_slider_default_value: number,
filter_annotations_on_load: boolean,
switch_subtask_keybind: string,
toggle_annotation_mode_keybind: string,
create_bbox_on_initial_crop_keybind: string,
toggle_brush_mode_keybind: string,
toggle_erase_mode_keybind: string,
increase_brush_size_keybind: string,
decrease_brush_size_keybind: string,
mask_annotation_opacity: number,
default_brush_overlap_mode: BrushOverlapMode,
set_brush_overlap_none_keybind: string,
set_brush_overlap_exclude_keybind: string,
set_brush_overlap_overwrite_keybind: string,
fly_to_next_annotation_keybind: string,
fly_to_previous_annotation_keybind: string,
annotation_size_small_keybind: string,
annotation_size_large_keybind: string,
annotation_size_plus_keybind: string,
annotation_size_minus_keybind: string,
annotation_vanish_keybind: string,
fly_to_max_zoom: number,
n_annos_per_canvas: number
})string -- The value of the id attribute of the <div> element that ULabel is meant to occupy. This element must exist in the document at the time the constructor is called.
ULabel has primarily been tested inside of divs that have been styled with position=absolute;, and width, height, top, and left set. Stay tuned for official recommendations about this.
string OR array -- A reference to the image(s) to be annotated. In the case of a single image session, a simple URL to the image can be provided. It will be assigned directly to an <img> tag's src attribute.
In the case of a multi-frame annotation job, an array of URLs may be given. Note that for performance reasons, ULabel assumes that each image in the array has the same dimensions as the first image in the array.
string -- This is intended to be a unique ID for the user performing annotations. It will be assigned to each annotation that this user creates during the session.
A single async function may be provided for a submit button.
async function (obj) => {/* Your on submit behavior here */}If the hook alone is provided, the name will default to "Submit" and the button's color will be orange.
If either more than one submit button or more button customization is desired, then submit_buttons must be an array of submit_button objects. It may be an array of length 1 if only 1 button is desired but you want to change the text or color of the button.
submit_button Objects must be provided in the form of
{
name: "<Arbitrary Button Name>", // The button has a set height and width, so the name should be short
hook: async function (annotations) {
// Define submit behavior here
// ULabel instance is bound to this function, and so it can be accessed with this
// If behavior is to leave this page, use this.set_saved(true) to avoid warning to user
// If submit is unsuccessful and annotations edits should not be treated as "saved", return false
},
color?: "Arbitrary Color" // e.g. "#639", "#3AB890", "rgb(200, 0, 170)", "hsl(0, 100%, 50%)"
/**
* If true, will call ulabel.set_saved(true) before the hook is called,
* thus avoid the "unsaved changes" warning. Defaults to false.
*/
set_saved?: boolean
size_factor?: number // Transform the default button size by this factor.
row_number?: number // The row number of the button in the toolbox
// Buttons with lower row numbers will be higher in the toolbox
// If row_number is not provided, it will default to 0
// Buttons will be arranged left to right in the order they are provided in the array
}The argument to the hook is an object with the format:
{
"task_meta": <obj>, // The task_meta from the constructor
"annotations": {
"<subtask 1>": [/* subtask 1 annotations */],
"<subtask 2>": [/* subtask 2 annotations */],
...
}
}Where <subtask n> refers to the nth key in the object provided as the subtasks argument to the constructor.
As you can see, each subtask will have a corresponding list of annotation objects. Each annotation object has the following format:
{
// a unique id for this annotation
"id": "<uuidv4 string>",
// the provided username
"created_by": "<string>",
// timestamp when annotation was created
"created_at": "<ISO datetime string>",
// the username associated with the most recent modification to the annotation
"last_edited_by": "<string>",
// timestamp of the most recent modification to the annotation
"last_edited_at": "<ISO datetime string>",
// true if annotation was deleted
"deprecated": "<bool>",
// indicates what/who deprecated the annotation, eg { human: false }
"deprecated_by": "<object>",
// which type of annotation
"spatial_type": "<string>",
// (nullable) e.g. [[x1, y1], [x2, y2], ...]
// For "bitmask" annotations this is instead a run-length-encoded object:
// { "counts": <number[]>, "size": [<height>, <width>] }. See Bitmask annotations.
"spatial_payload": "<array | object>",
// The class associated with the annotation
"classification_payloads": [
{
"class_id": 10,
"confidence": 1
},
{
"class_id": 11,
"confidence": 0
},
{
"class_id": 12,
"confidence": 0
}
],
// (nullable) frame ann was created for
"frame": "<int>",
// certain spatial types allow text
"text_payload": "<string>",
// as provided to constructor
"annotation_meta": "<object>"
}object -- Configuration for each subtask in the annotation session.
In certain cases, you may want to divide your annotations among different tasks. For example, if you are visualizing annotations from two different sources (e.g., different annotators, or one from a model, another from a human). ULabel supports this natively through what we call "subtasks".
Every annotation session requires at least one subtask. Each subtask has its own configuration, which is specified with a JSON object. See below for an example from the frames.html demo.
{
"car_detection": {
"display_name": "Car Detection",
"classes": [
{
"name": "Sedan",
"color": "blue",
"id": 10,
"keybind": "1"
},
{
"name": "SUV",
"color": "green",
"id": 11,
"keybind": "2"
},
{
"name": "Truck",
"color": "orange",
"id": 12,
"keybind": "3"
},
],
"allowed_modes": ["bbox", "polygon", "contour", "bbox3"],
"resume_from": null,
"task_meta": null,
"annotation_meta": null,
"read_only": false,
"inactive_opacity": 0.6
},
"frame_review": {
"display_name": "Frame Review",
"classes": [
{
"name": "Blurry",
"color": "gray",
"id": 20
},
{
"name": "Occluded",
"color": "red",
"id": 21
}
],
"allowed_modes": ["whole-image"],
"resume_from": null,
"task_meta": null,
"annotation_meta": null,
"read_only": false
}
}The "keybind" argument allows the user to select a class for existing annotations (when hovered), for new annotations, or for annotations that are actively being drawn.
The full list of "allowed_modes" that are currently supported is:
"bbox": A simple single-frame bounding box"bbox3": A bounding box that can extend through multiple frames"polygon": A series of points that define a simple or complex polygon"polyline": A series of points that does not define a closed polygon"tbar": Two lines defining a "T" shape"contour": A freehand line"whole-image": A label to be applied to an entire frame"global": A label to be applied to the entire series of frames"point": A keypoint within a single frame"bitmask": A raster (per-pixel) segmentation mask, painted with the brush. See Bitmask annotations."delete_polygon": Allows drawing a polygon around an area, and all annotations within that area will be deleted"delete_bbox": Allows drawing a bounding box around an area, and all annotations within that area will be deleted
The "bitmask" mode enables raster (per-pixel) segmentation. Each bitmask annotation stores a single binary mask the size of the image.
Interaction
- Painting uses the brush, shared with the
polygonbrush. Toggle the brush withtoggle_brush_mode_keybind(defaultg) or the Brush toolbox item, erase withtoggle_erase_mode_keybind(defaulte), and resize the brush withincrease_brush_size_keybind/decrease_brush_size_keybind(defaults]/[) oralt+scroll. - Starting a paint stroke over an existing bitmask of the currently-selected class adds to that mask; otherwise (a different class is selected, or you start over empty space) a new bitmask annotation of the selected class is created. Erasing is class-agnostic — it removes from whichever mask is under the brush. (This class-aware joining differs from the
polygonbrush, which joins any polygon under the brush.) - With the brush off, hovering a mask surfaces the usual edit dialogs: change its class via the ID dialog, or move/delete it like any other spatial annotation. Erasing a mask entirely deprecates the annotation (ULabel's delete semantics).
- Requires the
Brushtoolbox item (AllowedToolboxItem.Brush) to be present.
Overlap modes
When painting, the brush can enforce mutual exclusivity with other undeprecated bitmask annotations. The mode is a single global value, persisted to localStorage, and is chosen via the Brush toolbox item (shown in bitmask mode) or the overlap keybinds. Its initial value comes from default_brush_overlap_mode.
"none"(default): painting only adds to the active mask; other masks are untouched (pixels may be owned by multiple annotations)."exclude": newly-painted pixels never cover pixels owned by other bitmask annotations (existing masks win)."overwrite": newly-painted pixels are removed from any other bitmask annotation that owned them (the new mask wins); a mask fully carved away is deprecated.
Resolution is deferred to the end of a stroke and only affects the pixels the stroke adds (pre-existing overlaps are left alone). Erase strokes are unaffected. These modes govern new strokes only — they do not retroactively de-overlap already-imported masks.
Serialization
A bitmask's spatial_payload is a COCO-style, uncompressed run-length encoding:
{
// Alternating run lengths in column-major (Fortran) order, always starting
// with a background (0) run. A leading foreground pixel is a leading 0.
"counts": [<number>, ...],
// [height, width] of the mask (matches COCO's size convention)
"size": [<height>, <width>]
}Note this is the uncompressed form (counts as an integer array), not the LEB128-packed string used by pycocotools. Masks import from and export to this same object shape.
The render opacity of bitmask annotations is configurable via mask_annotation_opacity.
The resume_from attributes are used to import existing annotations into the annotation session for each subtask, respectively. Existing annotations must be provided as a list of annotations of the form specified above.
object -- Meta about the annotation session to be saved at the task and annotation levels, respectively.
These are provided for convenience. They simply pass their contents to the global output object and to each annotation, respectively.
number -- The ratio of rendering resolution to image resolution.
In some cases, you may want the annotations to render at a higher or lower resolution than the underlying image. For example, for very low resolution images like CT scans, you may want to specify a value of 2-4 for aesthetic purposes, whereas for very high resolution images that will only be annotated at a very coarse level, you may want to specify a value of 0.25 - 0.5 for performance purposes.
InitialCrop -- A definition for a bounding box that the viewer should fit to at the beginning of the session. Units are pixels in the underlying image.
{
"top": <number>,
"left": <number>,
"height": <number>,
"width": <number>
}The line width with which new annotations are drawn initially. Units are pixels in the underlying image. When this value is not included, the default value of 5 is used.
Defines how annotation line size is adjusted based on the zoom level. The following modes are supported:
"fixed": Line size remains constant regardless of zoom level. (Default. Use for best performance)"match-zoom": Line size increases with increased zoom level."inverse-zoom": Line size decreases with increased zoom level.
URL to a page that gives annotation instructions.
An array of numbers that defines the vertical order of items in the toolbox. At least one item must be included in the array. Any excluded items will not be displayed in the toolbox.
The supported toolbox items are:
enum AllowedToolboxItem {
ModeSelect, // 0
ZoomPan, // 1
AnnotationResize, // 2
AnnotationID, // 3
RecolorActive, // 4
ClassCounter, // 5
KeypointSlider, // 6
SubmitButtons, // 7
FilterDistance, // 8
Brush, // 9
ImageFilters, // 10
AnnotationList, // 11
Keybinds, // 12
ConfidenceSlider, // 13
}You can access the AllowedToolboxItem enum by calling the static method:
const AllowedToolboxItem = ULabel.get_allowed_toolbox_item_enum();The AnnotationResizeItem can be used to programmatically control annotation size for subtasks. It can be accessed with the static method:
const AnnotationResizeItem = ULabel.get_resize_toolbox_item();Using the class, you can call any of its static methods by passing in your ULabel instance, the key of the subtask you want to modify, and any other required arguments. For example:
AnnotationResizeItem.toggle_subtask_vanished(ulabel, subtask_key);Note that in order to work, the AnnotationResize toolbox item MUST be present in your ULabel instance.
Configuration object for the FilterDistance toolbox item with the following custom definitions:
type DistanceFromPolyline = {
distance: number // distance in pixels
}
type DistanceFromPolylineClasses = {
"closest_row": DistanceFromPolyline, // value used in single-class mode
[key: number]?: DistanceFromPolyline // values for each polyline class id, used in multi-class mode
}
type FilterDistanceConfig = {
"name"?: string, // Default: Filter Distance From Row
"component_name"?: string, // Default: filter-distance-from-row
"filter_min"?: number, // Default: 0 (px)
"filter_max"?: number, // Default: 400 (px)
"default_values"?: DistanceFromPolylineClasses, // Default: {"closest_row": {"distance": 40}}
"step_value"?: number, // Default: 2 (px)
"multi_class_mode"?: boolean, // Default: false
"disable_multi_class_mode"?: boolean, // Default: false
"filter_on_load"?: boolean, // Default: false
"show_options"?: boolean, // Default: true
"show_overlay"?: boolean, // Default: false
"toggle_overlay_keybind"?: string, // Default: "p"
"filter_during_polyline_move"?: boolean, // Default: true. Set to false for performance boost,
// since it will not update the filter/overlay until polyline moves/edits are complete.
}Configuration object for the ImageFilters toolbox item with the following custom definitions:
type ImageFiltersConfig = {
"default_values"?: {
"brightness"?: number, // Default: 100 (0-200%)
"contrast"?: number, // Default: 100 (0-200%)
"hueRotate"?: number, // Default: 0 (0-360 degrees)
"invert"?: number, // Default: 0 (0-100%)
"saturate"?: number // Default: 100 (0-200%)
}
}This toolbox item provides CSS filter controls that apply only to the image, not to the UI elements. Users can adjust brightness, contrast, hue rotation, inversion, and saturation using sliders. The filters are hardware-accelerated by modern browsers for optimal performance.
The AnnotationList toolbox item displays all annotations in the current subtask in a scrollable list. This toolbox item provides several features:
Display Features:
- Shows each annotation with its spatial type icon (bbox, polygon, point, etc.) and class name
- Displays annotation index (0-based) for easy reference
- Collapsible interface to maximize canvas space
Filtering Options:
- Show Deprecated: Toggle to show/hide deprecated annotations (default: hidden)
- Group by Class: Organize annotations by their classification for easier management
Navigation:
- Click any annotation in the list to fly-to and zoom on that annotation
- Toast notification appears showing current position (e.g., "3 / 10") when navigating
Bidirectional Highlighting:
- Hover over an annotation in the list to highlight it on the canvas with the ID dialog
- Hover over an annotation on the canvas to highlight its corresponding entry in the list
This toolbox item requires no configuration and can be added to the toolbox_order array using AllowedToolboxItem.AnnotationList.
The ConfidenceSlider toolbox item (added to toolbox_order via AllowedToolboxItem.ConfidenceSlider) deprecates (hides) or shows spatial annotations based on their confidence values. Unlike the deprecated KeypointSlider, it works with all spatial annotation types that have a confidence payload (bbox, bbox3, polygon, polyline, contour, tbar, and point), across every subtask.
It supports two modes:
- "all" mode (default): a single slider applies one confidence threshold to every targeted spatial annotation across all subtasks, using each annotation's highest confidence value.
- Per-class mode: one slider is shown per targeted class id. Each slider only filters annotations whose assigned (highest-confidence) class matches that slider, using that class's confidence value.
The class_filter_mode config controls whether these modes are user-toggleable:
"toggle"(default): a checkbox lets the user switch between the two modes."all-only": only the single global "all" slider is shown."class-only": only the per-class sliders are shown.
Any annotation with a confidence at or above the threshold is shown; any below is deprecated. Thresholds are expressed as percentages, and the slider's range and increment are configurable via filter_min, filter_max, and step_value.
Configuration object with the following custom definitions:
type ConfidenceSliderClasses = {
"all": number, // percentage threshold (0-100) used by the single global slider
[classId: string]?: number // per-class-id thresholds (class id as a string key)
}
type ConfidenceSliderConfig = {
"name"?: string, // Default: "Confidence Filter"
"filter_min"?: number, // Default: 0 (%)
"filter_max"?: number, // Default: 100 (%)
"default_values"?: ConfidenceSliderClasses, // Default: {"all": 0}
"step_value"?: number, // Default: 1 (%)
"class_filter_mode"?: "toggle" | "all-only" | "class-only", // Default: "toggle"
"filter_on_load"?: boolean, // Default: true
// The spatial types to filter. Defaults to all confidence-filterable spatial types.
"target_spatial_types"?: ULabelSpatialType[],
// The class ids to create sliders for in class-only/toggle mode. Defaults to all class ids.
"target_class_ids"?: number[],
}Keybind to reset the zoom level to the initial_crop. Default is r.
Keybind to set the zoom level to show the full image. Default is shift+r.
Keybind to create a point annotation at the mouse location. Default is c. Requires the active subtask to have a point mode.
Keybind to delete the annotation that the mouse is hovering over. Default is d.
Keybind to delete a vertex of a polygon or polyline annotation. The vertex must be the one currently being hovered (showing an edit suggestion) or actively being edited. For polylines, if only one point remains after deletion, the entire polyline is deleted. For polygons, if fewer than 3 points remain in a layer after deletion, that layer is removed. Default is x.
Default value for the keypoint slider. Must be a number between 0 and 1. Default is 0.
Deprecated: The
KeypointSlidertoolbox item only filterspointannotations. Use theConfidenceSlidertoolbox item (confidence_slider_toolbox_item) instead, which filters all spatial annotation types and supports per-class targeting and multiple sliders.
If true, the annotations will be filtered on load based on the keypoint_slider_default_value. Default is true.
Keybind to switch between subtasks. Default is z.
Keybind to toggle between annotation and selection modes. Default is u.
Keybind to create a bounding box annotation around the initial_crop. Default is f. Requires the active subtask to have a bbox mode.
Keybind to toggle brush mode for polygon and bitmask annotations. Default is g. Requires the active subtask to have a polygon or bitmask mode.
Keybind to toggle erase mode for polygon and bitmask annotations. Default is e. Requires the active subtask to have a polygon or bitmask mode.
Keybind to increase the brush size. Default is ]. Requires the active subtask to have a polygon or bitmask mode.
Keybind to decrease the brush size. Default is [. Requires the active subtask to have a polygon or bitmask mode.
The fill opacity (0-1) used when rendering bitmask (raster segmentation) annotations. Default is 0.4.
The initial brush overlap mode for bitmask painting: "none" (default), "exclude", or "overwrite". The live value is global and persisted to localStorage, so a user's last choice takes precedence over this default on subsequent sessions.
Keybind to set the brush overlap mode to none. Default is shift+n.
Keybind to set the brush overlap mode to exclude. Default is shift+e.
Keybind to set the brush overlap mode to overwrite. Default is shift+o.
Keybind to set the zoom to focus on the next annotation. Default is Tab, which also will disable any default browser behavior for Tab.
Keybind to set the zoom to focus on the previous annotation. Default is shift+tab. Supports chord keybinds (e.g., shift+p, ctrl+alt+n).
Keybind to set the annotation size to small for the current subtask. Default is s.
Keybind to set the annotation size to large for the current subtask. Default is l.
Keybind to increment the annotation size for the current subtask. Default is =.
Keybind to decrement the annotation size for the current subtask. Default is -.
Keybind to toggle vanish mode for annotations in the current subtask. Default is v.
Keybind to toggle vanish mode for all subtasks. Default is shift+v
Maximum zoom factor used when flying-to an annotation. Default is 10, value must be > 0.
The number of annotations to render on a single canvas. Default is 100. Increasing this number may improve performance for jobs with a large number of annotations.
If true, the user can click and drag to contiuously place points for polyline and polygon annotations. Default is true.
When false, new annotations will be limited to points within the image, and attempts to move annotations outside the image will bounce back to inside the image. Default is true.
Display utilities are provided for a constructed ULabel object.
(string, int) => string -- Changes the image source for a given frame. Returns the old source.
(string) => string -- Changes the background color for the annotation box. Returns the old color.
() => string -- Returns the key of the current subtask.
() => object -- Returns the current subtask object.
(string) => array -- Gets the current list of annotations within the provided subtask.
(array, string) => void -- Sets the annotations for the provided subtask.
(bool) => void -- Allows js script implementing the ULabel class to set saved status, e.g., during callback.
() => void -- Removes persistent event listeners from the document and window. Listeners attached directly to html elements are not explicitly removed. Note that ULabel will not function properly after this method is called. Designed for use in single-page applications before navigating away from the annotation page.
Sets the zoom to focus on a non-deprecated, spatial annotation in the active subtask's ordering that is an <increment> number away from the previously focused annotation, if any. Returns true on success and false on failure (eg, no valid annotations exist, or an annotation is currently actively being edited).
Sets the zoom to focus on the provided annotation id, and switches to its subtask. Returns true on success and false on failure (eg, annotation doesn't exist in subtask, is not a spatial annotation, or is deprecated).
Sets the zoom to focus on the provided annotation, and switches to its subtask if provided. Returns true on success and false on failure (eg, annotation doesn't exist in subtask, is not a spatial annotation, or is deprecated).
() => number | null -- Returns the current keypoint slider value as a number between 0 and 1. Returns null if the KeypointSlider toolbox item is not active or the slider element is not found.
Deprecated: Prefer
get_confidence_slider_value()with theConfidenceSlidertoolbox item.
() => object | null -- Returns an object mapping class identifiers to their distance filter values (in pixels). The object always includes a closest_row key for the single-class slider. In multi-class mode, additional keys correspond to each class ID. Returns null if the FilterDistance toolbox item is not active or no sliders are found.
() => object | null -- Returns an object mapping class identifiers to their confidence threshold values (as percentages, 0–100). The object always includes an all key for the single global slider. In per-class mode, additional keys correspond to each class ID. Returns null if the ConfidenceSlider toolbox item is not active or no sliders are found.
Callbacks can be provided by calling .on(fn, callback) on a ULabel object.
For example:
let ulabel = new ULabel(...);
ulabel.on(ulabel.begin_annotation, () => {
// Define some custom behavior here
console.log("The user just began a new annotation.");
});