iaa_od.metrics

iaa_od.metrics.all_granularities(result, /, *, iom_thr=0.5)

Compute Count and Size Granularities for all annotators in the provided Ground Truths, using a new approach based on “pseudo-units” and lenient IoU matching.

Parameters:

• result (Result) – The Result object containing the Ground Truths and Units to compute granularities on.

• iom_thr (float) – The Intersection over Minimum threshold to consider an annotation as “contained” within a unit. Defaults to STD_IOM_THR.

Returns:

An object containing the computed granularities for all annotators, along with the median values and benchmarks.

Return type:

Granularities

iaa_od.metrics.all_raters_vitality(gts, /, *, iou_threshold=0.5, use_iom=False, precomputed_result=None)

Compute the vitality for all raters in the provided Ground Truths.

Parameters:

• gts (list[GroundTruthProtocol]) – A list of GroundTruth objects representing the annotations.

• iou_threshold (float) – The IoU threshold to use for matching. Defaults to STD_IOU_THR.

• use_iom (bool) – Whether to use IoM matching. Defaults to False.

• precomputed_result (Optional[Result]) – An optional precomputed Result object containing the overall alpha value.

Returns:

A dictionary mapping each rater’s name to their vitality score.

Return type:

dict[str, float]

iaa_od.metrics.alpha(gts, /, *, iou_threshold=0.5, use_iom=False, precomputed_result=None, collapsed_categories=None)

This function computes Krippendorff’s alpha for a set of ground truths containing bounding box annotations.

Parameters:

• gts (list[GroundTruth]) – A list of GroundTruth objects containing annotations.

• iou_threshold (float) – The IoU threshold to consider two annotations as overlapping.

• use_iom (bool) – Whether to use IoM matching when finding units. Defaults to false.

• precomputed_result (Result | None) – An optional precomputed Result object, used to merge results from multiple algorithms.

• collapsed_categories (dict[str, list[int]] | None) – An optional dictionary to collapse categories into macro-categories, used for Scale Complexity computation.

Returns:

A Result object containing the computed alpha value and other relevant information.

Return type:

Result

iaa_od.metrics.c_score(result, /, *, iom_thr=0.5)

Function which computes the C-Score metric for a given dataset. C-Score represents the agreement of annotators on categories, relaxing the spatial constraint given by object detection tasks.

Parameters:

• result (Result) – The Result object containing all the information about the dataset and its annotations.

• iom_thr (float, optional) – The threshold for the grouping of bounding boxes based on their Intersection over Minimum.

Returns:

The computed C-Score metric for the dataset and for each individual unit.

Return type:

CScore

iaa_od.metrics.iom(lhs, rhs, /)

Computes Intersection over Minimum (IoM), which divides the intersection area of two bounding boxes by the area of the smaller bounding box.

Parameters:

• lhs (AnnotationProtocol) – The first annotation.

• rhs (AnnotationProtocol) – The second annotation.

Returns:

The IoM value between the two bounding boxes.

Return type:

float

iaa_od.metrics.iou(lhs, rhs, /)

Compute the Intersection over Union (IoU) between two bounding boxes.

Parameters:

• lhs (AnnotationProtocol) – The first annotation.

• rhs (AnnotationProtocol) – The second annotation.

Returns:

The IoU value between the two bounding boxes.

Return type:

float

iaa_od.metrics.iou_sweep(gts, start=None, stop=None, step=None, /, *, use_iom=False)

Compute Kappa and Alpha metrics for different IoU thresholds in the specified range.

Parameters:

• gts (list[GroundTruthProtocol]) – List of Ground Truths to compare.

• start (Optional[float]) – Starting IoU threshold (inclusive). Defaults to 0.5 if None.

• stop (Optional[float]) – Ending IoU threshold (inclusive). Defaults to 0.95 if None.

• step (Optional[float]) – Step size for IoU thresholds. Defaults to 0.05 if None.

• use_iom (bool) – Whether to use IoM matching. Defaults to False.

Returns:

List of Result objects corresponding to each IoU threshold.

Return type:

list[Result]

iaa_od.metrics.l_score(gts, /, *, iou_threshold=0.5, use_iom=False, exclude_mismatched=False, precomputed_result=None)

Computes the L-Score metric on a given dataset. The L-Score metric represents how much annotators agree on where there are objects, regardless of what labels they assigned to those objects. This is essentially Localization agreement, as opposed to Categorisation agreement.

Parameters:

• gts (list[GroundTruthProtocol]) – List of Ground Truths to compare.

• iou_threshold (float) – The IoU threshold to use for matching. Defaults to STD_IOU_THR.

• use_iom (bool) – Whether to use IoM matching. Defaults to False.

• exclude_mismatched (bool) – Whether to exclude mismatched units from the L-Score calculation. Defaults to False.

• precomputed_result (Optional[Result]) – An optional precomputed Result object from a previous Alpha computation to reuse its observations matrix. Defaults to None.

Returns:

The computed L-Score object.

Return type:

LScore

iaa_od.metrics.per_image_granularities(result, /, *, iom_thr=0.5, benchmark=None)

Compute Count and Size Granularities separately for each image in the dataset.

This mirrors all_granularities, but instead of aggregating the per-unit contributions over the whole dataset, it restricts the computation to the units belonging to a single image at a time and returns one Granularities object per image.

Normalisation behaviour is controlled by benchmark:

• When benchmark is None (default), each image is self-contained: the per-annotator values are normalised by the median computed within that image. A value of 1.0 therefore identifies the median annotator on that image, and the benchmark annotator may differ from one image to the next. Use this to inspect who annotates by group/instance relative to the others on a given image.

• When benchmark is supplied (typically the result of all_granularities(result)), every image is normalised by the global medians (benchmark.count_median and benchmark.size_median). Values are then comparable across images and against the global result, which is the appropriate choice for tracking how a single annotator’s behaviour drifts image by image (e.g. for labelling-fatigue analysis).

Images whose units are all singletons carry no granularity information and are omitted from the result. For a given image, only annotators that contribute to at least one non-singleton unit on that image appear in the returned dictionaries.

Parameters:

• result (Result) – The Result object containing the Ground Truths and Units.

• iom_thr (float) – The IoM threshold to consider an annotation as “contained” within a unit. Defaults to 0.5.

• benchmark (Granularities | None) – If provided, its count_median and size_median are used as the normalisation divisor for every image instead of each image’s own median, and its benchmark annotators are carried over. Defaults to None.

Returns:

A mapping from image filename to the Granularities computed from that image’s units alone.

Return type:

dict[str, Granularities]

iaa_od.metrics.rater_vitality(gts, /, *, annotator_name, iou_threshold=0.5, use_iom=False, precomputed_result=None)

Compute the vitality of a specific rater (annotator) in the provided Ground Truths.

Parameters:

• gts (list[GroundTruthProtocol]) – A list of GroundTruth objects representing the annotations.

• annotator_name (Optional[str]) – The name of the annotator whose vitality is to be computed.

• iou_threshold (float) – The IoU threshold to use for matching. Defaults to STD_IOU_THR.

• use_iom (bool) – Whether to use IoM matching. Defaults to False.

• precomputed_result (Optional[Result]) – An optional precomputed Result object containing the overall alpha value.

Returns:

The vitality score of the specified rater.

Return type:

float

iaa_od.metrics.scale_complexity(gts, /, *, collapsed_categories, iou_threshold=0.5, use_iom=False, precomputed_result=None)

Compute the scale complexity metric as defined in “Leadership of Data Annotation Teams” by McCulloh et al.

Return type:

ScaleComplexity

Parameters

gts (list[GroundTruthProtocol]): A list of GroundTruth objects representing the ground truth annotations. collapsed_categories (Optional[dict[str, list[int]]]): A dictionary mapping new category names to lists of original category IDs to be collapsed. If None, the function will prompt for interactive input. iou_threshold (float): The IoU threshold to use for matching. Default is STD_IOU_THR. use_iom (bool): Whether to use IoM matching. Default is False. precomputed_result (Optional[Result]): A precomputed Result object containing alpha value, IoU threshold, and lenient IoU settings. If provided, these settings will be used instead of computing them anew.

Returns

ScaleComplexity: The computed scale complexity value (global and per image), along with the alpha values before and after collapsing categories, both globally and per image.

Modules

alpha(gts, /, *[, iou_threshold, use_iom, ...])	This function computes Krippendorff's alpha for a set of ground truths containing bounding box annotations.
c_score(result, /, *[, iom_thr])	Function which computes the C-Score metric for a given dataset.
granularity
iou(lhs, rhs, /)	Compute the Intersection over Union (IoU) between two bounding boxes.
iou_sweep(gts[, start, stop, step, use_iom])	Compute Kappa and Alpha metrics for different IoU thresholds in the specified range.
l_score(gts, /, *[, iou_threshold, use_iom, ...])	Computes the L-Score metric on a given dataset.
rater_vitality(gts, /, *, annotator_name[, ...])	Compute the vitality of a specific rater (annotator) in the provided Ground Truths.
scale_complexity(gts, /, *, collapsed_categories)	Compute the scale complexity metric as defined in "Leadership of Data Annotation Teams" by McCulloh et al.