Fast computation of Krippendorff's alpha agreement measure in Python.
CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems
Inter-annotator agreement for Brat annotation projects
Exploring the inter-annotator agreement between ISIC Archive segmentation masks
This repository holds code to annotate textual data using LLMs, and calculate different measures of Inter-Annotator Agreement (IAA).
command line stuff to calculate inter annotator agreement
Official Repository of 'Leveraging Epistemic Uncertainty to Improve Tumour Segmentation in Breast MRI: An exploratory analysis' In SPIE Medical Imaging 2024
Reproducible audit of the MAD dataset (MAST failure taxonomy): three undocumented taxonomy versions, renumbered codes, and what that means for reported inter-annotator agreement.
Evaluation toolkit for multi-annotator human annotation research with agreement metrics, disagreement analysis, reports, and plots.
Measures whether your graders and your rubric are trustworthy: inter-annotator agreement, anchor diagnostics, grader calibration against gold, and an adjudication queue.
H-CAS: human-rated evaluation of dialectal and cultural competence in Hassaniya Arabic (code, annotation guide, ratings).
Production IAA engine — Krippendorff's Alpha, SBERT semantic agreement, Fleiss' Kappa, Shannon entropy, 3-layer collusion detection. Ran behind RawEval's 9-annotator workbench.
Concordia: reproducible inter-annotator agreement toolkit for human and machine cultural-heritage annotation
Audits the yardstick a judge leaderboard is scored against. Reproduces 39 published ASR benchmark-optimization scores exactly, then shows one documented threshold moves the mean by 38%. Standard library only; 109 tests in CI.
A Python Library to compute Disagreement Systematicity (σ)
Набор скиллов для оценки LLM-агентов и LLM-приложений: классификация, рубрики, согласие оценщиков, отчёты
Digita Literacy for School of Foreign Languages (NRU HSE, 2018)
[MICCAI ISIC 2024] Code for "Segmentation Style Discovery: Application to Skin Lesion Images"
Your experts agree. Are they right? Audits for every stage of an expert-rating quality pipeline, with control arms, real expert votes and a QA summary per batch.
Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"
To associate your repository with the inter-annotator-agreement topic, visit your repo's landing page and select "manage topics."