# Action recognition benchmarks

## Executive assessment

A credible evaluation for a small posture-based action model needs **three separate scoreboards**: recognition from pre-cut clips, localization in complete recordings, and causal recognition from an incoming stream. These measure different capabilities. Supplying the correct action boundaries removes a large part of the real application’s problem; processing a complete recording also permits future information that a live system cannot access.

The most useful core is NTU RGB+D 60/120 and FineGym for clip classification; PKU-MMD and BABEL-TAL for localization directly from body motion; and OAD plus a causal replay of a continuous skeleton dataset for streaming. Add THUMOS14 for comparison with the broader temporal-localization literature, and a segmentation dataset to measure repeated actions, transitions and over-segmentation. This prioritization is an assessment of fit to a posture-based recognizer, not a claim that those datasets exhaust human activity understanding.

The broader inventory below covers widely used benchmark families and closely related specialist tasks. It includes datasets that are immediately compatible with released pose annotations, datasets requiring a new skeleton adapter, and RGB datasets requiring pose extraction or a different input model. **A published dataset, an accessible download, and a runnable benchmark are three different states.** No finite list can certify “every possible benchmark,” including private collections and all task-specific derivatives.

Source availability was checked on September 15, 2026. Where current binary downloads, account approvals or exact derivative versions have not been verified, they are explicitly unresolved. Counts describe the cited release, not a promise that all original media URLs remain available.

## 1. What is being predicted?

| Task | Input supplied to the model | Required output | Suitable examples | What a good score does not establish |
|---|---|---|---|---|
| Trimmed action classification | A clip or skeleton span already containing an action | One class or class scores for the span | NTU, UCF101, HMDB51, Gym99, BABEL recognition | Finding the action among background frames |
| Untrimmed video classification | A whole recording with multiple actions | Which classes occur anywhere | Charades classification | Correct timing or actor assignment |
| Temporal action localization/detection | A full recording with unknown action boundaries | Class, start, end and confidence for each instance | THUMOS14, ActivityNet, HACS Segments, PKU-MMD, BABEL-TAL | Causal operation or low alert latency |
| Temporal action segmentation | A sequence containing successive actions | A label for each frame, then action segments | Breakfast, 50 Salads, GTEA, Assembly101 | Sparse-event detection under hours of background |
| Online action detection | Frames up to the present time only | Current action probabilities and/or live events | TVSeries, OAD, online THUMOS protocols | Accurate final boundaries unless also evaluated |
| Anticipation / early recognition | A prefix before an action or before its completion | Future action or early class prediction | EPIC-KITCHENS, Assembly101 | Detection under the same observation budget |
| Spatiotemporal action detection | Multiple people and unknown locations | Person boxes/tracks plus action labels in time | AVA, MultiSports, FineSports | Performance of a classifier given perfect tracks |
| Event spotting / sequencing | A longer recording | A timestamp for an event or ordered keyframes | SoccerNet, GolfDB | Interval overlap or whole-action recognition |
| Action quality assessment | A performance, often with its action type supplied | Quality score or execution assessment | FineDiving | Detecting which action happened |

These distinctions follow the original task definitions and evaluation implementations, including THUMOS14’s separate classification/detection tasks, the TVSeries online setting, and MS-TCN’s frame and segment metrics.[^2][^6][^7][^8]

### Pre-cut posture is especially privileged input

An evaluation can supply both **the correct person** and **the correct temporal crop**. A classifier receiving a 64-frame resampling of a known action does not have to decide whether an action exists, select the performer, discover its duration, or reject background. Its accuracy is still useful, but it is a component result. Keep this separate from the complete video → people → poses → action events pipeline.

For continuous evaluation, never cut the test stream at ground-truth starts and ends before inference. Ground truth is available only to the evaluator. Sliding windows are allowed, but they must be generated without knowing where the true actions are.

## 2. Pre-cut recognition inventory

The “pose route” column describes how a posture model could consume the dataset. It is not a claim that a conversion has already been implemented or validated.

| Benchmark family | Release / content | Pose route and protocol | Value for this model |
|---|---|---|---|
| **NTU RGB+D 60** | 60 classes; 56,880 original samples; RGB, depth, IR, Kinect skeletons | Native 25-joint 3D; cross-subject and cross-view. PYSKL also releases a separate HRNet 2D representation | Essential broad skeleton baseline; publish 2D and 3D rows separately.[^1][^3] |
| **NTU RGB+D 120** | 120 classes; 114,480 original samples | Native 3D and released 2D derivatives; cross-subject and cross-setup | Tests a larger vocabulary, hand-related distinctions and interactions.[^1][^3] |
| **NTU RGB+D 120, one-shot** | 100 auxiliary classes plus 20 novel evaluation classes | Exactly one specified exemplar per novel class; all its remaining examples are queries | Directly relevant to learning new action labels from little supervision. Different from X-Sub/X-Set and from retraining a 120-way classifier.[^66] |
| **Northwestern-UCLA** | 10 actions captured from three Kinect views | Native skeletons; use the exact published view split | Compact test of camera/view transfer; not a scale benchmark.[^4] |
| **UWA3D Multiview Activity II** | 30 actions, 10 subjects, four separately performed views | Kinect data; pin the cross-view protocol and downloaded representation | Recorded continuously, but distributed as segmented actions. Continuous capture does not by itself make the recognition protocol an untrimmed test.[^49] |
| **MSR Action3D** | 20 movement categories | Legacy depth/skeleton benchmark; identify the particular subject/subset protocol | Useful regression test, insufficient alone for a modern headline.[^4] |
| **MSR DailyActivity3D** | 16 daily activities; 320 sequences | Kinect-derived data; object use and sitting/standing variations | Probes what body motion can and cannot distinguish.[^4] |
| **UTD-MHAD** | 27 actions; 861 usable sequences; 8 subjects | Skeleton MAT files, plus depth/RGB/inertial modalities | Small, practical native-skeleton addition; keep skeleton-only separate from sensor fusion.[^5] |
| **Florence3D** | 215 samples, nine actions, ten subjects | Kinect pose adapter and explicit subject holdout | Small regression benchmark; original site permits nonprofit research use.[^51] |
| **HDM05** | Motion capture with named clips and longer recorded scenes | C3D or ASF/AMC adapter; pin label vocabulary and split | Clean kinematic data and repeatable motions; domain shift from video-estimated poses.[^9] |
| **BABEL-60 / BABEL-120 recognition** | Labeled spans of AMASS motion, at most five seconds for the challenge | Released recognition data; separate dense-only and dense-plus-extra label settings | Strong test of natural action distributions. This is the cropped recognition task, not BABEL-TAL.[^10][^11] |
| **UCF101** | 101 classes; over 13,000 video clips | Released HRNet COCO-17 poses; evaluate all three official splits | Familiar public reference for body-only recognition in varied scenes.[^3][^12] |
| **HMDB51** | 51 video action categories | Released HRNet poses; all three split pairs | Useful compact test of movie/web-video domain variation. The original homepage currently redirects; use the documented release and split manifest.[^3] |
| **J-HMDB** | 928 clips and 21 action classes in the original paper, with detailed pose/person annotations | Use the exact J-HMDB or sub-JHMDB subset and joint layout; recognition and person-localization are different evaluations | Useful for comparing annotated poses with estimated poses. It is not the same as the HMDB51 pose derivative.[^55] |
| **Toyota Smarthome, trimmed** | About 16K RGB-D clips, 31 activities performed by seniors | Recognition under the selected subject/view protocol; record pose-estimation source | Adds unscripted daily activities, class imbalance and population shift; distinct from TSU's untrimmed detection task.[^56] |
| **Kinetics-400** | 400 broad human-action categories; roughly ten-second clips | Original RGB or derived skeletons. PYSKL’s main pickle is only an index; pose shards are separate | Broad coverage, but many classes depend on objects/context; not an immediate single-file pose benchmark.[^3][^13] |
| **Kinetics-600 / 700 / 700-2020** | Successive expanded releases; the 2020 700-class edition replenishes clips | Separate RGB/derived-pose acquisition and exact version accounting | Expensive scale extensions. Do not conflate releases or pretraining sources.[^14] |
| **FineGym Gym99 / Gym288** | Hierarchical gymnastics events and fine-grained elements | Gym99 HRNet pose release; preserve exact temporal annotations and report mean-class accuracy as well as top-1 | Particularly valuable for small posture models and motion order; Gym99 and Gym288 are separate evaluations.[^3][^15] |
| **Diving48** | 48 fine-grained diving classes; over 18,000 clips in the original paper | Released HRNet poses; verify correspondence to corrected V2 labels | Good test of temporal structure. Unverified V1/V2 mixing invalidates comparisons.[^3][^16] |
| **Penn Action** | 2,326 sequences; 15 actions; human joint annotations | Adapt supplied joints rather than assuming COCO-17 indexing | Useful small pose/action dataset and motion-phase analysis.[^17] |
| **HAA500** | 500 curated atomic action classes | Human-centric clips; pose extraction/annotation format verification still required | Broad, relatively fine-grained actions; current project recommends v1.1 over the original v1.0.[^18] |
| **Something-Something V1/V2** | Object-manipulation templates; V2 has 220,847 clips | Primarily RGB/hand-object evidence; generic body COCO-17 is a poor complete representation | Useful temporal-reasoning challenge after adding hands/objects; lower priority for body-only claims.[^19] |
| **Moments in Time** | One million three-second clips; 339 event classes | RGB/audio; events also include animals, objects and natural phenomena | Measures wider event understanding, not purely human posture recognition.[^20] |
| **HACS Clips** | 1.5 million annotated short clips from 504,000 videos | RGB or newly derived poses; distinguish this from HACS Segments | Large classification/pretraining resource, with a separate localization counterpart.[^21] |
| **BoxingVI** | 6,915 punch clips, six punch types, 20 sparring sessions, 18 athletes | Validate the precise dataset identity, pose representation and session/athlete/view holdout | Direct sport fit. A leave-one-view-out result is not automatically comparable to every published split.[^22] |
| **Assembly101 recognition** | Fine-grained assembly segments with 3D hand poses | Hand-skeleton adapter; source-sequence separation across views | Important when expanding from body motion to manipulation.[^23] |
| **EPIC-KITCHENS-55/100 recognition** | Egocentric kitchen action segments; EK100 has 97 verbs and 300 nouns | Hand/object/RGB inputs; use official recognition splits | Body-only performance is limited by missing object identity and self-occlusion.[^24] |

### Why there is no single recognition ranking

NTU’s native Kinect skeletons and PYSKL’s estimated 2D poses are different observations of the task. Their coordinate systems, joint sets, noise, visibility and available depth differ. Likewise, a two-person model receives information a body-1-only classifier omits. An RGB model may exploit a cup, a racket or a scene that is absent from the skeleton. Compare matched inputs first, then show broader multimodal systems in a separately labeled table.

FineGym demonstrates a second comparison trap: its official documentation distinguishes **mean-class accuracy** from ordinary top-1 accuracy. A model can improve its overall accuracy by favoring frequent classes while failing rare elements. Report both, with the exact Gym99/Gym288 release.[^15]

## 3. Continuous body-motion benchmarks

| Benchmark | Temporal data and labels | What to measure | Access / implementation assessment |
|---|---|---|---|
| **PKU-MMD phase 1** | 1,076 long sequences, 51 classes, approximately 3–4 minutes per sequence; 25-joint 3D skeletons and start/end labels | Official cross-subject/cross-view detection evaluation and temporal mAP; causal event metrics as additional rows | Closest existing adapter. Use full sequences for detection, never ground-truth-cut test actions.[^25] |
| **PKU-MMD phase 2** | A separately released phase | Its own official split and evaluator | Separate acquisition/version audit needed; do not label a phase-1 score “PKU-MMD all.”[^25] |
| **BABEL-TAL** | Action localization in continuous 3D motion; published as a separate benchmark | Published localization metrics and split; evaluate a causal variant separately | High-priority addition after obtaining the exact benchmark conversion and motion assets.[^26] |
| **BABEL-TAL-20 / BT20** | The 20-class variant named in the 2022 LocATe preprint | Use the precise class mapping, split and evaluator from that release | Keep its identity explicit when comparing with the 2024 BABEL-TAL publication; do not silently combine versions.[^45] |
| **BABEL-TAL-60** | A separate, harder 60-class setting described in the same preprint | Full-sequence localization with the 60-class label mapping | Its longer action-label tail makes it a distinct test from BT20 and from cropped BABEL-60 recognition.[^45] |
| **BABEL dense frame labels** | About 43 hours of AMASS motion; action intervals can overlap | Multi-label localization/segmentation | Rich source for custom diagnostics, but custom splits are not the BABEL recognition challenge or BABEL-TAL.[^10] |
| **OAD** | Streaming skeleton dataset introduced with joint classification/regression recurrent networks | Online classification, action start/end detection and the original timing protocol | Strong causal test; locate and validate original streams and annotations before claiming a reproduction.[^27] |
| **UTKinect-Action3D** | Ten actions concatenated per recording; ten subjects, RGB/depth and 20-joint skeletons; action intervals supplied | Original recognition protocol plus a separately labeled continuous replay | Particularly cheap raw skeleton acquisition (about 3.3 MB). Missing tracked frames cause index jumps; preserve frame indices rather than assuming uniform retained samples.[^52] |
| **Watch-n-Patch** | Composite daily activities captured with Kinect v2 | Unsupervised segmentation/recognition and detecting omitted steps | Useful test of action order and missing-action reasoning. Supervised classification changes its original learning setting.[^53] |
| **Composable Activities** | 693 recordings; 16 composed activities and 26 atomic actions | Composition recognition and action-level timing under the original protocol | Tests concurrent body-part actions and long activity structure; current binary availability still needs verification.[^54] |
| **G3D** | Skeleton action data used in online detection research | Original event detection protocol | Valuable small streaming regression set; distribution and exact annotations still need access verification.[^27] |
| **IKEA ASM** | 371 assembly samples; three RGB views, depth, poses and atomic-action intervals; 33 action classes | Temporal segmentation and action recognition under official splits | Relevant continuous test. Only a subset of pose frames is manually labeled; the remainder includes pseudo ground truth.[^28] |
| **InHARD** | Industrial action sequences with synchronized RGB and BVH skeletons; action start/end fields | Declared label level, actor-disjoint recognition and continuous segmentation | Published repository describes 13 meta-actions and 74 finer actions; choose a level explicitly and convert BVH consistently.[^29] |
| **Assembly101 segmentation** | Long assembly/disassembly recordings, coarse/fine action segments and 3D hand poses | Official segmentation and anticipation tasks | The official site reports a May 2026 move of distribution to Hugging Face. Hands require a different joint adapter from body COCO-17.[^23] |
| **LARa** | Logistics picking/packing; version 1 has 758 labeled minutes, 14 people, optical motion capture, IMUs and RGB | Activity classification/segmentation and semantic-attribute prediction under a declared subject split | Native motion capture is relevant; sensor fusion is a separate input condition. The version-1 deposit flags a newer record, so acquisition must pin the chosen version.[^48] |
| **CAD-120** | 120 RGB-D videos of activities composed of subactivities and object interactions | Detection and anticipation; original work uses held-out-subject folds | Distinguish supplied ground-truth segments from inferred segmentation. Body-only input omits the object affordances in the original task.[^50] |
| **SHREC 2021 continuous hand gestures** | Heterogeneous gestures embedded in hand-skeleton streams | Online gesture detection | A hand-specific streaming test, requiring a hand-joint adapter.[^46] |
| **SHREC 2022 online gestures** | 288 sequences, 16 gesture classes, 26 tracked hand joints; irrelevant motions between gestures | Detection, false positives, latency and continuous-annotation Jaccard score | Especially relevant to live interaction. Frame timestamps are provided because sampling is irregular; retain them. Original site links train/test data, test labels and a MATLAB evaluator.[^47] |

**Highest-value addition: BABEL-TAL.** It directly tests whether a model recognizes and localizes actions from 3D motion. It complements PKU-MMD’s captured daily actions and the application’s sport-specific annotations. Its published offline localization performance must still be separated from a no-lookahead deployment claim.[^26]

**Do not turn every continuous dataset into a trimmed benchmark and count that as coverage.** It is reasonable to derive training crops from labeled ranges. At test time, supplying the true crop evaluates recognition only. Keep the full recording, transitions, repeated actions and background for the continuous row.

## 4. Continuous RGB, segmentation and multi-person benchmarks

| Benchmark / task | Defining property | Standard evaluation family | Suitability / required work |
|---|---|---|---|
| **THUMOS14 temporal detection** | Untrimmed sports videos; detection uses 20 classes, distinct from 101-class classification | Temporal mAP at explicitly listed overlap thresholds | Strong first external temporal-detection comparison; requires poses or RGB features over complete videos.[^2] |
| **ActivityNet temporal localization** | Broad long-video activity intervals; pin dataset version | Mean mAP across tIoU 0.50–0.95 in steps of 0.05, plus individual thresholds | Good broad localization test; official evaluator is available.[^30] |
| **HACS Segments** | 139,000 annotated segments in 50,000 videos, 200 classes | Temporal localization/proposal metrics | Larger, denser complement to ActivityNet; substantial ingestion cost.[^21] |
| **FineAction** | 103,324 instances, 106 classes in about 17,000 untrimmed videos; co-occurring actions | Temporal mAP, including 0.50–0.95; proposal AR where relevant | Good fine-grained stress test after THUMOS/PKU.[^31] |
| **MultiTHUMOS** | Dense, overlapping action labels over THUMOS video | Frame-level multi-label evaluation and specified localization protocol | Important for simultaneous actions; a single softmax label per frame is insufficient.[^32] |
| **Charades** | 9,848 indoor videos; 157 action classes; temporal and object annotations | Distinct official classification and localization mAP evaluators | Tests multiple concurrent activities and object-dependent labels.[^33] |
| **Toyota Smarthome Untrimmed** | Spontaneous daily activity, dense elementary/composite/object-interaction labels | Activity detection under its specified protocol | Useful deployment-like background and duration variation; pose-route assets require verification.[^34] |
| **Breakfast** | Long procedural cooking sequences | Frame accuracy, edit score, segment F1; four splits in the MS-TCN setup | Standard temporal segmentation; released I3D features are not posture data.[^7] |
| **50 Salads** | Continuous food-preparation procedures | Frame accuracy, edit score, segment F1; five splits in the MS-TCN setup | Particularly useful for fragmentation and repeated-action errors.[^7][^35] |
| **GTEA segmentation** | Egocentric daily actions | Frame accuracy, edit score, segment F1; four splits in the MS-TCN setup | Small segmentation test; hands/objects dominate, and GTEA/Gaze/Gaze+ must not be mixed.[^7] |
| **EPIC-KITCHENS action detection** | Actions must be localized in long egocentric recordings | Official verb, noun and action evaluation for the selected task | Different from its supplied-segment recognition challenge.[^24] |
| **Ego4D moment queries** | Find specified activities in long first-person video | Benchmark-specific retrieval/localization metrics | Broad long-video test, with its own vocabulary, split and input contract.[^36] |
| **AVA / AVA-Kinetics** | Person-localized, often multi-label atomic actions | Person-box action detection mAP; pin annotation/evaluator version | Requires actor localization/tracking. Not interchangeable with interval-only mAP.[^37] |
| **UCF101-24** | Person-localized 24-class derivative of UCF101 | Spatiotemporal frame/tube detection under a pinned annotation revision | Use corrected annotations; its detection score is not UCF101's 101-class clip accuracy.[^57] |
| **JRDB-Act** | Moving-robot observations with individual actions, social groups and group activities | Spatiotemporal action and group/activity detection | Strong crowded-scene extension; JRDB-Pose adds pose annotations, but coordinate/version alignment must be audited. Downloads require an account.[^58] |
| **Perception Test, action localization** | Densely annotated action intervals within a broader multimodal benchmark | Its action-localization mAP and task-specific challenge split | A relevant TAL track even though other Perception Test tracks evaluate questions, tracking or sound. Released TSP video features are not skeleton inputs.[^59] |
| **MultiSports** | Multiple people with spatiotemporally localized sports actions | Frame and video/tube action detection metrics | Relevant to sports archives; evaluate tracking and action classification together and separately.[^38] |
| **FineSports** | Fine-grained, hierarchical multi-person basketball actions | Spatiotemporal localization | Strong specialist extension; release agreement and data access are required.[^39] |
| **Ego-Exo4D** | Synchronized first/third-person recordings of skilled activities; multiple benchmark tasks | Choose the particular activity, proficiency or pose task | Useful future cross-view/skill benchmark; not one generic action-recognition score.[^40] |

### Online evaluation is a protocol, not just a dataset name

TVSeries explicitly asks for action detection as soon as an action happens. Its original release contains 27 episodes from six series and 30 action classes. LSTR’s official implementation evaluates online recognition with per-frame mAP or calibrated mAP and can break performance down by action progress.[^6][^41]

THUMOS14 and other recordings can support an online protocol, but a precomputed feature can already contain future context. A causal temporal head over bidirectional video features is not a fully causal system. Audit the feature extractor’s frame window, pose smoothing, person tracking, normalization, postprocessing and suppression, in addition to the action network.

For the application, replay complete test recordings in timestamp order. At time *t*, reveal only frames up to *t*. Record both the predicted event time and the time the prediction was actually emitted. Freezing the output history makes it impossible to improve an earlier alert retroactively after observing the action’s end.

## 5. Sport-specific and adjacent tests

**GolfDB** provides 1,400 golf swing videos with event-frame labels. It is a sequencing/keyframe benchmark, making it relevant to a P-system or swing-phase model. Report event timing under GolfDB’s own protocol; do not present it as full continuous action detection in unconstrained hours of golf footage.[^42]

**SoccerNet** is a benchmark suite covering action spotting and other soccer-video tasks. A spotting label identifies an event timestamp, whereas a punch or swing detector may predict an interval. The event definition and allowed timing tolerance matter as much as the model score. Ball events may also require visual evidence absent from a single person’s skeleton.[^43]

**FineDiving** focuses on procedure-aware action quality assessment. It is valuable if the product will judge how well someone performed, but rank correlation with judged quality is not recognition accuracy or temporal detection mAP.[^44]

### Additional gesture benchmarks: matched isolated and continuous tasks

| Family | Pre-cut task | Continuous task / important distinction | Acquisition and model implications |
|---|---|---|---|
| **Jester** | 148,092 short clips across 27 labels | Includes no-gesture labels, but clip classification alone does not establish continuous spotting | Original publisher now distributes through Qualcomm; hand details and RGB input matter.[^60] |
| **Montalbano V2** | Crops can be derived for auxiliary recognition | Official user-independent continuous gesture spotting, 20 Italian gesture categories, scored using Jaccard overlap | RGB, depth and 20-joint skeletons are supplied. Preserve complete recordings for the official task.[^61] |
| **IPN Hand** | An isolated recognition baseline is provided | 200 continuous RGB videos, 13 gesture types, natural non-gesture motion and adjacent gestures | About 4.6 GB of MP4s; 4,218 gesture instances, 50 people. No body skeleton is supplied by this original RGB release.[^62] |
| **IPN HandS** | A derived hand-skeleton dataset | The 2025 work adds frame-level hand skeleton annotations for continuous IPN sequences | Promising lightweight route; exact release files, revised labels and correspondence to the original split still need acquisition verification.[^63] |
| **EgoGesture** | 24,161 labeled gesture spans, 83 classes | 2,081 full RGB-D videos also support spotting; preserve provided subject-disjoint train/validation/test split | Original site explicitly requires a signed agreement and approval before download; no agreement was submitted in this task.[^64] |
| **ChaLearn IsoGD / ConGD** | IsoGD supplies isolated gestures | ConGD supplies continuous recordings; both use a 249-label vocabulary | A particularly clear paired benchmark. These RGB-D releases must not be confused with Montalbano's released skeletons; pose extraction is a separate step.[^65] |

Additional families to consider in a second catalog pass include SBU Kinect Interaction, Berkeley MHAD, CAD-60, MPII Cooking, SHREC 2017/DHG pre-cut hand gestures, NVIDIA Dynamic Hand Gestures, and sign-language datasets. Their original releases, active download paths and exact protocols have not all been independently validated here; they are an explicit acquisition backlog, not runnable or verified rows. Gesture/sign models also need fingers, hands and sometimes face channels that a 17-joint body model does not observe.

Pose-estimation datasets, motion-generation corpora, anomaly-detection benchmarks and video-language question-answering suites should not be counted automatically as action-recognition coverage. They become relevant only with a clearly defined action task, labels, held-out split and evaluator.

## 6. Measurement design

### Recognition

Report top-1, and top-5 when meaningful for the class count; include mean-class accuracy and per-class support. Separate random-seed variation from split variation. For UCF101/HMDB51, report each official split and the mean over splits; three seeds of split 1 do not replace the three official splits.

Use a predeclared training recipe and model selection rule. Final-epoch evaluation is straightforward; if selecting the best checkpoint, use a genuine validation set. Never tune thresholds, receptive fields or checkpoints on the final test set and present the resulting number as untouched held-out performance.

### Localization

A detection needs class, start time, end time and confidence. Temporal intersection-over-union is the length of overlap divided by the length of the union of the predicted and true intervals. A one-second prediction shifted by half a second overlaps a one-second target for half a second, but its union is 1.5 seconds: tIoU is 1/3, not 1/2.

Use the benchmark evaluator to enforce matching and duplicate rules. ActivityNet’s implementation averages per-class AP over a specified grid of overlap thresholds. Report that average and useful individual thresholds rather than replacing it with an unspecified “accuracy.”[^30]

### Segmentation

Frame accuracy gives long actions more weight. Edit score evaluates the sequence of action labels after repeated frame labels are collapsed. Segment F1 measures whether complete predicted segments match true segments at fixed overlap thresholds, exposing repeated fragments that frame accuracy can hide. The MS-TCN evaluator reports F1 at 10%, 25% and 50% overlap, alongside accuracy and edit score.[^8]

Keep single-label segmentation and multi-label action streams separate. A punch while stepping, or holding an object while walking, can produce valid simultaneous labels. Forcing those cases into one mutually exclusive label changes the task.

### Streaming

Add false alerts per minute, event precision/recall, onset timing error, emission latency and duplicate alerts per true event. State the permitted lookahead in milliseconds. If the system deliberately buffers 200 ms, call that 200 ms lookahead rather than zero-latency streaming.

Tune the alert threshold on validation recordings, then freeze it. Report recall at fixed false-alert rates as well as event F1. Include background-only periods, long gaps, camera cuts, people entering/exiting, tracking changes and consecutive actions without a pause. These are recommended product diagnostics, not substitutes for official benchmark metrics.

### Small-label learning

The landing page’s “a couple manual ground truths” claim needs a separate learning-curve experiment. Use, for example, 1, 2, 5, 10, 25 and all training examples per class, sampled only from training identities. Hold the evaluation set fixed. Repeat sampling, report variation, and count both positive and negative annotations plus the review effort used to acquire them.

A classifier trained on tens of thousands of benchmark clips cannot establish a two-example claim. Retrieval-assisted labeling is also additional supervision: record how many proposed examples were accepted, rejected or edited.

NTU-120 provides an established one-shot protocol: train a representation on 100 auxiliary classes, then recognize 20 novel classes from one fixed exemplar each. The original repository publishes the exact class IDs and exemplars. Train that representation afresh without the novel classes; today's fully supervised 120-class weights have already seen those labels and cannot serve as an unseen-class one-shot result. Report which auxiliary labels and pretrained pose models are available. Zero-shot language-based recognition, few-shot adaptation, and closed-set supervised classification are separate supervision conditions.[^66]

## 7. Small models for local recognition

The reason to keep the action classifier small is practical: target CPU inference in a browser or on a phone. The workflow is video → body poses → a compact action classifier → predictions the person watching can use or correct. Pose extraction still has its own compute cost; a small classifier alone does not establish real-time performance on every device.

This also creates a way to build a better training set. Review the proposed actions, correct their labels and start/end times, and save useful examples alongside the mistakes and background activity. Those reviewed recordings can support a heavier model later. The value comes from collecting varied, correctly labeled motion; accumulating unreviewed predictions would also accumulate the small model's errors.

[^1]: ROSE Lab, Nanyang Technological University. [NTU RGB+D and NTU RGB+D 120](https://rose1.ntu.edu.sg/dataset/actionRecognition/). Original papers 2016/2019; official dataset, modalities, counts and access terms.
[^2]: THUMOS organizers. [THUMOS Challenge 2014](https://www.crcv.ucf.edu/THUMOS14/home.html). 2014. Distinct untrimmed classification and temporal-detection tasks.
[^3]: PYSKL authors. [Dataset formats and download instructions](https://github.com/kennymckormick/pyskl/blob/main/tools/data/README.md). Official released-pose formats, split conventions and Kinetics shard requirement.
[^4]: Jiang Wang. [Action datasets](https://wangjiangb.github.io/my_data.html). Primary descriptions of MSR Action3D, DailyActivity3D and Northwestern-UCLA.
[^5]: Chen, Jafari and Kehtarnavaz. [UTD-MHAD](https://personal.utdallas.edu/~kehtar/UTD-MHAD.html). ICIP 2015. Modalities, actions, sample counts and files.
[^6]: De Geest et al. [Online Action Detection](https://arxiv.org/abs/1604.06506). ECCV 2016. TVSeries and online task definition.
[^7]: Abu Farha and Gall. [MS-TCN official implementation](https://github.com/yabufarha/ms-tcn). CVPR 2019. Breakfast, 50 Salads and GTEA features and split protocol.
[^8]: MS-TCN authors. [Segmentation evaluator](https://github.com/yabufarha/ms-tcn/blob/master/eval.py). Frame accuracy, edit and segment F1 definitions.
[^9]: Müller et al. [Documentation: Mocap Database HDM05](https://resources.mpi-inf.mpg.de/HDM05/07_MuRoClEbKrWe_HDM05.pdf). 2007. Motion formats, clips and longer scenes.
[^10]: Punnakkal et al. [BABEL](https://babel.is.tue.mpg.de/). CVPR 2021. Motion labels, overlapping intervals and dataset scale.
[^11]: Max Planck Institute. [BABEL Action Recognition Challenge](https://babel-evaluation.is.tuebingen.mpg.de/). BABEL-60/120, cropped-span input and permitted labels.
[^12]: Soomro, Zamir and Shah. [UCF101](https://arxiv.org/abs/1212.0402). 2012. Class and video scope.
[^13]: Kay et al. [The Kinetics Human Action Video Dataset](https://research.google/pubs/the-kinetics-human-action-video-dataset/). 2017. Original Kinetics-400.
[^14]: Smaira et al. [Kinetics-700-2020](https://arxiv.org/abs/2010.10864). 2020. Expanded/replenished release; [Kinetics-700](https://arxiv.org/abs/1907.06987), Carreira et al., 2019, describes the preceding 600/700-class expansion.
[^15]: Shao et al. [FineGym](https://sdolivia.github.io/FineGym/) and [official repository](https://github.com/SDOlivia/FineGym). CVPR 2020. Hierarchy, exact timestamps, Gym99/Gym288 and mean-class accuracy.
[^16]: Li, Li and Vasconcelos. [RESOUND / Diving48](https://openaccess.thecvf.com/content_ECCV_2018/html/Yingwei_Li_RESOUND_Towards_Action_ECCV_2018_paper.html). ECCV 2018. [MMAction2 Diving48 preparation](https://github.com/open-mmlab/mmaction2/blob/main/tools/data/diving48/README.md) specifies corrected V2 annotations.
[^17]: Zhang, Zhu and Derpanis. [Penn Action dataset README](https://github.com/dreamdragon/PennAction/blob/master/README). ICCV 2013. Actions and human joint annotations.
[^18]: Chung et al. [HAA500](https://www.cse.ust.hk/haa/). ICCV 2021. Atomic actions and v1.1 update.
[^19]: Twenty Billion Neurons. [Something-Something V2 release](https://medium.com/twentybn/something-something-v2-release-9107b4a8ce99). 2018. Original publisher’s release announcement.
[^20]: Monfort et al. [Moments in Time Dataset](https://arxiv.org/abs/1801.03150). 2018. Event classes, duration and modalities.
[^21]: Zhao et al. [HACS: Human Action Clips and Segments](https://arxiv.org/abs/1712.09374). ICCV 2019 version. Separate clip and interval datasets.
[^22]: Kumar et al. [BoxingVI](https://arxiv.org/abs/2511.16524). November 2025. Punch classes, source sessions and athletes.
[^23]: Sener et al. [Assembly101](https://assembly-101.github.io/). CVPR 2022; distribution update May 2026. Segments, 3D hand poses and task suite.
[^24]: Damen et al. [EPIC-KITCHENS official task definitions](https://epic-kitchens.github.io/2024). EK100 dataset and recognition/detection/anticipation distinctions; research terms.
[^25]: Liu et al. [PKU-MMD official repository and evaluator](https://github.com/ECHO960/PKU-MMD). 2017. Continuous skeleton format and annotations.
[^26]: Sun et al. [Localization and recognition of human action in 3D using transformers](https://www.nature.com/articles/s44172-024-00272-7). Communications Engineering, September 3, 2024. BABEL-TAL and LocATe.
[^27]: Li et al. [Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks](https://arxiv.org/abs/1604.05633). ECCV 2016. OAD, streaming skeleton detection and G3D evaluation.
[^28]: Ben-Shabat et al. [IKEA ASM](https://ikeaasm.github.io/). 2020. Modalities, atomic actions, pose annotation coverage and access terms.
[^29]: Havard and collaborators. [InHARD official repository](https://github.com/vhavard/InHARD). 2020. BVH format, hierarchical classes, temporal labels and download location.
[^30]: ActivityNet authors. [Official temporal detection evaluator](https://github.com/activitynet/ActivityNet/blob/master/Evaluation/eval_detection.py). Threshold grid, matching and mAP aggregation.
[^31]: Liu et al. [FineAction](https://deeperaction.github.io/datasets/fineaction). TIP 2022. Taxonomy, temporal instances, overlapping labels and evaluation.
[^32]: Yeung et al. [Every Moment Counts / MultiTHUMOS](https://arxiv.org/abs/1507.05738). 2015. Dense overlapping temporal labels.
[^33]: Sigurdsson et al. [Charades official project](https://prior.allenai.org/projects/charades). ECCV 2016. Classification versus localization and temporal annotations.
[^34]: Dai et al. [Toyota Smarthome Untrimmed](https://arxiv.org/abs/2010.14982). 2020. Natural continuous activity detection.
[^35]: McKenna and Stein. [50 Salads institutional dataset record](https://discovery.dundee.ac.uk/en/datasets/50-salads/publications/). 2013.
[^36]: Ego4D consortium. [Ego4D project](https://ego4d-data.org/) and [OpenTAD supported data](https://github.com/sming256/OpenTAD/blob/main/docs/en/data.md), including Ego4D moment queries and required assets.
[^37]: Google Research. [AVA and AVA-Kinetics](https://research.google.com/ava/). AVA CVPR 2018, subsequent v2.2/AVA-Kinetics releases. Person-localized action annotations.
[^38]: Li et al. [MultiSports official repository](https://github.com/MCG-NJU/MultiSports). ICCV 2021. Multi-person spatiotemporal sports detection.
[^39]: Xu et al. [FineSports official repository](https://github.com/PKU-ICST-MIPL/FineSports_CVPR2024). CVPR 2024. Task and release-agreement process.
[^40]: Grauman et al. [Ego-Exo4D](https://arxiv.org/abs/2311.18259). 2023/2024. Skilled activity, multiple views and distinct benchmarks.
[^41]: Xu et al. [Long Short-Term Transformer official implementation](https://github.com/amazon-science/long-short-term-transformer). NeurIPS 2021. Online per-frame and per-stage evaluation.
[^42]: McNally et al. [GolfDB](https://arxiv.org/abs/1903.06528). 2019. Swing sequencing and event-frame annotations.
[^43]: SoccerNet organizers. [SoccerNet benchmark suite](https://www.soccer-net.org/). Original 2018 release and subsequent task expansions.
[^44]: Xu et al. [FineDiving](https://arxiv.org/abs/2204.03646). CVPR 2022. Procedure-aware action quality assessment.
[^45]: Sun et al. [LocATe: End-to-end Localization of Actions in 3D with Transformers](https://arxiv.org/html/2203.10719v1). March 2022 preprint. Distinct BT20 and BABEL-TAL-60 settings; the later journal publication is source 26.
[^46]: Caputo et al. [SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild](https://arxiv.org/abs/2106.10980). 2021. Gestures embedded in continuous streams.
[^47]: SHREC organizers. [SHREC 2022 online gesture benchmark](https://univr-vips.github.io/Shrec22/); [original paper](https://arxiv.org/abs/2207.06706). 2022. Dictionary, streams, timestamps, labels and evaluator.
[^48]: Niemann et al. [LARa version 1 dataset deposit](https://zenodo.org/records/3862782). 2020. Motion capture, IMUs, recordings and annotation scope; points to a newer version.
[^49]: Rahmani et al. [UWA 3D Multiview Activity II dataset record](https://research-repository.uwa.edu.au/en/datasets/uwa-3d-multiview-activity-ii-dataset/). Institutional record, 2023. Continuous acquisition and subsequent segmentation.
[^50]: Koppula and Saxena. [Anticipating Human Activities using Object Affordances](https://www.cs.cornell.edu/~hema/papers/anticipation_PAMI2015.pdf). PAMI 2015. CAD-120 task, held-out-subject evaluation and ground-truth segmentation conditions.
[^51]: University of Florence MICC. [Florence 3D actions dataset](https://www.micc.unifi.it/resources/datasets/florence-3d-actions-dataset/). 2012/2013. Dataset scale, activities and research-use terms.
[^52]: Xia, Chen and Aggarwal. [UTKinect-Action3D](https://cvrc.ece.utexas.edu/KinectDatasets/HOJ3D.html). CVPR Workshops 2012. Concatenated recordings, action intervals, joint format and missing-frame caveat.
[^53]: Wu et al. [Watch-n-Patch: Unsupervised Learning of Actions and Relations](https://arxiv.org/abs/1603.03541). 2016. Composite activity data, segmentation and omitted-action task.
[^54]: Lillo et al. [Discriminative Hierarchical Modeling of Spatio-Temporally Composable Human Activities](https://openaccess.thecvf.com/content_cvpr_2014/papers/Lillo_Discriminative_Hierarchical_Modeling_2014_CVPR_paper.pdf). CVPR 2014. Composable Activities data and protocols.
[^55]: Jhuang et al. [Towards Understanding Action Recognition](https://openaccess.thecvf.com/content_iccv_2013/papers/Jhuang_Towards_Understanding_Action_2013_ICCV_paper.pdf), ICCV 2013; [author project page](https://pages.iai.uni-bonn.de/gall_juergen/projects/JHMDB/JHMDB.html). Joint/person annotations and original release scope.
[^56]: Das et al. [Toyota Smarthome: Real-World Activities of Daily Living](https://openaccess.thecvf.com/content_ICCV_2019/html/Das_Toyota_Smarthome_Real-World_Activities_of_Daily_Living_ICCV_2019_paper.html). ICCV 2019. Trimmed unscripted activity dataset.
[^57]: OpenMMLab. [UCF101-24 data preparation](https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ucf101_24/README.md). Documents the corrected annotation derivative; [original UCF101 task links](https://www.crcv.ucf.edu/research/data-sets/ucf101/).
[^58]: JRDB organizers. [JRDB and JRDB-Act](https://jrdb.erc.monash.edu/). CVPR 2022 action benchmark and subsequent JRDB2022 releases; task families and account-gated downloads.
[^59]: Google DeepMind. [Perception Test official repository](https://github.com/google-deepmind/perception_test). NeurIPS 2023 benchmark and subsequent challenges. Task-specific intervals, split manifests, features and evaluation.
[^60]: Qualcomm. [Jester dataset](https://www.qualcomm.com/developer/software/jester-dataset). Publisher's current distribution, labels, split counts and research license.
[^61]: ChaLearn organizers. [Montalbano V2](https://chalearnlap.cvc.uab.es/dataset/13/description/). ECCV 2014. Continuous spotting, skeleton format and Jaccard evaluation.
[^62]: Benitez-Garcia et al. [IPN Hand](https://gibranbenitez.github.io/IPN_Hand/). ICPR 2020/2021. Original continuous data, download sizes, annotations and paired baselines.
[^63]: [IPN HandS: Efficient Annotation Tool and Dataset for Skeleton-Based Hand Gesture Recognition](https://www.mdpi.com/2076-3417/15/11/6321). Applied Sciences, 2025. Derived frame-level skeleton annotations; binary distribution not verified here.
[^64]: Zhang et al. [EgoGesture dataset](https://nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html). TMM 2018. Paired segmented/continuous tasks, exact subject splits and access process.
[^65]: Wan et al. [ChaLearn Looking at People RGB-D Isolated and Continuous Datasets](https://openaccess.thecvf.com/content_cvpr_2016_workshops/w18/papers/Wan_ChaLearn_Looking_at_CVPR_2016_paper.pdf), CVPR Workshops 2016; [organizers' task description](https://chalearnlap.cvc.uab.cat/challenge/25/track/24/description/). IsoGD/ConGD vocabulary and temporal task split.
[^66]: NTU RGB+D authors. [Official NTU120 one-shot evaluation protocol](https://github.com/shahroudy/NTURGB-D#evaluation-protocol-of-one-shot-action-recognition-on-ntu-rgbd-120). Exact auxiliary/novel classes and fixed exemplar identities.
