Action recognition · Working examples

Teaching small models to spot human movement

Movement embeddings seem to carry too much video-specific information: with a small dataset, it is easy to learn the video instead of the movement. Tiny models using only 2D postures can get roughly 80% of the way to a useful workflow while enough movement examples are gathered for heavier models. Run them locally, correct the mistakes, and grow the dataset without a cloud bill for action inference.

Golf: video to compared swings

Your video

Your swing

Your swings

3
SWING 1 · 3.4s
SWING 2 · 2.9s
SWING 3 · 2.7s

Comparison

Fighting: what the jab cross model sees

—
frame 1/22 · +15/727ms
attn
4%
score
100%
drives: left ankle, right knee, left elbow

The animation from /move-classifier, showing a labeled jab cross. Larger, brighter joints indicate greater model sensitivity. The bars show attention and action score for each frame.

Open the jab cross model
The golf pipeline beside the existing jab cross explanation component. The explanation was computed from the stored model and labeled clip on September 16.

Follow a person. Follow the evidence.

Recognizing an action and recognizing its performer are separate problems. Step through real footage to see how face and body evidence connects sightings, keeps uncertainty visible, and repairs mistaken identities.

Who is who?

0:00 / 0:00frame 0, tick 0
State machine run0%
Boxes and poses0%

Part 1: Start with the movement

Finding a swing, a jab or another human movement in a video can start with a small set of labels. Posture means the positions of the body’s joints over time. Once those positions are extracted, a small action model can work on them locally in a browser. Pose extraction has its own processing cost. A few labeled examples start the search; reviewing the results supplies the next batch of labels.

General video representations such as V-JEPA 2 and DINO retain information about appearance as well as movement. With little labeled data, separating the action from the person, scene and camera is a difficult learning problem. In the local frozen-embedding alignment tests, repeated swings and nearly static poses caused trouble. Joint positions give the small model a more focused starting point.

Language models can describe video, but fine timing and short, repeated motions remain difficult. Sending frames for continual analysis also adds cost. The aim is an inexpensive recognition loop: run predictions, use them, and correct the useful mistakes into training data.

What the embedding comparison actually established

The local comparison tested motion alignment with frozen V-JEPA 2.1 and DINOv3 features. It was not a test of every video model, nor a controlled experiment isolating background information. Some reference alignments were drawn using the posture matcher’s own display, which favors that baseline. Resampling by movement progress helped with playback-speed differences; choosing the wrong repetition remained a problem. See the August 23 experiment, model versions and recorded results.

Pose extraction also makes mistakes and discards useful cues: objects, contact, fine hand shape and depth can distinguish actions that look similar as joint positions. Using joint positions reduces the input size and allows local action inference, but loses those cues. For related limits of fine-motion video description, see the BoxComm study.

Keep the postures. Change the action model.

For a system that already stores 2D joint coordinates, the practical question is simple: can a new movement be learned from those records, without retrieving and decoding the video again? Keep the joint sequences, timestamps, tracking identities and confidence scores, and new labels can train another action model on the same data. At inference, it receives successive frames of posture coordinates. Turning a new camera feed into those coordinates still requires pose extraction.

An embedding retains the features learned during training. A new movement label may depend on details it has discarded. Similar overall actions can differ in speed, direction, repetition or a brief joint transition. Those details may be difficult to recover from a frozen descriptor with a few examples. For example, V-JEPA 2 demonstrates substantial motion understanding, while DINOv2 learns general image features. Neither result guarantees a descriptor for every arbitrary sports movement.

Cached video embeddings

A new classifier can reuse existing embeddings without reopening videos. Changing or fine-tuning the image encoder, changing crops, or extracting missing detail generally needs the source frames again. New video also needs that encoder before the classifier can run.

3D skeletons and pretrained motion models

A method built for 3D joints needs compatible 3D data or an additional lifting step that estimates depth from 2D. Pretrained motion models add a representation and its training assumptions. These can help, but they are extra dependencies to evaluate when only 2D coordinates are available.

Separate detectors and video teachers

A detector-plus-classifier system adds boundary supervision and a detection policy. A video teacher adds feature extraction and a training objective; a posture student can still run without pixels afterward. Their extra processing is useful only if it improves recognition.

Other skeleton-based methods can also reuse stored coordinates. This model needs few steps to turn labels into a small local recognizer. The 2D posture is itself incomplete: depth, contact, objects, occluded joints and fine hand movements may require richer inputs. Retaining coordinates keeps new timing and geometry features available without committing the dataset to one frozen embedding.

MIL channels0 models · no posture
Figure 1Try the recognition example. Trained action models run in your browser as the video plays; posture extraction happened beforehand. The traces show the local recognizers’ scores and timing.

Part 2: What the compact posture model borrows

The benchmark classifier uses short filters over changing joint positions, then combines their responses to name an action. The complete classifier has roughly 26,000 to 33,000 learned parameters. The deployed recognizers and the event-timing experiments use related small models, with different outputs: a whole-window score, or an action’s time and duration.

This size makes local CPU inference practical, including browsers and phones. The benchmark weight files cover the action classifier; extracting postures is a separate step. The experiments below test which ideas from larger models help with limited inputs, compute and labeled data.

What helped, and what did not

Better descriptions of the body helped name a supplied clip. In the earlier NTU-120 experiment, adding bone coordinates and their motion raised accuracy from 65.69% to 67.19%. The next bundle of longer input, wider layers and filters spanning more time added only 0.47 points. Experiments and source references.

Finding the action needed a different objective. A fight analyzer must separate strikes from feints and footwork, then decide when to alert. Weighting the most informative frames improved accuracy averaged across classes from 79.3% to 84.9% while worsening detection of the action’s time span. Useful features can carry over from leading clip classifiers; their accuracy does not measure this added task.

Agreement was the clearest gain in the event study. Asking four models to agree on an action and its timing raised the share of labeled actions found from 42.3% to 54.3% at an estimated five false alarms per minute. Telling one model which joints were unreliable helped it, but did not improve the four-model combination. Extra training tasks for motion order and tempo, and a camera-view input, did not help. Pretraining on boxing clips had mixed results; a larger posture extractor alone did not help either.

More data could help if it adds missing evidence. Longer training did not improve the early model’s validation result. Pretraining on 1.74 million motion frames improved reconstruction without a consistent detection gain. More precisely timed strikes, rare actions and difficult background examples could help. Repeated ambiguous labels cannot supply precise boundaries. No controlled data-scaling study quantified the gain.

Finding an event here means finding the correct class inside a labeled window. Four folds held out tracks; false alarms were measured on labeled background windows. Five per minute still means one every 12 seconds. The event study used later frames and did not test the complete live system. These experiments establish neither a state-of-the-art ranking nor a universal streaming limit. Full protocols and the effects of label changes and output filtering are below.

What each architectural change achieved

August 28 to 30: pre-cut NTU-120 cross-subject, HRNet 2D poses, seed 42. The graph-model rows use a different baseline from the tiny-TCN rows. They are not streaming measurements. All rows and original run references are in the study record.

Borrowed model features · pre-cut top-1 accuracy
ChangeBefore → afterMechanism and interpretation
Bone coordinates and bone motion65.69 → 67.19%Adds relative limb geometry: +1.50 points.
Dilation, longer input and more width67.19 → 67.66%A bundled change: only +0.47 points; no isolated estimate for each part.
Ensemble + longer input and dilation65.69 → 74.38%104K total parameters. Joint/bone/motion members scored 66.2 / 66.9 / 64.4%; averaging reached 74.38%. The full baseline difference includes the other changes.
Adaptive graph + temporal branches + resized stages80.40 → 81.53%A bundled improvement on the larger single-actor graph baseline: +1.13 points, 1.19M → 0.96M parameters.
Channel-wise graph refinement81.53 → 81.44%Similar accuracy at 0.75M parameters; no accuracy gain in this seed.
Frequency filter + time-reversal feature81.44 → 79.46%The combined DCT/reversal experiment lost 1.98 points. Neither feature was isolated.

September 3 to 4: class-correct labeled-window recall at nominal 5 / 10 false alarms per minute. Values interpolate a threshold sweep; the historical evaluator clamps to endpoints outside its sampled range. These are operating-curve estimates, not independently tested fixed thresholds. No confidence intervals were established.

Changes to the continuous-event spotter · recall @5 / @10
ChangeBefore → afterWhat happened
Four-seed agreement ensemble42.3 / 52.5 → 54.3 / 60.8%Sapiens2 poses. Requiring agreement on a class and nearby peak reduced unstable detections; four inference passes.
Visibility from real joint confidence42.3 / 52.5 → 47.3 / 57.4%Signals unreliable joints. Adding it to the ensemble gave 50.2 / 57.3%, below the ensemble alone.
Order and tempo auxiliary losses45.5 / 53.4 → 40.6 / 51.9%DETRPose study. Learning shuffled-order and tempo tasks did not improve event recall.
BoxingVI pretraining45.5 / 53.4 → 43.5 / 56.4%2,656 clips: worse at the stricter false-alarm rate, better at the looser rate. Mixed transfer.
Suppress kick peaks inside punch spans50.7 / 58.7 → 51.9 / 61.1%DETRPose ensemble. A small gain from removing competing events; excluding nearby jab/cross peaks hurt.
Camera-view input42.3 / 52.5 → 41.4 / 52.1%No gain. Training separate view-specific models also lost recall as each received fewer examples.

The event head replaced a clip-level pooled score with per-frame class peaks and a predicted width. Background became low scores in every head, rather than a competing class. Dilated filters expanded the convolutional span to 65 frames. This aligned the output with the task, but no clean ablation assigns a gain to those changes separately. Speed-based pseudo-labels also confused extension with retraction; changing the event target was necessary before interpreting later results.

A separate 16-move study found that a shared graph/attention/transformer teacher helped classes with fewer than 20 positives: 71.3% versus 62.9% balanced clip accuracy for production MIL. Output distillation did not preserve that benefit. Sharing supervision helped in that setting; it did not establish better stream segmentation.

Cleaning labels produced successive single-model recalls of 42.3%, 48.2% and 49.3% at nominal five false alarms per minute, but changed the evaluation population. That is not a controlled learning curve. Likewise, storing 700 events after fixing the confidence gate, versus 12 before, measures recovered output rather than accuracy.

In the early speed-anchor recipe, extending training from 200 to 900 epochs raised training anchor heat from 0.300 to 0.540, while validation changed from 0.205 to 0.209 and recall fell from 0.178 to 0.156. Later target and threshold corrections superseded that setup. This shows that longer training did not solve that experiment; it does not prove that additional labeled data would fail.

Left-padded convolutions did not make this event pipeline strictly causal: GroupNorm used every frame in the supplied window, Gaussian smoothing read both directions, and peak selection used later samples. The 65-frame figure describes the convolutional span, not the complete system’s information horizon or alert delay. Causality and evaluation audit.

Why 95%+ clip recipes need changes for streaming segmentation

Spatial features can transfer; future context needs a new implementation

Bone vectors, per-frame pose heatmaps and spatial graph operations can run on arriving frames. Multi-scale temporal filters can be made causal with past-only padding; attention can use causal masks and cached history. The published PoseC3D recipe uses complete-clip temporal sampling, symmetric 3D convolutions and pooling. Replacing these changes the evidence available to its classifier and requires retraining and evaluation. PoseC3D recipe; temporal convolutions.

A learned graph can hide whole-clip aggregation

ProtoGCN averages over time when constructing its relation features. A streaming version would need a trailing window or prefix summary. Its prototype bank and contrastive training are reusable ideas; their original six-stream Gym99 result of 95.9% is not a streaming score. A causal ProtoGCN was not implemented or tested in this study. ProtoGCN §3.3; Table 2.

Its own NTU-120 ablation improves 87.8% → 88.1% → 88.5% → 89.0% as topology enhancement, prototype reconstruction and contrastive learning are added. Those are real classification gains; this project has no corresponding streaming ablation. ProtoGCN Table 3.

Clip pooling and repeated test crops do not supply event boundaries

Pooling answers whether a motion appears somewhere in the supplied window. Sliding that classifier over a recording does not add a training signal for starts, ends, duplicate alerts or background. Ten crops spread over a completed action cannot be reproduced before that action finishes; trailing-window crops are a different protocol. The local attention-pooling experiment improved balanced bag accuracy from 79.3% to 84.9% while worsening span detection. Matched pooling experiment; ST-GCN++ test sampling.

Ensembles and richer inputs cost computation, not necessarily lookahead

Joint/bone/motion ensembles can all process the same past frames. RGB crops, hand detail and another fighter’s pose could add evidence absent from one body skeleton. They increase processing and labeling requirements; none is inherently forbidden by streaming. The tested gains do not cover every such input or every recent model.

Precise event-spotting research is closer to strike timing

E2E-Spot targets events at one/few-frame precision, but its official GRU reads both temporal directions. T-DEED combines temporal scales, restores frame resolution and predicts event displacement; its code includes centered convolutions and a whole-window mean. These ideas inform an event detector, but their original scores do not describe zero-lookahead MMA recognition. E2E-Spot paper; bidirectional head; T-DEED method; temporal code.

Better supervision adds information that isolated clips lack

SF-Net trains localization from one annotated timestamp per action, an actionness head, expanded pseudo-labels and background mining. It provides a concrete alternative to guessing every event center from motion speed. Its published protocol is temporal localization; it does not establish strict streaming punch performance. SF-Net §§3 to 4.

Actual online literature measures the tradeoff. CMeRT (CVPR 2025) reports THUMOS’14 frame-mAP of 73.2 at zero added lookahead, 74.5 at 250 ms, and 76.6 at one second. Its stronger visual-feature experiment also improves performance. This supports a cost to deciding early, while showing that architecture and input quality can still help. These are RGB-action results, not punch recall or clip accuracy. Latency, Table 11; features, Table 10.

The practical limit is information available at the decision time: a feint and a strike can initially look alike, and a projected wrist trajectory does not establish contact with an opponent. That is a mechanism-based explanation, not a measured universal ceiling. Event-level recall, false alerts and emission delay must be measured together on complete held-out recordings.

Part 3: Where the small model falls short

The classifiers in this suite contain 25,680 to 32,664 learned parameters. That small footprint suits inexpensive recognition on pose data, but it comes with a substantial accuracy gap. These sizes cover the action classifier; extracting the joint positions is a separate step.

The gap on actions that are already cut

The tested ST-GCN++ and PoseC3D configurations label actions already cut out of a video. A live app must also find each action’s start and end, reject background movement and name the action using only frames that have arrived. Video Pose Distillation (VPD) adds a continuous detector, but it makes an offline action/background decision using future frames. VPD’s frame descriptor can use the current and previous frames; the future-reading restriction belongs to that detector and the released offline preprocessing. The skating clip results below are complete; detection and retrieval comparisons are still in progress.

Graph relationships, joint and bone motion features, and pretraining can carry into a model that reads only past and present frames. Full-clip future context cannot. The added timing task motivates the compact streaming workflow. Matched live evaluations can establish which changes improve it. Published 95%+ clip scores measure a different task.

Live recognition from a few examples has useful precedents. TOAD labels arriving video frames; Sabater and colleagues match a few 3D posture examples against a motion stream. UMEG-Net finds named sports events and their timestamps using later-frame context. These offer mechanisms to test within the browser model’s input and compute budget, rather than a direct ranking against its results.

Related work: few examples, live decisions and precise event timing

TOAD: online recognition with pretrained video and text features

TOAD reports 70.95% at four examples per class and 72.25% at eight on THUMOS14. The metric is per-frame mean average precision: it evaluates frame labels, not complete-event boundary accuracy. These runs use video pixels, CLIP features and Kinetics-400 pretraining. Its separate future-prediction objective uses future labels during training, without reading future frames at inference; that objective is omitted from these four/eight-shot results. Pretrained features and text-defined classes are the transferable ideas. Setup and Table III.

Sabater et al.: matching a small library of motion examples

A temporal convolutional network describes the preceding 32 frames of 3D skeletons, then compares that motion with one to three reference examples. It is pretrained on NTU-120. The therapy evaluation asks whether a patient imitates the therapist’s example in each game, with a detection tolerance tied to that 32-frame memory. The useful pattern is a learned motion descriptor plus a small reference library; reproducing its result here would require accounting for its 3D inputs and prior training. Method and therapy evaluation.

UMEG-Net: named events with precise timestamps

UMEG-Net learns from 15 to 100 annotated clips, each containing multiple events. Its graph combines people, ball and court keypoints where available; it also reports pose-only variants. Its temporal shift reads both earlier and later features (equation 3). Graph relationships and event-timing supervision are useful candidates for a live model, while that future-reading operation needs adaptation. The paper measures event type and timestamp together; its label budget, inputs and timing protocol differ from these experiments. Temporal shift, Table 1 and few-shot setting.

On the same 16,487 NTU-60 test clips, the saved LegibleTCN model scored 76.12%, ST-GCN++ 89.29% and PoseC3D 93.73%. All three received pre-cut actions, so this gap is not caused by segmentation. Model capacity, posture representation, the number of actors retained, training and repeated test views also differ. These runs do not isolate how many points each choice contributes. Measured runs and paired examples. The broader table below averages three compact-model seeds and cites published reference scores.

On the 99-action gymnastics benchmark, the small model scores 68.55% when every action class counts equally. ProtoGCN reports 95.9%, a 27.35-point gap. Its result combines six separately trained input streams, with a larger training and inference budget. ProtoGCN result; Six-stream protocol.

Six similar jumps, a few labeled examples

FSJump6 separates axel, lutz, flip, loop, salchow and toe loop within figure skating. Each trial provides four or eight examples per class (24 or 48 labeled actions total). These measured accuracies average five supplied support sets, each tested on all 520 held-out action cuts. They are separate from TOAD’s 20 broader sports-action categories and per-frame metric, and do not test live segmentation.

Measured FSJump6 clip accuracy · five support sets · 520 test actions
Method / raw results4 examples / class8 examples / class
LegibleTCN · 80 epochs17.15%19.69%
LegibleTCN · 2,000 epochs20.92%26.92%
Normalized 2D joints + DTW50.77%65.31%
VIPE* + DTW64.35%75.65%
VI-VPD + DTW65.92%79.19%

Longer training helped: a separate 2,000-epoch LegibleTCN run reached 20.92% / 26.92%, up from 17.15% / 19.69% at 80 epochs. It classified every training example correctly in all ten trials, while transfer to unseen footage remained weak. This tests the 21,606-parameter benchmark classifier, distinct from the production live trajectory-template matcher. More varied labeled movement is still worth testing; repeating the same examples does not measure what additional data would change.

The pipelines also differ before classification. LegibleTCN receives 64 sampled frames of image-centered 2D joints and confidence. The reference preserves variable-length sequences and uses dynamic time warping (DTW) to align them, with original and mirrored views. Its 2D baseline centers on the hips and scales by the torso; VIPE* learns a descriptor from 2D poses. Full VI-VPD instead embeds athlete pixels and optical flow from the previous frame, after pose-and-motion distillation on additional unlabeled sports video. Those pretrained features were supplied for this evaluation. The comparison changes normalization, confidence handling, representation, alignment and prior training together, so it cannot assign the gap to one cause. VPD feature design; Run protocol, variation and receipts.

All matched clip results and evaluation notes

The table aligns the released pose source and evaluation metric. Each reference column uses one joint stream; separately trained bone and motion ensembles are excluded from these columns.

Published reference results · same released HRNet 2D pose source
Dataset / metricLegibleTCNPoseC3DST-GCN++
NTU-60 · cross-subjectTop-1 · HRNet 2D76.39%93.7%89.3%
NTU-60 · cross-viewTop-1 · HRNet 2D82.28%96.5%97.4%
NTU-120 · cross-subjectTop-1 · HRNet 2D64.58%85.9%84.4%
NTU-120 · cross-setupTop-1 · HRNet 2D66.04%89.7%88.1%
FineGym Gym99Mean-class accuracy68.55%93.8%N/A
HMDB51 · split 1Top-1 · PoseC3D pretrained on Kinetics33.07%69.4%N/A
UCF101 · split 1Top-1 · PoseC3D pretrained on Kinetics49.97%86.9%N/A

HMDB51/UCF101 rows here use split 1 to match the reference; the full suite averages all three splits. Gym99 uses 68.55% mean-class accuracy, rather than its 74.11% sample-weighted accuracy. A dash means no reference is cited here. Diving48 stays outside the ranked comparison until its corrected-label version is verified. These are published reference measurements, not a claim that this table contains every current record.

Recent published context: ProtoGCN (CVPR 2025) reports 95.9% Gym99 mean-class accuracy, versus 68.55% here, a gap of 27.35 points. It uses the PYSKL poses and a six-stream ensemble, so its training and inference budget differs from this single classifier. Result, Table 2; Pose source and metric, §4.1; Six-stream protocol, §4.3.

What each benchmark exposes, and where more data could help

These are explanations to test against the observed gaps. The run compared whole training recipes; it did not isolate a cause for each dataset. More labels can teach a model to use its inputs better, but cannot restore information removed before the classifier sees it.

HMDB51: appearance and pretraining matter to the comparison

The matching split scores 33.07% versus PoseC3D’s 69.4%. The compact classifier starts from scratch; that pose reference is pretrained on Kinetics. This comparison therefore includes a representation learned from additional labeled video. Pose-only input also omits objects and scene cues available to the much higher-scoring RGB methods. More varied training examples could help learn viewpoint and motion variation, but missing object evidence requires richer inputs. The same-pose gap cannot itself be blamed on RGB access. Split and pretraining.

UCF101: broader activity labels with a very small classifier

The matching split scores 49.97% versus 86.9%. Again, the reference adds Kinetics pretraining, deeper processing and repeated test samples. Whole-body motion can identify many sports, while an object or interaction can distinguish otherwise similar poses. This recipe did not test how much more labeled video closes the gap. Repeating the existing training for longer would not be that experiment. Matching reference protocol.

NTU-60: people, viewpoints and interactions

The three-seed suite averages 76.39% across people and 82.28% across views, compared with published PoseC3D scores of 93.7% and 96.5%. Keeping only the first person discards another participant’s motion. Shallow joint mixing also gives the model fewer stages to build relationships between limbs. These are plausible limitations, not measured shares of the gap. More varied people and views could help; recovering the second actor requires changing the input. Actor selection.

NTU-120: more actions to separate with the same feature budget

Accuracy is 64.58% across people and 66.04% across recording setups, against 85.9% and 89.7%. The tiny model keeps its feature extractor while expanding the output vocabulary. Repeated graph mixing and processing at several time scales are reasonable improvements to test; the earlier experiments above show both gains and small or negative returns. More representative labels may help, but these two datasets are not a controlled experiment in data quantity.Classifier structure.

Gym99: subtle differences within gymnastics

Equal-weight accuracy across action classes is 68.55%, versus PoseC3D’s 93.8% and ProtoGCN’s 95.9%. Whole-action resampling and a small temporal stack can make detailed phases harder to distinguish. Richer joint relationships, longer temporal processing and several feature streams are candidate advantages. Labels covering those fine distinctions could help; their benefit has not been isolated here. Published comparison.

Diving48: a weak result with an unresolved label version

This run scored 9.05%. Fast rotations, depth and fine body configuration are plausible difficulties for sparse 2D joints and uniform sampling. But the released labels have not been matched to the corrected V2 benchmark, so a precise modern leaderboard gap would be misleading. Establishing the label and split correspondence comes before attributing the failure to architecture or adding training data. Run and label limitation.

Six architecture differences, with exact implementation references

The choices below are verified in the executed code and pinned reference implementations. Their effects on information flow are explicit; this run did not isolate how many accuracy points each choice costs.Explanations of the score gap are therefore hypotheses, not measured ablations.

Coordinates, confidence, and one actor

LegibleTCN: 17 joints × five channels (x, y, confidence, Δx, Δy); image-centered coordinates zeroed when confidence ≤0.01, 64 uniformly sampled frames, first actor only. Velocity is a difference between sampled positions, not a timestamp-normalized physical speed. Five-channel selection, L118 to 120; Normalization and features, L118 to 150; Actor selection, L315 to 318.

PoseC3D: confidence-weighted Gaussian heatmaps retain a spatial grid and combine visible people by pixelwise maximum. Heatmaps, L73 to 107. ST-GCN++: its configuration keeps up to two people. Person input, L13 to 20.

Potential consequence: the one-actor score omits the other person’s motion in interactions. Heatmaps also give the convolution an explicit image-grid neighborhood.

Independent joint filters, then one mixing layer

LegibleTCN: eight length-9 filters per joint produce 136 channels. A single 1×1 layer mixes these into 96 hidden channels. This mixing can combine any joints, but there is no skeleton adjacency matrix. Grouped filters and mixing, L91 to 106; Active widths, L449 to 451.

ST-GCN++: repeated graph layers mix joint features using trainable adjacency initialized from skeleton connections. Adjacency and multiplication, L37 to 81; Graph partitions, L161 to 166.

Potential consequence: repeated graph mixing can build joint relationships at several depths. The single mix keeps this classifier small for local inference. For NTU-60, that mixing layer contains 136×96 + 96 = 13,152 of the measured 26,844 parameters.

A small temporal filter versus deeper temporal processing

LegibleTCN: after mixing, one length-9 depthwise convolution processes each hidden channel separately. No dilated stack was enabled. The two length-9 convolutions span 17 sampled positions before pooling; that is the convolutional span, not a complete information limit. Depthwise layer, L116 to 119; Forward path, L121 to 139; Executed settings.

PoseC3D: a 3D ResNet uses three stages of 4, 6 and 3 residual blocks and 512 head features. Network configuration, L1 to 20; Temporal/spatial kernels, L135 to 171. ST-GCN++: temporal branches use dilations 1 to 4, pooling and a pointwise path across ten graph/temporal stages. Temporal branches, L38 to 105; Stages, L59 to 110.

Potential consequence: the references can compose motion features across more layers and temporal scales. This run does not quantify the gain from adding any one layer.

Whole-clip normalization and uniform pooling

LegibleTCN: ordinary GroupNorm pools statistics across channels and time in this noncausal configuration. The final head reads the temporal mean through a linear classifier; attention is disabled. Normalization, L100 to 106; Pooling and head, L141 to 148; Model construction, L311 to 313.

PoseC3D and ST-GCN++: these reference heads also use average pooling, over their learned spatial/temporal features and, for the graph model, people. Reference pooling, L74 to 94.

Interpretation: average pooling alone cannot explain the gap because it is shared. Temporal convolutions encode local order before pooling. Whole-clip GroupNorm also means the measured model is unsuitable for a zero-lookahead claim.

Training and test-time processing also differ

LegibleTCN: 80 epochs, class-balanced replacement sampling, AdamW 0.002, cosine decay, label smoothing 0.1, no augmentation; one 64-frame view per test clip and no seed ensemble. Training, L551 to 605; Run defaults, L446 to 473.

PoseC3D NTU-60: 48-frame samples, cropping/flips, 24 epochs of a 10× repeated dataset (240 dataset passes), and ten clips plus mirrored versions at test time. Exact pipeline and schedule, L26 to 74. ST-GCN++ NTU-60: 100-frame samples, ten test clips and 16×5 = 80 dataset passes with Nesterov SGD. Sampling and schedule, L31 to 56.

Interpretation: these differences confound an architecture-only comparison. ST-GCN++ does not have more dataset passes in this cited configuration. HMDB51/UCF101’s PoseC3D references additionally use Kinetics pretraining. Pretraining and protocol notes.

ProtoGCN adds learned graph prototypes and a contrastive objective

ProtoGCN reconstructs graph features through attention over a learned memory of joint-relationship prototypes. Its graph layers combine shared topology, joint correlations and pairwise differences. Prototype reconstruction, §3.2, equations 3 to 4; Motion topology, §3.3, equations 5 to 7.

A class-specific contrastive loss supplements classification loss, encouraging representations to approach their class memory and separate from other classes. The paper uses 150 training epochs; its quoted result combines six streams. Training objective, §3.4, equations 8 to 10; Training recipe, §4.2.

Potential consequence: learned relationship patterns, additional class separation and ensemble evidence provide capabilities absent from this LegibleTCN recipe. The 27.35-point gap is the total observed difference between recipes; it is not the measured contribution of prototype memory alone.

How other boxing and MMA analyzers find actions

What other fight analyzers do

BoxMind / BoxingWeb appeared in the earlier research. Its paper uses a TCN with probability and boundary-offset heads on 2D/3D poses, followed by punch-attribute classification. It reports 0.783 detection F1 at temporal IoU 0.5 and explicitly describes an offline system. It does find its own cuts; that result is not comparable to this study’s window-hit recall. Detection, Appendix B.2; offline scope, §4.4.

Compare the open-source implementations and related attempts

Source review, September 15, 2026. These are plausible matches and relevant alternatives; the exact remembered repository is unconfirmed. This is a scoped survey, not every combat-analysis project. Source review does not constitute a reproduction of their performance.

How each system finds and labels motion
Project / sourceApproachCompared with this spotter
BoxMind / BoxingWebThe paper detects boundaries; the public loader classifies supplied punch segments with pose and RGB.Closest published event formulation. Released classification code is narrower than the complete paper pipeline; its scores are not a live MMA benchmark.
ACM40960/BoxingYOLO pose → 25-frame trailing buffer → punch/block LSTMs. Motion gate, score threshold and 60-frame recovery control outputs.A rolling-window classifier with heuristic event handling. The punch head has no background class or learned boundary head; recovery can suppress close combinations.
imsoo/fight_detectionOpenPose / SORT → 32-frame angle-and-motion features → LSTMs. Standing, walking, punching and kicking; geometric collision checks.Coarse activity recognition trained from cut videos. Its collision rules use another person, but it does not distinguish fine-grained strike types and boundaries.
boxe.mlWrist-speed thresholds cut motion first; an LSTM labels straight/hook/uppercut, then geometry assigns lead/rear.An explicit alternative to the learned event head. Completed segments and centered smoothing make the inspected path offline; reported 83% is three-class clip accuracy.
RTM-FASS15-frame buffer → bidirectional LSTM and attention → eight Muay Thai strikes plus hit/miss; threshold and cooldown.Can classify a trailing window, despite bidirectionality inside it. No learned boundary head or explicit strike-background class; weights were not available from the documented link.
UFC Fight AnalyzerPython YOLO tracking, keypoints and skeleton display.A plausible name match, but the inspected implementation stops at pose tracking: no move classifier, action cutting or recognition benchmark.
TAPEAuthor describes proposal detection → classification → human verification, with fight-level evaluation and timestamped evidence.A relevant review-assisted product design. The writeup did not provide a public source repository or an independently reproducible streaming score.

Newer boxing literature: BoxComm (April 2026 preprint) describes causal TCN punch detection followed by VideoMAEv2/TCN classification. Its reported experiments evaluate commentary, not a matched punch curve comparing recall, false alarms and latency. It shows that modern video features can be combined with a streaming detector; commentary quality does not certify detector accuracy. BoxComm §5.1.

Other possible name matches: NeuroCombat’s linked repository was unavailable when checked, so its architecture could not be audited. MMAction2 is a general recognition/localization toolbox, rather than a ready-trained MMA analyzer. Full source review and links.

The local spotter learns event peaks and widths from labeled examples; several alternatives place motion thresholds and cooldowns around a clip classifier. Neither approach removes the need to test feints, overlapping actions, tracking errors and unseen fighters. The useful comparison is complete-event recall at the same false-alert and delay budget, not the highest accuracy printed by each project.

Nine examples: where each model misses

Three examples per outcome, comparing LegibleTCN with PoseC3D on pre-cut NTU RGB+D 60 X-Sub (2D HRNet) actions. The full test scored 76.12% and 93.73%, respectively.

Nine clips · three-second loops

Paused. Play or scrub any example.

PoseC3D gets it right; the small model misses

Human label
drop
Frame 1 / 71
LegibleTCN
pushing other personWrong
PoseC3D joint stream
dropCorrect
Human label
playing with phone/tablet
Frame 1 / 103
LegibleTCN
readingWrong
PoseC3D joint stream
playing with phone/tabletCorrect
Human label
pointing to something with finger
Frame 1 / 75
LegibleTCN
taking a selfieWrong
PoseC3D joint stream
pointing to something with fingerCorrect

Both models miss

Human label
put on a hat/cap
Frame 1 / 120
LegibleTCN
brushing hairWrong
PoseC3D joint stream
brushing hairWrong
Human label
touch chest (stomachache/heart pain)
Frame 1 / 106
LegibleTCN
check time (from watch)Wrong
PoseC3D joint stream
playing with phone/tabletWrong
Human label
throw
Frame 1 / 75
LegibleTCN
tear up paperWrong
PoseC3D joint stream
cross hands in front (say stop)Wrong

The small model gets it right; PoseC3D misses

Human label
writing
Frame 1 / 134
LegibleTCN
writingCorrect
PoseC3D joint stream
playing with phone/tabletWrong
Human label
pointing to something with finger
Frame 1 / 66
LegibleTCN
pointing to something with fingerCorrect
PoseC3D joint stream
taking a selfieWrong
Human label
check time (from watch)
Frame 1 / 81
LegibleTCN
check time (from watch)Correct
PoseC3D joint stream
pointing to something with fingerWrong
How to read these examples

A fixed crop fits the full movement in each clip; the original joint coordinates stay unchanged. Fading wrist and ankle trails show the preceding sampled frames. The first actor uses solid lines; a second actor uses dashed lines. The small model receives the first actor. Ten 48-frame clips plus flipped heatmaps; twenty views averaged. Batch four with cuDNN autotuning and unchanged TF32 settings. Prefix top-1 labels agree; probability differences accepted separately within 0.001 after the original 0.0001 check failed. Joints with confidence below 0.2 are omitted from the drawing.

Original playback speed is unavailable in this pose archive. Each preview maps the complete clip to three seconds; frame numbers refer to the source sequence. These animations do not measure recognition speed or alert delay.

The nine examples come from 16,487 test clips, selected by sample ID with distinct action labels. The compact model uses its offline, whole-clip checkpoint from seed 42; the benchmark summary averages three seeds.

xsub_val; pre-cut action classification · Released HRNet 2D COCO-17 poses from the same official test clips · Small-model seed 42. PoseC3D joint stream reference. Data version: 1. Full paired results and selection details.

Full six-dataset run: accuracy, model sizes and training recipe
Pre-cut clip recognition · 2026-09-15 · 3 seeds per split
Dataset / protocolTop-1Mean-classParametersCheckpoint
HMDB513 official splits34.19%± 0.38 across seeds34.19%25,971106.5 KiB101.4 KiB raw weights
UCF1013 official splits50.60%± 0.09 across seeds50.29%30,821125.5 KiB120.4 KiB raw weights
Diving48PYSKL train / test · V2 labels not yet cross-checked9.05%± 0.66 across seeds6.75%25,680105.5 KiB100.3 KiB raw weights
FineGym Gym99Train / validation74.11%± 0.14 across seeds68.55%30,627124.7 KiB119.6 KiB raw weights
NTU RGB+D 60Cross-subject · 2D HRNet76.39%± 0.19 across seeds76.38%26,844109.9 KiB104.9 KiB raw weights
NTU RGB+D 60Cross-view · 2D HRNet82.28%± 0.10 across seeds82.28%26,844110.0 KiB104.9 KiB raw weights
NTU RGB+D 120Cross-subject · 2D HRNet64.58%± 0.38 across seeds66.44%32,664132.7 KiB127.6 KiB raw weights
NTU RGB+D 120Cross-setup · 2D HRNet66.04%± 0.51 across seeds66.00%32,664132.8 KiB127.6 KiB raw weights

A fixed 80-epoch recipe, using released HRNet 2D poses, the first person in each clip, and 64 sampled frames. These classifiers receive the action boundaries and can use future frames within the clip. They do not measure continuous detection or live recognition.

HMDB51 and UCF101 average all three official splits. Variation is the standard deviation across three seed averages. Mean-class accuracy gives equal weight to each class present in the test split. Checkpoint sizes are measured for one trained classifier and exclude pose extraction. Diving48’s correspondence to corrected V2 labels has not been verified.

The continuous-video experiments measure a different task: finding actions and their timing among other movement. Their scores depend on missed events, false alerts and the allowed delay. They cannot be read as another clip-accuracy row, and the historical runs below do not establish a fully past-only video pipeline.

Earlier 3D and continuous-stream experiments
Recorded LegibleTCN results · July to August 2026
Dataset / metricOfflineCausal networkParametersFloat32 weights
NTU RGB+D 60Top-1 accuracyX-Sub · 3D Kinect · counts not auditedJuly 202671.8%70.4%37,900148 KiB
NTU RGB+D 120Top-1 accuracyX-Sub · 2D HRNet · 113,945 clipsAugust 2026 · 3 seeds65.94% ± 0.63Not measured32,700128 KiB
PKU-MMD phase 1Detection mAP @ IoU 0.5Cross-subject · 944 train / 132 test videosJuly 2026 · dilated model0.3770.213≈40,000≈156 KiB

Top-1 counts correctly labelled clips. Detection mAP also requires the predicted action to overlap the labelled interval. The PKU-MMD causal model recorded event F1 of 0.190 with a ±250 ms onset tolerance, precision of 0.151 and recall of 0.255. The 200 ms median latency covers onset-matched alerts within that tolerance. A causal network uses present and past frames; offline networks can also use future frames.

These are historical measurements from the recorded recipes. The NTU-60 record lists 40,091 clips without an audited total across train and test. NTU-120 reports mean ± standard deviation across seeds 42, 43 and 44; the other records use one seed. Weight sizes are estimates from parameter counts × 4 bytes, excluding the pose extractor, runtime, and file metadata.

The archived PKU preprocessing can use a later pose to fill missing starting frames. Its causal-network results therefore do not establish that the complete processing path works without future information.

The practical value is a model that can run beside playback in a browser or phone. Use its predictions where they are reliable; review the mistakes to create better training examples. That makes the small model useful now and creates data for testing heavier methods later.

Part 4: Turn predictions into the next dataset

Send video through posture extraction, run the small action model, and see the results on an ordinary machine. Use the clips directly, or review their labels and timing. Every corrected miss, false alarm and boundary is a candidate training example for the next model.

These existing golf examples show that workflow. They illustrate recognition and review; the benchmark results above are separate evaluations.

Watch the golf pipeline and boxing examples above

Where videos enter the pipeline

The input is a video of someone moving. This feed shows recent Reddit imports and their processing status, before their posture sequences become available to the recognizers.

intaker/GolfSwing → pose queue10 incoming
    Figure 3The latest 10 imported records. The feed animates through this loaded set.

    Label a few, then review what comes back

    Name an action and mark its start and end in a few clips. Search proposes similar movements. Accept the right examples, reject the wrong ones and adjust their boundaries. A few labels get retrieval started; the reviewed collection trains the classifier.

    Save ordinary background movement and confusing near-misses too. An unmarked interval may contain a real action, so it should not automatically become a negative example. Keep evaluation recordings separate from the growing training collection.

    the loopmark a few by hand

    Mark the start and end of each swing on the band below the video. Each saved example includes the movement label and its time range.

    Annotate a real video
    Figure 4Annotate, search and review. The annotation card and example wall use stored labels; the search ranking between them is illustrative.

    Open action discovery

    See why it fired, then fix the example

    Visible model responses help identify mistakes. Does the score rise during the swing, during the setup, or during the return to rest? The learned model shows which frames contributed. A trajectory template shows how closely a wrist follows the path in the labeled examples.

    Templates are useful for repeatable paths. The learned model can combine several joints over time. Both need checking against changes in speed and camera view; the video beside the scores makes those mistakes easier to notice.

    How the two recognizers in the example work

    Multiple-instance learning means training from labeled ranges without labeling every frame. The illustrated window model scores frames within 1.5 seconds, uses 32 hidden channels and advances by 500 milliseconds. Its window score does not by itself identify precise action boundaries.

    The trajectory template samples the wrist path at 64 points spaced by distance traveled. It compares progress through the movement without using playback speed as an input. It still depends on a recognizable path and reliable joint positions.

    how each one decidesplayback 1×

    MIL multiple-instance, per move

    one pose stream in

    input
    68 values per frame: 34 body-relative joint coordinates and 34 velocities
    reach
    9 taps × dilation 3 = 25 frames ≈ 830 ms
    pool
    early × late, so a prediction missing either end fails
    needs
    labeled time ranges; learns which frames contribute

    canonical line one trajectory per swing direction

    one wrist trail in

    input
    body-relative wrist positions without timing
    fit
    prefix 12→64 step 4, scored in σ of the corpus
    accepts
    deepest prefix under 1.0σ, fired past 0.85
    rate
    reported afterwards, never an input
    front swingcanonical line averaged over 41 labelled clips · 64 points · arc 11.2 torso units in the stored template
    Figure 5A window model and a trajectory template. The template uses persisted points from 41 labeled clips; the animated posture sequence is schematic.

    Keep the useful cuts; correct the rest

    Each detection becomes a label and a time range. Check whether it contains the action, whether the cut includes a neighboring movement, and whether another action was missed. Those corrections are more useful training material than another batch of unreviewed predictions.

    u/Morirz05#0 · 8.5s
    u/frankling54#0 · 2.9s
    u/Bowhunter1980#0 · 6.6s
    u/milesofmike#0 · 2.5s
    u/jbyler19#0 · 2.3s
    u/PlantainFair9392#0 · 1.9s
    u/MaybeMedium9876#0 · 2.7s
    u/AcademicAmount9283#0 · 8.6s
    Figure 6Detected actions, one per source video. Each card loops between its stored start and end times.
    71%Wesley Bryan iron back · SnooRadishes7417
    72%Grant Horvat Swing Back · SnooRadishes7417
    Figure 7Two performances aligned for comparison, with their stored match score. These are matched human movements, not predictions from a published reference model.

    Make reviewing the dataset part of using it

    Put a person’s detected actions together, beside their matched references. Repeated mistakes become easier to find, and the useful clips already have a place to go.

    Figure 8The two latest completed comparison walls approved by a reviewer.

    Reviewing the small model builds training data for larger models: varied people and viewpoints, accurately timed actions, and reviewed examples of when to avoid a detection.