# What transferred from action-recognition benchmarks to continuous video?

Research note · September 15, 2026 · Historical experiments: August 28–September 4, 2026

**Finding the action was a separate problem from naming a prepared clip.** Our small skeleton models learned useful motion signals, but a live-video product also has to reject background, distinguish feints from strikes, and decide when an event happened. Improvements on pre-cut benchmarks did not consistently improve that operating tradeoff. Ensembling helped; several more elaborate additions did not.

These results describe the limits we encountered with this implementation, representation and labeled corpus. They do not establish a universal ceiling for streaming recognition, or show that temporal localization explains every accuracy gap. The August NTU experiments and September 15 benchmark suite both classified pre-cut clips; their remaining gaps therefore cannot be attributed to finding the action in a stream.

## How to read the numbers

Three experiments answer different questions:

- **Pre-cut classification:** top-1 accuracy on NTU-120 X-Sub, using released HRNet 2D COCO-17 poses. The August feature ladder below used seed 42; the larger reference runs used 80 epochs. These are historical experiments, distinct from the fresh three-seed tiny-TCN suite.
- **Bag classification:** balanced accuracy for whether a labeled clip contains a move. The August 31 transfer study used three seeds across 16 move classes. This does not measure continuous-video event detection.
- **Continuous-video spotter development:** four folds grouped by entity track, evaluated on labeled action windows and labeled background windows with surrounding context. This was not an exhaustively annotated full-stream test, nor a held-out-fighter or held-out-video protocol.

In the spotter tables, **R@5 / R@10** means class-correct window-hit recall at the reported 5 / 10 false alarms per minute. A hit means a same-class event peak lies inside the labeled action window; it does not require an accurate start/end boundary. False alarms are counted only inside held-out historically labeled idle windows. The harness linearly interpolates a threshold sweep and clamps to its nearest endpoint outside the sampled range. These are descriptive curve summaries, not independently validated deployment thresholds or guarantees of full-video alert rates.

## The features we borrowed

### Pre-cut NTU-120: a useful architecture ladder

| Configuration | Top-1 | Parameters | Mechanism and measured interpretation |
|---|---:|---:|---|
| Tiny joint-input model, 64 frames | 65.69% | ~33k | Named joint filters followed by channel mixing. |
| Add bones and bone motion | 67.19% | ~33k | Parent-relative geometry supplied explicitly; +1.50 points. |
| Add dilation, 100 frames and width | 67.66% | ~60k | Longer temporal context and more channels together added only 0.47 points beyond bones. |
| Joint/bone/motion ensemble, 100 frames and dilation | 74.38% | 104k total | The full baseline difference bundles sampling/dilation changes. Within this configuration, individual streams scored 66.2%, 66.9%, 64.4%; averaging reached 74.38%. |
| Graph model plus reference recipe, two actors | 82.27% | 1.19M | Anatomical message passing plus a bundle of preprocessing, capacity, sampling and training changes. This is not an isolated graph-layer ablation. |
| Single actor, ten test clips | 80.40% | 1.19M | A separate complete-clip configuration; actor count and test averaging cannot be disentangled by subtracting these rows. |
| Adaptive graph, multi-scale temporal branches and resized stages | 81.53% | 0.96M | Learned connections, parallel temporal scales and stage widths 64/128/128/256→96/192/192/384: +1.13 points as a bundle. |
| Channel-wise topology refinement | 81.44% | 0.75M | Per-sample, per-channel joint connectivity produced similar accuracy at smaller size. |
| Add DCT and time-reversal difference | 79.46% | 0.80M | Spectral filtering plus forward-minus-reversed features lost 1.98 points. They were not ablated separately. |

The reference bundle used image-centered coordinates, low-confidence-joint masking, two actors, 100 frames, greater capacity and SGD/cosine training without the earlier augmentation. Several settings changed together; individual contributions were not measured. Later records corrected the matched published ST-GCN 2D joint reference to 80.1%, superseding an earlier 82.9% comparison.

**What transfers in principle:** anatomical graphs, bone/motion channels, parallel temporal scales and agreement between models. **What requires redesign for streaming:** complete-clip pooling, ten-clip averaging, temporal pooling inside dynamic topology, bidirectional processing, full-window frequency transforms and reversed-clip features. Reusing a block does not supply event-boundary supervision. Two-person geometry also answers a different question from a single-person detector.

### Move classification: larger models helped only in some regimes

An August 30 graph-MIL trunk swap lost on lead uppercut (.908→.779), rear uppercut (.934→.826) and lead hook (.921→.683), while winning downswing (.897→.999). This was a limited, incomplete paired bag experiment, not evidence against graph models generally.

The next model combined learned graph connections, joint attention, multiple temporal scales and a bidirectional transformer. Training it separately per move overfit. Sharing its trunk across 16 moves changed the outcome:

| Three-seed bag study | ≥20 positives | <20 positives | All moves |
|---|---:|---:|---:|
| Shared modern teacher | .815 | .713 | .764 |
| Production MIL | .850 | .629 | .739 |
| Production MIL, distilled | .848 | .621 | .734 |
| Legible-TCN | .740 | .622 | .681 |
| Legible-TCN, distilled | .767 | .621 | .694 |

Shared supervision helped scarce classes, but teaching the small student the teacher's bag scores and frame responses did not transfer that advantage. This tests a particular distillation method and student capacity. It does not prove that knowledge transfer is impossible. An initially dramatic one-seed improvement on a class with one held-out positive disappeared across three seeds.

### Event spotting: which changes moved the useful curve?

The spotter replaced bag-level decisions with per-class event heat and width heads, temporal residual blocks, peak selection and separation rules. Its roughly 11k-parameter trunk retained named joint filters. Background became “all event scores low.” That fixed the formulation; it did not automatically solve background discrimination.

| Addition | R@5 / R@10 | Comparison and interpretation |
|---|---:|---|
| Original-pose baseline | .455 / .534 | Reference for the following five rows. |
| Four-seed agreement ensemble | .507 / .587 | +5.2 / +5.3 points; requires multiple inference passes. |
| Ensemble plus kick/punch exclusion rule | .519 / .611 | A small additional gain from suppressing incompatible overlapping events. |
| Temporal order and tempo auxiliary losses | .406 / .519 | No gain under this recipe. |
| Estimated visibility channel | .388 / .503 | Heuristic reliability flags did not help. |
| BoxingVI pretraining, then fine-tuning | .435 / .564 | 2,656 pretraining clips; mixed −2.0 / +3.0 points, no consistent win. |
| Re-estimated-pose baseline | .423 / .525 | Better pose estimation alone did not improve the spotter. |
| Visibility flags derived from real joint confidence | .473 / .574 | +5.0 / +4.9 points within that pose corpus; confidence below .3 plus edge, duplicate-joint and teleport checks. |
| Four-seed ensemble on re-estimated poses | .543 / .608 | Most consistent architecture gain across the two pose corpora. |
| Ensemble plus kick/punch rule | .543 / .621 | Further improvement at the higher alert budget. |
| Ensemble plus confidence | .502 / .573 | Gains did not stack. |
| Camera-view input | .414 / .521 | No improvement over .423 / .525; with confidence, .403 / .503. |

Other decoder rules were mixed: relaxing cross detection after a jab produced .461/.543, while cross-class exclusion produced .429/.498. View-specific models also lost: side-only recall .31 versus pooled .50; front-only .32 versus .38 in that separate bucket experiment. Smaller training subsets and redundancy with existing skeleton geometry are plausible explanations, not isolated causal findings.

## Why more data sometimes meant no improvement

**More training:** under the original, subsequently superseded speed-anchor targets, 200→900 epochs raised training anchor heat .300→.540, while validation heat stayed .205→.209 and recall fell .178→.156. This rejected “just train longer” for that setup. It did not establish a data-scaling ceiling.

**More unlabeled motion:** masked-motion pretraining used 1.74 million frames, 6,928 tracks and 24,225 windows. Reconstruction loss improved .052→.0046. In a paired 441-clip downstream comparison, anchor/background AUC changed .907→.912 and anchor/flank AUC .865→.858. Learning reconstruction did not reliably improve discrimination. This leaves other objectives, larger models and better labeled data untested.

**More negatives:** 2,042 explicit negative labels lowered scores and alert rates, but class-correct recall remained approximately .54 at comparable 8–10 background false alarms/minute. Earlier automatic gap-mining worsened excess detections for jab 43→61 and cross 30→41: unlabeled gaps in sparsely annotated fights can contain genuine strikes. More incorrectly assumed negatives can teach the wrong boundary.

**Better labels:** pruning and cleaning produced successive single-model summaries .423/.525→.482/.574→.493/.589. The evaluated label population changed, so this is not a fixed-test-set ablation. Nevertheless, it shows why collecting consistent events, representative background and rare-class examples is a different intervention from merely collecting frames. No controlled labeled-data scaling curve was run.

## Two corrections that matter

An early diagnostic of the deployed model found 33/38 combination clips with flat scores and only 4/38 with ordered, non-flat peaks. A subsequent September 1 retrain of the same baseline configuration on the then-current labels reported 5% flat and 76% ordered/non-flat, versus the deployed model's 87% and 11%. The training population was not controlled. Neither diagnostic used independent event-timing ground truth; the early failure is not an architectural ceiling. A separate matched pooling experiment raised balanced bag accuracy .793→.849 while worsening span detection, demonstrating why the training objective and evaluation task must agree.

Speed peaks were poor event anchors: among 221 punch clips, only 17% coincided with maximum extension; 42% preceded it and 41% followed it. Replacing them exposed useful event/background separation. Early near-zero AP against the same pseudo-anchor rule used for training was subsequently identified as measuring proxy agreement, not independently annotated strike accuracy. It cannot support a claim that localization was impossible. Background false alarms also required correction to exclude unlabeled padding.

**“Causal convolutions” did not make the historical system strictly causal.** Retrospective code inspection finds GroupNorm statistics over every frame in the supplied bag, symmetric smoothing and future-neighbor peak selection. At σ=67ms, smoothing reads ahead up to 201ms; peak selection examines another 100ms neighborhood. Whole-window normalization adds a separate future dependency. The recorded results therefore describe windowed spotting, not validated zero-lookahead streaming or measured live latency.

## Evidence and implementation references

Historical records reviewed: **“move recognition”** (August 28 UTC); **“music”** (August 30–31 UTC, benchmark and transfer work); **“bleeding edge action rec”** (September 1–3 UTC); **“action rec research”** (September 3–4 UTC). Only experiment facts are reproduced here.

Raw benchmark outputs were verified at August 30 **01:36:34**, **02:53:10**, **04:12:17**, **08:07:56 UTC**, corresponding to archived console output from `reference.log`, `ref1.log`, `ref2.log` and `novel.log`. The three-seed transfer table was recorded August 31 **03:41:37 UTC**. Spotter tables were recorded September 3 **13:20:39** and **19:59:43 UTC**; label-cleaning results September 4 **14:02:56 UTC**. Historical spotter archives: `results/out`, `results2/out2`, `results3`, `results4/out4` under the experiment archive's `spot-standby` directory. These are historical outputs, not newly rerun trials.

Current implementation references are retrospective mechanism checks, not frozen historical execution snapshots:

- `scripts/lib/skeletonBenchCommon.py:217`: adaptive adjacency; `:244`: channel-wise topology; `:275`: DCT; `:305`: temporal branches; `:396`: pooling; `:406`: reversed-clip branch.
- `scripts/lib/pysklBenchmark.py:229`: reference recipe; `:268`: multi-clip averaging.
- `app/lib/moveClassifier/spotTcn.ts:136`: whole-bag normalization; `:400`: smoothing; `:479`: peak selection; `:560`: event agreement.
- `scripts/train-move-classifiers.ts:2796`: track folds; `:2892`: background-window counting; `:2917`: class-correct hits; `:3070`: interpolation and endpoint handling.

## What the published literature actually transfers

A 95%+ result on a selected action clip does not measure the ability to find unknown events in a recording. This does **not** mean the underlying models are unusable for streaming. The restrictions concern information access, training targets and the evaluation protocol.

| Published method | Reusable mechanism | What changes before a streaming claim |
|---|---|---|
| PoseC3D | Per-frame pose heatmaps and learned spatial/temporal filters | Its complete-clip recipe samples 48 frames, averages ten test clips and uses mirrored heatmaps. Trailing-window sampling, causal temporal filters and event heads would be a new evaluated configuration. [Pinned recipe, L26–74](https://github.com/kennymckormick/pyskl/blob/f2bf3a6b08e2e8dec744692d64efdb187fd6719a/configs/posec3d/slowonly_r50_ntu60_xsub/joint.py#L26-L74) |
| ST-GCN++ | Anatomical/adaptive spatial connections and multiple temporal scales | Centered convolutions and pooling require past-only variants or declared delay. Graph reasoning itself is compatible with streaming. [Temporal implementation, L8–105](https://github.com/kennymckormick/pyskl/blob/f2bf3a6b08e2e8dec744692d64efdb187fd6719a/pyskl/models/gcns/utils/tcn.py#L8-L105) |
| ProtoGCN | Motion relationships, prototype reconstruction and contrastive learning | Its relation construction averages over the temporal input. A causal version needs prefix/trailing-window aggregation. Its 95.9% Gym99 ensemble result does not establish segmentation accuracy. [§3.3](https://arxiv.org/html/2411.18941v2#S3.SS3), [Table 2](https://arxiv.org/html/2411.18941v2#S4.T2) |
| E2E-Spot | Direct supervision for precise event timestamps | The official GRU processes both temporal directions. It can process a completed trailing window, but predictions for earlier frames then have lookahead. [Paper](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136950033.pdf), [pinned head, L22–34](https://github.com/jhong93/spot/blob/edec4201471beed631bed374bd0b95fcdc8a2f4f/model/modules.py#L22-L34) |
| T-DEED | Temporal encoder–decoder, restored frame resolution and event displacement | Official code contains centered convolutions and a whole-window temporal mean. Its timing ideas are relevant to brief strikes; transferring performance to MMA remains untested. [Method](https://arxiv.org/html/2404.05392v1#S3), [pinned temporal code](https://github.com/arturxe2/T-DEED/blob/d953b4de8a2ad941c565e1683c5b07255cbe644e/model/modules.py#L122-L168) |
| SF-Net | One labeled timestamp per action, actionness, pseudo-label expansion and background mining | Adds temporal supervision absent from clip labels. Its original evaluation is temporal localization, not strict online MMA recognition. [Paper, §§3–4](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123490409.pdf) |

The online-action literature provides a measured latency tradeoff. CMeRT reports THUMOS’14 **frame-mAP 73.2 with zero added lookahead, 74.5 at 250ms, 75.4 at 500ms and 76.6 at one second**. Stronger visual features also improve its results. Neither experiment is a punch benchmark or a labeled-data scaling curve. Its separate event metrics reinforce why per-frame accuracy alone cannot establish good event segmentation. [CMeRT Tables 10–12](https://arxiv.org/html/2503.18359v1#S5.T11)

**Mechanistic inference:** if a feint and a strike have indistinguishable observed prefixes, a detector must wait, use another observable cue or accept uncertainty. More training cannot expose motion that has not happened. Richer inputs can, however, add evidence missing from a body skeleton; ensembles and spatial graph layers can improve the interpretation of past frames. Their costs are not automatically future-frame dependence.

The practical lesson is to transfer mechanisms and then measure the product's task. A strong prepared-clip classifier is a useful component. Reliable event detection additionally needs consistent event definitions, representative background supervision, temporal evaluation and an explicit latency budget.

Related: [source-audited open-source combat analyzers](/combat-action-analyzers.md).
