transformer-style attention · live training visualization
Watch the model's 32 hidden neurons specialize as training nudges 2,200 weights. Pick any neuron to see what input it listens to and how strongly it fires for each sample clip — positive AND negative — all the way through the pooled bag prediction.
active-learning loop · real clips, real model
The model is retrained from scratch on a growing subset of this move's real labels — each positive added one at a time, negatives in bundles — and its held-out accuracy responds. Every thumbnail is a real labelled clip.
loading…
what this is
overview · how it works · scaling up
overview
Arbitrary movement can be recognized across thousands of hours of video on any laptop in about a day's work, with precision that scales to your dataset.
how it works
The small model is one attention layer over per-frame skeleton keypoints — attention scores which frames in a clip belong to the move and pools them into a single prediction (attention-based MIL, Ilse et al. 2018), so it trains from clip-level labels alone (UntrimmedNets, Wang et al. 2017) and handles any-length actions. You label a few clips, run it back over the footage to mine its false positives, retrain, and repeat.
scaling up
Once the dataset is large enough, you swap in a multi-layer transformer whose self-attention runs across frames, capturing the longer, order-dependent movements a single layer can't.
pick a move to study
top 0 by positive count
architecture · how the model reads a clip
static — order-aware (mil-tc) moves add the conv stage
stage 1
17 KEYPOINTS
(x, y, conf) × 17 joints
One frame at a time. RTMO pose model gives you this for free.
Think of 32 little detectors. They all stare at the same 68-number pose vector and each one writes down a single number: "how strongly do I see my pattern?"
wait — what's a "neuron", actually?
Forget brain cells — that's just borrowed jargon. A "neuron" is a tiny piece of arithmetic. It takes a list of inputs, multiplies each one by its own importance weight, sums them up, adds a small offset (called a "bias"), and runs the answer through a simple rule (here, ReLU). out = ReLU(w₁·in₁ + w₂·in₂ + … + w₆₈·in₆₈ + b) That's the whole thing. Each "detector" we just described IS one neuron. 32 neurons = 32 of these arithmetic boxes side-by-side, each with its own w₁…w₆₈ and b. Stack many layers of these and you get a "neural network". This model has just one such layer, which makes it about the smallest thing that still earns the name.Yes, weights can be negative. A positive wj means "when input j goes up, my answer goes up". A negative wj means "when input j goes up, my answer goes down" — i.e., this input suppresses me. Training is free to pick either sign for any (neuron, input) pair. Most weights end up small and irrelevant; a handful end up large.
4 rows × 8 = 32 detectors
Where the patterns come from: nobody writes them by hand. Training picks each detector's pattern — one might end up firing on "right wrist above the head", another on "fast hip rotation", another on "knees bent". Whatever helps tell positive clips apart from negative ones.
What ReLU does (and doesn't do): if a detector's answer comes out negative, just write down 0. If it's positive, keep it as is — could be 0.4, 3.7, 12.0, anything ≥ 0.⚠ ReLU does NOT squash to 0-or-1. The output hk is any non-negative number. The "squash to 0-1" thing happens later, in the attention and score heads (stages 5 & 6), using a different function called σ (sigmoid). ReLU's only job is "kill the negatives".
the same idea in math
h = ReLU(W_e · x + b_e)
W_e is a 32×68 table — one row per detector. Each row holds how much that detector cares about each of the 68 inputs. b_e is a small offset per detector (its "threshold"). The dot product takes one row × the 68 inputs and sums them. The output h = a list of 32 numbers, one per detector.
→
depthwise 1-D conv
stage 4 · order-aware (mil-tc) only
TEMPORAL CONV g
g = ReLU(W_c ⊛ h + b_c)
Each neuron gets its own tiny 3-tap filter over time — it mixes that neuron's value at t−1, t, t+1. This turns the order-blind per-frame h into an order-aware g: "rising into a pose" now scores differently from "falling out of it". Base MIL skips this stage (g = h); combos like jab cross use it.
t−1
t
t+1
↓
fork → both heads read g (g = h for base MIL)
stage 5 · attention head
ATTENTION a
a = σ(w_a · g + b_a)
One attention head (not the many of a transformer). Weight each neuron's output by w_a[k], sum, add bias, squash with σ into (0, 1). Answers: "is THIS frame the move?"
∥
parallel both read g
stage 6 · score head
CLASS SCORE s
s = σ(c · g + b_c)
Same recipe, different weights (c[k]) + its own σ. This is where 0-to-1 squashing happens, NOT in ReLU. Answers: "if it is, how strongly?"
↓
p = Σ(a·s) / (Σa + 1)
stage 7 · pool
POOL ACROSS FRAMES
For the WHOLE clip: weight each frame's score by its attention, then average. The +1 is a fake background frame that keeps the model from cheating by zeroing every attention.
p = Σ(aᵢ·sᵢ) / (Σaᵢ + 1)
The whole model fits on a napkin
Take a frame's 17 (x, y) joints, flatten + add velocities → a vector x of length 68.
Multiply by a matrix W_e(32×68), add a bias, ReLU. Result: a 32-dim hidden vector h.
Run h through two small read-out vectors:
a = σ(w_a · h + b_a) → "is THIS frame the move?"
s = σ(c · h + b_c) → "if it is, how strongly?"
For the whole clip: p = Σ(aᵢ·sᵢ) / (Σaᵢ + 1). The +1 is a fake background frame with zero score that keeps the model honest.
That's it. About 2,200 numbers in W_e + w_a + c + biases. Training nudges them so positive clips score near 1 and negatives near 0.
trajectory shape vs speed · why slow motion breaks the model, and the featureVariant fix
featureVariants.ts · applied inside forwardTC when the model params carry featureVariant
The wrist traces the identical arc both times. But the model never sees the arc — it sees one 68-dim vector per frame, and in the slow version every velocity entry is ÷4 and there are 4× more frames. Nearly every number the model reads has changed.
Why “shape” is hard. A swing is a shape traced through pose space — 17 joints = 17 coupled curves. What stays constant across playback speeds is only the shape of those curves; every sample of them changes. Formally, recognizing the shape means recognizing an equivalence class under time re-parameterization (any monotone re-timing of the same path), which is an infinite family of disguises for one movement. A linear encoder + temporal conv reads raw per-frame coordinates, so each disguise looks like a different input. It has no object called “curve” — that is the entire difficulty. The mathematics for quotienting out re-timing exists (elastic shape analysis / SRVF, path signatures below) but needs geometry a 32-neuron legible net can't represent, so we approximate the invariance in the features instead.
What the model reads. Per frame: [34 body-frame joint positions | 34 velocities], velocity = displacement per 33ms reference frame. k× slow motion leaves positions intact and divides all 34 velocity dims by k — to the model a slow-mo swing is literally a different (near-idle) movement.
The three transforms (a model trained with one applies it to its own input inside forwardTC — training and every inference path stay consistent by construction; untagged models are untouched):
vnorm
v ÷ max(p90 bag speed, floor)
relative velocity: the clip's own tempo cancels; the floor keeps idle small. Replicated win on back swing (IoU .403→.477 and .490→.535 across two seeds, fewer false fires).
vsqrt
v ÷ √‖v‖
SRVF square-root velocity: k× tempo shrinks to √k, no bag statistics needed. Within noise on our data.
vdir
v ÷ ‖v‖
pure direction, exactly speed-invariant — but it erases the idle/swing magnitude gap and over-fires badly on slow footage (measured ~3× spurious detections). Kept as a cautionary arm.
What features cannot fix. Inference scores frames in a fixed 1500ms sliding window and the temporal conv spans 25 frames — at 8×+ slow motion (a 20s swing) the window holds a sliver of near-static arc no matter how velocity is expressed. That regime needs re-timing at inference (score with timestamps ÷ k, keep the best k) or windows measured in accumulated motion (arc length) instead of milliseconds.
gradient descent · the actual method, in plain words
✗ not how it works
Random weights + pick the best? Close intuition, but hopelessly slow. With 2,200 weights, even sampling a billion random sets per second, you'd need longer than the age of the universe to stumble onto a good one. We need a smarter way to search.
✓ actual method: gradient descent
Start random. All 2,200 weights begin as small Gaussian noise.
Run ONE clip through. Compare prediction to its label. Compute a loss — a single number: "how wrong are we?"
Compute the gradient. Calculus tells you, for EACH weight, the direction + size of the nudge that would reduce the loss most.
Nudge every weight a tiny step in its downhill direction.
Next clip. Repeat. All clips = one epoch.
Run 150 epochs. Weights settle into a configuration that classifies the training set.
🏔 the analogy
You're blindfolded on a hilly landscape, looking for the lowest point. You can't see, but you can feel the slope under your feet — that's the gradient.You take a small step downhill. Repeat. Eventually you reach a valley.The "valley" here is a 2,200-dimensional weight configuration that minimizes the loss. The model never sees the whole landscape — it just keeps stepping downhill.
what you're watching on this page
The red loss curve goes down because gradient descent finds lower-loss weight configurations.
The W_e heatmap morphs because those 2,176 numbers are getting nudged every epoch.
The detector specialties emerge because weights drift into useful patterns that help training labels match.
Two terms you'll see elsewhere: backpropagation = the chain rule trick for computing all 2,200 gradients efficiently; SGD = "stochastic gradient descent", meaning we use one clip at a time instead of all of them at once.
seeing the calculus · gradient descent in pictures
the marble + the tangent line are everything you need to picture how a model learns
① the loss landscape · a marble rolls downhill
marble = current weights · red arrow = nudge direction (downhill)
Pretend there are only 2 weights (instead of 2,200). At every point in the plane the model has a loss — how wrong it is. Lower = better. The contour rings show "same-loss" loops, like a topographic map.The yellow marble = the current weights. The red arrow at each step points downhill — that direction is what calculus computes from the slope. Each step nudges the marble that way by a tiny amount. Eventually it settles in the green minimum.In the real model the same thing happens, but in 2,200-dimensional space — impossible to draw, identical math.
② what's a derivative? · slice through one weight only
w = -1.80 · loss = 9.64 · increase w
Now hold every weight fixed except one. Plot loss as that single weight moves. The yellow tangent line touches the curve at the current weight — its slope IS the gradient for this weight.
Slope positive → loss goes UP when w increases → decrease w.
Slope negative → loss goes DOWN when w increases → increase w.
Slope ≈ 0 → can't improve further; you're at (or near) a minimum.
Drag the slider to feel it: the tangent and the nudge direction update live. Now imagine doing this for all 2,200 weights at once — that's one step of gradient descent.
③ the full recipe, mapped to those pictures
1. Start random. Marble is placed at a random point on the landscape. All 2,200 weights begin as small Gaussian noise.
2. Run ONE clip. Forward pass through the model. Get a prediction. Compare to label. Compute the loss. (The marble's height on the landscape.)
3. Get the slope. Calculus computes the gradient — the slope of loss w.r.t. every weight. (The tangent line, but in 2,200 dims.)
4. Nudge. Every weight steps a tiny amount in its downhill direction. Tiny because if you take a big step you'll overshoot the valley.
5. Next clip. Repeat 2-4. All clips through once = one epoch.
6. Repeat 150 epochs. The marble settles in a valley. The weights now classify most training clips correctly.
The clever bit — backpropagation. The model has many layers (W_e → ReLU → w_a/c → σ → pool). To get the slope of the loss w.r.t. an early weight like W_e[5][10], you'd seemingly have to redo a huge calculation. The chain rule says: take the slope at the output, multiply by the local slope at each layer going backward, and the product is your gradient. That's "backpropagation" — it's the chain rule applied layer-by-layer, and it lets you compute all 2,200 gradients in about the same time as one forward pass. In this codebase the math is hand-derived; you can read it in app/lib/moveClassifier/milAttention.ts lines 213-297.
What this means for the curves above: the red loss line you see further down the page is the marble's height plotted against step number. As gradient descent works, that line slopes downward — that's the marble rolling toward the valley, projected onto one axis.
why did switching to pytorch change the accuracy?
it didn't — the optimizer did. a note on a common trap
the trap: "new framework = better model"
The model's training was moved from hand-rolled TypeScript (manual backprop) to PyTorch on the Apple-Silicon GPU. Accuracy on some moves went up. The natural conclusion — "PyTorch trains better models" — is wrong. A framework is just a calculator; it doesn't change what's being computed. The same architecture, the same math, the same 2,200 weights.
the proof: parity = 0.00000016
Take the PyTorch-trained weights and run them through the oldTypeScript forward pass. The predictions match the PyTorch forward to 1.6 × 10⁻⁷ — floating-point dust. If the framework were doing anything "smarter," this would diverge. It doesn't, because the forward math is byte-for-byte the same. (This is also why the model is drop-in: inference never changed.)
what actually changed: one line — the optimizer (SGD → Adam)
And it happened almost by accident, in a chain:
To use the GPU, you batch. The old TS loop did one bag at a time — that's ~240 weight updates per epoch × 150 epochs ≈ 36,000 gradient steps. The GPU version trains all bags in one step per epoch → only 150 steps.
So plain SGD suddenly under-trained. 150 steps with a fixed learning rate isn't enough — jab's accuracy fell to 0.62.
To compensate, the optimizer was switched to Adam (lr 0.03 SGD → lr 0.01 Adam). And Adam didn't just claw back the lost steps — it overshot past what the old SGD ever reached.
The gain is Adam-vs-SGD, not PyTorch-vs-TypeScript. You'd get the identical bump by writing Adam into the old TS trainer — it'd just be slow and miserable to hand-derive (which is exactly what you switch to autograd to avoid).
why Adam is fundamentally better for this model
Plain SGD uses one global learning rate for all 2,200 weights. But this model is badly ill-conditioned — its parts live on totally different scales:
the attention headw_a (32 weights),
the encoderW_e (2,176 weights reading 68 inputs each),
the biasesb_a, b_c, b_e.
Their gradients differ by orders of magnitude, so a single step size that's right for one is far too big or too small for the others — SGD limps. Adam keeps a separate, adaptive step size per weight (running averages of each weight's gradient and its variance), so every part of the network moves at its own correct pace. It settles into a lower-loss, better-fitting minimum that one global rate can't reach. That's the whole gain — a smarter search, not a smarter model.
⚠ don't over-read the numbers
Real gains are only on the moves with enough labels to measure honestly: jab 0.78 → 0.82, kick stayed ~0.91(tight ranges, ~100 positives each). Those are trustworthy.
The eye-catching jab cross 0.68 → 0.88 is mostly a mirage. Jab cross has only 23 positives → ~4–5 validation clips, and Adam fits the train set to 0.99 (it overfits harder than SGD). On one lucky split that overfit happens to land at 0.88; the multi-seed sweep showed jab cross still bouncing 0.71–0.76 with huge ranges. A single-split number on 5 val clips is noise, not a breakthrough.
what the migration actually bought
Speed. All 16 models train in 18s on the GPU (vs 88s CPU, vs the old per-bag JS loop). The sweeps and retrains stop being a coffee break.
Maintainability. Autograd computes the gradients, so there's no hand-derived backprop to keep correct when the architecture changes.
Nothing for inference. The weights are written back as the same JSON; the pure-TS forward pass runs them unchanged on the x86-64 server. The accuracy story lives entirely in how the weights were found, not in what runs them.
where does "attention" factor in?
the one thing this model actually shares with transformers
attention = the attention head's output (stage 5)
"Attention" is just one number per frame, between 0 and 1. It's the model's vote on:a ≈ 1 → "this frame MATTERS — look HERE" a ≈ 0 → "ignore this frame — windup / standing / recovery"Same formula you already saw in the attention head (stage 5):a = σ(w_a · h + b_a)The 32 detectors fire on a frame → their outputs get weighted by w_a → summed → squashed to (0, 1) by sigmoid → that's the attention for THAT frame.
why it's called "attention"
Because the pool literally weights each frame's score by its attention. The final bag-score is:p = Σ(aᵢ·sᵢ) / (Σaᵢ + 1)High-a frames dominate the average. Low-a frames contribute almost nothing. The model gets to focus on whichever frames it decided are "the move" — that's the analogy with paying attention.
the magic: nobody told it which frames matter
Every labelled clip carries ONE label — "this clip contains a jab" or "it doesn't". No frame-level labels. And yet, after training, attention spikes on the 3-4 frames where the jab actually happens. The model figures out which frames are the move all by itself, just because pooling rewards being right on a few high-a frames and the sparsity regularizer punishes high attention everywhere.This is called weakly-supervised temporal localization. It's how you train a "where is the action?" model from "is there action?" labels.
How transformer attention differs: a transformer's attention is much richer — each frame's "query" dot-products every frame's "key" to mix information across frames, with multiple heads doing this in parallel. This model's attention is the simplest version: one scalar per frame, no cross-frame mixing. Same intuition ("focus on what matters"); different math (much, much smaller). See the side-by-side comparison panel at the bottom of the page.
training run
no run yet
loss + accuracy as training progresses
click anywhere on the chart, or drag the slider, to scrub to that epoch
no snapshots yet — start training to populate
epoch 0/0
encoder matrix W_e at this epoch
watch bright bands emerge as neurons specialize
scrub to an epoch with a checkpoint…
how could a detector encode "hand over foot"?
concrete weight recipes for relational features — what one neuron CAN and CAN'T capture
① relative position · "hand over foot"
A weighted difference between two joints IS a relative-position feature. Set opposite-sign weights on two joints' y-coordinates:
+1.0 L wrist y (pos) −1.0 L ankle y (pos) ~0 on the other 66 dims
h = ReLU(wrist_y − ankle_y + b) Fires when the two values differ in the "right" direction. The pair on x instead measures horizontal gap; pair on mirrored joints to measure left-vs-right symmetry.
② direction of motion · "moving outward"
"Moving outward" = velocity in the right direction. The 68-input vector already includes velocity for every joint, so a single neuron can do it:
+1.0 R wrist x (vel) +1.0 R elbow x (vel) −1.0 L wrist x (vel) −1.0 L elbow x (vel)
Fires when right limb moves in +x (rightward = outward) AND left limb moves in −x (leftward = outward) — i.e., both arms extending out at once. No integration over time needed — velocity is pre-computed and served up directly as input.
③ symmetric pose · "both arms raised"
Same-sign weights on mirrored joints = a symmetric pose detector. Sums fire when both sides agree.
+0.6 L wrist y (pos) +0.6 R wrist y (pos) −0.4 L shoulder y (pos) −0.4 R shoulder y (pos)
Effectively measures "wrists relative to shoulders" on both sides at once — fires when BOTH wrists are above their respective shoulders. One-sided poses give it a weaker response because only half the inputs contribute.
Products of two inputswrist_y × ankle_y — a multiplicative gate, which a linear neuron can only approximate.
Multi-frame memory — "wrist was high THREE frames ago" requires looking backward in time, which one neuron can't do (velocity gives you only Δ-from-previous).
BUT 32 detectors plus the read-out heads approximate many of these. Distance ≈ a sum of L¹ pieces ("wrist far right of hip" + "wrist far above hip"). Angles ≈ multiple direction detectors voting. The class-score head's c[k] weights combine detectors, so "wrist FAR from hip" emerges from several specialists co-firing — not from one neuron.
The big picture: a single linear neuron + ReLU can encode any weighted sum or difference of the 68 inputs. That's already enough for relative positions, motion directions, and symmetric pose patterns — the bread-and-butter of describing body language. With 32 such neurons, each specializing on different (joint, joint, axis) combinations, the model assembles a vocabulary of body-language primitives. The read-out heads (a for attention, s for class score) then combine those primitives into "is this frame the move?" and "is this the right move?". The architectural ceiling is multi-frame temporal patterns — which is exactly what transformers and RNNs add. At your label scale, this trade-off is the right one.
detector roster · what each of the 32 actually learned
auto-derived from W_e at this epoch · click any detector to deep-dive
The model never told us what each detector is for — but we can figure it out from its weights. Each detector has 68 importance weights (one per input number). We look at which inputs it cares about most, group them by body part (head / arms / legs / torso) and by kind (current position vs. how fast it's moving), and that's its "specialty". The two little arrows on the right say which way it pushes the two read-out heads when it fires:a ↑/↓ = raises/lowers attention ("frame matters") · s ↑/↓ = raises/lowers class-score ("really IS the move")A detector with both ↑ is a "yes this is the move" voter. With both ↓ it's an "ignore-this-frame" voter (negative pattern). Mixed signs mean it's nuanced.
reading a top-driver line · example: +R ankle x (pos)
+ or −
Sign of the weight. +: input ↑ → answer ↑. −: input ↑ → answer ↓ (suppresses).
R ankle
Which of the 17 body joints (nose, shoulders, elbows, wrists, hips, knees, ankles, etc.).
x or y
Axis: x = horizontal, y = vertical. Body-frame coords, not screen pixels — normalized so the torso is roughly unit size.
(pos) or (vel)
pos = joint's current location. vel = how fast it's moving this frame (current − previous, scaled by Δt).
So +R ankle x (pos) reads: "positive weight on the right ankle's horizontal position — when the right ankle is far to one side, this input pushes the detector's answer up." The card shows only the top 3 by |weight| out of 68; click any card for the full ranking in the deep-dive.
What are "top weights"? Each detector has 68 weights (one per input number). Most end up small and barely matter. We sort the 68 by |weight| (absolute value — so −0.42 ranks the same as +0.42: sign tells direction, magnitude tells how much it matters) and call the largest 5 the detector's "top weights" — its loudest voices. The "% of top-5 weight strength lands here" number is what fraction of those 5 magnitudes points at the dominant body-part-group. Above ~50% = clean specialist. Below ~30% = generalist (scattered).
scrub to an epoch with a saved checkpoint to see the detector roster.
sample clips · how the model reads each one
3 positive + 2 negative · click any neuron cell to deep-dive
a · attention
Per-frame value in (0, 1). Answers "is THIS frame the move?" — high = move, low = idle/wind-up.
s · class score
Per-frame value in (0, 1). Answers "if it IS the move, how strong?" — kept high on textbook poses regardless of timing.
a·s · gated
The actual signal that gets pooled. Both heads must fire on the same frame to count. Final bag-score = Σ(a·s) / (Σa + 1).
32-neuron grid · click any cell
Each square is one of the model's 32 hidden neurons h_k for the current frame. Brightness = firing strength. Click any cell to open the deep-dive: what that single neuron listens to (its W_e row), how much it pushes attention (w_a[k]) vs score (c[k]), and its firing curve across all 5 sample clips.
no sample-clip snapshot at this epoch — try a multiple-of-5 epoch
full network trace · put a real clip through the model and watch every number
production weights · forward pass runs live in your browser
This traces a real labelled clip through the production model, stage by stage: each posture's 68 numbers flow into the 32 neurons (watch the products fly along the edges), the dilated conv taps slide across the series — the exact postures they read light up in the filmstrip — then the attention and score heads assemble per-frame decisions, and pooling accumulates the final bag score. Hover any heatmap cell to see which posture and which body parts that neuron is reading; click to pin the neuron.
attribution tree · legible TCN (BoxingVI, causal streaming)
the actual computation animated: skeleton → per-joint filters → named units → verdict, edge = weight×activation per frame
loading legible-tcn-demo.json…
this attention-MIL vs a real transformer
What you're watching
One encoder layer + ReLU. Every frame is encoded independently.
One attention scalar per frame: "does THIS frame matter?"
~2,200 parameters total. Fits 10–200 labels per move.
Sparsity + smoothness regularizers concentrate attention onto a contiguous burst (the trimmed move).
No frame can directly look at another frame.
What a real Transformer adds
Multi-head self-attention: each frame's Q dots every frame's K → softmax → mixes information across the whole clip.
Stacked blocks (often 6–96) with residuals, FFNs, layer norm.