Stage 4 · Custom tracker
Turning per-frame fish detections into persistent fish identities, built up one mechanism at a time from a naive baseline and measured at every step.
Abstract
The detector from Stage 3 says where the fish are in a single frame, but not which fish is which. Stage 4 builds the tracker that links detections from one frame to the next, so that each fish keeps the same identity from the start of a video to its end. The tracker is built from scratch as an ablation ladder: it starts from the simplest possible rule, where each fish takes the nearest detection in the next frame, and adds one mechanism at a time, such as motion prediction, one-to-one assignment, a distance gate and a memory for fish that are briefly hidden. Every level runs on the same video clip with the same detector and reports the same counters, so the value of each mechanism is measured rather than assumed. (Results go here once the ladder is complete.)
Introduction
Stage 3 ended with an object detector that answers one question for every frame: where are the fish? It does not say which fish is which. The detector treats every frame on its own, with no memory of the one before: it can find five fish in frame 100 and five in frame 101, but cannot tell whether the fish in the top left of frame 101 is the one that was there in frame 100. To the detector, every frame is a new picture of strangers.
Behaviour, though, belongs to individuals. If two fish swap identities halfway through a video, the distance one of them swam is split between two animals, and a chase can be credited to the wrong fish. Not every measure needs identity for equally long: a chase or a feeding strike lasts a few seconds, and only needs the fish involved to keep their identities for that long. The larger ambition is to follow each fish as an individual across a whole recording, and eventually across days, so that its activity, its boldness or its rank in the group can be compared with the others. That reaches beyond this project, but the tracker is built as a step towards it.
Keeping identities is the job of a tracker: it holds a memory of every fish it follows, and for each new frame decides which detection belongs to which fish. At first sight this looks easy. At 60 frames per second, two frames are only 17 milliseconds apart, which seems too short for a fish to move far, so each fish could simply take the detection closest to where it was. But the decision is made 60 times a second, for every fish, and a single wrong decision is enough to swap two identities for the rest of the video.
This tank adds its own difficulties. The fish are all one species, of similar size and shape, and fish of the same colour morph are practically indistinguishable on camera, with no spot or scar to recognise an individual by. They swim in a loose group and constantly cross paths; when two overlap, the detector often returns one box for both, and during feeding the whole group can pack into a single spot. The plant and the shelter pipes hide fish for seconds at a time. Fish also dart, and in a tank only 35 cm wide a single burst can carry one fish right next to another within a few frames. Reflections in the glass can still be detected as fish that do not exist. Closeness alone is not enough to keep identities reliably.
Well-known trackers such as SORT, DeepSORT and OC-SORT were designed mostly for pedestrians and cars, which are large, move slowly relative to their size and rarely overlap for long. SORT, for example, links boxes by how much they overlap between frames, and a darting fish often leaves no overlap at all. Early attempts to borrow ideas from these trackers gave messy results, handing one fish’s detection to its neighbour. Adopting one whole would also hide which parts actually matter for small, fast fish.
This stage therefore builds the tracker from the ground up, one mechanism at a time. It starts from a deliberately naive baseline, nearest-detection matching with no prediction and no memory of hidden fish, and measures how often it fails. Each following step adds one idea, most of them a more mathematical way of predicting where a fish will be or of deciding fairly between fish that want the same detection, and is measured against the step before it on the same clip. The goal is to find out how far a tracker can get on position and motion alone, fed by the baseline detector from Stage 3, and what each of its parts is worth along the way.
Aim
Measure how well fish identities can be held on position and motion alone, using the Stage 3 baseline detector, and how much each mechanism added to the tracker contributes.
Methods and results
The tracker in the pipeline
The tracker sits between the detector and everything that studies behaviour. Upstream, each
video is registered in the MySQL videos table in Stage 1,
its sampled frames are labelled in Stage 2, and those
labels train the YOLOv8 detector of Stage 3, which
is stored in the MLflow Model Registry. The tracker does not use the sampled frames: it reads
the full video directly, all 60 frames per second, and loads the detector from the registry by
its alias, so a run always names the exact model it used.
Inside the tracker, every frame goes through the same loop. The detector finds the boxes, the boxes labelled as reflections are dropped, and each remaining box is reduced to its centre. The tracker then decides which centre belongs to which fish it is following, updates each fish’s position and motion, and draws the result on a copy of the frame. In the finished tracker this decision is made by the whole ladder below: each fish’s position is predicted from its recent motion, the detections are shared out so that no two fish take the same one, matches that would need an impossible jump are refused, fish that disappear behind the plant are kept in memory until they come back, and two fish merged into a single box are carried through the crossing on their own momentum. Before any of this, the tracker waits until it has a clean view of every fish in the tank and only then gives out the identities.
Each run writes its own folder: the annotated video, a copy of the settings it used, and a
summary of its counters. The finished tracker also writes tracks.parquet, one row per fish
per frame with its identity, position, confidence and whether it was hidden, together with a
small image crop of every fish. These are what the later stages build on. The moments where
the tracker struggles, such as crossings and fish lost behind the plant, are pulled out and
relabelled to improve the detector in Stage 5.
The crops and the stretches of video where an identity is certain become training material for
re-identification in Stage 6. The tracks themselves are the
raw data for the behaviour measures: chasing in Stage 7 and
feeding strikes in Stage 8. Tracks are kept as a file
per run rather than in MySQL, so each run stays self-contained next to its video and settings.
Building it as an ablation
This stage is an ablation, so method and result alternate: each level’s result is the reason for the next level’s method. The setup and the measurements below are shared by every level. Each level then follows the same order: the idea it adds, how it works, what happened on the clip, and what still breaks, which leads to the next level.
Setup shared by every level
Every level of the tracker runs on the same material, so that the only thing that changes
between two runs is the mechanism being tested. The clip is the first minute of
IMG_1839.MOV (0 to 60 s), with five fish in the tank, and every level sees exactly the same
boxes: the Stage 3 detector is loaded from the MLflow Model Registry under a fixed alias
(@baseline), and only boxes of class danio_rerio are passed to the tracker, so reflections
are dropped before matching.
The first 10 seconds of the clip are a warm-up. The tracker runs and draws as usual, but
nothing is counted; every counter is scored on the same 50 seconds, from 10 to 60 s (2,986
frames), whether or not a level needs the warm-up. This keeps the comparison fair once later
levels spend those first seconds settling on their identities. Each mechanism is switched on by
its own flag in config.yaml, the level is written into the name of the run folder, and each
run folder keeps a copy of the settings it used and a summary of its counters, so any level can
be re-run and checked later.
What is measured
At the end of every run the tracker prints the same summary and saves it next to the video. Four counters are computed automatically, frame by frame:
| Counter | Counted when | What it reveals |
|---|---|---|
| Collisions | a fish takes a detection another fish already took in the same frame | two fish sitting on one box |
| Unmatched detections | a detection is left over after every fish has chosen | a real fish that no identity follows |
| Teleports | a fish moves further than max_distance (150 px) in one frame |
a fish grabbing a far, wrong detection |
| Empty frames | the detector finds no fish at all | detector failures, not tracker failures |
Counters say that something went wrong, not whether an identity was lost. The identity switches themselves are counted by eye, on the annotated video, at a list of crossings fixed once by timestamp before any level was run. The annotated video separates the three steps of the pipeline. Every fish the detector found is drawn as a thin yellow outline; every box it labelled as a reflection is drawn in pink, as in the labelling tool, and never reaches the tracker; and every fish the tracker follows is a thick box in its own colour, with its ID above it. A yellow box with no colour on it is a detection nobody claimed, two IDs on one yellow box are a collision, and a colour jumping from one fish to another is an identity switch. A fish with no yellow box at all was missed by the detector, not by the tracker.
L0 · A naive nearest-neighbour baseline
The idea. The simplest rule a tracker can follow: each fish takes the detection closest to where it was in the previous frame. There is no prediction, no rule against two fish taking the same detection, no distance limit and no memory of fish that disappear. L0 exists to be beaten. It shows how far the plain intuition gets, and where it breaks.
How it works. Before the first identity is given out, the tracker waits for a frame in which the detector finds exactly as many fish as there are in the tank, and then creates one identity per detection. A frame with an extra box (a reflection the detector mistook for a fish) or a missing one (a fish hidden behind the plant) is skipped, so neither can become a permanent identity. On this clip that clean frame came after four frames, well inside the warm-up.
From then on, every frame goes through the same three steps. First, the detector returns the boxes in the frame, and each box is reduced to its centre point. Then each fish looks at all the new centres and takes the one closest to where it was in the previous frame; each fish chooses on its own, one after the other, without knowing what the others chose. Finally, each fish remembers the step it just made, how far it moved sideways and up or down, as its velocity, and its position moves to the new centre. If the detector found nothing in a frame, every fish stays where it was.
The velocity is stored at L0 but not yet used for anything. That is deliberate, and it is where L1 starts.
What happened. L0’s counters over the 50 scored seconds (2,986 frames, video 10 to 60 s):
| Collisions | Unmatched detections | Teleports | Empty frames | ID switches at the fixed crossings |
|---|---|---|---|---|
| 11,944 | 7,422 | 200 | 0 | … / … |
These counters are added up over every scored frame, and together they describe a complete collapse. With five fish, at most four can land on a detection someone else already took in each frame, so the ceiling for collisions over 2,986 frames is 11,944. L0 sits exactly on it: from the first seconds onward, all five identities share one detection in every scored frame. The 7,422 unmatched detections, about two and a half per frame, are the other fish, still seen by the detector but no longer followed by any identity. The 200 teleports are the moments when the fish carrying the identities lost its own detection for a frame, and the nearest remaining one belonged to a fish far away. There were no empty frames, so every failure here belongs to the tracker, not the detector. Identity switches at the fixed crossings are counted by eye on the video and are still to be scored for L0.
What still breaks, and why. The failures share one cause: every fish looks for itself in the place it was, not in the place it is going. When two fish pass close to each other, the nearest detection to a fish’s old position can belong to its neighbour. Both fish can then pick the same box, which is a collision, and the other fish’s box is left with no one, which is an unmatched detection. A fish that darts away loses its own detection in the same way.
The next step. The tracker already knows each fish’s last step, its velocity, and does nothing with it. A fish that moved 10 px to the right in the last frame will most likely be about 10 px further right in the next one. L1 asks each fish to look for its detection there, at its predicted position, instead of where it was. Prediction comes before any fairer assignment rule on purpose: assigning detections fairly does not help if every fish is looking in the wrong place.
L1 · Motion prediction
(To be written.)
L2 · Velocity smoothing
(To be written.)
L3 · One detection per fish (Hungarian assignment)
(To be written.)
L4 · Distance gate
(To be written.)
L5 · Memory for hidden fish
(To be written.)
L6 · Merge-aware handling
(To be written.)
Summary of the ladder
(One row per level: the counters and the identity switches at the fixed crossings.)
Discussion
(What each mechanism bought, where the remaining errors sit, and what they point to for the following stages.)