AI Surf Project

For training I used 62 clips of wakesurfing filmed from a boat. This comes out to be around 38 minutes (68,595 frames) at 1080p and 30 fps..

A YOLO11 pose model finds the rider and their joints; the ankles show where the board is. A second YOLO11 model, trained only on surfboards, (My model) finds the board..

Detectors are chosen in a settings file, so you can switch models without changing the code. An RT-DETR plugin is built in and runs automatically on CUDA or Apple chips, so you can use it on a MacBook on the boat.

Instead of marking the board in every frame, I click the rider and board in the first frame, and Meta's SAM 2 tracks them through the video. Using this method I created labels in a standard format. I use every frame to measure performance, but train the model on every fifth frame that shows the board to avoid using nearly identical frames.

I measured each clip's coverage and dropped the six weakest validation clips. That left 54 training and 8 validation clips: 10,746 labeled frames out of 68,595.

The training was set for 100 rounds but stopped at round 70 to avoid overtraining. The saved model was from round 50. It scored 0.684 on one accuracy measure, which counts a board as correctly detected when the predicted box overlaps at least half of the actual board. It scored 0.533 on a stricter measure. Precision was 0.76 (how often its detections were correct), and recall was 0.72 (how often it found the board).

Training loss kept falling while validation loss rose, a sign of overfitting: hence the early stop.

I wrote a tool to compare the model's results with SAM 2's labels. It measures how often the rider and board are found, how accurate their detected positions are, how much those positions jump between frames, how long the model loses track, and how fast it runs. Here are the results for one 613-frame test clip:

From the pose joints, the program measures the rider's form. These are 2D measurements in the camera image, so they're best read as trends:

• Knee bend: the angle at each knee (180° is straight).
• Waist bend: the angle at each hip between shoulder and knee.
• Lean: torso tilt from vertical.
• Stance width: ankle gap divided by the rider's height.
• Foot position: where each ankle sits along the board.

• Knees straighter than 165°: "Bend your knees."
• Hip angle under 120°: "Chest up."
• Leaning over 20°: "Stack over the board."
• Stance under 0.20 or over 0.45 of height: "Feet wider" or "Feet closer."

Cues only run while the rider is surfing. One appears after 8 bad frames in a row (about a quarter second), clears after 20 good ones, and only the most important cue shows at a time.

After a run, a report gives each rule's in-range percentage and worst moment, and lists up to three things to work on. Runs are saved as logs, so old runs can be re-scored with new thresholds.