Skip Intro: Audio Fingerprinting, False Positives, and the Star Trek Problem
You sit down to watch the next episode of your favorite show. The cold open plays. Something happens. Someone gasps. A title card crashes in with a familiar musical sting, then a sixty-second theme song in a style that was very fashionable in whichever year the show premiered. You have heard this theme song nine times already this week.
In the lower-right corner of the screen, a quiet button appears that says Skip Intro. You press it. You are back in the episode. The whole thing took half a second.
That is the feature. It is small, it is polite, it is the kind of thing people do not think about until it is not there. Getting it to work across a whole library of TV shows without a human labeling a single one of them turned out to require a remarkable amount of audio signal processing, some database archeology, and a pitched battle with Star Trek.
This is the story of how ShowShark, a self-hosted media server that runs on macOS and streams to iPhone, iPad, Apple TV, Mac, and Vision Pro, learned to find the opening title sequence in any TV show on disk. It is a story about audio fingerprinting, about the quiet gap between "the algorithm worked in testing" and "the algorithm worked in production," and about why a musical episode of a Star Trek spin-off forced us to rewrite our search window.
Why Intro Detection Is Harder Than It Sounds (ha ha!)
The naive solution is to ask the user to label the intro on the first episode and then assume it is the same on every other episode of the season. This is terrible for several reasons.
First, users hate configuration. If a feature requires the user to scrub a timeline and click "intro starts here" and "intro ends here" for every show, the feature has failed before it shipped. The feature needs to be automatic.
Second, even within one season, the intro does not always start at the same timestamp. TV editors give each episode whatever cold-open length the script demands. A murder mystery might have a ninety-second teaser; the season finale might skip the cold open entirely. The content before the title sequence varies, so the theme song rolls in at different wall-clock seconds in every episode.
Third, every show's intro is different. A sitcom might have a thirty-second theme; a prestige drama might have a two-minute orchestral sequence with a voiceover over guest credits. We cannot hard-code the duration.
What is true, though, is this: within a single season, every episode shares the same theme music as every other episode. The audio is identical across episodes because it is the same master recording. The picture may have different guest-credit overlays, but the underlying audio track does not care about that.
That invariant is the opening through which the whole feature becomes tractable.
Audio Fingerprinting, Explained For People Who Did Not Go to Music School
The goal is to look at a one-hour block of audio from episode one, look at a one-hour block of audio from episode two, and ask: "is there a stretch of audio, anywhere, that appears in both?"
Doing that naively would require comparing every sample to every other sample, and that is hopeless. A single minute of 44 kHz audio is 2.6 million samples. For two minutes, you would be running 6 trillion comparisons. Even if you could make each comparison free, loading the data would bankrupt you.
So we do what every modern media app does for this problem: we convert each chunk of audio into a compact fingerprint, and we compare fingerprints instead.
A fingerprint is a sequence of short numeric summaries, one per small window of audio. The algorithm we use takes eight summaries per second. Each summary is 32 bits. So a ten-minute clip of a TV episode becomes roughly 4,800 numbers, weighing about 19 kilobytes. Comparing two ten-minute clips is now 4,800-against-4,800 comparisons, which is nothing.
Conceptually, the pipeline is this:
Raw audio (mono, downsampled to 11025 Hz, 16-bit)
│
▼
[ sliding window, 4096 samples at a time, ]
[ one new window every ~0.124 seconds ]
│
▼
[ fast Fourier transform ]
[ turn time-domain audio into frequency content ]
│
▼
[ project 2049 frequency bins down to 12 musical ]
[ pitch classes: C, C#, D, D#, E, F, F#, G, G#, A, ]
[ A#, B -- the same twelve notes as a piano octave ]
│
▼
[ smooth across time, normalize, quantize ]
[ pack the result into a single 32-bit integer ]
│
▼
0xa42cb12c one frame
0xa42c906c next frame (124 ms later)
0xa424c0ec next frame
. . .
If you are not a signal-processing person, the one thing to understand is that this sequence of 32-bit numbers is highly characteristic of the music. Two different orchestra pieces produce very different fingerprint sequences. The same orchestra piece, compressed through two different encoders and mastered by two different engineers, tends to produce nearly identical fingerprints; the differences show up as a few flipped bits per frame, not as wholesale divergence.
That "a few flipped bits" property is what makes the matching step work.
Finding the Shared Run
Given fingerprints for episode one and episode two, the matcher's job is to find the longest stretch during which the two fingerprints agree.
We define "agree" as: at this frame position in episode one, and this other frame position in episode two, the two 32-bit numbers differ by at most three bits. (Three bits out of thirty-two means about a ten percent tolerance. Any more permissive and we get false positives; any stricter and real matches drop.)
The algorithm looks like this:
Episode 1 fingerprint: ═══════════════════════════════
Episode 2 fingerprint: ═══════════════════════════════
Step 1: pick an alignment "shift" (how far to slide E2 left/right relative to E1).
Step 2: at this shift, walk both fingerprints in lockstep. For each pair of
aligned frames, count the number of differing bits. If three or fewer,
that frame is a match.
Step 3: stitch consecutive matches into "runs". Allow small gaps of a few
missing frames (an encoder glitch should not break a run).
Step 4: record the longest run at this shift. Try the next shift.
Step 5: across all shifts, the overall longest run is the intro.
The shift search matters. Episode one's theme might start at second 60; episode two's at second 90. The matcher cannot know that up front. It handles this by building an inverted index: for each 32-bit value that appears in episode one, it records every frame where it appears. Then for each frame in episode two, it looks up which frames in episode one contain the same value, and computes what shift would align them. It tries that shift, plus or minus two frames of slack.
Across a full season of ten episodes, that is 45 unordered pairs, and for each pair we keep the longest run. Then we aggregate: for episode one, what is the longest run it participated in across its nine partners? That becomes episode one's intro. The same logic runs for every episode.
For a show with a clear shared intro, the answer converges hard. All ten episodes walk away with essentially the same detected range, give or take a second.
For our first real-world test, it converged beautifully. And then we ran it against Star Trek.
The Star Trek: Strange New Worlds Incident
Star Trek: Strange New Worlds is a currently-running Paramount+ show. It is a nearly perfect test case. Every episode has the same orchestral theme by Jeff Russo. The music is identical across episodes within a season. The detection algorithm should devour it.
We ran the matcher against season one. It matched. Every episode reported an intro. All ten runs agreed within a few seconds. We wrote a celebratory comment in the test log.
Then we looked at the numbers.
Detected intro for S01E01: 0.00s - 22.43s (22.4s)
Detected intro for S01E02: 0.00s - 23.05s (23.0s)
Detected intro for S01E03: 0.00s - 23.05s (23.0s)
Detected intro for S01E04: 0.00s - 23.54s (23.5s)
...
Every episode was reporting a twenty-three-second intro starting at zero. Anyone who has ever watched Strange New Worlds will tell you: the intro is not twenty-three seconds; it is nearly two minutes (?!?), and it does not start at the beginning of the episode. It starts after the cold open, deep into the runtime.
Something was terribly right and terribly wrong at the same time. The matcher had found a shared piece of audio; it had just found the wrong one.
We watched one of the episodes. There it was: at the very start of every episode, Paramount airs a short studio-logo bumper, with its own little musical sting. That sting is about twenty-two seconds long and is identical across every episode of the show. The matcher was finding that and reporting it as the intro. It was technically correct. It was also completely useless.
The real main title, the one you actually want to skip, sits much further in. How much further depends on the cold open:
S01E01: [bumper][ 6:30 cold open ][ main title, 1:45 ][ body ]
0 22s 6:50 8:35
= 410s = 515s
S01E02: [bumper][ cold open, 11:00 long ][ main title, 1:45 ][ body ]
0 22s 11:30 13:15
= 690s = 795s
Episode one's main title starts at 6:50. Episode two's starts at 11:30. The cold-open length is variable, which is fine, because the matcher is shift-invariant. But here is the thing that killed us: we were not looking for it.
The Search Window, and Why It Was Too Small
Inside the matcher, there is a parameter called the "search window." It says: "the start of a detected intro must fall within the first 25 percent of the audio we analyzed, or within the first 600 seconds, whichever is smaller."
That parameter is a safety rail. Without it, the matcher would happily match any shared audio at any position in the episode. A show with a running gag that replays a catchphrase in every episode would have its intro reported as the timestamp of that catchphrase. A serialized drama with a recap montage would get its "intro" reported as the montage. The search window restricts where a real intro can live, which dramatically cuts false positives.
There was one small problem.
What we were analyzing: [ first 600 seconds of each episode ]
0 600s
│
▼
Where the main title is: NOT HERE
... actually, it is at 690s ...
Our extract window was six hundred seconds. Episode two's main title starts at second 690. We were never even feeding it to the matcher. The fingerprints stopped at 600 seconds, so the 690-second main title might as well not have existed.
The matcher had found the only shared audio inside its 600-second horizon: the 22-second Paramount bumper at the very start. That was the correct longest-shared-run for the input it was given. The input was wrong.
Two Numbers, Both Wrong
We widened the audio extraction window from ten minutes (600 s) to twenty minutes (1200 s). Then we ran the diagnostic harness, which dumps every candidate matching run between a pair of episodes, at every alignment shift, whether or not it passes the search window.
E1 vs E2, all matching runs 15s+, sorted by duration
shift E1 range E2 range duration in window
----- ---------------- ---------------- -------- ---------
-2249 [ 410.98s, 516.42s) [ 689.63s, 795.07s) 105.44s OUT
-2248 [ 411.47s, 516.54s) [ 690.00s, 795.07s) 105.07s OUT
-2247 [ 452.48s, 493.62s) [ 730.89s, 772.02s) 41.13s OUT
-2250 [ 411.35s, 437.74s) [ 690.12s, 716.52s) 26.39s OUT
+0 [ 0.00s, 22.30s) [ 0.00s, 22.30s) 22.30s IN
-2250 [ 456.08s, 474.66s) [ 734.85s, 753.44s) 18.59s OUT
-2250 [ 499.69s, 516.29s) [ 778.47s, 795.07s) 16.60s OUT
-2247 [ 500.80s, 516.54s) [ 779.21s, 794.94s) 15.74s OUT
There it was. A 105.44-second shared run starting at 6:51 in episode one and 11:29 in episode two: a difference of 278 seconds, captured as the "shift." That is the main title. The matcher had found it. It just said OUT, because its start lives well past the 25 percent / 600 second search window.
And the 22-second Paramount bumper at the very start was sitting there labeled IN. Of course it was. The search window did exactly what it was designed to do: it let the bumper through and rejected the main title.
The fix was two numbers. The extract window went from a 600-second ceiling to a 1200-second ceiling. The search window went from "first 25 percent, capped at 600 seconds" to "anywhere inside the extracted fingerprint, capped at 1200 seconds."
We re-ran it. Season one: ten out of ten episodes detected the main title, with durations clustered tightly between 105.4 and 108.3 seconds. Every episode's detected range landed squarely on the musical theme.
Season two: eight of ten detected the main title. The other two are fascinating. Episode nine is "Subspace Rhapsody," the musical episode. Its "theme" sequence uses different instrumental arrangements than a normal episode, so its fingerprints drift past our three-bit Hamming threshold against every other episode. The matcher cannot find the main title there, so it falls back to the Paramount bumper. That is the correct behavior; the matcher's rule is "find shared audio," and the musical episode genuinely does not share the normal theme with the rest of the season. Episode seven is the crossover with the animated show Lower Decks, which does a clever thing with its opening; the matcher reports a shorter 68-second partial match. Again, correct. The algorithm is not producing wrong answers, only the answers that the shared audio actually supports.
Season three: seven of ten got the full main title; three fell back to partial matches on stylistic episodes. Same story.
Across three seasons and thirty episodes, 25 got the correct 105-second main title. The remaining five fell back to whatever shared audio the matcher could actually find, which is exactly what it is supposed to do.
A Sixty-Minute Feature That Looked Like A Five-Minute Feature
At this point the algorithm was working. But the user experience was still broken in two specific ways, and one of them was a mystery until we read the log files.
The first problem was the Library tab in ShowShark Server, which displays a live counter for each episode's intro-detection status. On a fresh server launch the user watched the analyzer tick through episodes and expected the counts to move in the UI. The counts did not move. They stayed frozen at whatever they were when the tab opened.
The cause was embarrassingly simple. The panel ran its SQL query once when the tab appeared and never again. On a library with hundreds of episodes, the background analyzer could update the database for an hour straight without the UI refreshing once.
The fix was a fifteen-second periodic refresh, active only while the Library tab is visible:
View .task {
loadOnce()
while not cancelled {
sleep(15s)
refreshJustTheAnalyzerStats()
}
}
SwiftUI cancels the task automatically when the view disappears, so there is no polling while the user is on a different tab. Fifteen seconds is a compromise: fast enough to feel live, slow enough that a quiet library does not pay for the SQL round-trip every second.
The second problem was scarier. The log showed lines like this:
Failed to reset intro_analysis_attempts for episode #8466:
no such column: intro_analysis_attempts
Thousands of them. The analyzer was working, the status column was being updated, but an auxiliary counter column was simply missing from the database. The migration that should have added it had not run.
When we dug in, we found the cause. During development the migration for the intro-detection feature was version 27. The column we wanted to add, intro_analysis_attempts, was originally left out of that migration and added later, still under version 27. Users who had installed the intermediate build had a database stamped at version 27 without the column. On the next launch, the migrator looked at the database, saw "already at 27, nothing to do," and skipped right past the missing column.
This is a familiar class of bug for anyone who has shipped a schema-versioned application. The rule is: never edit a migration after it has shipped, even in development. Every schema change, no matter how small, gets its own new version. The migrator is idempotent per version; if you add something to an already-shipped migration, you break every user who ran the older copy.
The fix was a new migration, version 28, whose entire job is to add intro_analysis_attempts if it is missing and do nothing otherwise. We added a regression test that constructs the exact corrupt database shape (stamped at 27, missing the column) and asserts that version 28 repairs it. Now no matter how messy the ecosystem got during development, a freshly-launched server will heal itself.
The Focus Engine, and Not Being Rude
The last UX puzzle was Apple TV. The Skip Intro button lives in the lower-right corner of the video player. It is visible whenever the viewer is currently inside a detected intro range. That much is easy.
The hard part on tvOS is the focus engine. Apple TV has no touchscreen. Every interactive element on screen has a concept of whether it is "focused," meaning that the directional swipes on the remote will act on it. Normally you want newly-appearing elements to take focus, so the viewer can interact with them immediately.
But think about watching TV. You pressed play. The cold open is over; the theme song is about to start. You are settled in. Do you actually want the remote to silently redirect to a Skip button you did not ask for? Probably not. Most of the time, you want to watch the intro; it is part of the show.
The answer we landed on is passive focus: the button is focusable, so a user who swipes toward it can engage it, but it does not reach out and grab focus when it appears. The focus engine's default target stays wherever it was, usually on the player controls. Swipe up to the button if you want to skip; otherwise the button sits quietly and does not interrupt your evening.
In SwiftUI on tvOS, that is just:
Button(action: { skipIntro() }) {
Label("Skip Intro", systemImage: "forward.fill")
}
.buttonStyle(.plain)
.focusable()
Without .focusable() the button is not reachable by remote at all. Without .buttonStyle(.plain) it takes focus aggressively. With both, it behaves like a polite roommate.
What The Feature Looks Like Now
After a fresh install, a user who adds a TV library and lets the server run overnight will find, the next morning, that the Apple TV experience has changed subtly. Every episode of every show gets a little Skip Intro button that appears at the right moment and disappears after the musical theme is over. Press it and you jump past the intro. Ignore it and you watch the intro like a normal person.
The server's Library tab shows a live counter of how many episodes have been analyzed, how many had an intro detected, how many did not (short specials, single-episode seasons, things with no shared opening), and how many failed in transient ways that the weekly re-run will pick up. No configuration. No labeling. No manual work.
The whole thing runs as a background task, paced to leave plenty of CPU for whatever the user is actually asking the server to do. If a library scan starts mid-analysis, the analyzer yields. If the server restarts mid-season, the analyzer resumes on the next season on the next launch, because it writes status after every season and skips seasons it has already finished.
Pipeline summary
TV show files on disk
│
▼
[ extract first 20 min of audio to mono PCM at 11025 Hz ]
[ (uses GStreamer with a DTS workaround path) ]
│
▼
[ compute 32-bit audio fingerprint per 0.124 sec frame ]
[ (Accelerate / vDSP, runs half the CPU cores in ]
[ parallel, cache results in SQLite as BLOB) ]
│
▼
[ match pairwise across each season; find the longest ]
[ shared run at any alignment shift; aggregate longest ]
[ per episode ]
│
▼
[ write (intro start, intro end, status) to episodes. ]
[ row in SQLite; stats table picks it up for the UI ]
│
▼
[ when the client asks for playback, the server sends ]
[ the boundaries back on StartPlaybackResponse; the ]
[ client shows Skip Intro at the right moment ]
Everything above the client is invisible to the viewer. All anyone ever sees is that little polite button.
What We Learned
A few things worth remembering, in no particular order.
Algorithms have assumptions, and those assumptions look like parameters. Our search window was a parameter. It encoded the assumption "intros are near the start of the episode." That assumption is correct for a sitcom and wrong for any show with a long cold open. The bug was not in the matcher; the bug was in the set of shows the matcher had ever been tested against.
Your test library needs to be weird. We did most of the early testing against content with predictable structure. The algorithm worked beautifully. The first time we ran it against a show with long cold opens, it produced plausible-but-wrong answers that could easily have shipped. A fake-friends test library is insufficient; you need the awkward stuff.
A diagnostic harness is worth ten algorithm rewrites. Once we wrote the tool that dumps every candidate run at every shift, the root cause became obvious in thirty seconds. Before that tool existed, we had been slowly convincing ourselves that the audio was somehow not matching, and that we would need deeper changes to the fingerprint pipeline. The problem was nothing that exotic. The problem was that we were not looking at the right number.
The focus engine is not a bug. It is asking a legitimate question: "does this new thing deserve the user's attention more than what they were already doing?" Most of the time, the answer is no.
"Automatic" is a promise. Any feature that could be automatic and is not is one more thing the user did not ask to have to do. Over years of shipping software the accumulated weight of those "one more things" is the main reason users give up on products. The work of making things automatic is the work of not being the reason.
ShowShark is an open project focused on providing a polished, self-hosted experience: you bring the media, ShowShark serves it, the clients on your devices stream it, and the feature list grows without the configuration burden doing the same. Intro detection is one more thing the server now does quietly in the background so that the person watching a show does not have to think about anything except whether to press one button.
Or not press it. That is also fine.