Synthesizing a soundtrack timed to video cuts

Yes, you can synthesize a soundtrack that lands on a video’s cuts with numpy and scipy in about 25 lines. The trick is to treat the cut list as absolute times, turn each into a sample position with one rounding step, and write notes into a single long array. In our run, five note onsets that we detected from the finished audio landed within 3.3 ms of their cuts. A video frame at 30 frames per second lasts 33 ms, so that error is about a tenth of a frame.

Everything below was run with Python 3.13.18, numpy 2.5.3, scipy 1.18.1 and ffmpeg 6.1.1 on Linux. The outputs shown are the real ones. We did not listen to the result on speakers. We measured it.

What are we building?

A short soundtrack for an 11-second video with five shots. The video cuts at 0, 2.4, 3.9, 6.5 and 9.1 seconds, and we want a new note to start exactly on each cut, with the previous note ending cleanly. The sound is a plain sine tone plus one overtone with a quick attack and a decay, which is enough to make the onset measurable.

What does the code look like?

import numpy as np
from scipy.io import wavfile
from scipy.signal import find_peaks

SR = 44100
CUTS = [0.0, 2.4, 3.9, 6.5, 9.1]   # seconds where the video cuts to a new shot
TOTAL = 11.0

def note(freq, dur, sr=SR, attack=0.005, release=0.25):
    n = int(round(dur * sr))
    t = np.arange(n) / sr
    tone = np.sin(2*np.pi*freq*t) + 0.5*np.sin(2*np.pi*2*freq*t)
    env = np.minimum(1.0, t/attack) * np.exp(-t/release)
    return tone * env

def build(cuts, total, sr=SR, fade_out=0.05, release=0.25):
    out = np.zeros(int(round(total*sr)))
    freqs = [220.0, 261.63, 329.63, 392.0, 440.0]
    for i, c in enumerate(cuts):
        end = cuts[i+1] if i+1 < len(cuts) else total
        n = note(freqs[i % len(freqs)], end - c, sr, release=release)
        start = int(round(c*sr))
        k = int(fade_out*sr)          # fade the tail so the note never ends mid-wave
        if k:
            n[-k:] *= np.linspace(1, 0, k)
        out[start:start+len(n)] += n[:len(out)-start]
    return out

x = build(CUTS, TOTAL)
print("raw peak:", round(float(np.max(np.abs(x))), 3))
x = x / np.max(np.abs(x)) * 0.8
wavfile.write("track.wav", SR, (x*32767).astype(np.int16))

Three choices in there are doing real work. Each note’s start comes from round(c*sr) on the absolute cut time. Each note is written into the shared array with +=, so overlaps add. And the whole track is normalized once at the end. We test each of those below.

Does it land on the cuts?

To check, we read the WAV file back and found the onsets from the audio alone, without using the cut list. We computed a 2 ms moving RMS (root mean square, a smoothed loudness measure), looked at how much it rose over 2 ms, and picked the peaks with scipy.signal.find_peaks.

sr, y = wavfile.read("track.wav")
y = y.astype(float)/32767
win = int(0.002*sr)                      # 2 ms RMS window
env = np.sqrt(np.convolve(y**2, np.ones(win)/win, mode="same"))
jump = env[win:] - env[:-win]             # rise over 2 ms
pk, _ = find_peaks(jump, height=0.15, distance=int(0.5*sr))
found = pk/sr
for c, f in zip(CUTS, found):
    print(f"cut {c:5.2f}s  onset {f:7.4f}s  error {1000*(f-c):+6.2f} ms")

Output:

raw peak: 1.272
rate 44100 samples 485100 duration s 11.0
onsets found: 5 expected: 5
cut  0.00s  onset  0.0023s  error  +2.34 ms
cut  2.40s  onset  2.4018s  error  +1.77 ms
cut  3.90s  onset  3.9010s  error  +1.02 ms
cut  6.50s  onset  6.5033s  error  +3.29 ms
cut  9.10s  onset  9.1028s  error  +2.81 ms

All five onsets arrive within 3.3 ms after the cut. The small positive error is the detector at work, not the synthesis. Our notes have a 5 ms attack ramp, so the loudness rise peaks a little after the start. Because the track has exactly 485,100 samples, it is exactly 11.0 seconds long at 44,100 Hz.

Our first detector attempt used a plain rectified-signal peak finder with a 64-sample hop. It reported a spurious peak at 0.5166 s and shifted every later match. Smoothing the signal first, as above, fixed it. If you write your own check, test the check too.

What happens if you skip the normalize step?

Five notes with overtones and ring-out sum to more than full scale. The raw peak printed above was 1.272. Here is what that does to a 16-bit file:

x = build(CUTS, TOTAL)
over = int(np.sum(np.abs(x) > 1.0))
wrapped = (x*32767).astype(np.int16)
good = np.clip(x, -1, 1)
print("samples above full scale:", over, "of", len(x))
print("worst wrapped-vs-clipped difference (int16 steps):",
      int(np.max(np.abs(wrapped.astype(int) - (good*32767).astype(np.int16).astype(int)))))

samples above full scale: 2786 of 485100
worst wrapped-vs-clipped difference (int16 steps): 65534

A difference of 65,534 steps is the full range of a 16-bit integer. The cast does not clamp the value. It wraps around, so a peak slightly above the top comes out near the bottom, which sounds like a harsh crackle. Normalizing the whole track to a peak of 0.8 before the cast avoids it. We read the finished WAV back and measured a peak of 0.79998.

What causes clicks where one note ends and the next begins?

Our default notes decay quickly, so the old note is almost silent at the cut and nothing clicks. To test the risk, we gave every note a 3-second ring-out so the previous note was still loud at each cut, then measured the largest sample-to-sample step across the cut, with and without a 50 ms linear fade on each note’s tail.

fade    0s: biggest sample-to-sample step at a cut = 0.3292
fade 0.05s: biggest sample-to-sample step at a cut = 0.0004

Without a fade, the waveform jumps by 0.3292 in a single sample, which is a third of the way from silence to full scale. That jump is what you hear as a click. With the fade, the biggest step drops to 0.0004. A fade of a few tens of milliseconds is enough, and it is short enough that nobody hears it as a fade.

Why compute positions from absolute times?

The tempting way to build a track is to make each segment, round its length to whole samples, and concatenate. Every segment then carries its own rounding error, and the errors add up. We tested 1,000 segments of 1/24 second each at 44,100 Hz:

segment-by-segment: 1838000 samples; absolute: 1837500 ; drift: 11.34 ms

A 1/24 s segment is 1,837.5 samples long, and Python rounds half to even, so each segment came out as 1,838. That half-sample error repeated 1,000 times is 500 samples, or 11.34 ms. Eleven milliseconds is about a third of a frame at 30 frames per second. It is a deliberately bad case with a long cut list, and an 11-second video with five cuts would see almost none of it. A long montage or a whole reel could.

The fix is what the code above already does: start each note at round(absolute_time * sr), so each position carries at most half a sample of error, and the errors do not add up.

How do we put the track on the video?

We made a test video with ffmpeg’s testsrc pattern and muxed the audio on with explicit stream selection. We used -map because a video that already has an audio stream, even a silent one, makes ffmpeg skip the new file, as we showed in Muxing a Soundtrack onto a Silent Video with ffmpeg.

ffmpeg -f lavfi -i testsrc=duration=11:size=320x180:rate=30 -c:v libx264 -pix_fmt yuv420p video.mp4
ffmpeg -i video.mp4 -i track.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 128k out.mp4
ffprobe -v error -show_entries stream=codec_type,codec_name,duration -of csv=p=0 out.mp4

h264,video,11.000000
aac,audio,11.000000

Both streams run for exactly 11 seconds. A volumedetect pass over the output reported a mean volume of -18.6 dB and a max volume of 0.0 dB. The WAV we fed in peaked at 0.79998, about -1.9 dB, so AAC encoding added headroom use. We did not investigate why, but if you need strict peak limits, leave a larger margin than 0.8 or measure the encoded file as we did.

What would we do in practice?

Keep the cut list as seconds in one list, or in one file the video code reads too. Two lists drift apart.
Convert each time to a sample index with one round, and write notes into one array.
Normalize once at the end, then convert to integers.
Fade every note’s tail, even if the sound seems to decay on its own. Check by printing the largest step at each boundary.
Detect onsets from the finished file and print the errors. The check takes about ten lines and found a mistake in our own first detector.

Limitations

We measured timing from the audio file, and from ffprobe stream durations. We did not check lip-sync or beat alignment by ear, and we did not render a video with real cuts.
The sound is a sine with one overtone. Real music has softer onsets that are harder to detect, so a 3 ms result here will not carry over to a recorded instrument.
The 11.34 ms drift figure comes from a deliberately long, 1,000-segment case. It scales with the number of rounded segments.
We tested one Python, numpy, scipy and ffmpeg combination on Linux, at 44,100 Hz and 16 bits only.
The AAC peak overshoot is an observation from one encode. We did not test other bit rates or encoders.

Frequently asked questions

How do I make audio line up with video cuts in Python?

Convert each cut time to a sample index with round(seconds * sample_rate), then write each note starting at that index in one long array. Do not build the track by concatenating segments, because each segment's rounding error accumulates. In our test, onsets detected from the finished audio landed 1.0 to 3.3 ms after the cuts.

Why does my generated audio sound clipped or crackle?

Summing notes can push the signal past full scale. When we mixed five overlapping notes, the raw peak was 1.272 and 2,786 samples went over 1.0. Casting that straight to 16-bit integers wraps the values around instead of clamping them. Normalizing the whole track to a peak of 0.8 before conversion removed the problem.

What causes clicks at the start of a new note?

A click comes from a sudden jump in the waveform, usually where the previous note is cut off while still loud. In our test, the biggest sample-to-sample step at a cut was 0.3292 without a tail fade and 0.0004 with a 50 ms linear fade. A short fade on the end of each note is enough.

How much does sample rounding drift matter?

It depends on how many segments you add. Rounding each 1/24-second segment to whole samples at 44,100 Hz added 0.5 samples per segment, which came to 11.34 ms of drift after 1,000 segments. Computing each position from the absolute time avoids that growth.

How do I add the generated track to the video?

Use ffmpeg with explicit stream selection: -map 0:v -map 1:a, copy the video with -c:v copy, and encode the audio as AAC. In our test, both streams in the output measured 11.000000 seconds. The -map flags matter if the video already contains an audio stream.