tips

Browser-Side Video Remuxing: fMP4, ISOBMFF, and Why We Don't Transcode

What actually happens when you 'merge' streaming segments into an MP4 in the browser — the ISOBMFF box structure, why fragmented MP4 concatenates but regular MP4 doesn't, and why remuxing is 50x faster than transcoding.
FlowPick Team
14 min read
# mp4 # isobmff # fmp4 # remux # wasm # ffmpeg # deep-dive

When FlowPick merges 300 HLS segments into a single MP4 in your browser, it's not transcoding. It's not re-encoding. It's not even looking at the pixels. It's remuxing — rewriting the container without touching the codec bitstream.

This matters because remuxing is roughly 50x faster than transcoding, runs in real-time even on a 5-year-old laptop, and produces zero quality loss. The catch: you have to actually understand what an MP4 file is, not treat it as a black box.

This article is the black-box removal.

What an MP4 file actually is

MP4 is a brand name for ISO Base Media File Format (ISOBMFF), defined in ISO/IEC 14496-12. It's a tree of nested "boxes" (also called "atoms"), each with a 4-character type and a length prefix:

ftyp   (file type — identifies this as MP4, declares brand)
moov   (movie metadata — codec config, track layout, timing)
  mvhd   (movie header — timescale, duration)
  trak   (track — one per video/audio/subtitle stream)
    tkhd   (track header)
    mdia   (media info)
      mdhd   (media header — track timescale)
      hdlr   (handler — declares "this is video" or "this is audio")
      minf   (media info)
        stbl  (sample table — where the actual frames live)
          stsd  (sample descriptions — codec config like SPS/PPS)
          stts  (time-to-sample — PTS for each frame)
          stsc  (sample-to-chunk — which chunk has which samples)
          stsz  (sample sizes)
          stco  (chunk offsets — where each chunk sits in the file)
mdat   (media data — the actual H.264/AAC bitstream)

The moov box tells players how to interpret the mdat box. Without moov, you have a pile of compressed bytes that no player can decode.

A regular MP4 has all of moov upfront (or at the end, requiring a second seek), and the mdat is one contiguous chunk. To add a frame, you'd have to rewrite stts, stsc, stsz, stco and shift bytes around. This is why "edit an MP4" was historically painful.

Fragmented MP4: the streaming-friendly variant

Fragmented MP4 (fMP4) splits the file into self-contained "fragments":

ftyp
moov   (still has codec config — stsd, but stts/stsc/stsz are empty)
moof   (movie fragment — sample table for THIS fragment only)
  tfhd  (track header for this fragment)
  trun  (track run — sample sizes, PTS, durations for this fragment)
mdat   (media data — samples for THIS fragment)
moof   (next fragment)
  ...
mdat
...

Each moof+mdat pair is a complete, self-describing unit. To concatenate two fMP4 fragments, you just append the bytes — no rewriting, no offset math. The trun box inside each moof carries the sample table for that fragment's samples only.

This is why HLS and DASH both moved to fMP4. Each segment is one moof+mdat pair (plus an init segment with ftyp+moov). Concatenating segments produces a valid fMP4 file.

What the init segment is

ftyp
moov
  mvhd
  trak (video)
    tkhd
    mdia
      mdhd
      hdlr (vide)
      minf
        stbl
          stsd  ← avc1 config (SPS/PPS for H.264)
          ...   (empty sample tables — they'll be filled by moof)
  trak (audio)
    ...

The init segment is the "header" — it declares codecs, track layout, and timescale, but contains no actual frames. Without it, the moof/mdat segments are uninterpretable.

This is why DASH's <SegmentTemplate initialization="init.mp4"> and HLS's #EXT-X-MAP:URI="init.mp4" both exist — they're pointing at the fMP4 init segment.

The remux algorithm

To merge fMP4 segments (DASH case — already fMP4):

  1. Write the init segment bytes (your output's ftyp+moov)
  2. For each media segment, write its moof+mdat bytes in order
  3. Done

That's it. The output is a valid fMP4 file. No codec touched, no re-encoding, no quality loss. CPU usage is roughly "memory bandwidth limited" — you're literally just memcpy-ing bytes.

async function mergeFmp4(initUrl, segmentUrls) {
  const init = new Uint8Array(await (await fetch(initUrl)).arrayBuffer())

  const chunks = [init]
  for (const url of segmentUrls) {
    chunks.push(new Uint8Array(await (await fetch(url)).arrayBuffer()))
  }

  // Compute total length
  const total = chunks.reduce((s, c) => s + c.byteLength, 0)
  const out = new Uint8Array(total)
  let offset = 0
  for (const chunk of chunks) {
    out.set(chunk, offset)
    offset += chunk.byteLength
  }
  return new Blob([out], { type: 'video/mp4' })
}

This 12 lines of code is most of what FlowPick does for DASH streams. The complexity isn't the merge — it's parsing the MPD to get the URLs in the first place (covered in the DASH deep dive) and handling the edge cases below.

The HLS case: MPEG-TS to MP4

HLS traditionally uses MPEG-TS, not fMP4. MPEG-TS is a different container — 188-byte packets, multiplexes audio and video together, has its own timing (PCR/PTS). You can't just concat .ts files into an MP4; you have to demux the TS, extract PES packets, parse PTS, and write fMP4.

This is where FFmpeg WASM earns its keep. FFmpeg's mpegts demuxer + mp4 muxer handles:

  1. Reading 188-byte TS packets, filtering by PID (video PID, audio PID)
  2. Reassembling PES packets from TS payloads
  3. Extracting PTS/DTS from PES headers
  4. Grouping access units (one video frame = one AAC frame for audio)
  5. Writing moof/mdat boxes with correct trun sample tables

The command-line equivalent:

ffmpeg -i input.ts -c copy -f mp4 output.mp4

-c copy is the key — it tells FFmpeg to copy the codec bitstream verbatim, not decode and re-encode. CPU usage is minimal.

In the browser, FFmpeg WASM does the same thing:

import { FFmpeg } from '@ffmpeg/ffmpeg'

const ffmpeg = new FFmpeg()
await ffmpeg.load()

// Write segments to FFmpeg's virtual FS
for (let i = 0; i < segments.length; i++) {
  await ffmpeg.writeFile(`seg${i}.ts`, segments[i])
}

// Concat list
const concatList = segments.map((_, i) => `file 'seg${i}.ts'`).join('\n')
await ffmpeg.writeFile('list.txt', new TextEncoder().encode(concatList))

// Remux
await ffmpeg.exec([
  '-f', 'concat', '-safe', '0', '-i', 'list.txt',
  '-c', 'copy',
  '-f', 'mp4', 'out.mp4'
])

const output = await ffmpeg.readFile('out.mp4')

For a 30-minute 1080p video with 300 segments, this runs in ~15-30 seconds on a modern laptop. The same operation via transcoding would take 10-20 minutes.

Why remux instead of transcode

Speed. Remuxing is memory-bandwidth-bound; transcoding is CPU-bound. A 4-core x86 transcodes 1080p H.264 at roughly 1-3x real-time (so a 30-min video takes 10-30 mins). Remux runs at 50-100x real-time.

Quality. Transcoding introduces generation loss — every re-encode degrades quality, even at "high" bitrates. Remux preserves the original bitstream byte-for-byte.

Battery. Transcoding burns CPU; remux doesn't. On a laptop, transcoding a 4K stream will drain your battery in 20 minutes. Remux barely registers.

Predictability. Transcoding time depends on the source's complexity (dark scenes encode faster than detailed ones). Remux time depends only on file size — linear and predictable.

The only reason to transcode is if you actually need a different codec — H.265 source to H.264 target for older devices, for instance. FlowPick doesn't do this because (a) every modern device supports H.264/AAC, and (b) the latency cost isn't worth it.

Edge cases that actually break naive concatenation

1. Discontinuities. HLS ad breaks use #EXT-X-DISCONTINUITY to signal that PTS resets. Concatenating segments across a discontinuity produces an MP4 where the player jumps backward in time. FFmpeg handles this; naive byte concatenation doesn't.

2. Codec changes between periods. DASH multi-period MPDs (ad + main content) can use different H.264 profiles per period. Concatenating these fMP4 fragments produces a file where the init segment's stsd doesn't match later fragments. Players either fail or show corruption.

3. Init segment mismatch. If you fetch the wrong init segment (e.g., for a different Representation), the codec config doesn't match your segments. Output file is unplayable.

4. Segment ordering. With parallel fetching, segments arrive out of order. You must reorder before concatenation. The DASH parallel fetch pattern handles this with an indexed array.

5. Moov atom at end of file. "Progressive download" MP4s (rare in streaming, common in direct downloads) put moov at the end. Without moov, you can't play the file until it's fully downloaded. FFmpeg's -movflags +faststart moves moov to the front. Always use this flag when muxing for download.

What's actually in an moof box

If you want to go deep, here's the byte layout of a typical moof:

moof (size: 0x00000400)
  mfhd (size: 0x00000010) — movie fragment header
    sequence_number: 0x00000001
  traf (size: 0x000003e0) — track fragment
    tfhd (size: 0x00000018) — track fragment header
      track_id: 1
      default_sample_duration: 0x000003e8 (1000 in timescale units)
    tfdt (size: 0x00000014) — track fragment decode time
      baseMediaDecodeTime: 0x00000000
    trun (size: 0x000003c0) — track run
      sample_count: 60
      sample 0: duration=1000, size=12345, flags=0x02000000
      sample 1: duration=1000, size=11876, flags=0x02000000
      ...

The flags field encodes whether each sample is:

  • A keyframe (0x02000000 = sync sample)
  • A non-keyframe (0x00010000 = non-sync)
  • Discontinuity (0x00000100)

The tfdt box gives the absolute decode time of the first sample in this fragment. Adding up durations in trun gives you each sample's PTS within the fragment.

This is what FFmpeg parses when it remuxes. If you ever want to write your own remuxer (don't, just use FFmpeg), this is the structure you'd need to write.

Original opinion: don't write your own muxer

There's a cottage industry of "lightweight" browser muxers that try to skip FFmpeg and write ISOBMFF by hand. I've looked at the source of most of them. Without exception, they handle ~80% of cases and break on:

  • HEVC streams (different NAL unit packaging)
  • Audio with gapless playback metadata
  • Streams with B-frames (DTS != PTS, ordering matters)
  • Variable framerate content
  • HDR metadata (Content Light Level, Mastering Display Color Volume)

FFmpeg is ~30 years of accumulated edge-case handling. Reimplementing even 50% of it in JS would take years. The cost of pulling in FFmpeg WASM (~25MB gzipped) is dramatically less than the cost of getting muxing wrong on real content.

The exception: if you're building a minimal "concat fMP4 segments" tool with zero edge cases, the 12-line merge function above works. FlowPick has both paths — simple concat for DASH, full FFmpeg remux for HLS with discontinuities.

References

Summary

The "merge segments into MP4" step looks like magic from the outside. It isn't — it's a structured remux that takes fMP4 fragments (which are designed to concatenate) and writes them sequentially with an init segment as the header. For HLS, you additionally demux MPEG-TS into fMP4 via FFmpeg WASM. Either way, you're copying codec bytes, not re-encoding them, which is why it's fast and lossless.

The hard parts aren't the remux — they're parsing the manifest, handling discontinuities, getting the init segment right, and dealing with DRM. Those are covered in the HLS and DASH deep dives.

The next article covers the lower-level browser APIs — WebCodecs, Web Workers, and OPFS — for when FFmpeg WASM isn't the right tool and you want to do real video processing in the browser.

Related articles

Recommended reading

Previous

How to Batch Download Images from Any Website

Next

Browser Video Processing Performance Benchmarks: FFmpeg WASM vs WebCodecs vs Native