MP4 and the ISO Base Media File Format
MP4 is a container derived from the ISO base media file format that stores encoded media with track descriptions and timing.
You drag the playback position forward in an Emby movie. The player now needs pictures and sound for a new time, but an MP4 file is a sequence of bytes. How does it get from that time request to the right encoded data?
An MP4 carries the media together with descriptions of its tracks, timing, and data locations. Following a seek makes the purpose of those descriptions visible. The example below uses a conventional nonfragmented layout.
Logical layout of a simple conventional file. The boxes contain descriptions and data; their drawn widths do not represent actual byte sizes.
1. Identify the selected track and its decoder configuration
The reader first needs to know which video and audio tracks are selected, how they are encoded, and how to configure their decoders. The movie description in the moov box contains the track information. Within each track, a sample-description table such as stsd identifies the coding arrangement and its configuration.
H.264 video and AAC audio are common in MP4, but the filename extension does not select a codec for every track. The actual sample entry supplies that identification.
A sample here is a coded media unit in the file-format sense. A coded audio sample can represent many decoded PCM amplitude samples. The reader needs the encoded unit's size and timing, then the audio decoder expands it into the corresponding sound.
2. Relate the requested time to a usable starting sample
The track has timing information for its samples. A conventional stts table describes their durations on the decoding timeline; ctts, when required, supplies offsets for the presentation timeline. These can differ because predicted pictures may need to be decoded in a different order from the order in which they appear on screen.
The reader uses that information to relate the requested time to the track's samples. For video with reference dependencies, it also needs a suitable starting point. A sync-sample table such as stss, where present, provides information about access points, while the codec's rules determine the required reconstruction path.
If that starting point is earlier than the requested picture, the decoder reconstructs the necessary intervening pictures. The player can then begin presentation at the intended position. Audio is located on its corresponding timeline so that both tracks reach the same presentation time.
This is why a seek does more than multiply the movie's average bitrate by the number of elapsed seconds. Encoded sample sizes vary, several tracks can be interleaved, and a requested picture may depend on data from an earlier point.
3. Turn the selected sample into a byte location
Sample-to-chunk information such as stsc describes how the track's samples are grouped into chunks. A chunk-offset table, stco or co64, gives the chunks' locations in the file. Sample-size information such as stsz supplies the size of each coded unit.
Consider a small constructed example. The reader has chosen sample 15 as its starting sample. The tables establish that chunk 5 begins at byte offset 40,000 and contains samples 13 through 16:
| Sample in chunk 5 | Encoded size | Calculated start offset |
|---|---|---|
| 13 | 1,000 bytes | 40,000 |
| 14 | 1,500 bytes | 41,000 |
| 15 | 900 bytes | 42,500 |
| 16 | 1,100 bytes | 43,400 |
To locate sample 15, add the lengths of the preceding samples in that chunk:
40,000 + 1,000 + 1,500 = 42,500
The reader can now obtain 900 bytes beginning at that offset. It did not need to decode samples 13 and 14 merely to discover where sample 15 was stored. Their lengths and the chunk location supplied the navigation information.
That byte lookup is separate from the decoder's reference requirements established in step 2. Locating a coded picture does not make it independently decodable. The tables get the right data to the right decoder; the codec determines how that data reconstructs a picture.
4. Decode the data and restore the presentation order
The selected samples are passed to the configured decoder in the necessary order. Decoded pictures are presented using their presentation times, and decoded audio is placed on the same presentation timeline.
For the user, the result is a new playback position. Underneath, the reader has connected four pieces of information: track configuration → timing and access point → byte locations → decoded presentation.
A damaged table can break that chain even when much of the encoded media remains intact. A wrong offset directs the reader to the wrong bytes; missing configuration can prevent decoding; incorrect timing can affect duration, ordering, or synchronization. Each failure concerns a different relationship in the file.
A small dictionary for the file structure
MP4 organizes information into boxes, each with a size and type. Some boxes contain others, producing a hierarchy. The ISO base media file format defines the shared structure on which MP4 is built.
| Box or table | What to look for |
|---|---|
| ftyp | Brands declaring file-format conventions |
| moov | Movie and track descriptions |
| mdat | Stored media data |
| stsd | Sample coding descriptions and configuration |
| stts / ctts | Decode durations and presentation offsets |
| stsc / stsz | Sample grouping and sizes |
| stco / co64 | Chunk locations |
| stss, where present | Sync-sample information |
The shared ancestry with QuickTime explains many familiar structures in MOV and MP4. The specific file conventions and supported coding arrangements still determine how a reader uses them.
What changes when the layout changes?
In a conventional file, the main description can be placed after the bulk media data. A sequential reader then needs access to that later description before it can interpret the presentation. Fast start moves the description earlier and updates affected offsets. It makes navigation information available sooner.
Fragmented MP4 supplies initialization data followed by descriptions and media for successive fragments. The lookup information for later samples arrives with those fragments, which suits segmented delivery and media still being produced. The conventional table walkthrough above therefore does not describe every fragmented-file lookup.
For an Emby item, inspect the packaging actually delivered to the app as well as the stored file. A server-prepared stream may organize the same encoded media differently. If the playback problem concerns duration or seeking, follow the timing and location descriptions. If the descriptions lead to the correct data but decoding fails, inspect the selected codec and its configuration. That follows the same chain the reader uses to reach the requested playback position.
See Also
- Containers and Codecs
- Codec Identifiers and FourCC
- MP4 Fast Start
- Fragmented MP4
- H.264 / AVC Video
- AAC-LC Audio