Searching MP4E Videos

Overview

An exported MP4E video that has been analysed carries a search card: a small, public, unencrypted description of what the video is about — its title, summaries, topics, the things in it, chapters, key moments, the phrases people would use to look for it, the questions it answers and the catalogue facts the creator publishes (tags, licence and rights). Hosts, search engines, media libraries and AI assistants can read it without the MP4E player and without any key.

The card carries descriptions only. The video's interactivity — overlays, plugins, rules, variables — stays in the protected MP4E metadata and is never part of the card.

Search inside a video

Viewers can also search inside a video while they watch: the official Search plugin (mp4e:search) adds a Search entry to the player's settings menu, finds chapters, key moments, questions, things, people, scenes and what is said, and jumps to the moment the viewer picks. It runs entirely in the player — on your site, offline, with no MP4E server — and reads only what the creator allowed. Nothing is needed on your side: a video the creator added Search to carries it.

• To narrow what Search (and every other plugin) may read on your site, set config.permissions.intelligence.
• To know when a viewer jumps to a result, listen for the plugin's search:selected event through onEvent — see Actions and events.
• To build your own in-video experience over the same data, write a plugin that reads it with mp4e.getIntelligence().

The rest of this page is about the other direction: finding the video itself, from outside the player.

What the creator controls

The card is built from the video's analysis (Make interactive in the Studio, or Analyse a video in the Agent API). A video that has not been analysed has no card. Two switches in the Studio, under Library panel → Settings → Search card, decide what an export carries:

• Include a search card in exported videos — on by default.
• Include the full transcript — off by default. When on, the card gains a transcript field with the spoken text, one line per sentence, and transcriptTimes with the time each line is said.

The Export dialog starts from these two switches and lets the creator leave the card, or just its transcript, out of a single export without changing the project. The card is written only into exported files. When a card is written, the exported file's title, description and long description tags are set from the project name and the analysis summaries, replacing any such tags the uploaded file carried. The analysis's chapters are added as a chapter track without changing any other track (an export with audio tracks is rebuilt from the first video track and the project's audio tracks, so data tracks such as timecode are not carried there); if the uploaded file already has chapters of its own, those are kept and the analysis chapters are not added to it (the card and XMP still list them). The project's stored original is never changed. With the card switched off, or without an analysis, an export carries no card: any card, XMP and chapters that MP4E itself wrote into the file before are removed, and the file's own tags and chapters are left as they are, apart from the private details an export leaves out (below).

What an exported file leaves out. Unless the creator turns on Include private details for that export, an exported file leaves out the project's private details — inside the protected metadata (private catalogue fields, owner/client/project, the recording place, device and time of day, hidden or private analysis items, the edit history; the text of scene dialogue and the analysis diagnostics is emptied) nor in the file's own tags: the camera's location, make, model and software, GPS telemetry tracks and any XMP packet, MP4E's own and foreign, are removed, and the file's creation times keep only their date. The frames and sound are not touched. Telemetry a camera writes as an ordinary subtitle track, and vendor data stored outside the movie header, are not recognised. A .mov upload also loses its own artist, comment, copyright and encoder tags (an .mp4 keeps them). Images and videos in the project's media library are carried exactly as they were uploaded, with any location or camera data of their own, and under their original file names; clean those files before adding them if that matters. A file whose own metadata cannot be cleaned with certainty is not exported; the creator exports it with private details instead.

Where the card lives

The card is written in four places in the MP4, so each kind of reader finds it where it already looks:

The card's fields

The JSON in the freeform atom. All times are seconds from the start of the video. The card reflects the creator's corrections to the analysis, and leaves out anything the creator hid or kept private. Its catalog holds only the catalogue facts that are public: a custom field appears only when the creator marks it public.

{
  "cardVersion": 2,
  "generatedFrom": "2026-10-09T10:00:00Z",
  "title": "Grandma's pierogi",
  "oneLine": "Grandma makes pierogi from scratch.",
  "paragraph": "In her kitchen a grandmother rolls dough, fills it with potato and cheese, and boils the dumplings.",
  "language": "en",
  "contentType": "cooking",
  "duration": 10.0,
  "topics": ["pierogi", "polish cooking"],
  "people": [{ "name": "Grandma", "role": "cook" }],
  "things": [{ "name": "Rolling pin", "category": "tool" }],
  "chapters": [
    { "title": "Making the dough", "start": 0.0 },
    { "title": "Filling the pierogi", "start": 4.0 }
  ],
  "keyMoments": [{ "title": "Adding the butter", "time": 6.5 }],
  "searchPhrases": ["grandma's pierogi recipe", "polish dumplings"],
  "questionsAnswered": [{ "question": "How long do pierogi boil?", "time": 8.0 }],
  "catalog": {
    "tags": ["family", "recipes"],
    "rights": { "license": "CC-BY-4.0" },
    "fields": [],
    "recordedOn": "2024-05-02"
  }
}

Reading it today

Every placement can be read with standard tools. Each command below was run against an exported MP4E video.

Chapters (ffprobe)

The chapter track, as any player sees it.

ffprobe -v error -show_chapters -of json video.mp4

{
    "chapters": [
        {
            "id": 0,
            "time_base": "1/1000",
            "start": 0,
            "start_time": "0.000000",
            "end": 4000,
            "end_time": "4.000000",
            "tags": {
                "title": "Making the dough"
            }
        },
        ...
    ]
}

Title and summaries (ffprobe)

The iTunes tags: ©nam as title, desc as description and ldes as synopsis.

ffprobe -v error -show_entries format_tags=title,description,synopsis -of json video.mp4

{
    "format": {
        "tags": {
            "title": "Grandma's pierogi",
            "description": "Grandma makes pierogi from scratch.",
            "synopsis": "In her kitchen a grandmother rolls dough, fills it with potato and cheese, and boils the dumplings."
        }
    }
}

XMP and tags (ExifTool)

The XMP packet — subjects and the chapter and key-moment markers in milliseconds — next to the iTunes tags.

exiftool -G1 -XMP-dc:all -XMP-xmpDM:all -ItemList:Title -ItemList:Description -ItemList:LongDescription video.mp4

[XMP-dc]        Title                           : Grandma's pierogi
[XMP-dc]        Description                     : In her kitchen a grandmother rolls dough, fills it with potato and cheese, and boils the dumplings.
[XMP-dc]        Subject                         : pierogi, polish cooking, grandma's pierogi recipe, polish dumplings
[XMP-xmpDM]     Tracks Track Name               : Chapters, Key moments
[XMP-xmpDM]     Tracks Track Type               : Chapter, Cue
[XMP-xmpDM]     Tracks Frame Rate               : f1000, f1000
[XMP-xmpDM]     Tracks Markers Start Time       : 0, 4000, 6500
[XMP-xmpDM]     Tracks Markers Name             : Making the dough, Filling the pierogi, Adding the butter
[XMP-xmpDM]     Tracks Markers Type             : Chapter, Chapter, Cue
[ItemList]      Title                           : Grandma's pierogi
[ItemList]      Description                     : Grandma makes pierogi from scratch.
[ItemList]      Long Description                : In her kitchen a grandmother rolls dough, fills it with potato and cheese, and boils the dumplings.

The whole card (Python)

The freeform atom with mutagen; prints the card JSON shown in The card's fields.

pip install mutagen

python3 -c "import json,sys; from mutagen.mp4 import MP4; print(json.dumps(json.loads(bytes(MP4(sys.argv[1]).tags['----:com.mp4e.search:card'][0])), indent=2))" video.mp4

The whole card (Node.js)

No dependencies: walks the MP4 boxes to the freeform atom and prints the card, or No search card in this file.

1// read-search-card.mjs — prints the search card of an MP4E video: node read-search-card.mjs video.mp4
2import { openSync, readSync, fstatSync } from 'node:fs';
3
4const fd = openSync(process.argv[2], 'r');
5const fileSize = fstatSync(fd).size;
6const read = (pos, len) => { const b = Buffer.alloc(len); readSync(fd, b, 0, len, pos); return b; };
7
8function* boxes(start, end) {
9  let pos = start;
10  while (pos + 8 <= end) {
11    const h = read(pos, 16);
12    let size = h.readUInt32BE(0), header = 8;
13    if (size === 1) { size = Number(h.readBigUInt64BE(8)); header = 16; } else if (size === 0) size = end - pos;
14    if (size < header) return;
15    yield { type: h.toString('latin1', 4, 8), start: pos + header, end: pos + size };
16    pos += size;
17  }
18}
19const child = (box, type, skip = 0) => box && [...boxes(box.start + skip, box.end)].find((b) => b.type === type);
20
21const moov = [...boxes(0, fileSize)].find((b) => b.type === 'moov');
22const ilst = child(child(child(moov, 'udta'), 'meta'), 'ilst', 4); // `meta` has 4 bytes of version/flags
23let card = null;
24for (const item of ilst ? boxes(ilst.start, ilst.end) : []) {
25  if (item.type !== '----') continue;
26  const parts = Object.fromEntries([...boxes(item.start, item.end)].map((b) => [b.type, read(b.start, b.end - b.start)]));
27  const mean = parts.mean?.toString('utf8', 4), name = parts.name?.toString('utf8', 4);
28  if (mean === 'com.mp4e.search' && name === 'card') card = JSON.parse(parts.data.subarray(8).toString('utf8'));
29}
30console.log(card ? JSON.stringify(card, null, 2) : 'No search card in this file.');