Agent API

Overview#

The Agent API lets a script, server, or agent of your own do — over plain HTTP, with an API key and no browser — everything a person does in the Studio's Agent panel: upload a video (which creates a project), ask for a change in plain English, follow the run to completion, answer a question it asks along the way, read what it changed, and download the resulting interactive MP4. A project you create or change this way is an ordinary MP4E project — open it in Studio with a browser and you see the exact same project, the exact same run history, side by side with anything a collaborator does there.

A project created through this API keeps version history on, so every change the agent makes is a revision you can list and undo. A project created in Studio may have history switched off; the agent cannot author one of those, and asks for a run on it come back 409 (see Errors).

Everything here builds on the same key you already use for project:read access — an API key just needs the additional scopes below.

Authentication#

Send the secret as a Bearer token, exactly as described on the API Keys page — no cookie, no query parameter. Every route below requires the scope listed for it; a request needs every scope its route requires.

curl "https://api.mp4e.com/api/v1/capabilities" \
  -H "Authorization: Bearer mp4e_sk_..."

A key restricted to specific projects (see API Keys → Creating a key) can never be used to create a NEW project — a project that does not exist yet can never be in that restriction, so the key could never use it either. Restrict a key after you know which project it is for, or leave it unrestricted for the account's own uploads.

Walkthrough#

One full pass — check the key, upload, start a run, follow it, answer its question if it asks one, and download the result. Every response below is shown trimmed to the fields this walkthrough uses; see Reference for the complete shape of each.

1. Check what the key can do

curl "https://api.mp4e.com/api/v1/capabilities" \
  -H "Authorization: Bearer mp4e_sk_..."

2. Upload a video — this creates the project.

curl -X POST "https://api.mp4e.com/api/v1/projects" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -F "file=@video.mp4"

{
  "videoId": "6c2a1a10-...",
  "title": "video.mp4",
  "revision": 2
}

revision is where the project starts; pass it as baseRevision on the next step if you want the run refused should anything change underneath you. Version history is turned on for the new project, and its first listed revision is the agent's first change — GET .../history is empty until then. A file exported with streamTimeline=true cannot become a project and is refused with 422; upload the original video, or an export made without it.

3. Start a run — describe the change in plain English.

curl -X POST "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../runs" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"request": "Add a button at the end that links to our site"}'

{
  "id": "6f2b7e2a-...",
  "status": "queued",
  "baseRevision": 2,
  "currentRevision": 2,
  "changes": [],
  "question": null,
  "source": "api",
  "createdAt": "2026-09-06T12:00:00Z",
  "updatedAt": "2026-09-06T12:00:00Z"
}

4. Follow the run — either poll from a cursor, or open a live stream. Both carry the same events.

curl "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../runs/6f2b7e2a-.../events?after=0" \
  -H "Authorization: Bearer mp4e_sk_..."

curl -N "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../runs/6f2b7e2a-.../stream" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -H "Accept: text/event-stream"

5. Answer, if the run asks a question — a run in awaiting_input carries a question and waits here until you reply.

curl -X POST "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../runs/6f2b7e2a-.../answer" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"text": "The speaker on the left"}'

6. Read the finished run

curl "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../runs/6f2b7e2a-..." \
  -H "Authorization: Bearer mp4e_sk_..."

{
  "id": "6f2b7e2a-...",
  "status": "completed",
  "stage": null,
  "request": "Add a button at the end that links to our site",
  "summary": "Added a button overlay in the final scene.",
  "error": null,
  "baseRevision": 2,
  "currentRevision": 3,
  "changes": [
    {
      "changeGroupId": "b1e2c3d4-...",
      "revision": 3,
      "description": "Added a \"Visit our site\" button overlay",
      "touched": ["layers", "overlays"],
      "undoneBy": null,
      "undoable": true
    }
  ],
  "question": null,
  "hints": null,
  "source": "api",
  "createdAt": "2026-09-06T12:00:00Z",
  "updatedAt": "2026-09-06T12:00:14Z"
}

7. Export and download

curl "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../export" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -o out.mp4

Want the same sequence as a script instead of curl? scripts/v1_walkthrough.py (in the MP4E backend repository) runs exactly these steps end to end — upload, run, poll, answer if asked, export — reading the key from an environment variable.

Reference#

Base URL https://api.mp4e.com. Every path below is relative to it; {videoId} and {runId} come from the upload and create-run responses above. Listing your projects and reading a project's header/summary are documented on the API Keys page (project:read) — not repeated here.

Media added this way is referenced from a project by media://<mediaId> — the reference the upload returns as ref, and the one the listing carries alongside each item's mediaId — never by a URL, so it keeps working in the exported file.

An upload accepts images (any image/* except image/svg+xml), audio (audio/*), video/mp4 and application/pdf, up to 25 MiB per item — anything else is a 415, anything larger a 413.

GET /api/v1/capabilities — the operations this API has, and which of them the calling key may use. No scope of its own: a caller must be able to see what it lacks without already holding it.

{
  "operations": [
    {
      "id": "create_run",
      "method": "POST",
      "path": "/api/v1/projects/{videoId}/runs",
      "scope": "agent:author",
      "meaning": "Ask the agent to change the project, in plain English.",
      "allowed": true
    }
  ],
  "pluginPolicy": "official",
  "understandingProvider": "gemini"
}

pluginPolicy is which marketplace plugins the key's account may use — official (the mp4e namespace, the default), signed (official plus plugins signed by a signer MP4E trusts — today the same set as official, since only the platform signer is trusted) or any (every published plugin). A plugin outside it is left out of what the agent is shown, and installing one is refused whatever scopes the key holds. understandingProvider is the model that analyses the account's videos (see Analyse a video) — set per account by MP4E, gemini by default; it cannot be chosen per request.

POST /api/v1/projects — upload a video and create a project from it, multipart form data. A project-restricted key is refused before the upload is even read. The new project has version history on, so the agent can record and undo its changes there.

multipart/form-data
  file: <the video>

{
  "videoId": "6c2a1a10-...",
  "title": "video.mp4",
  "revision": 2
}

GET /api/v1/projects/{videoId}/history — the project's revisions, newest first. Optional ?limit= (default 50, max 200). A project created through this API has no revisions until its first change lands, so this is an empty list until then.

{
  "versions": [
    {
      "revision": 3,
      "description": "Added a \"Visit our site\" button overlay",
      "createdAt": "2026-09-06T12:00:14Z",
      "source": "api",
      "runId": "6f2b7e2a-..."
    }
  ]
}

POST /api/v1/projects/{videoId}/runs — ask the agent to change the project.

{
  "request": "Add a button at the end that links to our site",
  "baseRevision": 2,
  "hints": null
}

request is required (plain text, up to 4000 characters). baseRevision is optional — when given, it must match the project's current revision or the run is refused with a 409; when omitted, the run starts against whatever revision is current. hints is not used by an external caller — leave it out. Response is the run shape shown in the walkthrough above, with status: "queued".

GET /api/v1/projects/{videoId}/runs — this project's runs, oldest first. Optional ?limit= (default 50, max 200). Response: { "runs": [<run>, ...] }, each entry the same run shape as GET .../runs/{runId}.

GET /api/v1/projects/{videoId}/runs/{runId} — one run's current status, changes, summary, and pending question (see the walkthrough's step 6 for the full shape). A run in awaiting_input carries a non-null question:

{
  "question": "Which product should the button link to?",
  "options": ["Speaker", "Lamp", "Headset"]
}

GET /api/v1/projects/{videoId}/runs/{runId}/events — poll the run's progress from a cursor. Optional ?after= (default 0) and ?limit= (default 500, max 1000). Always send the response's cursor back as the next request's after — it is not always the same as the last event's seq.

{
  "events": [
    { "seq": 12, "type": "status", "payload": { "status": "running" }, "createdAt": "2026-09-06T12:00:01Z" },
    { "seq": 15, "type": "progress", "payload": { "text": "Reading the project" }, "createdAt": "2026-09-06T12:00:03Z" },
    { "seq": 20, "type": "change", "payload": { "changeGroupId": "b1e2c3d4-...", "revision": 3, "description": "Added a \"Visit our site\" button overlay", "touched": ["layers", "overlays"] }, "createdAt": "2026-09-06T12:00:13Z" },
    { "seq": 21, "type": "status", "payload": { "status": "completed" }, "createdAt": "2026-09-06T12:00:14Z" }
  ],
  "cursor": 21
}

Event type is one of status, progress, change, question, answer, or error (an error payload is reduced to { "message": "..." }).

GET /api/v1/projects/{videoId}/runs/{runId}/stream — the same events, live, as server-sent events. Optional ?after=; a Last-Event-ID header (for reconnecting) takes priority over it. A run snapshot is sent first, then every event as it lands, then a fresh snapshot after every status change; the connection closes once the run reaches a terminal status. Opening a stream for a run that is already terminal replays its remaining history once and closes immediately.

POST /api/v1/projects/{videoId}/runs/{runId}/answer — answer the question a run in awaiting_input is waiting on. Body { "text": "..." } (required, up to 4000 characters). Response is the run shape, now back to queued.

POST /api/v1/projects/{videoId}/runs/{runId}/cancel — stop a run. No body. Changes already made stay. Response is the run shape.

POST /api/v1/projects/{videoId}/runs/{runId}/changes/{revision}/undo — undo one revision this run made, restoring the document from before it as a new revision. No body. Only the run's LAST landed revision can be undone, only once the run has finished, and only while that revision is still the project's current one — anything else is a 409. Response is the run shape, reflecting the new current revision.

GET /api/v1/projects/{videoId}/export — download the interactive MP4, with the same content protection (encrypted and signed metadata) as a Studio export. Optional ?withAudio=true, ?embedEngine=true|false, ?searchCard=true|false and ?searchCardTranscript=true|false — the last two override the project's settings.searchCard for this export only (omitted, the project decides; the transcript is never written without the card). ?includePrivate=true keeps the project's private details in this export (private catalogue fields, the recording place, device and time of day, hidden and private analysis items, the edit history, the camera's own tags and GPS tracks — except that an export with withAudio=true is rebuilt from the video and audio tracks and never carries the camera's tags or GPS tracks); omitted or false, they are left out. A file whose own metadata cannot be cleaned with certainty is refused with 422 — export it with includePrivate=true instead. ?streamTimeline=true|false overrides the project's settings.streamTimeline (off by default) for this export only: on, the file's metadata is a small signed start document and the object tracking is stored beside it as encrypted chunks rather than inside the metadata. The export is refused with 503 when the server cannot encrypt and sign it, and with 422 when the stored chunks do not read back exactly as written. Response is the file itself (video/mp4, Content-Disposition: attachment) — no JSON envelope.

Analyse a video#

MP4E can watch and listen to a project's video and write down what it is about: a summary at three lengths, its topics and any content warnings, the scenes described in prose (what happens, who is there, what is said), chapters grouping those scenes, the people in it, the key moments worth a marker or a card, and the things on screen or mentioned, each with its category (such as tool or ingredient, or one of the lenses the creator asked it to look for, such as Animals, with the details that lens asks about). Every time in it is checked against the video's shot cuts and spoken sentences before it is saved. The result is saved into the project's intelligence section, where the agent reads it — so a request like "add chapter markers" or "show a card when the blender appears" can use what the analysis found. When the project has no subtitles yet, the transcript is added as its original subtitle track.

An analysis is paid work: start it only when it is wanted. It runs in the background, typically for about a minute, and only one analysis can be queued or running per project at a time. The model that does it is set for your account by MP4E (understandingProvider in GET /api/v1/capabilities); there is no field to choose one. A new analysis keeps the lenses (what to look for) of the previous one; the creator sets them in the Studio.

POST /api/v1/projects/{videoId}/understand — start an analysis. No body. Answers 201 at once with the queued job. While another analysis is queued or running for the project it answers 409 instead, and the body's activeJobId is that job's id — poll it rather than starting again.

curl -X POST "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../understand" \
  -H "Authorization: Bearer mp4e_sk_..."

{
  "id": "9d41c0e2-...",
  "status": "queued",
  "stage": null,
  "progress": 0,
  "message": null,
  "error": null,
  "createdAt": "2026-10-08T12:00:00Z",
  "updatedAt": "2026-10-08T12:00:00Z",
  "finishedAt": null,
  "result": null
}

GET /api/v1/projects/{videoId}/understand/{jobId} — the job as it stands. Poll it every few seconds until status is terminal.

{
  "id": "9d41c0e2-...",
  "status": "succeeded",
  "stage": "done",
  "progress": 100,
  "message": "The analysis is ready",
  "error": null,
  "createdAt": "2026-10-08T12:00:00Z",
  "updatedAt": "2026-10-08T12:01:04Z",
  "finishedAt": "2026-10-08T12:01:04Z",
  "result": {
    "intelligenceVersion": 7,
    "diagnostics": [
      { "code": "time.snapped", "detail": "chapters[2].startTime 83.60 s snapped to the skeleton anchor 83.40 s" }
    ]
  }
}

• status goes queued → running → succeeded, failed or cancelled; those three are terminal.
• While it runs, stage and progress move through skeleton (10), analyse (40), validate (80) and save (95) to done (100), and message says what is happening in words.
• A failed job carries its reason in error as a sentence, and nothing was written to the project.
• On success, result.intelligenceVersion is the project revision the analysis was saved as, and result.diagnostics lists the corrections made to the model's times. Read the analysis itself with GET /api/v1/projects/{videoId}/document?sections=intelligence.

When the job has succeeded the response also carries intelligence: the video's analysis as you (the owner) see it, with your own corrections applied and items you hid left out. Items you marked private are included with private: true; every item says whether it came from the analysis (source: "ai") or from you ("human"). intelligence.catalog is the project's catalogue as its owner sees it: tags, rights, owner, client and project, the camera data (when and where it was recorded, and on what), and every custom field, private ones included — or null when the project has none.

Make a video interactive#

Once a video has been analysed, MP4E can build interactive features into it from what the analysis found: chapters to jump between, a step checklist that ticks itself off, highlight markers, info and glossary cards, product cards, a quiz written from the transcript, a call to action, subtitles. A recipe is a ready-made combination of those features for a kind of video (How-to / Recipe, Lesson / Training, Product / Shopping, Tour / Showcase). You choose which features to build and a look for them; MP4E builds them in one agent run, each on a layer of its own, using plugins from the marketplace. The Studio's Make interactive page does the same through a wizard.

A build is paid work, like any agent run: it spends agent credits, and is refused for the same reasons a run is. It needs the video to have been analysed first (Analyse a video above) and keeps the project's version history on — a build turns it on when it is off and saves a version first if the current state has none, so the Studio can undo the whole build.

GET /api/v1/projects/{videoId}/make-interactive/recipes — what this video can have. Answers 400 ("Analyse the video first.") until the video has an analysis.

curl "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../make-interactive/recipes" \
  -H "Authorization: Bearer mp4e_sk_..."

{
  "suggested": "recipe-howto",
  "analysedAt": "2026-10-10T12:00:00Z",
  "recipes": [
    {
      "slug": "recipe-howto",
      "name": "How-to / Recipe",
      "description": "Chapters to jump between, a step checklist that ticks itself off, and tips as they come up.",
      "features": [
        { "id": "chapters", "label": "Chapter navigation", "plugin": "mp4e:cuts",
          "reason": "Chapters found: 6", "available": true, "count": 6,
          "defaultOff": false, "note": null },
        { "id": "steps", "label": "Step checklist", "plugin": "mp4e:steps",
          "reason": "Steps found: 9", "available": true, "count": 9,
          "defaultOff": false, "note": null },
        { "id": "info-cards", "label": "Info cards", "plugin": "mp4e:info-card",
          "reason": "Facts found: 0", "available": false, "count": 0,
          "defaultOff": false, "note": null }
      ]
    }
  ],
  "catalogue": [ { "id": "quiz", "label": "Quiz", "plugin": "mp4e:quiz",
                   "reason": "Written from the transcript; chapters: 6", "available": true, "count": 6,
                   "defaultOff": false, "note": null } ],
  "themes": [
    { "id": "clean", "label": "Clean", "look": "Light glass surfaces, soft shadows",
      "swatch": { "surface": "rgba(250, 251, 252, 0.86)", "text": "#0e1621", "accent": "#1d5fd6" } }
  ]
}

• analysedAt (string or null) is when the analysis this answer was computed from was made (intelligence.generatedWith.at). After a new analysis, ignore an answer whose analysedAt is older.
• suggested is the slug of the recipe that fits the video best (by its content type, else by how many of its features the video can have); null when no recipe offers anything.
• Each recipe lists its features in order. A feature is available only when the analysis holds what it needs — a step checklist needs steps, product cards need products, a quiz needs chapters and a transcript, subtitles need a transcript short enough to be written out in full. reason says what was found, and count how many; for subtitles withheld because the transcript is too long, it says to add them from the Localization panel instead. Items you hid in the analysis do not count.
• defaultOff (boolean) says the feature is offered but should start unticked, and note (string or null) says why. This applies to subtitles on a video whose picture already shows captions, where note reads "This video already shows captions". Such a feature is still available and can be built when you ask for it; leave it out of features to follow the default.
• requires (a list of feature ids, often empty) names the features a feature needs: they must be built with it or already be in the project. The shopping cart (cart) requires product-cards, and is offered when the analysis read a price for at least one product.
• catalogue is every feature the video can have from any recipe, each once — the pool to add from when you want a feature the chosen recipe does not include.
• themes are the looks you can ask for, each with the colours of its surface, text and accent.

POST /api/v1/projects/{videoId}/make-interactive — start a build. Answers 201 at once with the run, in the same shape as POST …/runs, plus kind ("build") and verification (null until the run completes).

curl -X POST "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../make-interactive" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"recipe": "recipe-howto", "features": ["chapters", "steps"], "theme": "clean"}'

{
  "id": "7094cf9e-...",
  "status": "queued",
  "stage": null,
  "request": "Make this interactive: Chapter navigation, Step checklist",
  "summary": null,
  "error": null,
  "baseRevision": 3,
  "currentRevision": null,
  "changes": [],
  "question": null,
  "source": "api",
  "kind": "build",
  "verification": null,
  "createdAt": "2026-10-10T12:47:29Z",
  "updatedAt": "2026-10-10T12:47:29Z"
}

A build that tracks products first answers 201 with { "trackJob": { "id", "status", "stage", "progress", "message", "result", "error" } } instead of the run. Follow it with GET …/make-interactive/track/{jobId}, which answers { "job": { … }, "runId": "…" } — runId appears once tracking has ended and started the build run; from there the run is followed like any other. Tracking that fails does not stop the build: the run starts without objects. The recipes answer says what tracking would do, in tracking: { "eligible": [{ "entityId", "name" }], "category": "product", "estimateSeconds": … }, and each feature carries canUseObjects. When the account may not track (video analysis is not enabled for it, or the install's monthly analysis budget is spent), tracking also carries refused, that reason as a sentence. A build asked to track then still starts: it runs without tracking, and its run carries trackingSkipped, one sentence saying why.

When product cards are among the features and tracking found products, each tracked product opens its own card when a viewer taps it, and by itself while it is the only product on screen — the card shows its name, description and price, with no button, in the chosen theme, and the products show the chosen cue. This is set up before the build run starts; the run then adds timed cards only for the products that were not tracked. Card settings you made yourself for the products group are kept. If the cards cannot be set up, the build carries on and gives every product a timed card.

With the shopping cart (cart) among the features, the cart is also set up before the build run starts: the Cart on a layer of its own, connected to the product cards, whose button becomes Add to cart for every product with a price (checkout happens on your site — see In-video shopping). If it cannot be set up — for example the video already has a Cart of its own — the build still runs, without it, and says why: the run answer carries { "cart": { "written": false, "reason": "…" } } (for a build that tracks first, the tracking job's result.cart does).

baseRevision is the revision the project had just before the build. The run is then followed exactly like any other — GET …/runs/{runId}, its events or stream, answer if the run stops to ask a question, cancel. Undoing the whole build in one step is a Studio action (Undo the whole build); over the API you revert the build change by change with the run undo route, latest first. One project runs one agent run at a time, a build included.

When a build completes, verification says whether it did what was asked: { "ok": true, "missing": [], "errors": [] }. A feature that did not end up on a layer of its own is listed in missing (by id) — tracked product cards count when the products still open their own card, plus a layer for any products that were not tracked — and any problem the build introduced into the project is in errors as a sentence — among them a second scene group of the analysis' chapters when the project already had one (chapter navigation reuses it). A build whose check finds a problem still ends completed — read verification, and refine the result with an ordinary run. A build may also report warnings, which do not make it fail: { "code": "productCards.trackedAlsoTimed", "names": [ … ] } means timed cards were also added for products that already open their own card on tap.

A build is refused before anything changes when:

• the recipe slug is not one of the recipes (404);
• features is empty, repeats an id, names an unknown feature, or names one this video cannot have (400, the message names them);
• the video has not been analysed, or theme is not one of the themes (400);
• a chosen feature needs another that is neither chosen nor in the project (400, e.g. "Shopping cart needs Product cards: choose it too");
• a chosen feature is already on the project from an earlier build (400 — remove it in the Studio's What was added list to build it again);
• expectedRevision is stale (409);
• a run is already in flight, or the account is not eligible to run the agent or is out of credits (409 / 403, as for a run);
• an earlier build of the project is still tracking its products (409; the body's activeJobId is that tracking job — follow it with the track route).

Search your library#

GET /api/v1/search?q=<words> (scope project:read) finds your analysed videos by what happens in them. It matches the words of q against each video's title, summaries, topics, search phrases, chapters, key moments, people, things (their names, categories and lens details) and the questions it answers, and against its catalogue: tags, owner, client and project, and custom fields; videos that hold more of the words rank first. Only videos that have been analysed can be found, transcripts are not searched, and a key restricted to some projects only finds those. The Studio's search-card switch does not affect this search — it only controls what exported files publish.

curl "https://api.mp4e.com/api/v1/search?q=melt%20the%20cheese&limit=3" \
  -H "Authorization: Bearer mp4e_sk_..."

{
  "query": "melt the cheese",
  "results": [
    {
      "videoId": "d92313d1-...",
      "title": "acceptance-grilled-cheese-CC-BY-The-Food-Shop",
      "oneLine": "A home cook demonstrates how to make the ultimate classic American grilled cheese sandwich using sourdough bread and a blend of three cheeses.",
      "contentType": "cooking",
      "duration": 269.4,
      "score": 12.0,
      "matches": [
        { "kind": "chapter", "title": "Bread and Cheese Selection", "time": 11.71 },
        { "kind": "keyMoment", "title": "Cheese Trio Introduction", "time": 23.21 },
        { "kind": "chapter", "title": "Grating Cheese Hack", "time": 28.92 }
      ]
    }
  ]
}

• q is required (a blank one is 400) and at most 200 characters; limit is 1–50, default 10.
• results are best first; score only orders them. An empty list means nothing matched — try other words.
• matches holds up to three chapters (chapter), key moments (keyMoment), answered questions (question) or things (thing) that contain the words, each with its time in seconds. A thing matches by its name or its category (which it carries), at the time it is first on screen; a thing that is only mentioned is never a match.

Installing your own plugin#

When you already know exactly what a piece of interactivity should look like — its markup, its styling, its behaviour — write it yourself and install it, instead of spending the agent's turns having it author one. What you install is an ordinary custom user: plugin: the same entry the Studio's plugin editor writes, editable there afterwards, and carried inside the exported MP4 like every other plugin. Once it is in the project, a run uses it by ref.

POST /api/v1/projects/{videoId}/plugins (scope library:install) — the request body is the plugin definition, with one reserved key: expectedRevision, an optimistic lock. When you send it, the install is refused with a 409 if the project has moved on since that revision.

curl -X POST "https://api.mp4e.com/api/v1/projects/6c2a1a10-.../plugins" \
  -H "Authorization: Bearer mp4e_sk_..." \
  -H "Content-Type: application/json" \
  -d @badge.json

{
  "identifier": "badge",
  "name": "Hello Badge",
  "description": "A small pill that shows a line of text.",
  "namespace": "user",
  "type": "overlay",
  "category": "information",
  "version": "1.0.0",
  "content": {
    "html": "<div class='badge' id='badge'>{{text}}</div>",
    "css": ":root { --accent: {{accentColor}}; } .badge { display:inline-flex; align-items:center; padding: calc(8px * var(--s,1)) calc(14px * var(--s,1)); border-radius:999px; background:var(--accent); color:#001014; font: 600 calc(14px * var(--s,1))/1 system-ui, sans-serif; cursor:pointer; }",
    "script": "var badge = document.getElementById('badge'); badge.textContent = mp4e.getConfig().text; mp4e.onConfigUpdate(function (config) { badge.textContent = config.text; }); badge.addEventListener('click', function () { mp4e.emit('badgeClicked', { text: mp4e.getConfig().text }); });"
  },
  "configSchema": {
    "type": "object",
    "properties": {
      "text": { "type": "string", "title": "Text", "default": "Hello" },
      "accentColor": { "type": "string", "title": "Accent colour", "default": "#00e5ff" }
    },
    "required": ["text"]
  },
  "defaultConfig": { "text": "Hello", "accentColor": "#00e5ff" },
  "emits": {
    "badgeClicked": { "label": "Badge tapped", "description": "Fired when the viewer taps the badge." }
  }
}

{
  "ref": "user:badge",
  "revision": 12,
  "replaced": false
}

Installing the same ref again replaces that entry and answers replaced: true — that is how you iterate on a plugin. There is no delete from a key: removing a plugin is a Studio action.

content is what runs in the plugin's sandboxed iframe: {{configKey}} interpolation reaches the HTML and the CSS, while the script reads its configuration through the mp4e bridge (template syntax does not work there). See Creating Plugins for the definition's parts and sizing, Plugin API Reference for the mp4e calls, Events & Communication for emitting events and binding rules to them, and Template Syntax for the interpolation syntax.

A definition is checked before anything is written:

• namespace must be user, or left out — it is set for you. mp4e:, core: and community: plugins are published through the marketplace and are refused here.
• identifier is 1–64 characters of lowercase letters, digits, underscore or hyphen, starting with a letter or a digit. The plugin's ref is user:<identifier> — the key of the entry, and the type an overlay names to use it.
• type is one of overlay, object-display, modal, service, subtitle or controls.
• name is required; category and version are optional.
• content.html, content.css and content.script must be strings — a part you leave out is stored empty.
• configSchema, defaultConfig, events and emits must be JSON objects when present; actions, variables and outputs must be JSON arrays.
• id and source are set for you (user:<identifier> and user), and a key the format does not define is dropped rather than stored.
• The entry must match the format's shape for a plugin, and the project the change would produce must load in the engine.
• The serialised entry must stay under 256 KiB — the plugin travels inside the MP4.

Nothing is written unless every check passes — a refused definition leaves the project's revision exactly where it was.

GET /api/v1/projects/{videoId}/plugins (scope project:read) — every plugin the project carries, each with the ref an overlay names. Bodies are withheld, so this stays a listing. source says where an entry came from — user for one installed this way or authored in Studio, marketplace for one installed from the marketplace.

{
  "total": 2,
  "items": [
    { "ref": "user:badge", "name": "Hello Badge", "type": "overlay", "category": "information", "source": "user", "version": "1.0.0" },
    { "ref": "user:price-tag", "name": "Price Tag", "type": "object-display", "category": "commerce", "source": "user", "version": "1.2.0" }
  ]
}

Once it is installed, name the ref in your run request — the agent uses a plugin by ref: "add a user:badge overlay top-left for the whole video".

Author directly#

A project's interactivity is a JSON document, and everything the Studio's clicks produce is an edit to it. With the project:edit scope you can produce those edits yourself instead of describing them to the agent — useful when you know exactly what the document should say, or when you are generating many projects from a template. Direct edits and agent runs share one revision chain and mix freely.

The loop is four calls:

1. GET /api/v1/projects/{videoId}/document?sections=… — read the sections you want to change, and note the revision.
2. GET /api/v1/schema/{kind}/{name} — look up the exact shape of anything you are unsure of.
3. POST /api/v1/projects/{videoId}/changes — send the sections back, changed, quoting that revision as expectedRevision.
4. POST /api/v1/projects/{videoId}/diagnose — ask what the engine thinks, before or after committing.

Reading the document#

GET /api/v1/projects/{videoId}/document (scope project:read) returns the project's header, a count of what each section holds, and — for any section named in ?sections=a,b — its full content.

curl -s -H "Authorization: Bearer $MP4E_API_KEY" \
  "https://api.mp4e.com/api/v1/projects/$VIDEO_ID/document?sections=variables,layers"

{
  "videoId": "6f0c…",
  "title": "Product tour",
  "revision": 7,
  "historyEnabled": true,
  "summary": { "variables": 0, "layers": 1, "scenes": 3, "plugins": 2 },
  "sections": {
    "variables": { "complete": true, "content": [] },
    "layers": { "complete": true, "content": [ { "id": "layer_main", "name": "Main", "order": 0, "enabled": true, "overlays": [] } ] }
  }
}

Each named section says whether you are holding all of it. A very large section comes back with "complete": false, its content truncated to text and a note saying how much was dropped — and a section in that state cannot be replaced: a whole-section write would delete the part you never read, so the change route refuses it. Edit such a section in the Studio, or ask the agent for that change instead.

Two reads are separate from the document because they are paged: GET …/objects?offset=&limit= (the detected-object registry) and GET …/timeline?from=&to=&limit= (which objects are visible in which frames).

Looking up a shape#

GET /api/v1/schema/{kind} lists every name defined for one kind — action, condition, variableType, event, section or referenceType — and GET /api/v1/schema/{kind}/{name} returns one entry's full schema. Add ?names=a,b,c to the listing path to fetch up to twenty at once; an unknown name there comes back under unknown with suggestions rather than failing the call. These need no scope beyond a live key — the format's vocabulary is public.

curl -s -H "Authorization: Bearer $MP4E_API_KEY" \
  https://api.mp4e.com/api/v1/schema/variableType/sequence

The sections themselves are documented in prose under Metadata — variables, overlays, scenes, rules, objects, subtitles and the full schema.

Applying a change#

A complete example: a sequence variable that picks a step name from the current frame, and one core:text overlay whose config interpolates it. Both sections are sent whole — the project had no variables and one empty layer, which is what the read above returned at revision 7.

{
  "expectedRevision": 7,
  "description": "Add a step caption driven by the timeline",
  "sections": {
    "variables": [
      {
        "id": "var_step_name",
        "name": "stepName",
        "type": "sequence",
        "source": "currentFrame",
        "match": "range",
        "fallback": "",
        "entries": [
          { "from": 0,   "to": 149, "value": "Unbox it" },
          { "from": 150, "to": 299, "value": "Charge it" },
          { "from": 300, "to": 449, "value": "Pair it" }
        ]
      }
    ],
    "layers": [
      {
        "id": "layer_main",
        "name": "Main",
        "order": 0,
        "enabled": true,
        "category": "overlay",
        "overlays": [
          {
            "id": "overlay_step_caption",
            "name": "Step caption",
            "type": "core:text",
            "zIndex": 10,
            "position": { "x": 50, "y": 88, "xUnit": "%", "yUnit": "%" },
            "size": { "width": 320, "height": 48, "widthUnit": "px", "heightUnit": "px" },
            "visibility": { "fromFrame": 0, "toFrame": 449 },
            "config": { "text": "{{stepName}}" }
          }
        ]
      }
    ]
  }
}

{
  "revision": 8,
  "noChange": false,
  "findings": [],
  "touched": [
    { "kind": "variable", "id": "var_step_name", "op": "added" },
    { "kind": "overlay", "id": "overlay_step_caption", "op": "added" }
  ]
}

revision is the new revision — quote it as expectedRevision on your next change. touched says what moved, and findings carries any warnings the engine had about the result (a change that produces an error finding is refused instead — see below). A change that resolves to no edit at all answers 200 with "noChange": true and commits nothing: the revision it reports is whatever was already there.

Every change that commits lands as a numbered revision recorded assource: "api", so it appears in GET …/history and a Studio user can undo it like any other edit.

Checking a change first#

POST /api/v1/projects/{videoId}/diagnose (scope project:edit) reports what the engine would reject or silently skip. With no body it judges the project as it stands; with { "sections": … } it judges the project those sections would produce — the same check the change route makes, without committing anything.

{
  "findings": [
    { "severity": "warning", "code": "overlay_never_visible", "path": "layers[0].overlays[1]",
      "message": "This overlay's frame range ends before it starts" }
  ]
}

What gets refused#

• 400 — expectedRevision missing or not an integer; sections missing, not an object, or empty; description not a string or longer than 500 characters; allowObjectDeletion sent as anything but a real JSON boolean (the string "false" is refused, not read as consent); a section name the format does not define, or a section whose shape it rejects. The message names what to fix.
• 409 — a stale expectedRevision (the body carries the revision the project is actually at); an agent run in progress on the project (apply the change before starting a run, or wait for it to finish); a section too large to have been read whole, which is the same limit that made the read say "complete": false; a change that would delete detected objects without "allowObjectDeletion": true — the body names the ids, and those rows are deleted for good, so undo does not bring them back; or a project whose version history is off.
• 413 — the request body is over 1 MiB, or a section you are proposing is itself larger than a read could ever return whole (it would be unreplaceable from then on).
• 422 — the engine refuses to load the project this change would produce. The body carries findings alongside the message: fix what they name and send the change again. Nothing is written, and the revision does not move.

Errors#

Limits#

• An upload has a maximum size; a larger file is refused with 413.
• A project may have only one run in flight at a time — start the next one after the current run reaches a terminal status (completed, failed, or cancelled).
• An account has a monthly credit balance for agent runs; a run started with none left is refused with 403.
• A project may have only one analysis queued or running at a time — start the next one after the current job is succeeded, failed, or cancelled (a refused start's 409 carries the current job's id as activeJobId).

Use it from an MCP client#

Everything above is also available as a set of tools for any client that speaks the Model Context Protocol — Claude Code, Claude Desktop, or your own MCP client — instead of writing the HTTP calls yourself. The tools are a thin layer over the same routes documented above: same key, same scopes, same limits, same errors. It is available to pilot testers from a checkout of the repository; the server needs pip install mcp httpx and no other project dependency.

Claude Code — your client launches the server from whatever directory it happens to be in, so give it an absolute path to a Python that has mcp and httpx, and point PYTHONPATH at the backend directory (-m honours it):

claude mcp add mp4e \
  -e MP4E_API_KEY=mp4e_sk_... \
  -e MP4E_API_BASE=https://api.mp4e.com \
  -e PYTHONPATH=/abs/path/to/MP4E/studio/backend \
  -- /abs/path/to/MP4E/venv/bin/python -m mcpserver

Do not add --scope project: that writes the entry — key included — into a .mcp.json file in the repository. Note that claude mcp add -e stores the key in plain text in ~/.claude.json and leaves it in your shell history, so use a short-lived key and revoke it when you are done.

Claude Desktop — add an entry to claude_desktop_config.json. Same rule: command is an absolute path to a Python that has mcp and httpx, never a bare python.

{
  "mcpServers": {
    "mp4e": {
      "command": "/abs/path/to/MP4E/venv/bin/python",
      "args": ["-m", "mcpserver"],
      "cwd": "/abs/path/to/MP4E/studio/backend",
      "env": {
        "MP4E_API_KEY": "mp4e_sk_...",
        "MP4E_API_BASE": "https://api.mp4e.com"
      }
    }
  }
}

In both cases your client launches the server as a local process with the key you give it in env — the key never appears in a tool result or error message.

Connect a cloud agent — a client that runs in the cloud (an assistant with its own connector or plugin settings) cannot launch a local process, so point it at the hosted endpoint instead. It serves the same tools as the local server; give it the URL and one static header:

Put the key in the connector's header field, never in a prompt. Creating a project, adding media to one and exporting one read and write files on the machine running the server, so over this connection they go through the API routes above instead — upload a video with POST /projects, an asset with POST /projects/{videoId}/media, and download with GET /projects/{videoId}/export.

Available tools:

• See what the agent can change — the same capabilities listing the API returns.
• List or read projects — see every project the key can reach, or one project's details.
• Search your videos — find analysed videos by what happens in them, with the chapters, key moments, answered questions and things that match; see Search your library.
• Create a project — upload a local video file and start a new project from it.
• Add media to a project — upload a local image, sound or document so the agent can use it; the tool returns its media:// reference. Do this before starting a run: a run cannot upload anything itself.
• List a project's media — the images, sounds and documents the video already carries, each with its media:// reference.
• Install a plugin — hand over a custom user: plugin definition and the tool returns its ref. Do this before the run that should use it.
• List a project's plugins — every plugin the project carries, each with the ref an overlay names.
• Start a run — describe the change in plain English.
• Wait for a run — wait until it finishes or stops to ask a question, without polling yourself.
• Answer a run's question — reply so it can carry on.
• Cancel a run — stop one that is still going.
• Read a project's history — the revisions a run has committed.
• Undo a change — revert one committed revision (nothing is deleted; undoing writes a new revision).
• Export a project — download the finished interactive MP4 to a local path (private details are left out unless you ask to include them).
• Analyse a video — start the paid analysis described in Analyse a video; it returns the queued job at once.
• Read an analysis — the job's status, progress and result; call it again until the job is finished.
• List recipes — what a video can have built into it: the suggested recipe, each recipe's features with whether this video can have them and why, the catalogue of every feature, and the themes; see Make a video interactive.
• Make a video interactive — build the chosen features as one agent run; paid work, so start it only when it is wanted. It returns the run at once; wait for it like any run. Over MCP the build does not track products unless asked to (track=true); a tracked build starts when tracking ends, and you follow it in the Studio. cue (none, subtle or markers; subtle by default) sets how tracked products show they can be tapped.
• Read a project's document — its header, per-section counts and the full content of the sections you name.
• Read a project's objects or timeline — the detected-object registry, and which objects are visible in which frames, a page at a time.
• List or look up the format's schema — the exact shape of an action, condition, variable type, event or section.
• Apply a change — replace whole sections yourself, quoting the revision you read; the engine checks the result before it commits.
• Diagnose a project — what the engine would reject or silently skip, in the project as it stands or in the one a change would produce.

The usual sequence is the same one shown in the Walkthrough above: pick or create a project, start a run, wait for it, answer its question if it asks one, then export. The last five entries are the Author directly loop instead of a run — read, look the shape up, change, diagnose.