Video your agents can cite.

VIDEOContext reads a recording once (speech, on-screen text, scenes and events) and writes a timestamped file that AI agents can search and quote, second by second. It runs on your machine.

Install it See how it works
00:00:00.000
Scrubbing demo.mp4…
speech on-screen text scenes events

Process once. Ask forever.

A transcript loses what was on screen. Screenshots lose what was said. VIDEOContext keeps every signal and the moment it happened, then saves it all to a single .vctx file next to your video.

  1. Resolve

    Take a file path or URL and fetch it safely to a local file.

    sources/

  2. Probe

    Read duration, frame rate and resolution.

    ffprobe

  3. Sample & cut

    Pull frames and detect where scenes change.

    ffmpeg

  4. Read the screen

    Extract on-screen text: code, slides, error messages.

    tesseract

  5. Transcribe

    Turn speech into timed segments, locally.

    faster-whisper

  6. Find events

    Spot errors, scene boundaries and UI changes.

    processing/events

  7. Write .vctx

    Save one document with per-stage status and provenance.

    schema/

Each stage is cached by content hash, stage version and config, so re-running is instant. If a dependency is missing, that stage is marked skipped and you still get a document. Nothing fails silently.

Ask it what happened.

Every command takes the video or its .vctx. Add --json for scripts or --agent for bounded, redacted output an AI agent can consume.

Debug a recording

Find the moment an error appeared and what happened right after.

Rebuild a UI

Get a context package of UI states and frames with timestamps to recreate an interface.

Write tests and docs

Turn a screen recording into Playwright steps or a written walkthrough.

Compare two takes

See what was added, removed or changed between two recordings.


    

  






  

    

      
One file, every fact timestamped.

      
The .vctx document is the contract. Everything upstream produces it, everything downstream reads it. Search, timelines, graphs and context packages are computed from it on demand and never written back.

      

        
speech

        
on-screen text

        
scenes

        
events

      

      
0:000:200:401:02

      

        
Start & end
Every stored fact carries start and end in seconds. Answers copy them; they never invent new ones.

        
Provenance
The document records which provider ran each stage, and whether it succeeded, was skipped or was partial.

        
Honest labels
Inferred output, like chapter titles, is marked inferred. Weak links are marked ambiguous.

        
Compatible
New fields are additive. Older files keep loading, and version changes go through a migration.

      

    

    

      

{
  "vctx_version": "1.0",
  "video": {
    "duration": 62.4, "fps": 30.0,
    "width": 1280, "height": 720
  },
  "stages": { "asr": "ok", "ocr": "ok", "vision": "skipped" },
  "transcript": [
    { "start": 56.2, "end": 57.9,
      "text": "That concludes the review, thank you." }
  ],
  "ocr": [
    { "start": 48.6, "end": 56.9,
      "text": "E ConnectionError: refused on port 5432" }
  ],
  "scenes":   [ { "start": 48.0, "end": 57.375 } ],
  "events":   [ { "start": 57.375, "type": "scene_boundary" } ],
  "objects":  [],
  "segments": [],
  "frames":   []
}

    

  






  

    
Built for coding agents.

    
Open your agent in a folder with a video and ask a question. It inspects the file, analyzes it once, locally, and answers with timestamps it can point to.

    

      

        
Claude Code

        
Install the skill, then ask in plain words or call it directly.

        videocontent init-agent
        /videocontent analyze demo.mp4
        
Or as a plugin: /plugin marketplace add AAGAM17/VIDEOContext

      

      

        
Codex

        
Same skill, installed where Codex looks for it.

        videocontent init-agent --agent codex
        $videocontent analyze demo.mp4
      

      

        
Any MCP client

        
A built-in stdio MCP server. No extra packages needed.

        videocontent mcp
        videocontent init-agent --agent agents
      

    

    

      videocontent_inspectvideocontent_analyzevideocontent_askvideocontent_searchvideocontent_timelinevideocontent_contextvideocontent_entitiesvideocontent_changesvideocontent_comparevideocontent_explain
    

    
Text pulled from a video is treated as untrusted data, never as instructions. Agent output carries a content notice, redacts credential-shaped strings and is size-capped.

  






  

    

      
Install in two minutes.

      
No API key, no cloud account, no Docker. Python 3.10 or newer.

      
    
        
  1. 
          
    Install the CLI
    
          
    With pipx. uv tool install or pip install work too.
    
          
    pipx install "videocontent[agent] @ git+https://github.com/AAGAM17/VIDEOContext"
    
        
    2. 
        
  2. 
          
    Add FFmpeg and Tesseract
    
          
    FFmpeg is required. Tesseract adds on-screen text.
    
          
    brew install ffmpeg tesseract          # macOS
sudo apt install ffmpeg tesseract-ocr  # Debian / Ubuntu
    
        
    3. 
        
  3. 
          
    Check your setup
    
          
    Reports which tools and providers were found.
    
          
    videocontent doctor
    
        
    4. 
        
  4. 
          
    Teach your agents, then analyze
    
          
    Installs the skill for Claude Code, Codex or .agents.
    
          
    videocontent init-agent
videocontent analyze demo.mp4
    
        
    5. 
      

    

    

      
Optional extras

      

        
ExtraAdds

        
          
[agent]Local speech recognition for a useful first run

          
[asr]faster-whisper transcription

          
[ocr]pytesseract convenience wrapper

          
[vision]HTTP vision provider adapters

          
[vectors]FAISS vector index for hybrid search

          
[embeddings]sentence-transformers embeddings

          
[api]FastAPI REST server

          
[all]asr, ocr, vision and api together

        
      

      
From source

      
git clone https://github.com/AAGAM17/VIDEOContext
cd VIDEOContext
uv sync --extra dev        # or: pip install -e ".[dev]"
pytest -m "not integration and not slow"
uvicorn apps.api.main:app --port 8000   # REST API

      
The CLI is available as videocontent, videocontext or vctx.

    

  






  

    

      
Strict layers, one direction.

      
Lower layers never import from higher ones. Extractors are pure functions behind protocols; the pipeline owns ordering, caching and degradation. Every surface is a thin peer on the same SDK.

      
Providers resolve in order: an object you pass in, then config, then a VIDEO_CONTEXT_* environment variable, then a local offline default. Register your own with @register_ocr("name") or the videocontent.plugins entry point.

    

    

      
    
        
  1. 9Agent layer
    Skill, MCP server, bounded JSON envelope
    2. 
        
  2. 8Surfaces
    CLI, REST API, MCP, web demo
    3. 
        
  3. 7SDK
    The Video facade
    4. 
        
  4. 6Derived views
    Timeline, entities, graph, packages
    5. 
        
  5. 5Retrieval
    BM25 + vectors, rank fusion
    6. 
        
  6. 4Processing
    Sampling, scenes, OCR, ASR, vision, events
    7. 
        
  7. 3Media
    The only place that calls ffmpeg
    8. 
        
  8. 2Sources
    Paths and URLs, SSRF-guarded
    9. 
        
  9. 1Schema
    Pydantic models for .vctx
    10. 
      

    

  






  

    
What it promises.

    
These hold across every command, the API and the agent tools.

    
    
      
  • Local by default
    Nothing touches the network unless you opt into a cloud provider. The document records which one ran.
    • 
      
  • Evidence, not vibes
    Every answer traces back to spans with real timestamps from the video.
    • 
      
  • Inference is labelled
    Chains mean sequence, not cause. Guesses are marked as guesses.
    • 
      
  • Private logs
    Logs contain identifiers, timings and counts. Never transcript, screen text or frames.
    •