MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

Paper: MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning (v1) https://arxiv.org/html/2608.14015v1

MedClaw organizes long-video question answering around explicit evidence requests. Its text-only orchestrator never reads video pixels directly. Instead, frozen visual sub-agents inspect the inputs named by tools and return textual observations. The resulting trace records what was requested and what came back.

A neutral example starts with Phase A, Event B and Phase C. The question concerns the interval before B. Localizing B is useful, but an event-window observation should trigger a request for the preceding window. A matching excerpt from A supplies the relevant temporal evidence. The example’s times, symbols and tool fields are authored for explanation.

This separation gives retrieval planning a visible interface. A whole-video pass can establish a rough timeline, a segment request can revisit a candidate interval, and a frame tool can inspect a smaller visible detail. General background knowledge remains a different input from observations of this particular recording.

The system’s adjustable component is an external library of retrieval strategies. Its heuristic skill procedure uses 100 labeled training questions: 60 for mining candidates and 40 for validation. Each mined skill is installed alone and tested on the entire validation set against the empty-library baseline. Retention requires a mean gain greater than 0.05 on the 1–5 scale, with no individual answer dropping by two or more points from that baseline. The model parameters remain frozen.

MedClawBench contains 1,123 questions: 259 long-neurosurgery questions and 864 held-out lecture-video questions. Answers receive separate correctness, detail, contextual and temporal rubric scores. These measurements assess research question answering rather than clinical outcomes.

In the same-visual-backbone neurosurgery comparison, one-shot Qwen2.5-VL-7B reading averages 2.04 across the four dimensions, while the harness row averages 2.58. The comparison includes the orchestrator and tool loop. The film keeps that scope visible as it traces the requested window, returned excerpt and final interpretation.

Pinned source versions: Paper, v1: https://arxiv.org/html/2608.14015v1

#AIResearch #VideoUnderstanding #AIAgents