Lab

Recorded runs. The rubric below is a sanity check, not the scoreboard — it was wrong seven times during this build, each time in the direction that looks like a discovery, and the model’s own critiques proved the more reliable read of its renders. Treat the scores as a floor and the critiques as the evidence.

Two comparisons are wired up and awaiting data. E1: does K3 actually use the pixels, or would a measured text description do as well? E2: does dropping reasoning_content from its history degrade it, as Moonshot warn but never quantified? Both need a full grid; neither is answered yet, and the panels below say so rather than implying otherwise.

10 runs over 1 tasks, 0 errored. Roughly $3.02 at Moonshot list price — DigitalOcean does not publish a per-token rate for kimi-k3, so every cost here is an upper bound. Confidence intervals are percentile bootstrap; p-values are two-sided permutation tests.

Per-arm summary

E1 — is the vision load-bearing?

Both arms preserve thinking history. The only difference is whether the feedback is three screenshots or a measured description of the same DOM — box geometry, computed colours including the resolved background behind each element, font size and weight, and the document’s scroll size. Measurements, never conclusions: the description names no defect, so both arms have to find the problems themselves.

Read the pixel-only comparison first, and treat the whole-rubric one as uninterpretable. Every DOM-derived check is computable from the geometry the blind arm receives — and the one that discriminated in practice, legible-text (no text under 12px), has its answer printed verbatim in that input as font 11.52px. The blind arm reads the number; the vision arm has to notice small type in a downscaled JPEG. So a blind win on DOM checks says nothing about whether pixels help. The pixel checks are the only place the arms genuinely differ in information.

E2 — does dropping thinking history hurt?

Both arms see screenshots. The only difference is whether each assistant message goes back with its reasoning_content intact or with that field removed. Every request asserts the two message arrays are otherwise byte-identical, so any difference here has exactly one cause.

E2 again, without images

In the vision arm, screenshots accumulate across iterations and can come to dominate the context, shrinking reasoning_content’s share of it. These arms carry no image tokens at all, so this pairing isolates the thinking-history variable. If E2 looks null above but real here, dilution is why.

Regressions observed

A rubric check that passed at one iteration and failed at a later one. Computed mechanically from the render, not graded by a model.

Stopping early

Runs where K3 declared "done" on a render that still failed the rubric. Moonshot list “excessive proactiveness” as a known trait; this is what it looks like when measured.

Tasks