Harness and Meta-Harness 하네스와 메타 하네스

In the Week 01 lab you built a first agent by attaching two tools to a single while loop. Change what that loop feeds the model, when it stops, and how it passes errors back, and the success rate changes even though the model stays the same. All of the code wrapped around the model is called the harness.

ReAct, published by Shunyu Yao and colleagues in 2022, has the model alternate reasoning and acting inside one loop. Before it, the two lineages lived apart. One was pure reasoning, represented by Chain-of-Thought. The model walks through steps to a conclusion. If a fact in an early step is wrong, everything built on it is wrong. Yao and colleagues identified this as CoT's hallucination and error propagation problem. The other lineage was action: search and calculate, but leave no text saying why this tool is being called now.

ReAct splits every step into three pieces. The model first writes a Thought: what the situation is and what to do next, in natural language. Then it emits an Action: which tool to call with which arguments. When the tool runs, the environment produces an Observation: a search result, a computed value, an error message. That Observation enters the next step's Thought, and the model reasons again with it as evidence.

Pure reasoning is this figure without the Observation box. The model goes to the end on what it already knows. In ReAct the actual execution result comes back at every step. If the model wrongly assumes “this API's response will look like this”, the next Observation shows the actual response and the model corrects the assumption on the spot. Yao and colleagues reported that on HotpotQA and Fever, ReAct reduced CoT's hallucination and error propagation with a single Wikipedia API. Because the Thought stays as text, a person can read afterwards why the agent did what it did. The run log you commit to logs/ every week is exactly that text.

ReAct is a pattern. Making the model write a Thought does not make the Thought right. The model can write plausible reasoning and still emit the wrong Action. What ReAct guarantees is only that the mismatch surfaces in the next Observation. Catching it and rolling back is the job of the rest of the harness: termination conditions, error recovery, iteration caps.

A harness originally means the gear fitted to a horse so its strength goes in one direction. The definition Yoonho Lee and colleagues use in the 2026 Meta-Harness paper is this.

What a harness decides splits into five axes. The first is context management. The context window is finite, and what goes into it drives performance. Put in every Observation from the last twenty steps and the recent information gets buried; trim too hard and the model forgets what it just did. What to show, what to hide, and how to fold old records into summaries is the first decision. The second is tool granularity. Slice tools finely (read_line(n)) and the model's control gets precise while call counts and tokens grow; bundle them coarsely (read_file()) and one call does more, but when it fails it is hard to tell where.

The third is the termination condition. Using the moment the model declares an answer as the stop condition is risky. Models say they are done before they are. So working harnesses add a cap on iterations, a separate verification step that confirms the goal, or an explicit termination tool (finish(answer)). The fourth is error recovery. When a tool throws or the arguments are malformed, the harness can stop there, or it can return the error message as an Observation and let the model fix it. The latter fits ReAct. The fifth is the human intervention point (human-in-the-loop). Before an irreversible action such as a payment or a file deletion, decide whether to stop and ask for approval or to run autonomously to the end. The lab counts these interventions.

# Skeleton of a ReAct-style harness. Marks which line each of the five axes lives on
def run_react(task, tools, max_steps=8):
    history = [system_prompt(tools), user(task)]   # [axis 1] context management
    interventions = 0
    for step in range(max_steps):        # [axis 3] termination: iteration cap
        reply = model(history)                # generate Thought + Action in one call
        history.append(assistant(reply))
        if reply.tool_call is None:
            return reply.text            # [axis 3] model chose finish → exit
        if reply.tool_call.name in IRREVERSIBLE:
            if not ask_human(reply.tool_call):  # [axis 5] intervention point
                interventions += 1
                history.append(observation("Denied: human did not approve"))
                continue
        try:
            obs = tools.call(reply.tool_call)     # [axis 2] granularity lives in the tools definition
        except Exception as e:
            obs = f"Error: {e}"              # [axis 4] error recovery: errors are Observations too
        history.append(observation(obs))          # Observation → next Thought
    return "MAX_STEPS reached: incomplete"      # forced stop at the cap

None of the five values in this code is set by the model. People set all of them. The same model with different values is a different agent. The harness is a design space you can explore by experiment. The lab picks two points in that space and compares them.

Instead of a person turning the five knobs by hand and running A/B tests, the search itself can be handed to an agent. The harness reads its own run logs, decides “too many iterations here, bundle the tools more coarsely”, and writes the next version of the harness code.

The paper “Meta-Harness: End-to-End Optimization of Model Harnesses”, published in March 2026 by Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn, builds exactly this. The outer loop repeats three steps. A proposer, itself a coding agent, proposes new harness code; the harness is run on the tasks and scored; and the source code, scores, and execution traces are all written to a filesystem. On the next proposal, the proposer reads the code and traces of every prior candidate straight from that filesystem.

The paper's central claim is that existing text optimizers compress feedback too aggressively. They forget prior attempts and receive only a scalar score or a short summary. Lee and colleagues compared three conditions for what the proposer gets to see, on online text classification.

Numbers from Table 3 of Lee et al. (2026). Summaries do about as well as scores alone; only with full traces does the median candidate beat the best candidate of either other condition. These are the paper's numbers on the paper's task, not measurements from this course.

If the function that judges “did it improve” lives inside the loop, self-improvement drifts, because the model ends up grading its own answers. In Meta-Harness that judgment is the task score, which the proposer cannot touch. In Ouroboros below it is the acceptance criteria a person fixed in an interview, plus a deterministic verify command.

CASEThe meta-harness loop in Ouroboros

Ouroboros is the open-source Agent OS mentioned in the Week 01 introduction. Where Meta-Harness searches for the harness that maximizes a score, Ouroboros edits the execution rules until a run passes requirements a person has confirmed. In the repository the loop is split across four modules. SeedContract in core/seed_contract.py fixes the goal and acceptance criteria before execution. The orchestrator manages the run and records the trace. evaluation/pipeline.py collects the results and orchestrator/verifier.py decides pass or fail with the verify command. evolution/loop.py takes that verdict and the records as input to the next generation.

SeedContract        core/seed_contract.py     goal, acceptance criteria, ontology
OrchestratorRunner  orchestrator/runner.py    runtime calls, session events, run state
EvaluationPipeline  evaluation/pipeline.py    result collection and independent evaluation
Verifier            orchestrator/verifier.py  deterministic verdict from the verify command
EvolutionaryLoop    evolution/loop.py         per-generation edits, reruns, regression checks

This structure counts as a meta-harness because what EDIT changes is the rules that produce the next run, not the answer. The acceptance criteria and the verify command are fixed in the Seed, out of EDIT's reach. Cross that boundary and all that remains is the model grading itself.

LINEAGEThe lineage of self-improvement loops

Before the meta-harness there was already a line of work that goes beyond producing one more output and changes the conditions of the next run. The six methods are sorted by what they change and by what signal judges them. Self-Refine (Madaan et al. 2023) has the same LLM alternate draft, feedback, and revision. What changes is the output; the signal is self-feedback. Reflexion (Shinn et al. 2023) writes the cause of a failure in words and puts it in the episodic memory of the next trial. What changes is the context. DSPy (Khattab et al. 2023) declares a pipeline and has a compiler find the prompts and demonstrations that maximize a metric. GEPA (Agrawal et al. 2025) reads trajectories, reflects in natural language, and keeps prompt candidates on a Pareto frontier. Meta-Harness changes harness code, and the Darwin Gödel Machine (Zhang et al. 2025) keeps candidates that modify the agent's own code in an archive and validates them on coding benchmarks.

Five of the six, all but Self-Refine, keep the judging signal outside the thing being changed. In Self-Refine the same model produces the feedback, and that is why later work moved the signal to environment results, metrics, and benchmarks. The numbers each paper reports come from its own task, so they do not go into your lab report.

Hold the model, the task, and the tools constant, and vary only the harness. Give the same model the same task, once under a ReAct-style harness and once under a Plan-then-Execute harness, and measure four metrics side by side. Any multi-step task that needs tools will do. Something like “find the hour with the most errors in this log file and summarize it” is a good choice, since the answer requires reading a file, counting, and writing it up. The prerequisites are the same as Week 01: Python 3.10 or later, the anthropic or openai package, and one API key.

The ReAct harness runs Thought → Action → Observation at every step and re-decides what to do next each time. The Plan-then-Execute harness first produces the whole plan in one call (Plan), then runs the plan's steps in order (Execute). Because the plan is fixed up front it uses fewer tokens and its flow is predictable, but when a mid-run Observation contradicts the plan it is hard to change course. Which one wins depends on the task.

1. Put the shared tools and model wrapper in one module. The tools both harnesses share (read_file, count_pattern), the model call, and the token counter go in tools_shared.py. With different tools the harness comparison does not hold.
2. Implement the ReAct harness. Fill in the run_react skeleton above for your task. Terminate on the model's finish call or the max_steps cap. Count iterations and interventions during the run.
3. Implement the Plan-then-Execute harness. The first call returns a list of steps (the plan); subsequent calls execute each step. If the plan breaks, allow at most one replan so the flexibility cap is explicit.
4. Run each harness three times on the same task. Model output is stochastic, so a single run is luck. Run each harness at least three times and look at the success rate together with the spread of the metrics. Record the experimental conditions for every run: prompt, tool set, max_steps.
5. Record results in results.csv. One line per run: success, tokens, iterations, interventions. Keep the failed runs.
6. Write one paragraph of interpretation. Which harness won on which metric for this task, and why.

# Tools, model, and measurement shared by both harnesses. Same tools make a fair comparison
import os, anthropic, openai, re, json

OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
if OPENROUTER_KEY:                      # pick the vendor from whichever key is set
    client = openai.OpenAI(base_url="https://openrouter.ai/api/v1", api_key=OPENROUTER_KEY)
    MODEL = "anthropic/claude-sonnet-4.5"
else:
    client = anthropic.Anthropic()
    MODEL = "claude-sonnet-4-5"

def read_file(path):                    # tool 1
    with open(path, encoding="utf-8") as f:
        return f.read()[:4000]           # cap to protect the context

def count_pattern(text, pattern):        # tool 2
    return len(re.findall(pattern, text))

TOOLS = {"read_file": read_file, "count_pattern": count_pattern}

class Meter:                             # counts the four metrics in one place
    def __init__(self):
        self.tokens = 0; self.iters = 0; self.interventions = 0
    def add(self, n_in, n_out):
        self.tokens += n_in + n_out
        self.iters += 1

def call_model(messages, meter):
    if OPENROUTER_KEY:                 # OpenAI-compatible shape: choices, prompt/completion_tokens
        resp = client.chat.completions.create(model=MODEL, max_tokens=1024, messages=messages)
        meter.add(resp.usage.prompt_tokens, resp.usage.completion_tokens)
        return resp.choices[0].message.content
    resp = client.messages.create(model=MODEL, max_tokens=1024, messages=messages)
    meter.add(resp.usage.input_tokens, resp.usage.output_tokens)
    return resp.content[0].text

# Plan-then-Execute: fix the plan first, then run it in order
import json
from tools_shared import TOOLS, Meter, call_model

def run_plan_execute(task, max_replan=1):
    meter = Meter()
    # 1) PLAN: request the whole plan in one call
    plan_raw = call_model(
        [{"role": "user", "content": f"Answer only with a JSON list of steps for this task: {task}"}],
        meter)
    plan = json.loads(plan_raw)          # a parse failure is one failure mode too
    replans = 0
    # 2) EXECUTE: run each step of the plan in order
    context = []
    for step in plan:
        result = execute_step(step, context, meter)     # includes tool calls
        context.append(result)
        if result.get("off_plan") and replans < max_replan:
            plan = replan(task, context, meter); replans += 1   # flexibility cap
    return summarize(context, meter), meter, replans

run,harness,success,tokens,iters,interventions,note
# run: 1..6 / harness: react | plan_exec / success: O | X (by the criterion fixed before running)
# tokens: Meter.tokens / iters: Meter.iters / interventions: how many times a human approved or denied

Write the success criterion down as a sentence before you run. It has to be checkable, like “O if the answer names 14:00 as the hour with the most errors”. A criterion written after the run turns failures into successes.

Submission rules are the same as Week 01.

• REQS. Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models” (2022)The source text for ReAct. Read what the Thought, Action, Observation interleaving changes against CoT-only and act-only, together with the experimental design, and compare it with your ReAct harness from the lab. arXiv:2210.03629
• REQY. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, C. Finn, “Meta-Harness: End-to-End Optimization of Model Harnesses” (2026)The source text for the meta-harness. Read the definition of a harness, the three steps of the outer loop, and, with Table 3, why giving the proposer full traces beats giving it summaries. arXiv:2603.28052
• OPTA. Madaan et al., “Self-Refine: Iterative Refinement with Self-Feedback” (2023)Generate, feed back, revise, repeat. Check where the judging signal sits when the same model also produces the feedback. arXiv:2303.17651
• OPTN. Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning” (2023)Putting a verbal record of failure into the next trial. Compare it with the Ouroboros trace and the input to the next generation. arXiv:2303.11366
• OPTO. Khattab et al., “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines” (2023)A compiler that finds a pipeline's prompts and demonstrations against a metric. Earlier work by one of the Meta-Harness authors. arXiv:2310.03714
• OPTL. Agrawal et al., “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning” (2025)Reading trajectories to produce prompt candidates on a Pareto frontier. One of the methods Meta-Harness says ‘compresses feedback too aggressively’. arXiv:2507.19457
• OPTJ. Zhang et al., “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents” (2025)Keeping candidate agent code in an archive and validating it on coding benchmarks. The widest edit target in the lineage. arXiv:2505.22954