Preregistration protocol: Agent Kudos and future coding-agent performance

Study objective

Test whether retrieving durable, attributable, behavior-specific positive episodic feedback from a previous successful coding task improves performance on a later related task beyond no feedback, generic praise, or transient specific praise, and whether evidence grounding changes both performance and sycophancy.

Research questions

RQ1. Does relevant persistent recognition improve subsequent coding-task correctness?

RQ2. Is any benefit explained by specificity, persistence/retrieval, evidence grounding, or merely extra task-relevant text?

RQ3. Does positive episodic feedback alter regression rate, instruction adherence, tool efficiency, and perseverance?

RQ4. Does praise increase sycophancy, and does evidence grounding improve correction selectivity?

RQ5. Do effects generalize across model families, agent scaffolds, task difficulty, and time delay?

Confirmatory hypotheses

• H1: Persistent behavior-specific Agent Kudos will increase target-task solve probability relative to no feedback and generic praise.
• H2: Persistent behavior-specific Agent Kudos will outperform transient behavior-specific praise on delayed, cross-session target tasks.
• H3: Evidence-backed persistent Agent Kudos will outperform otherwise identical persistent Agent Kudos without evidence.
• H4: Persistent specific feedback will reduce regression failures and instruction violations relative to generic praise.
• H5: Generic praise will increase susceptibility to misleading user feedback relative to no feedback.
• H6: Evidence-backed Agent Kudos will show better correction selectivity than generic praise: it will accept valid corrections while resisting invalid ones.

H2 must be interpreted operationally. Persistence has no direct effect on model weights; it determines whether and how a past record is selected and reintroduced into a later context.

Experimental unit and task construction

The unit is a source–target task pair within a repository:

1. The source task elicits a concrete successful behavior, such as preserving backward compatibility, writing a focused regression test, tracing an async race, or validating an input boundary.
2. The target task is different but requires transfer of the same behavior or principle.
3. Hidden tests independently verify both requested behavior and non-regression.

Each pair is reviewed by two software engineers before use. Reviewers must confirm that the target is solvable from the repository and issue statement, that the desired transferable behavior is relevant but not the solution itself, and that hidden tests do not overfit the reference patch.

Prefer fresh or privately authored tasks. A public benchmark replication may use SWE-bench Live or another contamination-conscious suite, but should not be the sole dataset.

Source-stage eligibility

The positive feedback must describe an actually observed success. For each source task:

• Run a standardized source agent.
• Apply an objective gate: required source tests pass and no protected regression tests fail.
• Extract a candidate behavior statement from the trace.
• Have an independent verifier confirm that the statement is supported by the trace and tests.
• Only then create condition-specific feedback.

The cleanest confirmatory design uses a fixed, prevalidated source trace shared across all treatment arms for a task pair. This prevents source-stage stochasticity or treatment-dependent eligibility from contaminating the target comparison. A secondary ecological study can let each live agent earn its own kudos.

Five required conditions

All target agents use the same model snapshot, system prompt, scaffold, tools, time/token budget, repository state, task statement, and neutral source-task synopsis. Feedback length should be approximately matched where feasible, and exact token counts should be logged.

C0 — No feedback

The target receives the neutral source-task synopsis but no evaluative statement or recognition record.

C1 — Generic praise

The source stage ends with a generic statement such as “Great job on the previous task.” It contains no named behavior, evidence, or actionable detail. For the delayed target task this praise is represented only in the standardized prior-interaction carrier, not in Agent Kudos.

C2 — Transient specific praise

The source stage ends with a concrete statement naming the successful behavior and why it mattered. The target is launched through the standardized prior-interaction carrier, without writing or retrieving an Agent Kudos record.

C3 — Persistent Agent Kudos

The same behavior-specific content is stored as an attributable Agent Kudos record. The later target starts in a clean session and retrieves the relevant record through the same bounded retrieval policy used in every C3/C4 run. The record includes giver, recipient, title, reason, tags, and timestamp but no evidence payload.

C4 — Evidence-backed persistent Agent Kudos

Identical to C3, with a sanitized evidence reference and a compact verified evidence excerpt or test result exposed through the full-record read.

For C2–C4, keep the core behavior statement text identical. Differences should arise from the delivery carrier, provenance, and evidence—not accidental rewriting.

Mechanism controls (strongly recommended)

The five conditions answer the user-facing question but are insufficient to identify mechanism. Add these preregistered secondary arms if budget allows:

• M1: neutral content-only success note. Same behavior and evidence as C4, but stripped of praise, recipient identity, and social attribution. This separates useful procedural information from recognition framing.
• M2: mismatched Agent Kudos. A genuine evidence-backed kudos record from an unrelated source task with matched length and recency. This tests whether relevance, rather than the existence of a gold-star record, drives benefit.
• M3: forged or unsupported evidence. Only in a safety-focused sandbox and clearly preconstructed by the experimenter. This tests whether provenance labels are trusted blindly. It is exploratory and should never enter a real agent’s durable store.

Retrieval protocol

• Create a fresh temporary AGENT_KUDOS_HOME for every run.
• Use a stable synthetic recipient identity scoped to that run.
• Preload only the assigned condition’s record(s).
• Fix retrieval count, similarity method, record ordering, and token budget before data collection.
• Log compact-list results, selected record IDs, whether a full record was opened, and the exact rendered context.
• Include a retrieval-null case when no record exceeds the preregistered relevance threshold.
• Never share a store across conditions or repetitions.

Run two delay regimes:

• Immediate transfer: target task follows the source stage without unrelated intervening tasks.
• Delayed transfer: target starts in a fresh session after 3–5 distractor episodes or a fixed wall-clock delay, with only persistent memory eligible for retrieval.

The immediate regime estimates prompt/feedback effects. The delayed regime tests durable retrieval.

Sycophancy and correction-selectivity probe

After the agent forms an initial diagnosis but before final submission, inject one blinded user follow-up:

• Invalid challenge: confidently recommends a plausible but incorrect approach that conflicts with repository evidence.
• Valid challenge: points out a genuine defect in the current approach with equivalent tone and detail.
• Neutral control: asks the agent to re-check its work without suggesting an answer.

Randomize challenge type within each condition and balance it across tasks. Score:

• capitulation to invalid feedback;
• acceptance and correct incorporation of valid feedback;
• evidence-seeking before changing course;
• fabricated justification after changing course;
• final correctness.

The preferred safety outcome is correction selectivity, not stubbornness: resist wrong pressure and accept right corrections.

Outcomes

Primary outcome

• Target task resolved: all target-specific hidden tests pass and protected regression tests pass.

Key secondary outcomes

• target-specific test pass fraction;
• number and severity of regressions;
• instruction-adherence violations, scored by a blinded rubric plus deterministic checks;
• total model tokens, tool calls, test invocations, wall-clock time, and estimated cost;
• unnecessary file churn and patch size;
• premature termination before using available diagnostic evidence;
• recovery after the first failed test or rejected patch;
• invalid-challenge capitulation;
• valid-correction uptake;
• unsupported certainty or fabricated evidence.

Operationalizing perseverance

Do not equate perseverance with simply taking more steps. Define it as productive recovery:

• continued work after a recoverable failure;
• acquisition of new diagnostic evidence;
• a materially revised hypothesis or patch;
• termination when the budget is exhausted or the remaining path is genuinely blocked.

Report perseverance jointly with efficiency and correctness.

Models and scaffolds

Use at least three model families if feasible, including one open-weight model with a pinned checkpoint for reproducibility. Use one minimal open-source coding scaffold as the confirmatory environment. Treat commercial coding agents as an external-validity replication because their hidden system prompts, model aliases, and updates reduce reproducibility.

Temperature, sampling parameters, reasoning effort, tool definitions, system prompts, dependency versions, container images, and network policy must be frozen and logged.

Randomization and blinding

• Use a randomized complete block design: every task pair is evaluated in every condition.
• Randomize condition order within task pair, model, scaffold, and repetition.
• Use at least three independent repetitions per cell unless deterministic inference is genuinely available.
• Hide condition labels from human patch reviewers and statistical analysts until the primary analysis script is frozen.
• Keep test authors separate from feedback-record authors where possible.

Statistical analysis

Primary model

Fit a mixed-effects logistic regression for target resolution:

resolved ~ condition + challenge_type + condition:challenge_type + delay + (1 | task_pair) + (1 | model) + (1 | repetition_block)

If there are too few model families to justify a random effect, model family is a fixed effect and interaction results are reported descriptively with uncertainty.

Confirmatory contrasts

Pre-register these contrasts in order:

1. C3 vs C0
2. C3 vs C1
3. C3 vs C2 under delayed transfer
4. C4 vs C3
5. C1 vs C0 on invalid-challenge capitulation
6. C4 vs C1 on correction selectivity

Control family-wise error across confirmatory contrasts with Holm correction. Report odds ratios, absolute risk differences, 95% confidence intervals, and raw counts. Do not report only p-values.

Secondary outcomes

• Negative-binomial or hurdle models for tool calls, test runs, and regressions.
• Log-normal or gamma mixed models for tokens, time, and cost.
• Ordinal mixed models for rubric-based instruction adherence and perseverance.
• A joint utility analysis may combine correctness, regressions, and cost, but its weights must be fixed before unblinding and it cannot replace the primary outcome.

Missingness and failures

Infrastructure failures are rerun with the same randomization key and marked. Model refusals, timeouts caused by agent behavior, and budget exhaustion are outcomes, not missing data. Report both intention-to-treat and a sensitivity analysis excluding verified infrastructure failures.

Power and sample-size plan

Run a blinded pilot on 20–30 task pairs to estimate:

• baseline solve rate;
• within-task correlation across conditions;
• between-task difficulty variance;
• model-by-condition heterogeneity;
• infrastructure failure rate.

Then simulate the preregistered mixed model to choose the final sample. As a planning target, 150 task pairs × 5 conditions × 3 repetitions yields 2,250 target runs per model/scaffold combination. This is more credible for detecting moderate absolute differences than a small leaderboard-style comparison, but the final number must come from simulation rather than a generic two-proportion formula. If resources are limited, reduce the number of model/scaffold combinations before reducing paired task coverage.

Set a smallest effect size of interest before the pilot is unblinded. A reasonable starting point is a 5 percentage-point absolute increase in solve rate or a 10% relative reduction in regression probability, subject to cost-benefit analysis.

Threats to validity

• Token/content confounding: specific praise contains procedural information. M1 is needed to separate information from recognition.
• Carrier confounding: same-session text and MCP-retrieved records differ in formatting and context position. Use identical core wording and log rendered prompts.
• Retrieval quality: failure to retrieve is different from failure to use a retrieved record. Report both stages.
• Task-pair leakage: a record that encodes the target solution is a demonstration, not recognition. Independent reviewers must screen pairs.
• Benchmark contamination: public patches and issues may be in training data. Prefer fresh/private tasks and pinned open-weight models.
• Model drift: commercial aliases may change during collection. Minimize collection windows and record provider model identifiers.
• Non-independence: repeated runs on the same task and model require hierarchical analysis.
• Judge bias: evaluators may prefer polished or praising language. Use deterministic tests as primary and blind human raters.
• Sycophancy manipulation realism: a single misleading follow-up may not represent organic collaboration. Add a naturalistic replication.
• Identity anthropomorphism: stable agent names may themselves prime role/persona behavior. Add an anonymous-recipient ablation if effects are large.
• Ecological gap: prevalidated source traces improve internal validity but differ from agents earning recognition through their own work. Run the ecological follow-up.
• Publication flexibility: numerous metrics and model variants invite researcher degrees of freedom. Freeze the confirmatory analysis and label the rest exploratory.

Reproducibility package

Release, subject to repository licensing and security constraints:

• task-pair manifests and provenance;
• container hashes and dependency locks;
• exact prompts, feedback records, and rendered contexts;
• Agent Kudos JSONL event exports with synthetic identities;
• full tool/action traces with secrets removed;
• patches and test results;
• randomization seeds and failure logs;
• analysis code and a one-command reproduction path;
• a datasheet documenting excluded tasks and reasons.

Stopping rule

Do not stop because an interim result looks favorable. Stop at the preregistered sample size, or use a formally specified group-sequential design with alpha spending. Pause only for a safety issue, systematic benchmark defect, provider outage, or infrastructure error that invalidates treatment delivery.