Multi-objective search over an LLM repair pipeline

MSc Artificial Intelligence and Data Engineering dissertation, University College London, submitted September 2026.

Question

Multi-agent LLM pipelines expose many knobs (sampling temperature, top-p, retry budgets per stage), and their behaviour is stochastic. Can a principled search over those knobs improve the pipeline without retraining or prompt changes, and does a more sophisticated optimiser actually help at a realistic budget?

Why it matters

A search method has to earn its extra complexity under the same evaluation budget. Held-out performance and measurement reliability determine whether a training-front improvement is useful.

Method

• A four-stage analyse → repair → verify-by-execution → optimise pipeline, fixing real bugs in C and Python programs with a 1.5B-parameter model served locally.
• The joint decoding-and-retry policy is an 11-dimensional search problem under three objectives: test-pass fraction, cyclomatic complexity, and execution energy of the repaired program.
• Three search arms, NSGA-II, multi-objective Bayesian optimisation (qLogNEHVI over per-objective GPs) and uniform random sampling, compared against a repeated default policy at a matched 750-evaluation budget, five seeds per arm (791 host-hours).
• Search on Codeflaws (C); the resulting policies were transferred unchanged to RunBugRun (Python) and evaluated on held-out problems.

Findings

• Random sampling beat NSGA-II on held-out correctness, with complete separation across seeds (Cliff’s δ = 1, Holm-corrected p = 0.024).
• Selection matters more than the optimiser. Choosing from the Pareto front on training correctness raised held-out correctness by 3.4–5.2 points over the shipped defaults in every arm–benchmark cell, whereas a balanced three-objective selection rule did worse than picking at random from the same front.
• Correctness transfers across languages, energy does not. Policy rankings on correctness carried over from C to Python (ρ = 0.63 over 665 policies); energy rankings did not.

Measuring energy properly

Energy is a noisy objective. I designed a two-phase protocol that separates concurrent LLM generation from serial RAPL measurement of the repaired programs, which cut run-to-run spread from 4–17% to 2–3%. The protocol also exposed a measurement-epoch confound: a 1.68× shift in measured energy across a reboot, with correctness unchanged.

Analysis and engineering

• Statistics at the repetition unit: Mann–Whitney U, Cliff’s δ, Holm correction, task-grouped bootstrap, and front-quality indicators (hypervolume, IGD⁺, ε⁺) with cardinality control, plus a power analysis.
• 791 hours of unattended experiments with per-generation checkpoint/resume, pre-flight health checks, provenance manifests and hash-pinned evaluation sets; concurrency tuning gave 2.85× throughput over serial execution.
• Analysis notebooks are generated rather than hand-edited, and reports are exported from executed results so no reported number is transcribed by hand.

Limitations and uncertainty

The five repetitions per arm constrain the statistical evidence. The plotted energy fronts use repetitions 1–4 within the same measurement epoch; repetition 0 preceded a reboot and is excluded from this energy comparison. The animation highlights existing training points in presentation order and does not reconstruct the search chronology.

Next questions

Do the selection rules and cross-language correctness rankings hold for other models and repair benchmarks? The reported transfer concerns correctness; it does not establish transferable energy rankings.

Sources

The experiment repository contains the pipeline, analysis and provenance. The chart uses the real Codeflaws training fronts extracted from that repository.