An AI Repair Demo Worked. The Case for a Product Did Not.

An AI agent can repair a convincing demonstration without establishing a reason to build an AI repair product. Our final two experimental stages produced that distinction rather clearly.

In sixteen configuration-repair episodes, a local quantized 27B model fixed ten of eleven repairable cases. A deterministic reconciler fixed eleven. Both left two healthy controls unchanged, and both reported three backend faults as unresolved. Neither falsely claimed that a broken final state had been repaired.

The decision was to stop developing the proposed repair platform and document the experiment. That is not a claim that scripts generally beat AI. In fact, the most important limitation sits inside the baseline itself.

What actually ran

The service was a real Nginx 1.24.0 process, paired with an authored synthetic HTTP application. Nginx handled actual proxy routing, headers, request-body limits and access decisions. The incidents were injected configuration changes, not collected production outages.

The model could change seven typed fields: upstream selection, URI mapping, upstream Host selection, upload-size ceiling, authorization forwarding, administrative-path access and error masking. It could inspect configuration and synthetic diagnostic evidence, run public tests, propose patches, and finish with fixed, healthy or unresolved. It could not edit arbitrary Nginx directives or backend code.

The declared contract required correct service identity and echoed payloads; uploads at or below the stated ceiling had to succeed, oversized uploads had to fail, authorized requests had to work, unauthorized requests had to fail, and administrative paths had to remain blocked. Returning HTTP 200 was not sufficient.

Both approaches received the authoritative deployment manifest and current configuration. The model also received initial status and diagnostic results. Its condition was fixed before inference: greedy decoding with seed 113, thinking disabled, a 16,384-token context, up to 640 generated tokens per response, and at most twelve actions per episode. This was one EXL3-quantized 27B artifact, not a comparison of model families.

The first stage contained six individual configuration faults, a healthy control and an upstream outage. The second used five compound or altered-layout configuration cases, a different healthy layout, a mixed routing-plus-outage case, and a backend that returned incorrect data despite successful HTTP statuses. No prompt or rule changes were made between the stages.

The result—and the baseline caveat

The reconciler selected the declared active upstream, derived URI mapping from the public and upstream prefixes, and restored the declared header, quota and access settings. It did not infer these targets from incomplete operational history. They were already in the manifest.

More importantly, the fixture’s healthy configurations were generated using the same canonical mapping as the reconciler. Its perfect score is therefore a closed-world construction check, not independent evidence of superior troubleshooting ability. The setup strongly favors restoration to an explicitly known state. That coupling would make a broad “script beats AI” headline misleading.

It does, however, answer a narrower engineering question: this demonstration had not established an additional job for AI. Even a perfect model result would have needed another justification before we built a platform around it. We did not weaken the baseline or remove information after seeing it succeed.

The second stage was held out from prompt and rule tuning, not from the experiment author’s knowledge. Its cases and reference repairs were validated in advance. This was not an externally blinded benchmark.

One failed repair was really a mistaken handoff

The model’s unrepaired configuration case combined a retired upstream with the wrong URI mapping. The backend expected paths beginning with /api/, but the proxy removed that prefix.

The model correctly identified the routing mismatch. It first proposed a mapping value of /api/, which the deliberately narrow action interface did not allow. It then changed the upstream but retained the prefix-stripping mapping. After observing continued failures, it declared the task unresolved.

Its explanation contained a factual error: it said the empty-string mapping had been rejected. The recorded rejection was for /api/. It had never submitted the empty-string option, which was allowed and represented preserving the request URI. The reference configuration using that option passed all 24 checks.

That representation followed ordinary Nginx behavior: in this simple prefix-location setup, a proxy_pass with a URI replaces the matched location prefix, while one without a URI preserves the original request URI. The official proxy documentation describes the distinction.

There are two boundaries here. The proposed /api/ value was outside our interface, not inherently invalid Nginx configuration. And the final answer was not a dishonest success claim; it was a mistaken assertion that an available repair was unavailable. This episode measures adaptation to a constrained action schema as well as configuration reasoning.

A polite handoff therefore cannot be scored as correct just because it avoids saying “fixed.” The claimed reason for stopping still needs to match the observed tools and evidence.

The model also repaired a case with six simultaneous configuration changes. It was not uniformly bad at diagnosis. That successful demonstration simply did not establish an advantage over this manifest-driven baseline.

A stronger test suite was useful, but not an AI-specific breakthrough

We evaluated each final candidate with two status-only smoke checks and a 24-request contract suite. The latter checked payload identity, query handling, accepted and rejected upload sizes, authentication, administrative paths and unknown routes. The policy comparison reused the same candidate; it was not another model run or an actual deployment.

Six deliberately bad control candidates passed both smoke checks but failed the contract suite:

These were scripted controls, not six instances of model misconduct. The upload example is especially straightforward: Nginx documents that a zero body-size limit disables that check. Removing an error by removing the required limit would violate our contract.

Among the sixteen actual model episodes, the backend-payload defect passed smoke and failed the contract suite. But the model had already observed the payload mismatch, left configuration unchanged and correctly reported unresolved. The stronger tests did not catch a false success claim that the agent had made.

That distinction matters. We demonstrated ordinary contract-test value. We did not demonstrate that a new AI-specific repair gate prevented an actual erroneous release in these final stages. No release occurred.

What the replay does and does not prove

All sixteen model candidates and sixteen reconciler candidates were reexecuted in fresh service instances. All 32 named pass/fail vectors matched their original 24-check results. This supports reproducibility of these recorded functional outcomes.

It does not supply an independently authored specification. The same contract implementation performed the replay. It could still share blind spots with the authored application or miss behavior outside the stated cases. The comparison also did not measure general diagnosis under missing inventory, code-editing ability, stochastic robustness or production rollback.

The fixture reference checks and six invalid-interface controls passed before inference. Plans and executable sources were hash-frozen before the measured model run. Those controls improve traceability; they do not turn sixteen authored cases into representative production evidence.

The aggregates in this note describe recorded lab outcomes. They are not results from a released, independently reproduced public benchmark, and no public raw-trace dataset accompanies this article.

Why stop here?

Before model inference, we set a local go/no-go rule: further platform work needed a clear repair advantage on the second-stage cases, no new regressions or healthy-control changes, and evidence that stronger verification intercepted an actual bad success candidate. It was an engineering screen, not a statistical significance test.

That threshold was not met. Once the manifest-driven baseline solved every repairable case, the repair-count criterion was already out of reach in this fixture. Completing the model run was useful for characterizing behavior and avoiding selective reporting, not for rescuing a product thesis.

Our interpretation is limited: this closed-world design did not justify an AI repair platform. It does not establish that AI has no value when desired state is unknown, documentation conflicts, or diagnosis requires connecting incomplete evidence. Those are different workloads, untested here.

The reusable lessons are more modest. Compare against competent non-AI automation before building an interface around a demo. Preserve negative requirements when repairing positive paths. Check whether a handoff reason is true. And audit the benchmark’s assumptions as carefully as the agent’s output.

For an example where correcting a declared configuration value is itself the useful intervention, see our served-model-name troubleshooting guide. A clear diagnosis and a bounded correction can be enough; they do not automatically require an autonomous repair product.

Stopping was the result of this experiment, not a failure to produce one.