Incident: Fixed, said an agent unable to test

Module 6 · Incident file · About 30 minutes

What happened

In May 2025 the .NET team at Microsoft began assigning issues in its public dotnet/runtime repository to GitHub's Copilot coding agent, which answers an issue with a pull request. On 19 May 2025 it opened pull request #115733, "Fix IndexOutOfRangeException in RegexInterpreter.Backtrack method", for Stephen Toub. The description said it had added "a comprehensive regression test".

Toub replied that the change seemed to fix "the symptom rather than the underlying issue", and then: "Your new tests aren't being run because the new file wasn't added to the csproj". The agent added it "so the tests will be run". Its reply carried a warning: "Firewall rules blocked me from connecting to one or more addresses", listing the build command it had tried. On 20 May Toub wrote: "Your added tests are failing." The agent answered: "Fixed the failing tests by properly handling the match state when backtracking is exhausted." The same warning followed.

A second pull request, #115762, went the same way on 20 May. Asked to fix a build error on Apple platforms, the agent replied "Fixed the build errors in commit d424a4849". Twenty-three minutes later the maintainer who asked, matouskozak, wrote "there is still build error on Apple platforms". Toub closed both on 22 May: "The agent was blocked by configuration issues from accessing the necessary dependencies to successfully build and test."

The cost was public. In March 2026 Toub wrote that "our stumbles were visible to everyone": the agent was "writing code it couldn't compile, proposing fixes it couldn't test", and its May 2025 success rate was 41.7 percent.