Skip to content

Add cancellable deep permuter searches for one function - #137

Merged
RetroGameDeveloper merged 1 commit into
mainfrom
feat/single-function-deep-permuter
Oct 28, 2026
Merged

RetroGameDeveloper merged 1 commit into
mainfrom
feat/single-function-deep-permuter

Conversation

@RetroGameDeveloper

Copy link
Copy Markdown

Summary

Add Deep permuter · this function to the function drawer, with a time limit, candidate-compile limit, live counters and Stop button.

  • Uses the selected function's saved/scored C, in ROM and imported project modes.
  • Bypasses the background instruction ceiling and per-function turn limit for this explicit search only.
  • Visits the full currently generated local candidate list (the background's 20-candidate cap stays unchanged), then runs ordinary random permutations.
  • All improvements and exact matches use the normal compiler/link or imported object judge and save immediately.
  • Scoped cached misses are skipped across deep jobs; failed builds remain retryable, and cached improvements must be rejudged. Scalar cache entries cannot replace measured baseline instruction evidence.
  • Cancellation/time/compile limits release claims and close the generator. The UI retains logs and clears its progress indicator on success, error and cancellation.

The UI and docs explicitly state that arbitrary C rewriting cannot exhaust every possible source program or compiler option. This is a bounded deeper search, not an exhaustive proof.

Investigation

A private N64 project remained at 1,704 matches after the user's latest autopilot pass: zero new matches. Its latest recorded reuse phase ran 120 seconds with zero fresh candidate attempts. All 153 eligible normal near misses had reached their four-turn cap. This feature provides an explicit route beyond that cap without changing global settings or starting paid agents.

A separate warning-enabled compile on a private copy identified an uninitialized pointer in a saved near miss. Advisory compiler diagnostics and faster traversal of cached/indexed reuse work are useful next improvements; they are not claimed as implemented here. No game source, ROM, database or transcript is included.

Validation

  • 93 focused permuter, GUI lifecycle and route tests passed; the final baseline-evidence test also passed in the complete 14-test deep-search module.
  • Tests cover ROM/imported judge routing, keeping improvements, time/compile limits, cancellation, generator cleanup, busy/matched/unscored functions, cache behavior and retryable compile failures.
  • Real compiler/link replay on a private project copy: two four-compile jobs stopped at the requested cap; the second skipped previously judged misses and advanced to new candidates; claims were released.
  • Ruff and git diff --check passed.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 28, 2026
@RetroGameDeveloper
RetroGameDeveloper merged commit 070cc4f into main Oct 28, 2026
6 checks passed
RetroGameDeveloper pushed a commit that referenced this pull request Nov 1, 2026
)

* Xbox games start from their own default.xbe or disc image (beta)

An XBE is read and made a Windows program at the same addresses, which the PC
path imports as it is: Ghidra, the delinker, objdiff, the rebuild check and
every PC feature work on Xbox games. Kernel imports are named from nxdk's CC0
list, the XDK's library sections are set aside, and the compiler comes from
the XDK build in the library version table. A compiled demo (Visual C++ 7.1)
passes the console conformance tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Xbox in the table, the docs and the site: every feature ran on its demo

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Xbox: the rebuild gives back the game's XBE, early XDKs build with Visual C++ 7.0, Bink is set aside

The PE view keeps the XBE whole inside it and says where, so tools/rebuild.py
writes default.xbe beside the program (byte for byte your copy on a real
game). XDKs before 5233 shipped Visual C++ 7.0 builds: 4627's is the pack's
7.0, earlier ones get it as the nearest. RAD's Bink sections are licensed
middleware, set aside like the XDK's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Xbox: your own XDK sets its library code aside and builds early games with its own compiler

Setup takes the folder of your own Xbox Development Kit (checked and recorded,
never downloaded or copied). With it, the XDK's functions the linker put into
.text (the C runtime, XAPI) are found by its libraries' bytes and set aside:
432 of them in a real game, in seconds. XDKs before 5233 shipped Visual C++
7.0 builds of their own (13.00.9254 in 3911, not the pack's 9466): a game of
that XDK builds with it (MSVC/xdk, configure.py --xdk) where it's set up.
XDK 5849's compiler is byte for byte the msvc pack's 7.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Kits: a connector for the compilers and SDKs users bring, the Xbox XDK the first

A compiler that only ever came in a proprietary SDK, or an SDK a console's
games need, can't be pinned and downloaded. kits/ is its connector, like
consoles/ and agents/: a kit says what proves its folder, its build, its
compilers and the SHA-256 of the builds decompdecomp knows. Setup records the
user's folder (a local toolchain, nothing copied) and says whether the copy is
a known build; a console takes a kit by naming it (Console.kits) and its
compilers reach the build through configure.py --own. The XDK moves there from
toolchains.toml; the Xbox takes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Xbox: games built with Visual C++ 7.0 or an early XDK's compiler build without /arch:SSE

Their compilers don't have it (7.1 added it), and warned on every file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* PC projects can build a unit in LTCG harness mode, with Visual C++'s linker run through wine

Games built with /GL and /LTCG can't match one file at a time: the linker picks register conventions and inlines across files. A unit whose harness/<unit>.drv.cpp defines HarnessEntry now compiles with /GL, links with /LTCG into a DLL (tools/ltcg.py; link.exe and lib.exe go through wine, which wibo can't run them on) and tools/dll2coff.py cuts its functions into an object laid out like the delinked original, with an address-mapping check. Other units and ordinary PC projects build as before; LTCG is a kept setting when tools are brought up to date, and pc.create takes a plan hook for choices that depend on the analysed program.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Xbox imports tell link-time code generation apart and judge harness units

The import detects LTCG from call sites that load registers no standard convention uses (forks/convention.py), logs why, writes ltcg and its evidence in the manifest, uses the LTCG flags and writes a driver skeleton per game unit (forks/ltcg.py). The judge counts a function that fails the mapping or data check as not matched, repeat-compile detection sees the harness files, and Xbox builds are given wine. The demo builds with 'build.py ltcg' and tests/test_ltcg.py imports it and matches a register-convention function.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The docs say what Xbox has now: your default.xbe rebuilt, your own XDK, three real games imported

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* A piece another function jumps into stays a function of its own, so the program still rebuilds

merge_pieces counted only calls and addresses as references to a piece Ghidra
cut off. A linker that shares one tail between functions (link-time code
generation: an Xbox game of 23,000 functions had two) left the other
function's jump naming a symbol the join took away, and the rebuild check
could no longer place it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Agents can work a game built with link-time code generation: convention evidence in briefs, standins_for and unresolved names by kind

Stage 1 built LTCG units through a harness, but an agent had nothing to write one with. The brief of an LTCG project's function now shows its convention in the original (registers read before written, stack arguments, ret N, saves, a few call sites) and says whether the unit's compile goes through a driver yet, how to name a decorated function (`// FUNCTION: 0x<address>`) and where to start. The new standins_for tool (forks/standins.py) writes a starting harness for a unit: stand-ins for what it calls or reads elsewhere with their conventions, a HarnessEntry with each function's call sites, and the declarations. Names a harness link left unresolved reach compile_and_diff, the brief and a pushed-back release by kind (data, standard callee, register callee, inlined callee, library; forks/unresolved.py), and a function missing from the unit's object says what the build said about it. The Xbox guide now teaches harness mode instead of telling agents to release LTCG functions. tests/test_ltcg.py goes from claim to match through the tools an agent has, then checks the rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The import can force link-time code generation on or off for an Xbox game

Detection from the program is a guess a user may know better than: the import dialog now asks (auto, on, off) for consoles that declare import_options. The connector gets a small generic hook (Console.import_options, passed as keywords to create_project through import_game/create_scratch and the import route's `options` query), so nothing outside consoles/ compares console names. Paths picked by typing them stay on auto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The roadmap says link-time code generation works on Xbox games, and what's left of it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* LTCG briefs and standins_for say when a function is a member function, of which struct, with its fields and a declaration

Under link-time code generation `this` can arrive in eax, esi, edi, ebx or ecx, which the ECX-only struct reader
and the RTTI classes miss, so agents wrote member functions as free functions taking a pointer and guessed the
fields. forks/methods.py (ported from the prototype) reads it from the registers a function uses as the base of
fixed-offset accesses, notes "two objects alike" when two registers hit the same offsets, and groups functions by
shared evidence (vtable stores, callers passing the same global object). standins_for declares member functions
as such and writes the structs. Where RTTI already named the class, the brief only adds the register.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* An LTCG attempt whose function got another convention than the original's says which registers differ

A helper or driver call adds a call site, and the linker can then pick another convention for the callee, so
the bytes differ for a reason the C doesn't show. After an unmatched attempt the result compares the harness-made
object's function (convention.analyse) with the original's and names the registers, stack arguments or ret that
differ, with the hint to keep the original's number of call sites and check the driver's calls. The Xbox guide
gets the member-function and call-site paragraphs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The roadmap says all of the prototype's LTCG work is in

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Your own Visual C++ and several copies of any kit: games build with the compiler build they were made with

A game built with a Visual C++ build the downloadable packs don't have (a hotfix, 2012 to 2015, 2017 and newer)
could not match, and a kit held one folder only, so a user with two XDKs or several Visual C++ versions had to pick.
Setup now records any number of copies per kit (old single-folder manifests still read), lists each with its build and
a Remove button, and adds a new "Visual C++ (your own copy)" kit for PC that recognises VC98, VC and
VC/Tools/MSVC layouts. Each compiler carries its exact build, read from c2.dll's version resource, and a name unique
per copy (xdk-3911, own-9.0.30729). The Xbox picks the XDK copy whose build is the game's; the PC picks the copy
whose build the Rich header names (else the nearest of the same version), so games the packs can't build, 2017 and
newer included, are in scope with a matching copy. The kits' compilers are listed as compilers of their console so a
file can be switched to one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Adding a kit explains several copies and matching by build

Kits now hold any number of copies and consoles choose the copy whose build is the game's; contributors adding a kit
need to know parts read one copy only and version names must be unique per copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Your own Visual C++ is read by its cl.exe version, as its c2.dll says it

c2.dll's file version is cl.exe's (Visual C++ 6.0 is 12.0.8168, 2008 is
15.0.30729, 2017 is 19.16.27023), not Visual Studio's: read as Visual Studio's,
no copy matched a game's Rich header, and 2005 to 2010 ran through wine instead
of wibo. Checked against the msvc packs' own compilers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The docs say PC games of any Visual C++ build with your own copy of it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The site's PC card says any Visual C++ builds from your own copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Setup lists the kits users bring, and a filter chip for every platform it has

Compilers & tools showed only the compilers and the consoles' packs: the XDK and
your own Visual C++ never reached the page, so they couldn't be set up. Its
filter chips were a fixed list too, without Xbox, PS1 or GBA; now they come from
the platforms on the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* A kit's copies fit their card in Setup: the build, whether it's known, and the folder's end, wrapped

Each copy showed its whole folder path in the monospace installed mark, which
doesn't wrap: the cards ran into each other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* No game is named in the Xbox code and tests: a title ID example and a test's story say what kind of game

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* The fork-setup test's stand-in for PC's tool_args takes the kits it's now given

CI failed on every system: tool_args takes the console's kits since kits came in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* Your own Visual C++ 2.x, 4.x or 5.0 can build the games they compiled

A game built with Visual C++ 2.0 to 5.0 can't use the compiler packs (those compilers can't be pinned), and it has no
Rich header, so its linker's version (2.5x, 3.x/4.x, 5.x) now names the Visual C++ and picks the user's matching
copy; without a copy the message names the version to set up.

#137: the visualcpp kit reads MSVC20, MSDEV and DevStudio/VC copies (the version from c2.dll, c2.exe or cl.exe), and
rich_version maps cl 9.x to 2.0, 10.0/10.10/10.20 to 4.0/4.1/4.2 and 11.0 to 5.0. Visual C++ 1.x (16-bit) is refused.
They run through wibo, which is unverified for compilers this old: no copy was available to try.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

* A kit file rewritten within one tick of the clock is read again, not taken from the cache

file_version cached by path and mtime; Windows' coarse clock gave a stand-in rewritten
at once the same mtime, and CI on Windows read the old version. The size is in the
key too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant