Repository navigation
Add cancellable deep permuter searches for one function - #137
Merged
Merged
Conversation
RetroGameDeveloper
pushed a commit
that referenced
this pull request
Nov 1, 2026
) * Xbox games start from their own default.xbe or disc image (beta) An XBE is read and made a Windows program at the same addresses, which the PC path imports as it is: Ghidra, the delinker, objdiff, the rebuild check and every PC feature work on Xbox games. Kernel imports are named from nxdk's CC0 list, the XDK's library sections are set aside, and the compiler comes from the XDK build in the library version table. A compiled demo (Visual C++ 7.1) passes the console conformance tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * The Xbox in the table, the docs and the site: every feature ran on its demo Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Xbox: the rebuild gives back the game's XBE, early XDKs build with Visual C++ 7.0, Bink is set aside The PE view keeps the XBE whole inside it and says where, so tools/rebuild.py writes default.xbe beside the program (byte for byte your copy on a real game). XDKs before 5233 shipped Visual C++ 7.0 builds: 4627's is the pack's 7.0, earlier ones get it as the nearest. RAD's Bink sections are licensed middleware, set aside like the XDK's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Xbox: your own XDK sets its library code aside and builds early games with its own compiler Setup takes the folder of your own Xbox Development Kit (checked and recorded, never downloaded or copied). With it, the XDK's functions the linker put into .text (the C runtime, XAPI) are found by its libraries' bytes and set aside: 432 of them in a real game, in seconds. XDKs before 5233 shipped Visual C++ 7.0 builds of their own (13.00.9254 in 3911, not the pack's 9466): a game of that XDK builds with it (MSVC/xdk, configure.py --xdk) where it's set up. XDK 5849's compiler is byte for byte the msvc pack's 7.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Kits: a connector for the compilers and SDKs users bring, the Xbox XDK the first A compiler that only ever came in a proprietary SDK, or an SDK a console's games need, can't be pinned and downloaded. kits/ is its connector, like consoles/ and agents/: a kit says what proves its folder, its build, its compilers and the SHA-256 of the builds decompdecomp knows. Setup records the user's folder (a local toolchain, nothing copied) and says whether the copy is a known build; a console takes a kit by naming it (Console.kits) and its compilers reach the build through configure.py --own. The XDK moves there from toolchains.toml; the Xbox takes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Xbox: games built with Visual C++ 7.0 or an early XDK's compiler build without /arch:SSE Their compilers don't have it (7.1 added it), and warned on every file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * PC projects can build a unit in LTCG harness mode, with Visual C++'s linker run through wine Games built with /GL and /LTCG can't match one file at a time: the linker picks register conventions and inlines across files. A unit whose harness/<unit>.drv.cpp defines HarnessEntry now compiles with /GL, links with /LTCG into a DLL (tools/ltcg.py; link.exe and lib.exe go through wine, which wibo can't run them on) and tools/dll2coff.py cuts its functions into an object laid out like the delinked original, with an address-mapping check. Other units and ordinary PC projects build as before; LTCG is a kept setting when tools are brought up to date, and pc.create takes a plan hook for choices that depend on the analysed program. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Xbox imports tell link-time code generation apart and judge harness units The import detects LTCG from call sites that load registers no standard convention uses (forks/convention.py), logs why, writes ltcg and its evidence in the manifest, uses the LTCG flags and writes a driver skeleton per game unit (forks/ltcg.py). The judge counts a function that fails the mapping or data check as not matched, repeat-compile detection sees the harness files, and Xbox builds are given wine. The demo builds with 'build.py ltcg' and tests/test_ltcg.py imports it and matches a register-convention function. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The docs say what Xbox has now: your default.xbe rebuilt, your own XDK, three real games imported Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * A piece another function jumps into stays a function of its own, so the program still rebuilds merge_pieces counted only calls and addresses as references to a piece Ghidra cut off. A linker that shares one tail between functions (link-time code generation: an Xbox game of 23,000 functions had two) left the other function's jump naming a symbol the join took away, and the rebuild check could no longer place it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Agents can work a game built with link-time code generation: convention evidence in briefs, standins_for and unresolved names by kind Stage 1 built LTCG units through a harness, but an agent had nothing to write one with. The brief of an LTCG project's function now shows its convention in the original (registers read before written, stack arguments, ret N, saves, a few call sites) and says whether the unit's compile goes through a driver yet, how to name a decorated function (`// FUNCTION: 0x<address>`) and where to start. The new standins_for tool (forks/standins.py) writes a starting harness for a unit: stand-ins for what it calls or reads elsewhere with their conventions, a HarnessEntry with each function's call sites, and the declarations. Names a harness link left unresolved reach compile_and_diff, the brief and a pushed-back release by kind (data, standard callee, register callee, inlined callee, library; forks/unresolved.py), and a function missing from the unit's object says what the build said about it. The Xbox guide now teaches harness mode instead of telling agents to release LTCG functions. tests/test_ltcg.py goes from claim to match through the tools an agent has, then checks the rebuild. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The import can force link-time code generation on or off for an Xbox game Detection from the program is a guess a user may know better than: the import dialog now asks (auto, on, off) for consoles that declare import_options. The connector gets a small generic hook (Console.import_options, passed as keywords to create_project through import_game/create_scratch and the import route's `options` query), so nothing outside consoles/ compares console names. Paths picked by typing them stay on auto. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The roadmap says link-time code generation works on Xbox games, and what's left of it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * LTCG briefs and standins_for say when a function is a member function, of which struct, with its fields and a declaration Under link-time code generation `this` can arrive in eax, esi, edi, ebx or ecx, which the ECX-only struct reader and the RTTI classes miss, so agents wrote member functions as free functions taking a pointer and guessed the fields. forks/methods.py (ported from the prototype) reads it from the registers a function uses as the base of fixed-offset accesses, notes "two objects alike" when two registers hit the same offsets, and groups functions by shared evidence (vtable stores, callers passing the same global object). standins_for declares member functions as such and writes the structs. Where RTTI already named the class, the brief only adds the register. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * An LTCG attempt whose function got another convention than the original's says which registers differ A helper or driver call adds a call site, and the linker can then pick another convention for the callee, so the bytes differ for a reason the C doesn't show. After an unmatched attempt the result compares the harness-made object's function (convention.analyse) with the original's and names the registers, stack arguments or ret that differ, with the hint to keep the original's number of call sites and check the driver's calls. The Xbox guide gets the member-function and call-site paragraphs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The roadmap says all of the prototype's LTCG work is in Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Your own Visual C++ and several copies of any kit: games build with the compiler build they were made with A game built with a Visual C++ build the downloadable packs don't have (a hotfix, 2012 to 2015, 2017 and newer) could not match, and a kit held one folder only, so a user with two XDKs or several Visual C++ versions had to pick. Setup now records any number of copies per kit (old single-folder manifests still read), lists each with its build and a Remove button, and adds a new "Visual C++ (your own copy)" kit for PC that recognises VC98, VC and VC/Tools/MSVC layouts. Each compiler carries its exact build, read from c2.dll's version resource, and a name unique per copy (xdk-3911, own-9.0.30729). The Xbox picks the XDK copy whose build is the game's; the PC picks the copy whose build the Rich header names (else the nearest of the same version), so games the packs can't build, 2017 and newer included, are in scope with a matching copy. The kits' compilers are listed as compilers of their console so a file can be switched to one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Adding a kit explains several copies and matching by build Kits now hold any number of copies and consoles choose the copy whose build is the game's; contributors adding a kit need to know parts read one copy only and version names must be unique per copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Your own Visual C++ is read by its cl.exe version, as its c2.dll says it c2.dll's file version is cl.exe's (Visual C++ 6.0 is 12.0.8168, 2008 is 15.0.30729, 2017 is 19.16.27023), not Visual Studio's: read as Visual Studio's, no copy matched a game's Rich header, and 2005 to 2010 ran through wine instead of wibo. Checked against the msvc packs' own compilers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The docs say PC games of any Visual C++ build with your own copy of it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The site's PC card says any Visual C++ builds from your own copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Setup lists the kits users bring, and a filter chip for every platform it has Compilers & tools showed only the compilers and the consoles' packs: the XDK and your own Visual C++ never reached the page, so they couldn't be set up. Its filter chips were a fixed list too, without Xbox, PS1 or GBA; now they come from the platforms on the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * A kit's copies fit their card in Setup: the build, whether it's known, and the folder's end, wrapped Each copy showed its whole folder path in the monospace installed mark, which doesn't wrap: the cards ran into each other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * No game is named in the Xbox code and tests: a title ID example and a test's story say what kind of game Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * The fork-setup test's stand-in for PC's tool_args takes the kits it's now given CI failed on every system: tool_args takes the console's kits since kits came in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * Your own Visual C++ 2.x, 4.x or 5.0 can build the games they compiled A game built with Visual C++ 2.0 to 5.0 can't use the compiler packs (those compilers can't be pinned), and it has no Rich header, so its linker's version (2.5x, 3.x/4.x, 5.x) now names the Visual C++ and picks the user's matching copy; without a copy the message names the version to set up. #137: the visualcpp kit reads MSVC20, MSDEV and DevStudio/VC copies (the version from c2.dll, c2.exe or cl.exe), and rich_version maps cl 9.x to 2.0, 10.0/10.10/10.20 to 4.0/4.1/4.2 and 11.0 to 5.0. Visual C++ 1.x (16-bit) is refused. They run through wibo, which is unverified for compilers this old: no copy was available to try. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH * A kit file rewritten within one tick of the clock is read again, not taken from the cache file_version cached by path and mtime; Windows' coarse clock gave a stand-in rewritten at once the same mtime, and CI on Windows read the old version. The size is in the key too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kw52ayMLQzgUMTVxfzDfXH --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Deep permuter · this function to the function drawer, with a time limit, candidate-compile limit, live counters and Stop button.
The UI and docs explicitly state that arbitrary C rewriting cannot exhaust every possible source program or compiler option. This is a bounded deeper search, not an exhaustive proof.
Investigation
A private N64 project remained at 1,704 matches after the user's latest autopilot pass: zero new matches. Its latest recorded reuse phase ran 120 seconds with zero fresh candidate attempts. All 153 eligible normal near misses had reached their four-turn cap. This feature provides an explicit route beyond that cap without changing global settings or starting paid agents.
A separate warning-enabled compile on a private copy identified an uninitialized pointer in a saved near miss. Advisory compiler diagnostics and faster traversal of cached/indexed reuse work are useful next improvements; they are not claimed as implemented here. No game source, ROM, database or transcript is included.
Validation