Skip to content

fix(llm): pick CUDA only when an NVIDIA GPU actually answers, add engine Repair - #388

Merged
lightningpixel merged 2 commits into
devfrom
fix/llm-nvidia-detection
Oct 28, 2026
Merged

lightningpixel merged 2 commits into
devfrom
fix/llm-nvidia-detection

Conversation

@Lorchie

@Lorchie Lorchie commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Fixes #386.

Problem

has_nvidia_gpu() only checked that nvidia-smi was on PATH. On a machine where nvidia-smi.exe is present but no NVIDIA driver/GPU answers (the reporter has an RX 9070 XT), _pick_assets() installed the Windows CUDA build of llama.cpp, so llama-server could not use the GPU and ran on CPU.

Users who already got the wrong build were stuck: the Agent settings only show "Install engine" while the engine is missing.

Changes

  • Detection (api/services/llm_server.py): has_nvidia_gpu() now runs nvidia-smi -L and requires a zero exit code plus at least one GPU N: line. Cached, since /llm/status calls it on every poll. No usable NVIDIA GPU → Vulkan build, as before.
  • Repair (GET /llm/binary/install?repair=true, Settings › Agent › Engine › Repair): re-detects the GPU, downloads the build, extracts it into bin.new, stops the loaded llama-servers, then swaps bin → bin.old, bin.new → bin. The current engine is only touched once the new one is fully extracted, so a failed or cancelled download keeps a working install, and no stale backend DLL (ggml-cuda.dll) survives the swap.

Testing

  • api/tests/test_llm_gpu_detection.py (11 tests): nvidia-smi missing / failing without driver / listing no GPU / crashing / working; Vulkan vs CUDA+cudart selection; repair replaces the old files, keeps them on failed download and on a locked folder, plain install untouched.
  • Real run on Windows (RTX 5070) into a temp MODLY_LLM_DIR seeded with a stale DLL: release b10075, CUDA 12.4 + cudart picked, stale DLL gone, llama-server --list-devices sees the GPU. The same release resolves to llama-b10075-bin-win-vulkan-x64.zip when no NVIDIA GPU is detected.
  • Also checked on Windows: renaming bin works even while an exe inside it is running, so the swap cannot leave a half-deleted engine; the old copy stays in bin.old and is removed on the next repair.
  • npm test (pre-push hook) passes. tsc adds no error, eslint clean.

Also in this PR: two mesh-ops tests that failed on a clean dev

Separate commit, they blocked the pre-push hook:

  • test_auto_orient_does_not_flip_a_mesh_over: trimesh's update_faces keeps the cut-away vertices, so the exported bowl still carried the whole sphere and the test measured the bowl's "center" on the orphan top cap. The fixture now drops unreferenced vertices; auto_orient itself was correct.
  • test_mesh_ops_run_without_undeclared_networkx runs repair_mesh, which needs pymeshlab: it is now skipped when pymeshlab is missing, like the other repair/inspection tests (all 42 mesh-ops tests pass with pymeshlab installed).
  • Not tested: the Repair button clicked in the packaged app, and an actual AMD machine.

…ine Repair

has_nvidia_gpu() only checked that nvidia-smi was on PATH. A leftover
nvidia-smi.exe with no NVIDIA driver/GPU behind it (e.g. an AMD machine) got
the CUDA llama.cpp build, and llama-server fell back to CPU. It now requires
`nvidia-smi -L` to succeed and list a GPU; the result is cached since
/llm/status calls it on every poll.

Machines that already received the wrong build had no way back: the UI only
offers "Install engine" while the engine is missing. Settings > Agent now has
an Engine > Repair action (GET /llm/binary/install?repair=true) that
re-detects the GPU, downloads and extracts the new build into a staging dir,
then swaps it in place of bin/. The working engine is only touched once the
new one is fully extracted, so a failed download keeps it intact.

Fixes #386
- test_auto_orient_does_not_flip_a_mesh_over: trimesh's update_faces keeps
  the cut-away vertices, so the exported bowl still carried the whole
  sphere's vertices and the "center" was measured on the orphan top cap.
  Drop unreferenced vertices; auto_orient itself was correct.
- test_mesh_ops_run_without_undeclared_networkx runs repair_mesh, which
  needs pymeshlab: skip it when pymeshlab is missing, like the other
  repair/inspection tests.
@lightningpixel
lightningpixel merged commit 66c1d3f into dev Oct 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants