fix: return the polished raw-forward PDIP iterate as the forward value (#571/#573 follow-up)

[Jammy2211]

Overview

The released mapper-less positive-only solver (nnls_preconditioning_no_mapper: raw, #572) stops on a data-scaled absolute KKT residual, so columns fnnls holds at zero keep x ≈ tol / z while reporting "converged": the euclid total_source_flux latent traces +5.76 % off under jax.jit and the euclid latent test has been red since 2026-09-25, invisible to logL, objective and KKT. Phase 1 (autolens_profiling #355) measured this on 81 systems — raw PDIP is converged 81/81 yet its source-column flux is >1e-3 off fnnls on 53/81, and it leaves 11.5 % of the reference amplitude on reference-inactive euclid columns — while applying the #573 polish to the forward value gave 0/81 unconverged, worst inactive-column flux 3.3e-4, source-flux misses 1/81 and euclid latent +7.5e-5 at a median cost of +5 iterations. Decision: return the polished iterate (≤ 10 tight warm-started Jacobi-system PDIP iterations, already in the library from #573 for the backward pass) as the forward value, in both the custom_vjp forward and the primal so jit, grad and eager agree.

Plan

• Factor the raw forward solve + fix: NaN JAX gradient on MGE positive-only solves after #572 #573 polish into one helper in autoarray/util/jax_nnls.py and return the polished iterate (falling back to the unpolished one when the polish is not ok) as the forward value, from both the custom_vjp forward and the primal; backward pass unchanged.
• Keep converged / iterations as the raw forward solve's; keep the RAW_BACKWARD_POLISH_MAX_ITER name (imported downstream) and add RAW_POLISH_MAX_ITER as the preferred alias.
• Update the module and function docstrings, the nnls_preconditioning_no_mapper comment in general.yaml, and the settings.py / inversion_util.py docstrings to describe the polished forward value and why.
• Add a regression fixture (the 8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems + the euclid vis_lp system, with fnnls x_ref) and a jax-only amplitude test gating on inactive-column flux, total flux and source flux at 1e-3 plus jit==eager and finite non-zero grad — run red-first on unfixed d4298445 and record the failing cases here before the fix.
• Keep test_nnls_mge_convergence.py and test_jax_nnls.py green unchanged; full PyAutoArray pytest.
• Verify downstream: the euclid latent jit test passes on the fixed library with no config override; autolens_profiling accuracy re-run (81/81 converged, inactive-column worst ≤ 3.3e-4) becomes the ledger row after the library merge; sweep workspace_test mapper-less likelihood pins for moved values (re-pin is a follow-up).

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary)
• autolens_profiling (downstream: ledger row in results/notes/linear_solver_accuracy_2026_09.md + campaign page + regenerated README stats, after the library merge)
• euclid_strong_lens_modeling_pipeline (verification only, no edit)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	main (d429844)	clean — parallel claim with streaming-p1-array-free-dataset, disjoint files (human-approved 2026-09-30)
./autolens_profiling	main (3ad68af)	1 untracked dir (dataset/abell_1201/)
./euclid_strong_lens_modeling_pipeline	main (26e4385)	clean

Suggested branch: feature/raw-pdip-forward-polish

Deviation from the prompt

The prompt's witness thresholds (amp_rel_max ≤ 1e-3, flux_rel ≤ 1e-4) are not achievable by any candidate: amp_rel_max_sig worst 0.219 is shared by every accurate candidate (fnnls reference noise on near-flat directions), and the euclid "source columns" metric is +4.96e-2 for polish only because its three source columns carry negligible reference flux (the end-to-end latent is +7.5e-5). The regression test therefore gates on inactive-column flux and total/source flux at 1e-3, per system, chosen from the phase-1 rows before the library test is written, and drops the amplitude-max criterion. Recorded as a rule weakness in the phase-1 ledger already.

Implementation Steps

1. autoarray/util/jax_nnls.py
   • Factor the raw forward + polish out of _raw_forward_backward_point (:196-234) into _raw_forward_polished(Q_pc, q_pc, Q, q, D, solver_tol, max_iter) returning (y_out, converged, pdip_iter, (yp, sp, zp), ok, polish_iter) where y_out = where(ok, yp, y) — the same ok rule as today (polish converged, finite, sp>0, zp>0), falling back to the unpolished iterate.
   • _raw_forward_backward_point calls it, runs solve_relaxed_nnls from (yp, sp, zp) as now, and returns (y_out, converged, pdip_iter) as out — the polished forward value.
   • _solve_nnls_raw_forward_with (:237-296): the primal (:276-281) must call the same _raw_forward_polished and return y_out / … exactly as the fwd does, so jit, grad and eager agree bit-for-bit. Backward unchanged.
   • converged stays the raw forward solve's flag; iterations stays the forward count (status already carries polish_iter, ok). Keep the name RAW_BACKWARD_POLISH_MAX_ITER (imported by autolens_profiling _solvers.py:248); add RAW_POLISH_MAX_ITER = RAW_BACKWARD_POLISH_MAX_ITER and prefer the new name internally.
   • Rewrite the module docstring (:30-34) and the _solve_nnls_raw_forward_with docstring (:262-263): the polish is now forward and backward; say why (complementarity under the data-scaled stop leaves tol/z on inactive columns; phase-1 numbers).
2. autoarray/config/general.yaml:11 comment for nnls_preconditioning_no_mapper and autoarray/settings.py:227-237 docstring: describe the polished forward value. inversion_util.py:362-364 note on converged/iterations if it mentions the unpolished iterate.
3. Tests
   • New fixture test_autoarray/inversion/inversion/files/mge_solver_reference_systems.npz (≈180 KB): the 8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems + the euclid vis_lp system copied from autolens_profiling results/lens/solver/corpus/{slam_fixture_571,euclid_vis_lp}.npz with x_ref (fnnls) and a meta JSON carrying source_column_index_list, max_abs_q, group, provenance (corpus commit 3ad68af). Add a paragraph to files/README.md.
   • New test_autoarray/inversion/inversion/test_nnls_raw_forward_amplitude.py (jax-only, mirrors test_nnls_mge_convergence.py skips/EPSILON guard). Per system, through the library entry point solve_nnls_primal_raw_forward (built as inversion_util.py:462-481 builds it) and through reconstruction_positive_only_from with preconditioning="raw":
     • converged == 1, iterations < 50, finite;
     • flux_inactive_rel = Σx[x_ref ≤ 1e-6·max x_ref] / Σx_ref ≤ 1e-3 (raw is red on euclid: 0.115);
     • |Σx − Σx_ref| / Σx_ref ≤ 1e-3 (raw red on euclid);
     • |flux_rel_source| ≤ 1e-3 on the 8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems (raw red on k1, k2, k3, k5: 1.6e-3…6.4e-3; polish ≤ 5e-5) — euclid excluded from this one metric with the reason above;
     • jax.jit(f)(…) equals eager f(…) (assert_array_equal) and jax.grad finite/non-zero on every system (extends the fix: NaN JAX gradient on MGE positive-only solves after #572 #573 tests to the new fixture).
     • Docstring records the thresholds, the phase-1 values they were chosen from, and which cases are red on d4298445.
   • Existing test_nnls_mge_convergence.py and test_jax_nnls.py must stay green unchanged (the objective/control/gradient assertions all tighten under polish; test__control__well_conditioned_pdip_unchanged requires Jacobi mode bit-identical — untouched).
   • Red-first: run the new test on unfixed d4298445 and record the failing cases on the issue before the fix commit.
4. Full PyAutoArray pytest, jax_grad-style guards: none new.

Verification beyond the library

• euclid witness (no edit): in the canonical lens/euclid_strong_lens_modeling_pipeline (clean, main), run pytest tests/test_compute_latent_variable.py -k traces_under_jax_jit with the worktree PyAutoArray first on PYTHONPATH; must pass at JIT_VS_EAGER_REL = 1e-3 with no config override (expected +7.5e-5). Also run the full file to confirm the NumPy path is untouched.
• SLaM 48/48 stays converged + corpus re-measure: in the autolens_profiling worktree, run scripts/lens/solver/accuracy.py --posthoc with the fixed PyAutoArray (its library-entry candidate calls solve_nnls_primal_raw_forward, _solvers.py:163-167); expect 81/81 converged, inactive-column worst ≤ 3.3e-4, euclid latent via euclid_latent.py within 1e-3. This is the downstream row.
• workspace_test pins: grep lens/autolens_workspace_test for mapper-less (MGE / light_lp linear) likelihood pins and run those scripts under the smoke profile with the fixed library; report any value that moved (re-pin is a follow-up ship_workspace, not this PR).
• Standard: ruff, black --check per repo convention, full pytest test_autoarray.

Key Files

• autoarray/util/jax_nnls.py — _raw_forward_backward_point, _solve_nnls_raw_forward_with, new _raw_forward_polished, module docstring
• autoarray/config/general.yaml — nnls_preconditioning_no_mapper comment
• autoarray/settings.py — preconditioning docstring
• autoarray/inversion/inversion/inversion_util.py — raw branch / converged note
• test_autoarray/inversion/inversion/test_nnls_raw_forward_amplitude.py — new amplitude regression test
• test_autoarray/inversion/inversion/files/mge_solver_reference_systems.npz + files/README.md — new fixture
• test_autoarray/inversion/inversion/test_nnls_mge_convergence.py, test_autoarray/util/test_jax_nnls.py — must stay green unchanged

Original Prompt

Click to expand starting prompt

Linear-solver programme phase 2: fix the raw-forward PDIP amplitude bias in PyAutoArray

Type: bug
Target: PyAutoArray
Repos:

• PyAutoArray
• euclid_strong_lens_modeling_pipeline
• autolens_profiling
  Difficulty: medium
  Autonomy: supervised
  Priority: high
  Status: formalised
  Filed: 2026-09-30
  Blocked-by: none (the PyAutoArray claim of sparse-data-none-guard cleared 2026-09-30 — complete/2026/09/sparse-data-none-guard.md, PyAutoArray#591 merged)
  Witness: a PyAutoArray regression test loads the phase-1 corpus npz (copied from autolens_profiling results/lens/solver/corpus/ as a fixture beside test_autoarray/inversion/inversion/files/mge_slam_nnls_systems.npz) and asserts, per system, converged AND amplitude agreement with fnnls (amp_rel_max ≤ 1e-3, source flux_rel ≤ 1e-4) — an amplitude assertion, not logL — red on unfixed main; plus euclid tests/test_compute_latent_variable.py::test_latent_euclid_variables_traces_under_jax_jit passes on library main with no config override.
  Review-minutes: 3
  Consequence: glance
  Unattended: needs-slicing

Epic linear-solver-programme, phase 2. Contract: complete/2026/09/linear-solver-accuracy-study.md (phase-1 record with the original programme prompt folded in; the phase-1 verdict blocker is satisfied — autolens_profiling#355 merged 2026-09-30).

Symptom

Raw PDIP (PyAutoArray#572, nnls_preconditioning_no_mapper: raw, the released mapper-less
default) stops on the objective (gap 4.7e-6 vs fnnls) with amplitudes 4.3 % off fnnls, so
euclid total_source_flux traces to 3.511 under jax.jit vs 3.320 eager. The euclid latent
test has been red since 2026-09-25. logL cannot see it (Δχ² ~1e-5); amplitude latents shift ~6 %.

Fix

Candidate: phase 1 verdict (autolens_profiling#355, complete/2026/09/linear-solver-accuracy-study.md): no candidate passes the pre-registered rule — raw PDIP reports converged on 81/81 (KKT ~3e-14) yet leaves 11.5 % of the reference amplitude on euclid columns inactive in the reference (total_source_flux +5.76 %); jacobi diverges on 29/81. Post-hoc, each of these turns the euclid latent test green: forward polish (+7.5e-5), tol 1e-5 (+5.1e-4), jaxnnls tol with cap > 50 (-3e-8); caps ≤ 16 are unsafe. The binding constraint is the stopping test, not the iteration budget. Choose among these on the post-hoc evidence and gate on inactive-column flux, not logL/KKT.

Implement exactly the candidate the phase-1 pre-registered rule admits (converged 100 %, 48/48
SLaM incl.; worst amp_rel_max ≤ 1e-3; worst source flux_rel ≤ 1e-4; KKT ≤ 10× pdip_jacobi's;
lowest median iterations). The three plausible shapes:

1. A tighter data-scaled tolerance constant in data_scaled_solver_tol
   (autoarray/util/jax_nnls.py).
2. Return the PyAutoArray#573 polished iterate (≤ 10 warm-started Jacobi iterations) as the
   forward value in _raw_forward_backward_point / _solve_nnls_raw_forward_with. The
   custom_vjp forward must return the same value the primal (solve_nnls_primal_raw_forward)
   returns, or jit/grad and eager diverge — polish both or neither.
3. A KKT / solution-based stop instead of the objective-gap stop.

Also touch the inversion_util.py raw branch if the candidate needs it, and update the
nnls_preconditioning_no_mapper comment in general.yaml to describe the new behaviour.

Constraint

The 48/48 SLaM source_lp[1] points must stay converged (#571); the existing
mge_slam_nnls_systems.npz test must stay green.

Rejected

• Loosening JIT_VS_EAGER_REL in the euclid test.
• A pipeline config stopgap to jacobi (reintroduces fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571).

Downstream

• workspace_test mapper-less likelihood pins: re-pin if the forward value moves.
• autolens_profiling: re-run scripts/lens/solver/accuracy.py post-fix and append the row to
  results/notes/linear_solver_accuracy_2026_09.md (+ campaign page
  wiki/campaigns/linear_solver_accuracy.md).

Ownership and order

PyAutoArray owns the solver; euclid_strong_lens_modeling_pipeline owns the witness test (no edit
expected there); autolens_profiling is the corpus fixture source. Library-first: ship PyAutoArray,
then re-verify the euclid test on library main.

🤖 Generated with Claude Code

[Jammy2211]

Progress — 2026-09-30: implemented locally, red-first verified, euclid witness green

Branch feature/raw-pdip-forward-polish (worktree ~/Code/PyAutoLabs-wt/raw-pdip-forward-polish/PyAutoArray, base bd03e09e), not yet pushed:

• 7e62fa4d test: red amplitude regression for the raw-forward PDIP forward value — new fixture test_autoarray/inversion/inversion/files/mge_solver_reference_systems.npz (8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems + euclid vis_lp, fnnls x_ref, from the phase-1 corpus at autolens_profiling 3ad68af) and test_nnls_raw_forward_amplitude.py (107 cases, both the library entry point and reconstruction_positive_only_from).
• 31b1c2d7 fix: return the polished raw-forward PDIP iterate as the forward value — _raw_forward_polished() shared by the custom_vjp primal and forward so jit/grad/eager agree bit-for-bit; backward unchanged; RAW_POLISH_MAX_ITER added, RAW_BACKWARD_POLISH_MAX_ITER kept as alias; docstrings, general.yaml and settings.py wording updated.

Red on the unfixed base (12/107, both call paths): euclid inactive-column flux 1.152e-1 and total flux 1.152e-1; source flux k1 1.638e-3, k2 1.220e-3, k3 4.044e-3, k5 6.442e-3 (threshold 1e-3). jit-vs-eager, primal-vs-vjp and gradient cases were already green.

After the fix: 107/107; every system converges in 16–24 forward iterations with the polish accepted in 5–6; euclid inactive-column flux 3.31e-4, others ≤ 3.4e-7; source flux on k0–k7 ≤ 4.93e-5. test_nnls_mge_convergence.py + test_jax_nnls.py unchanged green; full pytest test_autoarray 1890 passed; black clean on the changed files.

Euclid witness (no edit, no config override): tests/test_compute_latent_variable.py 19/19 with the worktree library, including test_latent_euclid_variables_traces_under_jax_jit; total_source_flux jit-vs-eager +7.47e-5 (base +5.76e-2), magnification −7.47e-5, other latents ≤ 1.1e-8.

Notes: the jit-vs-eager check on the bare entry point builds the Jacobi scaling outside the traced function (XLA reorders 1/sqrt with x/D by 1 ULP on the base — not solver behaviour); the end-to-end path stays bit-exact. Euclid is excluded from the source-column flux metric: its single active source column carries ~0.4 % of the reference flux, so that metric reads +4.96e-2 even polished; the end-to-end latent test covers it.

Next: corpus re-measure (48/48 SLaM must stay converged) and workspace_test mapper-less pin sweep, then /ship_library.

[Jammy2211]

Summary

Closes #594 (epic linear-solver-programme, phase 2). The mapper-less positive-only solver (nnls_preconditioning_no_mapper: raw, #572) stops on an absolute infinity-norm KKT residual with a data-scaled tolerance, so complementarity s·z is judged against a threshold scaled by max|q| and a column fnnls holds at zero can keep x ≈ tol/z while reporting converged. Phase 1 (autolens_profiling#355) measured this on 81 captured systems: raw PDIP is converged 81/81 with objective gap ~1e-12, yet its source-column flux is >1e-3 off fnnls on 53/81 systems and on the euclid vis_lp system 11.5 % of the reference amplitude sits on reference-inactive columns (total_source_flux +5.76 %, the red euclid latent test since 2026-09-25). Log-likelihood, objective and KKT are all blind to it.

The fix returns the #573 (PR #574) polished iterate — ≤ 10 tight-tolerance PDIP iterations on the Jacobi-preconditioned system, warm-started from the raw iterate — as the forward value, not only as the start of the backward relaxed-KKT solve. _raw_forward_polished() is shared by the custom_vjp primal and forward so jit, grad and eager return the identical value; the backward pass is unchanged; the polish falls back to the raw iterate under the same ok rule as before. Measured on the 81-system corpus with this branch: pdip_raw is now bit-identical to phase 1's pdip_raw_polish candidate on 81/81 (all 48 SLaM points converged), worst inactive-column flux 3.3e-4 (was 0.115), source-flux misses 1/81 (was 53/81; the remaining one is the euclid source-column proxy, see below), polish accepted every time in 1–7 iterations (mostly 5). Euclid total_source_flux jit-vs-eager is now +7.47e-5 (was +5.76e-2) and tests/test_compute_latent_variable.py passes 19/19 on the pipeline's main with no config override.

Not admissible under phase 1's pre-registered rule as written (criteria 2–4 still fail: the euclid source-column proxy at 4.96e-2 because its single active source column carries ~0.4 % of the reference flux, one flat-direction system at amp_rel_max_sig 0.219 shared by every accurate candidate incl. tight solves, and 5/52 KKT ratios at the 3e-16 floating-point floor). Those are recorded rule weaknesses, not solver defects; the regression test gates on the flux metrics that the phase-1 evidence supports.

API Changes

No public signature changes. The raw-mode forward reconstruction (reconstruction_positive_only_from(..., preconditioning="raw") and solve_nnls_primal_raw_forward) now returns the polished iterate, so mapper-less (MGE / linear light profile) JAX reconstructions and every latent derived from them move at the ~1e-4 relative level on affected systems (euclid total_source_flux −5.4 %; log-likelihoods move at ~1e-11 relative — autolens_workspace_test pins unchanged, mge_group.py JIT now equals NumPy to 6e-12). converged / iterations still describe the raw forward solve; raw_forward_backward_status still reports the polish flag and count. RAW_POLISH_MAX_ITER added; RAW_BACKWARD_POLISH_MAX_ITER kept as an alias.
See full details below.

Test Plan

• Red-first: test_nnls_raw_forward_amplitude.py fails 12/107 on the base bd03e09e (euclid inactive-column + total flux 1.152e-1; source flux k1 1.638e-3, k2 1.220e-3, k3 4.044e-3, k5 6.442e-3 vs 1e-3), both call paths; green 107/107 after the fix
• test_nnls_mge_convergence.py + test_jax_nnls.py unchanged green (173 with the new file); full pytest test_autoarray 1890 passed; black clean on changed files
• jit == eager bit-for-bit through reconstruction_positive_only_from; jax.vjp output == plain call; jax.grad finite and non-zero on all 9 fixture systems
• euclid_strong_lens_modeling_pipeline tests/test_compute_latent_variable.py 19/19 with this branch (no edit, no override); total_source_flux jit-vs-eager +7.47e-5
• autolens_profiling scripts/lens/solver/accuracy.py (+--posthoc) on 81 systems: pdip_raw == pdip_raw_polish 81/81, 48/48 SLaM converged
• autolens_workspace_test smoke: imaging jax_likelihood/{lp,mge_group,rectangular_mge,rectangular_mge_rtu,delaunay_mge}.py, interferometer jax_likelihood/delaunay_mge.py, jax_grad/pixelization.py all pass; no pin moved
• CI green on the PR

Full API Changes (for automation & release notes)

Changed

• autoarray.util.jax_nnls.solve_nnls_primal_raw_forward(...) / reconstruction_positive_only_from(..., preconditioning="raw") — the forward value is now the polished iterate (≤ RAW_POLISH_MAX_ITER tight warm-started PDIP iterations on the Jacobi system, falling back to the raw iterate when the polish does not converge or leaves the interior). Reconstructions and derived latents on mapper-less positive-only JAX inversions move at ~1e-4 relative on ill-conditioned MGE systems; log-likelihoods at ~1e-11. converged and iterations unchanged in meaning (raw forward solve).
• autoarray/config/general.yaml nnls_preconditioning_no_mapper comment, Settings.nnls_preconditioning_no_mapper docstring — describe the polished forward value.

Added

• autoarray.util.jax_nnls.RAW_POLISH_MAX_ITER = 10 (RAW_BACKWARD_POLISH_MAX_ITER kept as an alias)
• autoarray.util.jax_nnls._raw_forward_polished(Q_pc, q_pc, Q, q, D, solver_tol, max_iter) (private; shared by the custom_vjp primal and forward)
• test_autoarray/inversion/inversion/files/mge_solver_reference_systems.npz — 8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems + euclid vis_lp with fnnls x_ref (from the autolens_profiling phase-1 corpus at 3ad68af); test_nnls_raw_forward_amplitude.py

Removed

• none

Migration

• none required. Pinned mapper-less reconstruction/latent values captured with raw mode before this change move by up to ~5 % on systems where the raw stop left flux on inactive columns (euclid-like); re-pin against the polished value.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

[Jammy2211]

Library PR Created — 2026-09-30

PR: #595 (head 31b1c2d, label pending-release), opened under the Heart RED development override recorded below and in the PR body. Merge stays with /prm on green checks; no release.

Workspace impact: option (ii) — autolens_profiling needs its phase-2 ledger row, campaign page and README stats from the post-fix accuracy run (results re-measured in the task worktree, same-stamp collision v2026.8.17.1 to resolve), and scripts/lens/solver/euclid_latent.py's validation leg is pinned to the pre-fix jit value 3.511 and must be re-based to the eager 3.3199. euclid_strong_lens_modeling_pipeline: no edit (its latent test already passes with this branch). autolens_workspace_test: no pin moved, no edit.

• Authorization: live human in the Claude Code CLI session, 2026-09-30: "ok do phase 2" launched this task after the phase-1 hand-back, and at the ship gate the human pushed feature/raw-pdip-forward-polish directly (~16:35 BST) in response to the reported RED reasons, after the auto-mode classifier denied the agent push. Scope: development shipping (push + this pending-release PR); merge only via a separate /prm on all-green required checks; no release.
• Heart readiness at the 16:25 BST gate (verbatim): RED release validation FAILED (stage integrate); YELLOW workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py); YELLOW manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml. None involve PyAutoArray. This PR does not repair Heart; Heart remains RED for release.
• Passed branch gates at 31b1c2d: red-first regression (12/107 red on base, 107/107 after); test_nnls_mge_convergence.py + test_jax_nnls.py unchanged; full pytest test_autoarray 1890 passed; black clean on changed files; euclid pipeline test_compute_latent_variable.py 19/19 with this branch; corpus 81/81 converged incl. 48/48 SLaM; autolens_workspace_test 7 mapper-less/MGE scripts pass with no pin moved; in-session review (supervised, human-approved plan).

[Jammy2211]

Library PR merged — 2026-09-30

PyAutoArray#595 merged (merge commit 7a89e19, head 42c5235) via /prm on green CI (unittest 3.12 / 3.13 / no-jax). Pending release: not on PyPI yet. Heart remained RED for release throughout (release validation FAILED (stage integrate), unrelated).

Issue stays open for the workspace phase in autolens_profiling: commit the post-fix corpus re-run with a non-colliding label (the cells stamp v2026.8.17.1 on source checkouts and overwrote the pre-fix files), re-base scripts/lens/solver/euclid_latent.py's validation leg from the pre-fix jit 3.511 to the eager 3.3199, append the phase-2 ledger row + campaign page + README stats, then /ship_workspace behind the library-first gate.

[Jammy2211]

Workspace phase ready — autolens_profiling, held at Heart RED ship gate (2026-09-30 ~18:50 BST)

Branch feature/raw-pdip-forward-polish in autolens_profiling (worktree ~/Code/PyAutoLabs-wt/raw-pdip-forward-polish/autolens_profiling), 4 commits on 3ad68af, head ad365df, all gates green locally, NOT pushed, no PR.

Heart readiness (verbatim): RED PyAutoGalaxy: CI failure; RED release validation FAILED (stage integrate); YELLOW workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py); YELLOW manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml. None in autolens_profiling. Human override + push needed (auto-mode denies the agent's first push of a branch).

Drafted PR body

Summary

Phase 2 of epic linear-solver-programme on the profiling side (PyAutoArray#594; library fix PyAutoArray#595 merged 2026-09-30 as 7a89e19a0, pending release). The library now returns the #573 polished iterate as the raw-forward PDIP forward value. This PR records that in the solver programme: every scripts/lens/solver/ cell re-run against merged library main (device.provenance.library_revisions names PyAutoArray 7a89e19a0), the euclid latent cell's validation leg re-based from the pre-fix jit value 3.511093374 to the post-fix 3.320127604 measured on main, the ledger's Phase 2 section, campaign page + index row, and regenerated README tables.

Measured on the 81-system corpus: fixed pdip_raw equals phase 1's pdip_raw_polish on every metric on 81/81 systems — converged 81/81 incl. 48/48 SLaM, forward iterations median 18 / max 24 (+1–7 polish, accepted on all), worst inactive-column flux 3.31e-4 (pre-fix 0.115), worst total-flux error 3.31e-4 (0.115), worst source-flux error 4.96e-2 on euclid only (16.4; ≤ 4.9e-5 on the #571 fixtures), objective gap 8.2e-16, KKT 3.2e-16. Euclid latent +7.47e-5 vs eager (pre-fix +5.76e-2); the euclid pipeline latent test passes on library main. Under the pre-registered rule the fixed pdip_raw passes criterion 1 and still fails 2–4 for the recorded rule weaknesses (euclid source-column proxy far stricter than the latent it stands for; one flat-direction system at 0.219 shared by every accurate candidate; 5/52 KKT ratios at the 3e-16 floor) — the ledger says so rather than re-basing the rule.

Artefacts: the cells stamp al.__version__ = 2026.8.17.1 on a source checkout, so the re-run overwrote the *_v2026.8.17.1.{json,png} artefacts in place; they are "current library main" by contract, and the pre-fix versions live at 3ad68af and in the ledger's phase-1 tables. pdip_raw_tol_1e-2 (direct solve_nnls, no polish) now reproduces the pre-fix pdip_raw; the phase-1 sentence claiming it reproduces pdip_raw is annotated, not deleted.

Upstream PR

• PyAutoArray#595 — fix: return the polished raw-forward PDIP iterate as the forward value (#594) #595 (MERGED, pending release)

Scripts Changed

• scripts/lens/solver/euclid_latent.py — PDIP_RAW_JIT_EXPECTED re-based to the post-fix: return the polished raw-forward PDIP iterate as the forward value (#594) #595 library-main jit value (comment names the PR, merge sha and the replaced pre-fix value); the JSON now records both the released and the library-main jit values
• scripts/lens/solver/_solvers.py — pdip_raw / pdip_raw_tol_1e-2 descriptions: polished forward value, forward-only iteration count; tol_1e-2 has no polish
• scripts/lens/solver/README.md, results/README.md (via build_readme.py) — regenerated tables; scripts/misc/tooling/build_readme.py — euclid renderer shows both jit values
• results/lens/solver/{accuracy,accuracy_posthoc,early_stopping}_summary_{all,slam_fixture_571}_v2026.8.17.1.{json,png}, euclid_latent_by_candidate_v2026.8.17.1.json — re-run against PyAutoArray 7a89e19a0
• results/notes/linear_solver_accuracy_2026_09.md — Phase 2 section; wiki/campaigns/linear_solver_accuracy.md, wiki/index.md — status, verdict, PRs, phase-2 row + journal entry

Test Plan

• ruff check . && ruff format --check . (250 files)
• build_readme.py --check, check_wiki.py --check, check_results_layout.py --check, build_dashboard.py --check, check_submits.py --check
• pytest scripts/misc/test -q — 984 passed, 5 skipped
• AUTOLENS_PROFILING_SMOKE=1 — the 7 lint.yml smokes + early_stopping.py + euclid_latent.py, all rc=0
• euclid_latent.py validation leg VALIDATES on library main (x_ref → eager rel −3.3e-8; pdip_raw → jit rel −2.0e-8)
• lint.yml green on the PR (incl. lychee, not run locally)

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

[Jammy2211]

Workspace PR opened — 2026-09-30

PR: PyAutoLabs/autolens_profiling#357 (head ad365df), opened under the Heart RED development override below (also in the PR body). Library-first gate satisfied (PyAutoArray#595 MERGED). Merge via /prm on green lint; no release.

• Authorization: live human in the Claude Code CLI session, 2026-09-30 ("do the workspace phase"; at the ship gate the human pushed feature/raw-pdip-forward-polish directly, ~18:55 BST, in response to the reported RED reasons). Scope: development shipping (push + this PR); merge only via a separate /prm on all-green required checks; no release.
• Heart readiness at the 18:50 BST gate (verbatim): RED PyAutoGalaxy: CI failure; RED release validation FAILED (stage integrate); YELLOW workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py); YELLOW manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml. None involve autolens_profiling. This PR does not repair Heart; Heart remains RED for release.
• Passed branch gates at ad365df: ruff check + format (250 files); build_readme / check_wiki / check_results_layout / build_dashboard / check_submits --check; pytest scripts/misc/test 984 passed / 5 skipped; 9 smokes rc=0; euclid_latent validation leg validates on library main; in-session review (supervised, human-approved plan).

[Jammy2211]

Shipped

Phase 2 of epic linear-solver-programme — both PRs merged 2026-09-30 by human /prm on green checks.

• Library: fix: return the polished raw-forward PDIP iterate as the forward value (#594) #595 — merged 7a89e19a0 (head 42c52358). Label pending-release: merged, not yet released.
• Workspace: docs(solver): record phase 2 — PyAutoArray#595 forward polish — cells re-run, euclid validation re-based (PyAutoArray#594) autolens_profiling#357 — merged 4652580b2 (head ad365df).

What changed

• PyAutoArray#595: the raw-forward PDIP solver (nnls_preconditioning_no_mapper: raw, fix(inversion): raw-forward PDIP for mapper-less positive-only JAX solves, surface convergence (#571) #572) now returns the fix: NaN JAX gradient on MGE positive-only solves after #572 #573/fix: NaN JAX gradient on MGE positive-only solves after #572 #574 polished iterate (≤10 tight warm-started Jacobi-system PDIP iterations) as its forward value, via _raw_forward_polished() shared by the custom_vjp primal and forward, so jit / grad / eager agree. Backward unchanged. RAW_POLISH_MAX_ITER added (old name aliased); docstrings/config wording updated. New fixture mge_solver_reference_systems.npz (8 fix(inversion): JAX PDIP positive-only solve fails to converge on SLaM MGE systems #571 systems + euclid vis_lp, fnnls x_ref) and test_nnls_raw_forward_amplitude.py (107 cases; 12 red on base bd03e09e — euclid inactive/total flux 0.115, source flux k1/k2/k3/k5 1.6e-3..6.4e-3; green after). End-to-end jit-vs-eager relaxed to rtol 1e-9 after the 3.12 Actions leg showed 5.4e-11 runner-dependent XLA reassociation.
• autolens_profiling#357: euclid_latent.py validation re-based to the post-fix jit 3.320127604; all 6 solver cells re-run against PyAutoArray 7a89e19a0 (provenance recorded; artefacts overwrote the v2026.8.17.1 stamp in place, pre-fix at 3ad68af); ledger Phase 2 section, campaign page + index, README regen.

Results

• Corpus 81/81 converged incl. 48/48 SLaM; fixed pdip_raw == phase-1 pdip_raw_polish on 81/81.
• Worst inactive-column flux 0.115 → 3.31e-4; source-flux misses 53/81 → 1/81 (euclid proxy only).
• Euclid total_source_flux jit-vs-eager +5.76e-2 → +7.47e-5; euclid pipeline latent test 19/19 on library main with no override.
• autolens_workspace_test: 7 mapper-less/MGE scripts pass, no pin moved (mge_group JIT now equals NumPy to 6e-12).
• Rule verdict: not admissible under the phase-1 pre-registered rule as written — criteria 2–4 fail on recorded rule weaknesses (euclid source-column proxy 4.96e-2 vs actual latent 7.5e-5; flat-direction system sig 0.219 shared by all accurate candidates; 5/52 KKT ratios at the 3e-16 floor).

Gates

• PyAutoArray: pytest 1890 passed + CI 3 legs green.
• autolens_profiling: ruff / 5 checks / pytest 984 / 9 smokes + lint green.
• Both PRs shipped under the Heart RED development override (recorded on this issue, the PRs, active.md and autonomy_log). Heart stayed RED for release throughout (release validation FAILED at stage integrate; later also a PyAutoGalaxy CI failure — unrelated to this change).

Pending release

PyAutoArray#595 is merged but unreleased; the Mind completion record carries pending-release: PyAutoArray@https://github.com/PyAutoLabs/PyAutoArray/pull/595 until /review_release clears it on a published release.

Next

Phase 3: draft/research/autoarray/mge_nnls_fix_pyautoarray_571_slam_60.md — GPU/A100 timing + parity rows.