feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3)

[Jammy2211]

Overview

Streaming epic phase 3 (Mind epic streaming-visibilities; Discussion https://github.com/orgs/PyAutoLabs/discussions/13). Phases 1–2 (#593, PyAutoLabs/PyAutoGalaxy#639, PyAutoLabs/PyAutoLens#758) let an array-free aa.Interferometer (no data / noise_map / uv_wavelengths / transformer) fit, save and reload. The go/no-go benchmark (2026-09-30) showed the in-memory sparse path runs out of memory at ~1e6 visibilities on a 16 GB laptop while streaming stays flat at 1.5–1.8 GB to 5e7, so the epic continues. A streamed fit is still usable only with visualization off: the dataset subplot reads six visibility-space quantities that raise, every fit.dirty_* panel goes through the absent transformer, subplot_of_mapper / subplot_mappings escape with InversionException because _recon_array reads the operated dict first (a live bug), and both visualizers' fallback logger(...) is a TypeError.

Plan

• Branch in place on is_array_free inside the existing plot functions (same names, same filenames): skip visibility-space panels, draw natural-weighted dirty image / beam / model / residual from the sparse terms; in-memory output unchanged.
• One autoarray helper for the natural dirty model image (W̃·m / Σw via operated_matrix_slim_from); ag/al fit properties model_image_natural, dirty_model_image_natural, dirty_residual_map_natural.
• Fix _recon_array (fall back to mapped_reconstructed_data_dict without a transformer) and the handlers; fix the two logger(...) calls; fix the stale autolens fits assertion.
• Tests in all three repos incl. visualize_before_fit + visualize end to end on an array-free fit; witness with visualization on.

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary), PyAutoGalaxy, PyAutoLens

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	main	clean
./PyAutoGalaxy	main	clean
./PyAutoLens	main	clean

PyAutoArray is also claimed by raw-pdip-forward-polish (#594, PR #595; NNLS/settings files only — disjoint). Parallel claim approved with the plan 2026-09-30. Suggested branch: feature/streaming-p3-visualizer

Design decisions (human-approved 2026-09-30, on the epic ledger)

Branch in place; in-memory byte-unchanged; skip rather than approximate visibility-space panels; natural-weighted panels from the terms with model image sum(galaxy_image_dict.values()); labels "… (Natural)" and EXTNAMEs DIRTY_IMAGE_NATURAL, DIRTY_BEAM, DIRTY_MODEL_IMAGE_NATURAL, DIRTY_RESIDUAL_MAP_NATURAL; no silent except DatasetException; no new skip switch (filed separately).

Implementation Steps

1. PyAutoArray autoarray/fit/fit_interferometer.py: dirty_model_image_natural_from(dataset, image).
2. autoarray/dataset/plot/interferometer_plots.py: array-free branches in subplot_interferometer_dataset, subplot_interferometer_dirty_images (1×2 natural dirty image / beam) and fits_interferometer (guard data reads; natural extensions).
3. autoarray/fit/plot/fit_interferometer_plots.py: optional model_image kwarg; natural panels on array-free.
4. autoarray/inversion/plot/inversion_plots.py: _recon_array fallback without a transformer; exc.InversionException in the two handlers.
5. PyAutoGalaxy: fit properties; subplot_fit, subplot_fit_dirty_images, subplot_fit_real_space, fits_dirty_images array-free branches; visualizer logger.warning.
6. PyAutoLens: fit properties; subplot_fit, subplot_fit_dirty_images, subplot_fit_interferometer_combined, subplot_fit_real_space, subplot_tracer_from_fit array-free branches; visualizer logger.warning + positions image → dirty_image_natural; stale dirty_images.fits assertion fixed.
7. Tests in all three repos (array-free plot outputs written; in-memory unchanged; _recon_array red-check; helper vs transformer path at 1e-10; model_image_natural == inversion.mapped_reconstructed_data on pixelization-only fits).
8. Witness: phase-2 MockSearch setup with visualization ON; png/fits listing for array-free vs in-memory; in-memory fits arrays identical to base.

Key Files

• autoarray/dataset/plot/interferometer_plots.py, autoarray/fit/plot/fit_interferometer_plots.py, autoarray/inversion/plot/inversion_plots.py, autoarray/fit/fit_interferometer.py
• autogalaxy/interferometer/{fit_interferometer.py,plot/fit_interferometer_plots.py,model/visualizer.py}
• autolens/interferometer/{fit_interferometer.py,plot/fit_interferometer_plots.py,model/visualizer.py}

Original Prompt

Click to expand starting prompt

Streaming phase 3: visualizer on array-free datasets (natural-weighted dirty panels, uv panels skipped)

Type: feature
Target: PyAutoArray
Repos:

• PyAutoArray
• PyAutoGalaxy
• PyAutoLens
  Themes:
• interferometer
• sparse-operator
• memory
  Autonomy: supervised
  Priority: medium
  Status: draft
  Epic: streaming-visibilities
  Phase: 3
  Difficulty: medium
  Consequence: glance
  Witness: visualize_before_fit and visualize (ag + al interferometer visualizers) on an array-free fit write every png/fits without raising — uv/visibility panels skipped, dirty image / dirty beam / dirty model / dirty residual present from the terms and labelled "(natural weighting)" — while in-memory output is byte-unchanged; inversion_plots._recon_array no longer performs a forward transform to type-check.
  Review-minutes: 6
  Unattended: ready
  Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
  Blocked-by: none (phase 2 merged 2026-09-30)

Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30 (decision (c)). After this phase ships, post the promised follow-up on the discussion (see epics.md notes).

What

1. autoarray/dataset/plot/interferometer_plots.py: guard uv/visibility/S-N panels on array presence (precedent: fits_interferometer
noise_map is not None guards); dirty image / beam from dirty_image_natural / dirty_beam when arrays are absent.
2. fit/plot/fit_interferometer_plots.py (+ ag/al mirrors): dirty model = W~·image via operated_matrix_slim_from; dirty residual =
   d~ − W~·(M s + i_p); skip normalized-residual / chi-squared dirty panels and the uv subplots when arrays are absent.
3. inversion/plot/inversion_plots.py _recon_array: replace the forward-transform type probe.
4. ag/al visualizers: no crash paths; tests mirror test_plotter_interferometer.py with an array-free fixture.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZZksyZ8LTA4LLxoZjQMNF

[Jammy2211]

Heart RED override — development only (recorded per AUTONOMY.md)

• Heart verdict at ship: RED, 2026-09-30T18:32:16Z. Exact reason strings: release validation FAILED (stage integrate); PyAutoLens: on branch feature/point-solver-padding-backend (not main); PyAutoLens: 2 uncommitted source change(s) (the last two are canonical-checkout drift from another session, not these branches). Unrelated to this work.
• Authorization: live human, in-session 2026-09-30, for this issue (feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #596) and the three feature/streaming-p3-visualizer branches (PyAutoArray, PyAutoGalaxy, PyAutoLens): "Authorize override for feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #596" — commit, push and the pending-release PRs. Merge remains a separate explicit human command with every required check green. No release, no rehearsal, no claim that these branches repair Heart.
• Branch gates passed before the ask: test_autoarray 1898 passed; test_autogalaxy 1293 passed; test_autolens 774 passed + 1 xfailed; red-checks (inversion-plot fix; array-free tests with source stashed); witness with visualization on; independent Codex gpt-6-astra review: FINDINGS (1) — model_image_natural summed galaxy_image_dict, whose entries are overwritten by a galaxy's linear reconstruction, so a galaxy with both ordinary and linear light lost its ordinary component (reproduced: rel error 1.0 in ag, 1.14/1.31 in al); fixed in-branch as profile_image + inversion.mapped_reconstructed_data, with mixed-galaxy tests against an independent transformer reference from fit.model_data (rel ≤ 2e-14), red-checked.

🤖 Generated with Claude Code

[Jammy2211]

Library PRs created

• feat(plot): interferometer plots on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #597
• feat(interferometer): visualizer on array-free datasets (streaming phase 3) PyAutoGalaxy#640
• feat(interferometer): visualizer on array-free datasets (streaming phase 3) PyAutoLens#761

Tests: test_autoarray 1898 / test_autogalaxy 1293 / test_autolens 776 + 1 xfailed

Shipped under the human-authorized Heart RED development override (comment above); merge stays a separate human command via /prm once CI is green (PyAutoArray → PyAutoGalaxy → PyAutoLens).

🤖 Generated with Claude Code

[Jammy2211]

Shipped

• PyAutoArray: feat(plot): interferometer plots on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #597 (merge c1d8581) — natural-weighted dirty model helper, array-free dataset/fit plot branches, inversion subplot fix.
• PyAutoGalaxy: feat(interferometer): visualizer on array-free datasets (streaming phase 3) PyAutoGalaxy#640 (merge 6d522ce4) — natural fit properties and plot branches; visualizer logger fix.
• PyAutoLens: feat(interferometer): visualizer on array-free datasets (streaming phase 3) PyAutoLens#761 (merge a2fbe881) — the lens mirror; both visualizers' logger fix; stale test asset removed.

All pending-release. Streaming epic phase 3 of 5: a streamed (array-free) dataset now fits, saves, reloads and visualizes end to end.

🤖 Generated with Claude Code