Small-datasets cap: read SMALLSHP so capped datasets stop re-simulating, honour PYAUTO_DISABLE_JAX in the interferometer sparse operator, cap mesh shapes with the data - #529
Merged
Conversation
…sh shapes
Three findings of the phase-8 diagnosis of the user-workspace smoke timings,
all of them the same shape: a harness-level fact the library did not act on.
1. Capped datasets whose shape cannot corroborate their stamp were deleted and
re-simulated on every run. `_is_capped_at_the_current_cap` required
`SMALLDAT = T` *and* a measured `data.fits` at exactly the cap, because
`SMALLDAT` records "the env var was set at write time", not "capped at
today's cap". Interferometer data is `(n_visibilities, 2)`, multi_dataset
prefixes its FITS and datacubes nest theirs per channel, so none of them
could ever corroborate: 5.2-6.5 s per script, ~26 s per autolens_workspace
CI run and ~28 s per autogalaxy_workspace one, plus every local run.
The `SMALLSHP` card (PyAutoNerves#159) states the proposition directly, so
corroboration is no longer needed and those families are reachable. Two
things keep the safety property intact. The card is read *with* the
both-axes contradiction guard the full-resolution branch already applies --
both cards record the writing *process*, so a 180x180 image written in a
shell exporting `PYAUTO_SMALL_DATASETS=1` carries them while being full
resolution, and reusing it in a capped run is the shape-mismatch class the
capped branch exists to prevent. And an absent or malformed card falls back
to the shape heuristic rather than being coerced, so every pre-card dataset
behaves exactly as it did.
The path resolution (`_capped_data_paths`) is widened for the capped branch
only, by exact suffix at two known levels -- `data.fits`, then
`{waveband}_data.fits`, then `channel_*/data.fits`, never a `*.fits` glob.
The full-resolution branch still reads `data.fits` alone: widening the
capped branch can only move a dataset from delete to keep, whereas widening
the destructive one is its own change with its own review.
`test__small_regime__interferometer_dataset__is_always_regenerated` asserted
the conservative behaviour this issue exists to change, and is rewritten to
pin the half that stands: without a cap card, that family still regenerates.
2. `apply_sparse_operator(use_jax=True)` ignored `PYAUTO_DISABLE_JAX=1`, so the
smoke profile paid 2.3-3.2 s of JIT for a backend it had asked to disable.
The interferometer method now lets the harness win over an explicit `True`,
through `autonerves.test_mode.disable_jax()`.
The imaging method is deliberately left alone, and says why in its docstring:
it has no `use_jax` argument to override -- it *is* the JAX implementation,
and its NumPy sibling `apply_sparse_operator_cpu` returns a different
operator class and requires numba. Routing to that under an env var would
change the returned type based on the environment, which is a larger and
(on these measurements) unmotivated change.
3. Pixelization mesh `shape=` did not honour the cap the data does, so a capped
run reconstructed 1600-2500 source pixels from ~80 image pixels. The
rectangular meshes and `image_mesh.Overlay` now cap per axis under
`PYAUTO_SMALL_DATASETS=1`, with the same `respect_small_datasets=True`
escape hatch `Grid2D.uniform` carries. Measured on the four HowTo chapter-3
scripts whose guards this subsumes: 43.6 s -> 17.3 s.
`disable_jax` is imported defensively: an autonerves too old to carry the
predicate is exactly the case the sibling literal in `dataset_util` exists for,
and a hard import would turn a safe degradation into an `ImportError` at module
load.
Refs #528
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0151gQm9fk3XGLi5f18Urdba
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #528. Phase 8c (library leg) of the
ci-timing-fast-testsepic (PyAutoMinddraft/feature/pyautoheart/ci_timing_fast_tests_epic.md): the three PyAutoArray findings of the phase-8 user-workspace smoke diagnosis (autolens_workspace#536). Merge after PyAutoNerves#160 (library-first): it reads theSMALLSHPcard and thedisable_jax()predicate that PR adds, both imported defensively so an older autonerves degrades to today's behaviour.Summary
dataset_utilgainsSMALLSHPreaders,_capped_data_paths(resolvingdata.fits→{waveband}_data.fits→channel_*/data.fitsby exact suffix — the file's own "Known gap"), and_is_capped_at_the_current_caprewritten around them. One deliberate deviation from the issue's diff: the card records the writing process, not the array, so the diff as written would have kept a 180×180 image written in a capped shell (two existing tests caught it). The card is therefore read with the both-axes contradiction guard the full-resolution branch already applied (_shape_contradicts_the_cap), which still leaves interferometer(n_vis, 2)reachable. Path resolution is widened for the capped (delete → keep) branch only; the destructive branch is unchanged. One existing test that encoded the old always-regenerate contract is rewritten to pin the half that still stands (no card → still regenerates); nothing skipped or loosened.apply_sparse_operatorhonoursPYAUTO_DISABLE_JAX=1on the interferometer dataset viaautonerves.test_mode.disable_jax(). Backed off on imaging with evidence: that method has nouse_jaxargument — it is the JAX implementation, its NumPy siblingapply_sparse_operator_cpureturns a different operator class and needs numba, and every measured JIT cost was interferometer. Documented in its docstring.shape=honoursPYAUTO_SMALL_DATASETSlikeGrid2D.uniform/Mask2D.circular:cap_mesh_shape_for_small_datasets(per axis,min) in the rectangular mesh family's singleself.shapeassignment andimage_mesh.Overlay, with therespect_small_datasets=Truehatch threaded through the subclasses (not stored on the instance, so__eq__/ serialization are untouched). This subsumes the four script guards HowToLens#77 / HowToGalaxy#73 added (they now resolve to the same(16, 16)and can be reverted) and reaches ~28 further call sites.Validation (this container, branches of the three libraries installed editable)
_testJAX scripts run uncapped, so their pins did not move).should_simulateTrue → False on the second run ofautolens_workspace interferometer/modeling.py(12.40 → 6.69 s warm),multi_dataset/modeling.py(10.48 → 6.32 s),autogalaxy_workspace interferometer/start_here.py(10.93 → 6.52 s). Estimated ~26 s per autolens_workspace CI run and ~28 s per autogalaxy_workspace run once the dataset cache carries a stamped set.Follow-ups (not here)
try/exceptonce the PyAutoNerves#160 release exists.Test Plan
python3 -m pytest -q -n auto— 1449 passed_testgates green with the branch installedlib-tests.ymlon the PR (both Python legs; the chain clones the same-named PyAutoNerves branch)🤖 Generated with Claude Code
https://claude.ai/code/session_0151gQm9fk3XGLi5f18Urdba
Generated by Claude Code