Repository navigation
2026-06-25 Engineering Team #114
Description
Activity
Open PRs on ActivitySim
-
Stable sort for school escorting #1085 - opened yesterday, under review
-
Fix for simulation-based shadow pricing segmentation #1083 - Opened this week, marked as draft, needs tests added
-
Adding nullable_nonnegative Decode Filter #1081 - Under review by @jpn--
-
Workplace Location Logsum Overwritten in Estimation #1077 - fixes a prior merge problem, need testing to ensure regression does not return. @dhensle
-
Explicit error terms production code #1064 - Under review by @dhensle
-
Implement deletion of temporary variables in summarize.py #1062 - No testing included, @i-am-sijia to coordinate with @ziocolaandrea to include testing
-
Add trip scheduling choice explicit chunking #1041 - Reviewed by copilot and @jpn--, requested some testing changes. @m-richards are you willing/able to address?
-
Adding ignore list to skim load for unused skims while running with sharrow #1036 - Reviewed by copilot and @jpn--, awaiting minor changes @dhensle
-
Access to global constants and removing chooser col filtering #1017 - reviewed as incomplete and marked as draft in December, no further progress since @dhensle
-
Park-and-ride lot choice and capacity #1001 - review complete, changes requested @dhensle
-
ActivitySim Engineering Meeting Notes
Meeting Overview
Agenda
- Open pull request review and coordination
- EET inconsistency investigation update and coordination plan
- Small example model / test dataset development update (CS)
- Upcoming meetings and schedule
Participants
- Jeff Newman (@jpn--) — Driftless
- Joe Castiglione (@joecastiglione) — Zephyr Foundation
- Jan Zill (@janzill) — Outer Loop
- Sijia Wang (@i-am-sijia) — WSP
- Stefan Coe (@stefancoe) — PSRC
- Joe Flood (@JoeJimFlood) — CS
Duration
Approximately 45 minutes
Executive Summary
The group conducted a systematic review of all open pull requests, identifying next steps and owners for each. Key items include the park and ride lot choice PR (close to done, sent back to RSG for bug fixes and test coverage), the large EET PR (Jan has addressed AI review comments and it still awaits human review from David), and a newly opened PR for stable sorting in school escorting (a fix identified independently by both Outer Loop and WSP as part of the EET inconsistency investigation). The simulation-based shadow pricing fix was discussed at length — the group agreed to pause adding a unit test until SANDAG can be consulted on what validation metrics are appropriate. Joe Flood provided a brief update on the small SANDAG sub-area example model, which is being developed to run on a 32GB laptop.
Meeting Notes
Open Pull Request Review
Park and Ride Lot Choice
This PR is relatively close to done. Jeff conducted an extensive review (with AI assistance) and flagged a few minor bugs and gaps in test coverage. The PR implements two approaches to handling oversubscribed lots: random selection and selection by latest arrival time. The latest-arrival approach is the one all agencies that are using this are actually using; the random selection code path has bugs that need to be resolved before it can be considered available. Jeff has returned the PR to RSG for these fixes.
TransLink funded the development and is using the feature; PSRC is in the process of implementing it. Stefan Coe confirmed PSRC has already implemented the feature and has not encountered bugs so far. Jeff noted that real-world user feedback is in many ways more valuable than other code review — if a user implements it and it works, that is meaningful validation.
Access to Global Constants / Remove Chooser Column Filtering
This PR was marked as draft by David in December after Jeff's earlier review requested additional work. There has been no subsequent work. Jeff tagged David in the PR comments; this should be followed up with David to determine whether progress is forthcoming or whether it should be handed off.
Unused Skims Bug Fix (Sharrow)
Jeff reviewed this PR and left comments; minor changes are needed before it can be merged.
Trip Scheduling Choice Explicit Chunking (VLC)
This PR was submitted by Matt Richards (VLC). Jeff reviewed the copilot comments, and also this PR needs tests. The plan is to wait to see if Matt is available and able to address the comments on his own; if he cannot, the team will decide next steps at that point.
WSP Australia PR
Sijia confirmed that the contributor is still working on it and has been in contact. Progress is expected.
Explicit Error Terms (EET) in Production Code — Large PR
Jeff noted he re-ran AI review on this PR and a batch of comments came through this morning. Jan had already reviewed the AI output and found mostly minor issues: spelling, documentation build items, and a type annotation fix. Jan had also separately run the PR through Opus privately, which caught a couple of additional things (e.g., a
0whereNoneshould be the default argument). Jan will incorporate these fixes as part of finalizing the PR. Jeff reiterated that he still wants David to do a human review in addition to the AI pass.Estimation Mode CI Test (RSG)
This PR fixes a merge issue that Jeff inadvertently introduced. David had planned to add a CI test to prevent regression. Jeff tagged David to clarify whether David will add the CI test or whether someone else should take it on.
Non-Nullable Decode Filter
Jeff started reviewing this today and expects it to be relatively straightforward, though it may need some test tweaks.
Simulation-Based Shadow Pricing Fix
Sijia opened this PR as a draft this week. The code fix is in place; a unit test has not been added yet. Sijia confirmed she reran one of the ET/SANDAG model runs with the fix applied and shadow pricing ran additional iterations as expected.
Jeff noted that an existing regression test is now failing — which is actually encouraging, because it's failing in exactly the right place (shadow pricing), and approximately 34% of work zone assignments are different from the previous golden file. Jeff pointed out this is likely an undercount of the real impact because many "work zones" are
-1(non-workers), so the actual fraction of workers with changed work zones is substantially higher.The group discussed what validation is needed before adding a unit test. Jeff and Joe agreed that simply blessing the new output as the new golden file is not sufficient. What's needed is an analysis of whether the new results are actually correct — e.g., do worker totals by job category match employment targets? Does the trip length distribution look reasonable? These are distributions and aggregate metrics, not individual zone-level checks.
Joe suggested reaching out to SANDAG, since this bug directly affects their operational model. Sijia noted she had already emailed SANDAG but had not heard back. Joe mentioned he had spoken briefly with Bhargava Sana (SANDAG) and would follow up more formally to ask whether SANDAG can assess the impact on their own calibration. The group acknowledged that SANDAG's current model has likely been calibrated on top of the incorrect shadow pricing behavior, so correcting it will change their results — the question is how much and in what way.
Jeff recommended holding off on adding the unit test for a week or two while waiting for SANDAG input, since understanding the consequences of the bug is important for writing a meaningful test.
On the underlying fix: Sijia explained that her implementation removes the need for users to separately specify a shadow pricing segmentation that matches their workplace location choice segmentation. Instead, the code automatically reads the segmentation from the workplace location settings. The root cause at SANDAG was that they originally segmented by income but later switched to occupation segmentation for workplace location choice, without knowing they also needed to update the shadow pricing settings — two separate config locations with different naming conventions.
Stable Sorting for School Escorting Trips (new PR)
A PR was opened recently to fix non-stable sorting in the school escorting model. Jan explained that this fix emerged from the EET inconsistency investigation: both Outer Loop and WSP independently found that in the transit scenario, there were unexpected changes in non-mandatory tour frequencies. Tracing this back led to school escorting, where small differences in how chauffeur-child bundles are formed within a household (depending on sort order of the input data) were propagating through to downstream model components.
The PR submitted by Will (RSG, not present) uses a
stable=Truesort argument. Jan had also independently worked on a fix, preferring to introduceperson_idas an additional sort key rather than relying solely on stable sorting. Jeff agreed with Jan's approach: using a stable sort alone just preserves an existing arbitrary ordering, which moves the instability risk elsewhere in the code (e.g., it could change with chunking or multiprocessing). Addingperson_idas an explicit tiebreaker ensures deterministic ordering regardless of input order.Jan will post his alternative implementation in the PR comments for further discussion. The group agreed it would be ideal to have a test that fails without the fix, but acknowledged this may not be straightforward given the non-deterministic nature of the original bug.
EET Inconsistency Investigation — Coordination Update
Jan summarized the current state: both teams (Outer Loop and WSP/RSG) have been investigating independently and found overlapping issues. The immediate next steps are:
- WSP and RSG (David and Will) will focus on the transit scenario, starting by applying the school escorting stable sort fix, rerunning, and then investigating remaining unexpected changes in school location choice.
- Outer Loop (Jan and team) will focus on the employment scenario, tracing through remaining differences in workplace location choice and other components.
- An internal check-in across teams is planned for July 8, one week before the July 14 consortium presentation, to assess where things stand and whether results from any fixes can be incorporated before the final report-out.
Joe confirmed the July 14 date as the target for the main presentation. The June 30 consortium meeting will include only a brief update on the EET investigation, with the bulk of that meeting focused on Sijia's explicit telecommute presentation.
Small Example Model / Test Dataset Development (CS)
Joe Flood reported he is actively working on this again. The approach he is taking is to use a larger geographic sub-area of the SANDAG region but reduce the synthetic population to bring memory and runtime down. This aligns with Jan's suggestion to prioritize more zones over more people, since skim memory scales with zones.
Jeff reiterated the design intent: more zones drives memory usage, more people drives runtime. Both dimensions need to be tuned independently to hit the target of running on a 32GB laptop in a reasonable amount of time. Jan also noted that having more people (even if still sub-sampled) enables more meaningful statistical validation of model outputs — a larger sample supports distribution-level comparisons rather than just golden-file exact-match tests.
Jeff noted that memory efficiency is becoming increasingly important as RAM costs rise. Purchasing high-RAM servers to support ActivitySim is becoming much more expensive.
Upcoming Meetings
- Next Tuesday's consortium meeting: Sijia will present on the explicit telecommute (EET) frequency model specification and the new model structure. Joe asked if any materials could be shared in advance; Sijia said she would try, but noted the presentation will include a detailed walkthrough so prior reading is not required.
- Next Thursday's engineering call: Is there are no pressing issues that arise before Wednesday, next week's Thursday call can be cancelled.
Action Items
- RSG (@dhensle): Fix the remaining bugs in the park and ride lot choice PR (pick-random code path) and ensure test coverage includes both code paths before resubmitting.
- Stefan Coe (@stefancoe): Continue PSRC's implementation of park and ride lot choice and report any issues found as GitHub issues.
- Jeff Newman (@jpn--): Follow up with David Hensle on the access to global constants / remove chooser column filtering PR — determine whether it will be completed or handed off.
- Jan Zill (@janzill): Incorporate AI review comments (spelling, doc build, type annotation, default argument fix) into the EET PR and finalize for human review.
- David Hensle (@dhensle): Conduct a human review of the EET PR.
- David Hensle (@dhensle) / Jeff Newman (@jpn--): Clarify ownership of adding the CI test to the estimation mode PR and get it merged.
- Sijia Wang (@i-am-sijia): Hold off on adding a unit test for the simulation-based shadow pricing fix for 1–2 weeks pending SANDAG input; continue to follow up with SANDAG for their assessment of the bug's impact on their calibration.
- Joe Castiglione (@joecastiglione): Follow up with @bhargavasana (SANDAG) to formally ask if SANDAG can assess the impact of the shadow pricing segmentation bug on their operational model, and suggest validation metrics (e.g., workers by occupation vs. employment by zone, trip length distribution).
- Jan Zill (@janzill): Post his
person_id-based tiebreaker implementation as an alternative on the school escorting stable sort PR for discussion. - Jeff Newman (@jpn--): Think through whether a failing test can be constructed for the school escorting sort issue; discuss on the GitHub issue.
- WSP / RSG (Sijia Wang, David Hensle, Will): Focus EET transit scenario investigation on school escorting fix first, then investigate remaining unexpected changes in school location choice.
- Outer Loop (Jan Zill and team): Investigate remaining differences in workplace location choice and other components in the EET employment scenario.
- All EET teams: Internal check-in on July 8; prepare for substantive EET report-out at the July 14 consortium meeting.
- Joe Flood (@JoeJimFlood): Continue development of the SANDAG sub-area example model; target running in under 30 minutes on a 32GB machine with a zone set large enough to exercise memory scaling.
- Sijia Wang (@i-am-sijia): Share explicit telecommute presentation materials in advance of Tuesday's consortium meeting if possible.
- Joe Castiglione (@joecastiglione): Decide by Wednesday whether to cancel next Thursday's engineering call (Jeff Newman will be traveling).
Agenda