Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1079 +/- ##
==========================================
+ Coverage 95.24% 95.36% +0.12%
==========================================
Files 59 60 +1
Lines 5548 5825 +277
==========================================
+ Hits 5284 5555 +271
- Misses 264 270 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hi @barjin and @Pijukatel, FYI: I assigned Jindra to the JS PR and Pepa to the Python one. Both PRs implement the same feature. Please also let me know WDYT about naming and whether you think adding just one method to Thanks! |
Pijukatel
left a comment
There was a problem hiding this comment.
I wonder if this should be rather implemented inside get_items_iterator as an optional stop_condition_callback(placeholder name), which would be None by default (current behavior) or a user-defined callback that checks whether we should stop or not.
For this specific feature, we would then define this user-defined callback to check if the actor run is finished.
run.iterate_dataset_items would than be just thin wrapper, something like:
pseudo Python code:
def iterate_dataset_items(...):
def is_finished():
run = self.wait_for_finish(wait_duration=poll_interval, timeout=timeout)
return run is None or run.status in _TERMINAL_STATUSES`
yield from self.dataset().iterate_items(...,stop_condition_callback=is_finished)
Adds
RunClient.iterate_dataset_items()and its async twin. They yield a run's dataset items while the run is still pushing them and return once the run has finished and the dataset is drained.Each poll reads the dataset's
itemCountand fetches pages whoselimitends at it. The endpoint scans exactlylimitrows, so offsets stay exact even whenclean,skip_emptyorunwindchange how many items a page returns. Between polls,wait_for_finish()waits up topoll_intervalfor the run to finish.itemCountlags about 5 s, so after the run finishes, the rows past it are read page by page until a page comes back empty. Withclean,skip_emptyorunwind, one unfilteredlimit=1read confirms the end. Polling goes on throughABORTINGandTIMING-OUT, since such a run can still push items.The arguments follow
DatasetClient.iterate_itemswithoutdescandsignature, pluspoll_interval.Docs: a new section in the Retrieve Actor data guide, and a mention on the convenience methods and pagination pages.
Closes #1065
✍️ Drafted by Claude Code