feat: Make list methods of clients iterable

[Pijukatel]

Description

• All collection clients list method returns an iterator as well.
• All async collection clients list method returns an async iterator as well.
• List of modified clients (same for async clients):
  • ActorCollectionClient
  • BuildCollectionClient
  • RunCollectionClient
  • ScheduleCollectionClient
  • TaskCollectionClient
  • WebhookCollectionClient
  • WebhookDispatchCollectionClient
  • DatasetCollectionClient
  • KeyValueStoreCollectionClient
  • RequestQueueCollectionClient
  • StoreCollectionClient
  • ActorEnvVarCollectionClient
  • ActorVersionCollectionClient
• Additionally, the following storage-related list methods were modified to support iteration as well:
  • DatasetClient.list_items (and marking Dataset.iterate_items as deprecated)
  • KeyValueStoreClient.list_keys (and marking Dataset.iterate_items as deprecated)
  • RequestQueueClient.list_requests

Example usage

...
# Sync
datasets_client = ApifyClient(token='...').datasets()

# Same as before
list_page = datasets_client.list(...)

# New functionality
individual_items = [item for item in datasets_client.list(...)]

...
# Async
datasets_client = ApifyClientAsync(token='...').datasets()

# Same as before
list_page = await datasets_client.list(...)

# New functionality
individual_items = [item async for item in datasets_client.list(...)]

Issues

• Implements: Add better support for pagination #539

Testing

• Unit tests
• Manual API tests

Checklist

• CI passed

[Pijukatel]

Naming parity with JS is not 100% as there it was possible to use generics, but the typing there allows more flexibility.

JS name:
PaginatedIterator<ActorCollectionListItem>

[codecov]

Codecov Report

❌ Patch coverage is 97.02602% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.92%. Comparing base (ff9817c) to head (fec16e9).

Files with missing lines	Patch %	Lines
src/apify_client/_pagination.py	81.81%	16 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master     #769    +/-   ##
========================================
  Coverage   95.92%   95.92%            
========================================
  Files          48       50     +2     
  Lines        5226     5600   +374     
========================================
+ Hits         5013     5372   +359     
- Misses        213      228    +15     

Flag	Coverage Δ
integration	95.92% <97.02%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Pijukatel]

I tried different approaches with generics, but this turned out to be the best approach:

• It truly describes all the attributes
• It has both static safety, and it benefits from run-time validation when these dataclasses are created from Pydantic models
• It has as little code duplication as possible due to the autogenerated composable init of dataclass
• It gives a clear overview of which endpoints are "weird" - not having the normal pagination metadata

[Pijukatel]

Test setup is somewhat complex and this is due to test code being nearly identical for each endpoint, but different endpoints not being consistent in which pagination it supports (different params and different pagination metadata).

Through this parameter definition, over 200 test cases is generated based on just a few actual tests

[Pijukatel]

These tests are only about pagination logic and constructing each item correctly is totally out of scope, so in all Pydantic models the items: list[whatever] validation is replaced by items: list[dict]

[vdusek]

Sorry, but I'm really not sure about merging pagination and iteration into a single list method. These are two distinct use cases - you either want a page (with metadata), or you want to stream items - and choosing the right method is a way to express that intent.

I would say this also goes against the single responsibility principle - the method now serves two different concerns, which makes the API worse in terms of DX.

If we look beyond the interface and go to the implementation, it becomes even worse. The current approach introduces a 14 IterablePageOf... classes plus mixins just to make the typing work. The return type carries both responsibilities; the async variant is no longer a coroutine but a lazy awaitable-iterable wrapper, and each resource client ends up using a closure-over-callback pattern (where there used to be two simple lines before).

Could we keep list() for pagination and iterate() for iteration? It's a more intuitive interface; callers don't have to learn the quirks of a hybrid object, it will be more understandable for type checkers, and the implementation would be significantly simpler without this mess.

[vdusek]

Do we still have ListPage? Could you ask Claude to also review and update the docs?

[Pijukatel]

Could we keep list() for pagination and iterate() for iteration? It's a more intuitive interface; callers don't have to learn the quirks of a hybrid object, it will be more understandable for type checkers, and the implementation would be significantly simpler without this mess.

If we agree on having it split into two methods in Python client, then it will be way simpler and cleaner internally.
@janbuchar , what do you think?

[janbuchar]

Could we keep list() for pagination and iterate() for iteration? It's a more intuitive interface; callers don't have to learn the quirks of a hybrid object, it will be more understandable for type checkers, and the implementation would be significantly simpler without this mess.

If we agree on having it split into two methods in Python client, then it will be way simpler and cleaner internally. @janbuchar , what do you think?

Well, in https://github.com/apify/apify-client-js, we have a list (or listItems for dataset items) method that returns a PaginatedIterator. This object can either be await-ed to receive a single page of the results or async-iterated to process all items in all pages.

Our current doctrine is pretty much "1:1 parity unless we have a clearly superior approach that we want to eventually use in the other flavor as well". Also, we mostly tend to sacrifice maintainability if it means that the API will become easier to use.

Thus the one thing to consider from @vdusek response is if returning a "polymorphic" object that is both an async iterable and a simple promise is strictly worse DX than having two distinct methods for each list-shaped endpoint. If it is, then we should also decide if this is something that we'd like to adjust in the JS version or if there is a compelling argument for letting the JS version and the Python version diverge.