dlt.extract.concurrency
FuturesPool Objects
class FuturesPool()
Worker pool for pipe items that can be resolved asynchronously.
Items can be either asyncio coroutines or regular callables which will be executed in a thread pool.
submit
def submit(pipe_item: ResolvablePipeItem) -> TItemFuture
Submit an item to the pool.
Arguments:
pipe_item- The pipe item to submit.pipe_item.itemmust be either an asyncio coroutine or a callable.
Returns:
The resulting future object
resolve_next_future
def resolve_next_future(
use_configured_timeout: bool = False) -> Optional[ResolvablePipeItem]
Block until the next future is done and return the result. Returns None if no futures done.
Arguments:
use_configured_timeout- If True, use the value ofself.poll_intervalas the max wait time, raisesconcurrent.futures.TimeoutErrorif no future is done within that time.
Returns:
The resolved future item or None if no future is done.
resolve_next_future_no_wait
def resolve_next_future_no_wait() -> Optional[ResolvablePipeItem]
Resolve the first done future in the pool. This does not block and returns None if no future is done.