Problems

Recognise it when: lots of independent, I/O-bound work: crawling, fetching, calling APIs, processing files.

with ThreadPoolExecutor(max_workers=8) as pool:
    pending = {pool.submit(fetch, start)}
    while pending:
        done, pending = wait(pending, return_when=FIRST_COMPLETED)
        for f in done:
            for nxt in f.result():
                if nxt not in seen:
                    seen.add(nxt)
                    pending.add(pool.submit(fetch, nxt))

• A coordinator thread owns seen, so there's no lock. If workers update shared sets themselves, guard them with a Lock.
• Use ProcessPoolExecutor for CPU-bound work (the GIL). Functions must be picklable, meaning top-level.
• For asyncio, asyncio.Semaphore(k) bounds concurrency and asyncio.gather fans out.