• 放入队列中，多进程消费，每个进程中还是单线程执行下载；
• thread 多线程下载；
• 如果是curl，可以用 libcurl multi 并行请求下载，也相当于多线程；

...应该差不多这个思路

还是有点看不懂,大佬,能帮我代码化一下吗?万分感谢

Something like this

import time
import threading
import requests

def download(url, index):
    response = requests.get(url)
    # print(f'{index:0>2d}:{url} downlaoded.')

urls = [f'https://learnku.com/' for i in range(100)]

now = time.time()

threads = []
for i, url in enumerate(urls):
    thread = threading.Thread(target=download, args=(url, i), daemon=True)
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join()

seconds = time.time() - now

print(f'All URLs downloaded in {seconds:.2f} seconds')

Note:

• The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting"). A Retry-After header might be included to this response indicating how long to wait before making a new request.
• Should try to download it again if failed.

可以用协程，更轻便一点

使用协程就需要把同步的操作都换成异步处理，引用aiohttp、aiofiles

import aiohttp
import asyncio
import aiofiles


async def download(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            await resp.text()
            # to do something
            async with aiofiles.open('a.txt', mode='w', encoding='utf-8') as f:
                await f.write('write something')


async def main():
    urls = ['url1', 'url2', 'url3']
    task = []
    for url in urls:
        task.append(asyncio.create_task(download(url)))
    await asyncio.wait(task)

if __name__ == '__main__':
    asyncio.run(main())