Multithreaded Web Scraping via VBA

[verance]

At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.

I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.

For Each url In URLList
ScrapeWebsite(url)
Next url

Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...

 

[John_w]

I've used the 'swarm' method - multiple VBScript processes launched by Excel VBA - to scrape websites and it is a very fast method if you have hundreds of URLs, much faster than sequential retrieval by VBA alone. See Multithreaded VBA – An Approach To Processing Using VBScript | Excel & VBA – Databison.

There are other ideas at multithreading - Multi-threading in VBA - Stack Overflow, particularly the VBA Multithreading Tool which looks interesting.