This is a discussion on Multithreaded Web Scraping via VBA within the Excel Questions forums, part of the Question Forums category; At the moment I'm using VBA to create a list of URLs as a collection and then loop through them ...
At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.
I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.
For Each url In URLList
Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...
I've used the 'swarm' method - multiple VBScript processes launched by Excel VBA - to scrape websites and it is a very fast method if you have hundreds of URLs, much faster than sequential retrieval by VBA alone. See Multithreaded VBA – An Approach To Processing Using VBScript | Excel & VBA – Databison.
There are other ideas at multithreading - Multi-threading in VBA - Stack Overflow, particularly the VBA Multithreading Tool which looks interesting.