Multithreaded Web Scraping via VBA

verance

New Member
Joined
Dec 22, 2015
Messages
4
At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.

I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.

For Each url In URLList
ScrapeWebsite(url)
Next url

Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...
 

Some videos you may like

Excel Facts

When they said...
When they said you are going to "Excel at life", they meant you "will be doing Excel your whole life".

John_w

MrExcel MVP
Joined
Oct 15, 2007
Messages
6,207

Watch MrExcel Video

Forum statistics

Threads
1,095,151
Messages
5,442,681
Members
405,191
Latest member
wedloski

This Week's Hot Topics

  • Copy entire row if CountA <>0 to another sheet
    [B]I want to copy entire row if CountA <>0 for column J7:AM7 (headers on J6:AM6) and so on till the last used cell is column D and paste the...
  • Select last used Row in Table
    I have created a Table in a Worksheet which is locked to prevent user errors and protect formula. Some of the cells require freetext entries which...
  • excel workbook: do not allow certain file name
    Hello all, Don't think this has ever been asked before, but how do I restrict file save [Before_Save Event] if the name of the file being saved...
  • fixing problem autofilter
    hello i need help about my code when i search by code in textbox it doesn't show anything this is my data [ATTACH type="full"...
  • “Weight”
    Hi, i’ve got a long sheet filled with weights such as kg,g,L & ml. i can build a formula to convert kg into g and liter into ml. How ever, my...
  • How to capitalize everything before a certain character?
    In column A, I have some text: Hello good day.mp3 Hello good day.flac etc. I'd like to capitalize everything before the period. I don't need the...
Top