Multithreaded Web Scraping via VBA

Thanks Thanks:  0
Likes Likes:  0
Results 1 to 2 of 2

Thread: Multithreaded Web Scraping via VBA

  1. #1
    New Member
    Join Date
    Dec 2015
    Posts
    4
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Default Multithreaded Web Scraping via VBA

    At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.

    I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.

    For Each url In URLList
    ScrapeWebsite(url)
    Next url

    Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...

  2. #2
    Board Regular
    Join Date
    Oct 2007
    Posts
    4,797
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Multithreaded Web Scraping via VBA

    I've used the 'swarm' method - multiple VBScript processes launched by Excel VBA - to scrape websites and it is a very fast method if you have hundreds of URLs, much faster than sequential retrieval by VBA alone. See Multithreaded VBA An Approach To Processing Using VBScript | Excel & VBA Databison.

    There are other ideas at multithreading - Multi-threading in VBA - Stack Overflow, particularly the VBA Multithreading Tool which looks interesting.

User Tag List

Like this thread? Share it with others

Like this thread? Share it with others

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

 

DMCA.com