Thanks Thanks:  0
Likes Likes:  0
Results 1 to 2 of 2

Thread: Multithreaded Web Scraping via VBA

  1. #1
    New Member
    Join Date
    Dec 2015
    Posts
    4
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Default Multithreaded Web Scraping via VBA

    At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.

    I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.

    For Each url In URLList
    ScrapeWebsite(url)
    Next url

    Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...

  2. #2
    Board Regular
    Join Date
    Oct 2007
    Posts
    5,575
    Post Thanks / Like
    Mentioned
    6 Post(s)
    Tagged
    2 Thread(s)

    Default Re: Multithreaded Web Scraping via VBA

    I've used the 'swarm' method - multiple VBScript processes launched by Excel VBA - to scrape websites and it is a very fast method if you have hundreds of URLs, much faster than sequential retrieval by VBA alone. See Multithreaded VBA An Approach To Processing Using VBScript | Excel & VBA Databison.

    There are other ideas at multithreading - Multi-threading in VBA - Stack Overflow, particularly the VBA Multithreading Tool which looks interesting.

Some videos you may like

User Tag List

Like this thread? Share it with others

Like this thread? Share it with others

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •