Results 1 to 2 of 2

Multithreaded Web Scraping via VBA

This is a discussion on Multithreaded Web Scraping via VBA within the Excel Questions forums, part of the Question Forums category; At the moment I'm using VBA to create a list of URLs as a collection and then loop through them ...

  1. #1
    New Member
    Join Date
    Dec 2015
    Posts
    4

    Default Multithreaded Web Scraping via VBA

    At the moment I'm using VBA to create a list of URLs as a collection and then loop through them and scrape some basic info. It all works fine but takes ~3 seconds per URL. Sometimes I have 50+ urls to go through so it takes longer than ideal.

    I use a for each loop as below where ScrapeWebsite is a sub which loads a url and retrieves the required info.

    For Each url In URLList
    ScrapeWebsite(url)
    Next url

    Is there any easy(ish) way to run all (or a portion) of the urls at the same time to save time? I've looked up a bit about swarms but it seems a little complicated for me...

  2. #2
    Board Regular
    Join Date
    Oct 2007
    Posts
    4,358

    Default Re: Multithreaded Web Scraping via VBA

    I've used the 'swarm' method - multiple VBScript processes launched by Excel VBA - to scrape websites and it is a very fast method if you have hundreds of URLs, much faster than sequential retrieval by VBA alone. See Multithreaded VBA An Approach To Processing Using VBScript | Excel & VBA Databison.

    There are other ideas at multithreading - Multi-threading in VBA - Stack Overflow, particularly the VBA Multithreading Tool which looks interesting.

Like this thread? Share it with others

Like this thread? Share it with others

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


DMCA.com