Scraping Data using Selenium

[hicksi]

Now that I've been able to access Web sites, I want to collect the data from a table on that site into my Excel spreadsheet.

In particular, I am trying to collect the list of studio albums for various bands from Wikipedia.

For example, I need to:

1. Access the page Rolling Stones Discography The Rolling Stones discography - Wikipedia
2. Find a <h2> that looks like "Studio Albums"
3. Within the code between this <h2> and the next <h2>
   1. Find a Table
   2. Select the whole table
   3. Paste the table into the Excel spreadsheet
4. Repeat for another known band's discography page.

Any thoughts and possible code would be appreciated.

 

[Dan_W]

Glad you managed to get Selenium working. I can find you some really good web scraping tutorials etc if you like (Wise Owl is always good).
Just a few points:
- If you're scraping pure HTML (like Wikipedia), you really wouldn't need to use Selenium at - it'd be quicker not to use Selenium! That said, it's useful to have it ready in your toolkit should you encounter some tricky javascript sites.
- You could try importing the data into Excel by using the Data tab -> "Get Web" function. You should try recording a macro as you do it, and have a look at the code that is produced. You might be able to to adjust it so that you can use it as a quick means of webscraping going forward.
- Also, finally, here's a poor man's webscraping technique - try typing this into the Immediate Window in the VBA EDitor and press enter. See what happens....

 

[Kerryx]

Or just use Data -> Get Data-> From Web

Select the table you want then press Transform Data and edit it to what you need in Power Query

 

[Dan_W]

Or just use Data -> Get Data-> From Web

Select the table you want then press Transform Data and edit it to what you need in Power Query

I feel like I did say that...

- You could try importing the data into Excel by using the Data tab -> "Get Web" function.

 

[Kerryx]

my apologies if offended of course you did mention it

 

[Dan_W]

No, no - not offended at all. I was just momentarily confused and suddenly filled with existential panic ("Did I really say what I think I said?", "Am I really who I think I am?", "Am I even real"?)

 

[Kerryx]