Extract all the links of a website

[stefanoste78]

I found this macro:

https://stackoverflow.com/questions/39222082/vba-to-find-text-from-webpages

I did not understand if it is possible to extract all the links of a web page or if it is possible to extract all the links of a website.

I would like to extract all the links of the pages of a website.

It could?

Thank you

 

[stefanoste78]

Hello. Is it possible to slightly modify this macro in such a way that it extrapolates for each link listed in the column the link of the next level to the first that contains the word indicated as column header for each heading inserted?
Thank you

 

[stefanoste78]

Good evening.
Can you tell me if it is possible to use the macro to be able to extrapolate for each url indicated in the column to only the links that have within them the string indicated in the column header (from column b onwards)?
Thank you

 

[Worf]

Hello

Place the desired string on the second row of each column.

Sub Read_Linksb()
Dim c00, myimg, i%, Lk, it, lr%, c%
With CreateObject("MSXML2.XMLHTTP")
    .Open "Get", "https://www.MrExcel.com", False
    .Send
    c00 = .responseText
End With
Cells(1, 1) = "List"
With CreateObject("htmlfile")
    .Body.innerHTML = c00
    i = 1
    Set myimg = .getElementsByTagName("img")
    For Each Lk In myimg
        i = i + 1
        Cells(i, 1) = Lk.src
    Next
    For Each it In .Links
        i = i + 1
        Cells(i, 1) = it.href
    Next
End With
For c = 2 To Cells(2, Columns.Count).End(xlToLeft).Column
    Cells(1, c) = Cells(1, 1)
    Cells(2, c) = "*" & Cells(2, c) & "*"
    Range("a1:a" & Range("a" & Rows.Count).End(xlUp).Row).AdvancedFilter _
    xlFilterCopy, Range(Cells(1, c), Cells(2, c)), Cells(3, c), False
Next
End Sub

 

[stefanoste78]

I tried the macro and noticed that the site must be inserted in the code. I would like the macro to automatically extract links for all the urls indicated in the column. I enclose an example.

https://www.dropbox.com/s/x0hvee4ab03on4h/research link.xlsm?dl=0

 

[Worf]

Buonasera

Sub Read_Linksb()
Dim c00, myimg, i%, Lk, it, lr%, c%, nu%
i = 1
For nu = 3 To Range("b" & Rows.Count).End(xlUp).Row ' start at B3
    With CreateObject("MSXML2.XMLHTTP")
        .Open "Get", Cells(nu, 2), False
        .Send
        c00 = .responseText
    End With
    Cells(1, 1) = "List"
    With CreateObject("htmlfile")
        .Body.innerHTML = c00
        Set myimg = .getElementsByTagName("img")
        For Each Lk In myimg
            i = i + 1
            Cells(i, 1) = Lk.src
        Next
        For Each it In .Links
            i = i + 1
            Cells(i, 1) = it.href
        Next
    End With
Next
For c = 3 To Cells(2, Columns.Count).End(xlToLeft).Column
    Cells(1, c) = Cells(1, 1)
    Cells(2, c) = "*" & Cells(2, c) & "*"
    Range("a1:a" & Range("a" & Rows.Count).End(xlUp).Row).AdvancedFilter _
    2, Range(Cells(1, c), Cells(2, c)), Cells(3, c), False
Next
End Sub

 

[stefanoste78]

I tried the macro. In essence it extrapolates links as the previous macro and then transfers only the links that contain the string in the header.
You could avoid the links coming out in the "a" column because if the urls are so many then the excel sheet would not be enough. At most you could transfer all the links extracted in the other sheet occupying the first column then the second etc.
Then, at the moment the macro is started, the domain level of the links to be transferred can be established (for example, only the links of the second or third level, etc.).


My final intention is to extract from the link the emails of the page to which the column titles refer.


Thank you

 

[Worf]

I am not sure if I understand what you want. Do you need to explore all site levels?
This would mean accessing all subpages, until exhausting all internal site links.
It should be possible to list all site links, including subpages.

 

[stefanoste78]

It would be useful to have them all and subsequently to be able to decide what link the link inserts into the dedicated column at the correspondence of the string given.
So there will be: 1) the extraction of all links;
2) selection by the user of what link insertion (if the maximum level of domains and 6 and user chosen 2 in the columns will only be inserted into link to the second level).
Could you? Of course, the user could write it from 1 to Num Massimo.

 

[stefanoste78]

Hi wolf. Is difficolt to do it?

 

[Worf]

I would say it is tricky. I am having a busy week but will work on this as soon as possible...