I’ve had great success in the past using VBA in Microsoft Access in conjunction with an Internet Explorer object and a Regex object to do web scraping. I typically set up the objects like this:
Dim RE As Object, Matches As Object
Dim oIE As InternetExplorer, hElm As IHTMLElement, vPr As Integer
Set RE = CreateObject("VBScript.RegExp")
Set oIE = New InternetExplorer
Then inside a loop, I read a URL from a table and load the page into the Internet Explorer object:
oIE.Navigate strURL
Do While (oIE.Busy Or oIE.ReadyState <> READYSTATE_COMPLETE)
DoEvents
Loop
Set hElm = oIE.Document.all.tags("html").Item(0)
varBuffer = hElm.outerHTML
I then have all of the HTML text in a variable that I can process with REGEX, and what I pluck out of the text can be immediately deposited into an Access table. Using IE 11 and Access 2013 last September, it worked beautifully for a run of more than 300 pages. But using the same two packages this year for the same run, IE always crashes with an error message between pages 150-159. I thought that it might be due to using Windows 10 instead of Windows 8, so I tried it on a virtual machine on a Windows 2012 server. I didn’t get the crash or any error message, but the loop simply stopped running between 120-130 pages. The Explorer object was open to the last page processed and completely responsive. Access seemed to be responsive also. However, task manager showed both IE and Access consuming nearly all of the processor cycles. Pressing the stop button in VBA stopped this activity, but didn’t highlight any line of code.
Can anybody figure out what’s going on and help me get back to the functionality I had last year? It would seem that some patch that Microsoft made to IE 11 has thrown a monkey wrench into the works. If it will help, this is the error event when IE suddenly crashes: