Importing from a web page that seems to expect specific browsers

Question

I've successfully and fruitfully imported data from web sites before, but now seem to have a particularly troublesome site that I would like to access through Mathematica on my MacBook. Import[site="https://auctionfiling.fcc.gov/form175/search175/index.htm"] returns the error: FetchURL::httperr: The request to URL https://auctionfiling.fcc.gov/form175/search175/index.htm was not successful. The server returned the HTTP status code 500 ("Internal Server Error"). Exploring further, the command URLExecute[site] returns an long exception report that includes the text: Further text indicates that the site requires either: Microsoft Internet Explorer 5.5 (or higher), Netscape 6 (or higher), Opera 6 (or higher), or Mozilla. Though I've successfully used it with Safari, Chrome and Firefox. Is there a way around this? I am wanting to pull down many thousands of FCC license records through this approach. Alternatively, perhaps there is an approach of running an external browser linked with Mathematica? Thank you for any suggestions. Steve

Alexey Popkov · Answer

Posting J.M.'s comment as an answer, you can specify the user agent string in HTTPRequest directly via "UserAgent" (here it is for Firefox 40.1 taken from this site):

URLRead[HTTPRequest[
  "https://auctionfiling.fcc.gov/form175/search175/index.htm", <|
   "UserAgent" -> 
    "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>]]

(note that the code in the comments is broken due to hidden Unicode characters).

It is worth to know that

The default "UserAgent" is typically "Wolfram HTTP Client xxxx", where xxxx is a version number.

Emilio Pisanty · Answer

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

This seems to be possible since v12.0, using WebExecute. You can find it in the new-features page of v12, under "Controlling Web Browsers" and described shortly as "execute commands in a web browser".
I don't know whether this would have been sufficient for your purposes (and, I would suspect, it's too late for your original task). But it's an interesting option to consider for similar problems.

Importing from a web page that seems to expect specific browsers

2 Answers

Add your own answers!

Ask a Question