Python Dryscrape - how to select child element

Question

I'm working on a project with www.NJMLS.com. I'm trying to extract email of agent name "Edward An": import dryscrape as d d.start_xvfb() br = d.Session() br.visit('http://www.njmls.com/members/index.cfm?action=dsp.results&city=&county=&nametype=firstname&name=Edward&x=70&y=33') #open url and try to extract the email of Edward An Now how to get email of "Edward An". If I try: for x in br.xpath("//*[@class = 'realtor-info']/*") # selecting all elements in

....

if "Edward An" in x: print x['herf'] #Not responding What's wrong with my script? if "Edward An" in x.text() print x Is also not working. Still No Error and No response

Amias · Answer

Have a look at the example here - https://dryscrape.readthedocs.io/en/latest/usage.html

Looks like you need to start the session object by giving it a base url

br = d.Session('http://www.njmls.com')

and then when you call visit give in the URI

br.visit('/index.cfm?action=dsp.results&city=&county=&nametype=firstname&name=Edward&x=70&y=33')

your if statement seems unlikely to work as its implying that X is is a string when its likely to be an element object but it seems there might be more issues.

Does the page require a login ?
Does the server let you visit that page ?
Does the server accept city and county being empty ?

You should be able to add some error handling or use a debugger (pycharm has awesome python debugger integration) or logging statements to see what else is happening.

My guess is that you are not getting the page you expect or it is not loading.

Python Dryscrape - how to select child element

One Answer

Add your own answers!

Ask a Question