I have a website, http://www.op.nysed.gov/opsearches.htm, for example where the user selects a **Profession** and enters a **Licensee Name** and clicks on the **Search** button which takes them to a new page to display the result.
I looked at scrapy, arachnode and other web crawlers on the web for this purpose but wasn't too convinced that is the right technology for it.
I was told that we have to crawl those search results from the page. Is it something that can be done?
Can crawler crawl as the user does the search?
I asked a question in SO here: http://stackoverflow.com/questions/23958408/how-to-crawl-external-web-search
↧
Generate dynamic results via search
↧