Crawling Querystring Parameters
« on: March 23, 2010, 06:04:33 PM »
I want to crawl links with Querystring parameters.  When I run the free generator, it does crawl the parameter I want, but with only three individual values.  I don't know how the generator found/picked those three.  I also don't know how I can manage this in the future as new values need to be added/removed.  There is one main parameter I would like to crawl and I need to crawl it for every value on the site.
 Any suggestions?

Thank you.
Re: Crawling Querystring Parameters
« Reply #1 on: March 23, 2010, 09:12:52 PM »
Hello,

there must be a way to reach that page by clicking links, starting from homepage. i.e. you should put that link somewhere on your site so that generator can find it.
Re: Crawling Querystring Parameters
« Reply #2 on: March 23, 2010, 09:51:35 PM »
The problem is the querystring is not tied to a link.  The user can enter data or select a state, this allows the user to navigate directly to a page or select from a list of links based on their selected state.  There is no static link to select.  Is there a was to cheat the system and provide the links 5000px off screen or some other trick?
Re: Crawling Querystring Parameters
« Reply #3 on: March 24, 2010, 05:06:07 PM »
You can create a special page with links to all inner pages handcoded, and then just use that page as starting URL for generator.