Crawling only 6 pages
« on: September 24, 2008, 03:45:35 PM »
Hi

I've just installed the software and when run it will only index 6 pages before writing the necessary files.  The site ( [ External links are visible to forum administrators only ] ) is crawlable which I confirmed by using your free online version where it indexed the 500 page limit.
Could you please advise me further.

Regards
Re: Crawling only 6 pages
« Reply #2 on: September 26, 2008, 03:31:31 PM »
Hi
I've overcome that problem by adding www into the site url.  However, I have a further problem as the server does not allow an HTTP request to be sent to the same server to avoid scripts potentially running in infinite loops.  This means that I can't index pages within the same site as the software is installed on.  I can index pages on another server but then I can't save the sitemap on the scanned site.  This situation of course does not allow me to run a cron job to automate updates.

Do you have any thoughts on how I can overcome this problem.

Regards
Re: Crawling only 6 pages
« Reply #3 on: September 26, 2008, 09:10:35 PM »
Hello,

depending on server configuration, possibly there is a proxy service running on the server that allows to open local pages for scripts?
Also, you can setup sitemap generator on external site and then create another cron task on main site just to download resulting sitemap files.