Sitemap Generator Forum

I just ran my generator for the first time and it is skipping a large number of pages.  On the changelog it says "400 Bad Request" next to all of them.  Anyone have any suggestions?

Hello,

perhaps your site blocks requests when there are too many of them coming in short time period. You can try to define 1 second delay after each 1 request in generator configuration to slow down the crawler.

I tried doing that just now and it still isn't working.  As it was running it showed it had scanned more pages than it had added to the sitemap for example:

Links depth: 2
Current page: products/17_AAG-1128-49.html
Pages added to sitemap: 391
Pages scanned: 836 (24,501.0 KB)
Pages left: 2733 (+ 595 queued for the next depth level)
Time passed: 0:16:08
Time left: 0:52:46
Memory usage: 4,804.5 Kb

In the end it has skipped 1000 pages every time I have run it, crawled 4164, and only indexed 431.

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.