Sitemap Generator Forum

I am trying to run a crawl on a site with about 20,000 identified pages but it keeps freezing at Links depth: 5  (13000 pages scanned and 8000 queued). i have run and re-run the crawl on a few occasions and it keeps freezing at this point. I have also set the crawl to delay for 60s after each  1000 requests if that helps?

Hello,

do you have "resume generation" checkbox when running the generator again?
There is no need in so long delays, something like 5 seconds every 200 request would be better.

Hi

Yes i have the 'resume generation' turned on and i have changed the delay to 6 seconds but it is still pausing at Link depth 5. I have tested the software to make sure that it is working and it seems to create sitemaps for them with no problem (they are a lot smaller though).

Hello,

please let me know your generator URL in private message.