Large site won't crawl
« on: April 21, 2011, 06:18:23 PM »
I installed the Unlimited PHP Sitemap Generator on a large site (100K+ URL's) The first time it crawled it "looked" like it may have tried to crawl the entire site. I got errors that it could not create sitemap1.xml - sitemap5.xml. I tried creating blank files, gave them the correct permissions, but now the Generator is only crawling less than 3000 pages.

Is there a cache that I need to clear to get it to grab all the URLs? I have
Maximum pages: 0
Maximum depth level: 0
Maximum execution time, seconds: 0
Maximum memory usage, MB: 256

I also have the following:
Number of URLs per file in XML sitemap and maximum file size: 40000

Any thoughts and suggestions are welcome.
Many thanks,

Re: Large site won't crawl
« Reply #1 on: April 22, 2011, 07:34:51 AM »

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.