XML Sitemaps Generator

Author Topic: Large site won't crawl  (Read 8439 times)

leslie

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Large site won't crawl
« on: April 21, 2011, 05:18:23 PM »
I installed the Unlimited PHP Sitemap Generator on a large site (100K+ URL's) The first time it crawled it "looked" like it may have tried to crawl the entire site. I got errors that it could not create sitemap1.xml - sitemap5.xml. I tried creating blank files, gave them the correct permissions, but now the Generator is only crawling less than 3000 pages.

Is there a cache that I need to clear to get it to grab all the URLs? I have
Maximum pages: 0
Maximum depth level: 0
Maximum execution time, seconds: 0
Maximum memory usage, MB: 256

I also have the following:
Number of URLs per file in XML sitemap and maximum file size: 40000

Any thoughts and suggestions are welcome.
Many thanks,
Leslie


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Large site won't crawl
« Reply #1 on: April 22, 2011, 06:34:51 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2