XML Sitemaps Generator

Author Topic: not crawling whole site  (Read 13676 times)

andy17

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
not crawling whole site
« on: October 22, 2008, 12:45:58 PM »
Each time I set the script crawling it stops after a few minutes and gives me the  message shown below, my site has 50,000 pages. I believe that I have set all the permissions correctly and I have link depth and everything else set to unlimited.

Any ideas what i am doing wrong.

Links depth: 1
Current page: directory/Desert_Blue_Jeans.php
Pages added to sitemap: 138
Pages scanned: 140 (10,322.8 KB)
Pages left: 56 (+ 7920 queued for the next depth level)
Time passed: 6:29
Time left: 2:35
Memory usage: 4,643.8 Kb
Resuming the last session (last updated: 2008-10-20 09:10:53)

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: not crawling whole site
« Reply #1 on: October 22, 2008, 09:40:53 PM »
Hello,

replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2