XML Sitemaps Generator

Author Topic: new - crawler is skipping folders - can you tell me why?  (Read 13975 times)

webmaster176

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
new - crawler is skipping folders - can you tell me why?
« on: October 03, 2008, 09:11:35 PM »
Hi!

When I run the generator, it crawls more than 1900 files correctly, and then skips a lot more without comment. As far as I can tell there's no difference. In particular, it doesn't seem to see the whole folder scienceforkids/, which is in my root directory.

Can anyone help?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: new - crawler is skipping folders - can you tell me why?
« Reply #1 on: October 03, 2008, 11:15:05 PM »
Hello,

looks like that section is restricted with your robots.txt
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

webmaster176

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: new - crawler is skipping folders - can you tell me why?
« Reply #2 on: October 03, 2008, 11:30:20 PM »
Okay, I'm a moron. Thank you so much. I forgot I had restricted it last year when I began working on the new section. I am so grateful to you for pointing that out.

I hate to appear even more moronic, but now I can't seem to log in to the generator on my site. I wrote down the username and password (I thought) but now they don't seem to work.

Karen

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: new - crawler is skipping folders - can you tell me why?
« Reply #3 on: October 06, 2008, 02:52:22 PM »
You can find out your login and password by viewing config.inc.php file via FTP:
    'xs_login' => '',
    'xs_password' => '',
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2