XML Sitemaps Generator

Author Topic: Configuration - Crawling Problem  (Read 7910 times)

insync

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Configuration - Crawling Problem
« on: July 23, 2011, 11:40:23 PM »
We've noticed that if the 'Starting URL' contains 'www', the sitemap generator does not stay within the starting directory, and jumps to directories it is not supposed to crawl.  Removing the 'www' fixes this.

Problem is google webmaster site is setup with www in the domain, so Google complains sitemap should have www in the url.  Please advise...thx

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Configuration - Crawling Problem
« Reply #1 on: July 24, 2011, 12:58:44 PM »
Hello,

you can use "Exclude URLs" setting to tell generator not to crawl specific directories.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

insync

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Configuration - Crawling Problem
« Reply #2 on: July 24, 2011, 01:13:34 PM »
We are using exclude urls.  Problem is this setting is ignored when we have www in starting url.  Remove www, and generator honors exclude urls, and doesn't go beyond the directory.

Also, this problem happens when we're crawling a subdirectory, and don't want to crawl below the directory...thx

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Configuration - Crawling Problem
« Reply #3 on: July 24, 2011, 01:25:16 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Make sure that you don't specify domain name in Exclude URLs, i.e. use it like:
folder1/
foder2/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

insync

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Configuration - Crawling Problem
« Reply #4 on: July 24, 2011, 03:50:56 PM »
sent private message

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2