XML Sitemaps Generator

Author Topic: General Questions  (Read 25102 times)

KellyW

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
General Questions
« on: June 19, 2007, 10:40:33 PM »
I just purchased this sitemap builder and have a general question. I didnt see any setting for it but does the crawler respect the robots.txt file?

I have added the files I dont want into the config as well.

Also it seems to hang alot. I had run one crawl and interrupted it then went back to run a clean crawl beause it wasnt creating the sitemap.xml file, only the html and txt file but on the second run it just started spitting out errors.

Warning: preg_match(): Compilation failed: nothing to repeat at offset 1 in /usr/local/psa/home/vhosts/4wheelers.com/httpdocs/sm_gen/pages/class.grab.inc.php(2) : eval()'d code on line 107

Anyone know what this error means?
« Last Edit: June 19, 2007, 11:07:45 PM by KellyW »

KellyW

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: General Questions
« Reply #1 on: June 20, 2007, 04:26:08 AM »
Ive also had an extremely slow crawl.

Links depth: 5
Current page: bestop-khaki-diamond-supertop-with-tinted-windows-wrangler-54709-36-p-1436.html
Pages added to sitemap: 3241
Pages scanned: 4680 (118,191.4 Kb)
Pages left: 168 (+ 403 queued for the next depth level)
Time passed: 281:02
Time left: 10:05
Memory usage: -

Over 4 1/2 hours to index 3241 and scan 4680?

I can scan it with an online scanner in 10 minutes. I only have about 5000 pages. The server isnt slow either because I can surf around the site, run other process and the CPU and memory usage is low.

Any ideas why this is so slow?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: General Questions
« Reply #2 on: June 20, 2007, 09:35:55 PM »
Hello,

let me know your generator URL in private message.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

KellyW

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: General Questions
« Reply #3 on: July 01, 2007, 07:54:50 PM »
I sent you the URL of the site and the generator, but dont know if it sent since theres no sent box in the messages.

The generator is creating weird URL's with double addresses like this on one line:

[external links are visible to admins only]

It is also adding HTTPS addresses to the sitemap which will cause Google to reject the sitemap.

KellyW

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: General Questions
« Reply #4 on: July 01, 2007, 08:09:55 PM »
I removed the program, reinstalled it and still get the error when I try to crawl:

Warning: preg_match(): Compilation failed: nothing to repeat at offset 1 in /usr/local/psa/home/vhosts/4wheelers.com/httpdocs/sm_gen/pages/class.grab.inc.php(2) : eval()'d code on line 107

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: General Questions
« Reply #5 on: July 01, 2007, 08:23:01 PM »
Hello,

replied to your private message.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2