General Questions

[KellyW]

I just purchased this sitemap builder and have a general question. I didnt see any setting for it but does the crawler respect the robots.txt file?

I have added the files I dont want into the config as well.

Also it seems to hang alot. I had run one crawl and interrupted it then went back to run a clean crawl beause it wasnt creating the sitemap.xml file, only the html and txt file but on the second run it just started spitting out errors.

Warning: preg_match(): Compilation failed: nothing to repeat at offset 1 in /usr/local/psa/home/vhosts/4wheelers.com/httpdocs/sm_gen/pages/class.grab.inc.php(2) : eval()'d code on line 107

Anyone know what this error means?

[KellyW]

Ive also had an extremely slow crawl.

Links depth: 5
Current page: bestop-khaki-diamond-supertop-with-tinted-windows-wrangler-54709-36-p-1436.html
Pages added to sitemap: 3241
Pages scanned: 4680 (118,191.4 Kb)
Pages left: 168 (+ 403 queued for the next depth level)
Time passed: 281:02
Time left: 10:05
Memory usage: -

Over 4 1/2 hours to index 3241 and scan 4680?

I can scan it with an online scanner in 10 minutes. I only have about 5000 pages. The server isnt slow either because I can surf around the site, run other process and the CPU and memory usage is low.

Any ideas why this is so slow?

[admin]

Hello,

let me know your generator URL in private message.

[KellyW]

I sent you the URL of the site and the generator, but dont know if it sent since theres no sent box in the messages.

The generator is creating weird URL's with double addresses like this on one line:

[external links are visible to admins only]

It is also adding HTTPS addresses to the sitemap which will cause Google to reject the sitemap.

[KellyW]

I removed the program, reinstalled it and still get the error when I try to crawl:

Warning: preg_match(): Compilation failed: nothing to repeat at offset 1 in /usr/local/psa/home/vhosts/4wheelers.com/httpdocs/sm_gen/pages/class.grab.inc.php(2) : eval()'d code on line 107

[admin]

Hello,

replied to your private message.