XML Sitemaps Generator

Author Topic: Crawler just churns and churns  (Read 8062 times)

rshandelman

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Crawler just churns and churns
« on: July 18, 2011, 08:46:41 PM »
I've upgraded to v5.0 and am running the sitemap generator as we speak but its just churning and churning (for hours) without results.  I "xed" out the browser and tried to log back the browser will never reload the /generator/ page.  I looked on the server and ror.xml and sitemap.xml have not been modified.

Can someone pls help and let me know how I can fix this?!

Thanks.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawler just churns and churns
« Reply #1 on: July 19, 2011, 10:47:38 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

rshandelman

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Crawler just churns and churns
« Reply #2 on: July 19, 2011, 04:30:55 PM »
Hi Oleg,

It finally worked after I left it alone for many hours. I set up a new crontab for the sitemap to recreate during a low activity time to hopefully increase the performance.

Code: [external links are visible to admins only]
/usr/local/bin/php /var/www/vhosts/mydomain.com/httpdocs/generator/runcrawl.php


I would prefer to recreate the sitemap every day, but for now I set to once a week to not overload the server.

Any suggestions?
« Last Edit: July 19, 2011, 04:35:04 PM by rshandelman »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawler just churns and churns
« Reply #3 on: July 19, 2011, 10:14:58 PM »
You can use the "Make delay for X seconds after each Y request" option - that will slow down the crawler, making the overall generation process longer, but reducing the server load.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

rshandelman

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Crawler just churns and churns
« Reply #4 on: July 20, 2011, 03:54:16 AM »
ok, cool. I'll try that. thx

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2