server kills sitemap generator
« on: June 05, 2008, 08:19:22 PM »
I have a site with about 4900 pages and 775,000 links on those pages.

I never had troubles until recently. Now the site map generator stalls on the 3rd level. My server provider indicates that the sitemap generator was using 100% of the cpu and had to be stopped.

This problem is consistent and does not changes from attemt to attempt.

The only change I have made has been to add about 700 pages (which is included in the 4900 number above). When the task is aborted, it is not crawling one of those pages.

Tech Support (Lexiconn server) seems to think that there is a bug in the sitemap generator cauksing it to go into a loop.

I'm not sure how to go about debugging this further.
Re: server kills sitemap generator
« Reply #1 on: June 05, 2008, 09:49:41 PM »
Hello,

I suggest you to use "Make delay after each X request" option to reduce the server load while crawling. In case if it interrupts the process, you can resume generation from that point.
Re: server kills sitemap generator
« Reply #2 on: June 06, 2008, 08:08:12 PM »
 I tried a 1 sec delay after 5 accesses  and 1 sec delay after 2 access…server shut us down both times..

 

Re: server kills sitemap generator
« Reply #4 on: June 23, 2008, 08:14:14 PM »
sorry for the delay in getting back to you..

server killing site map was fixed...software issue..

problem now is that your generator stops at level 3...indicates it is done...



any ideas?
« Last Edit: June 23, 2008, 11:03:43 PM by admin »
Re: server kills sitemap generator
« Reply #5 on: June 23, 2008, 11:03:25 PM »
Please install the latest version of Sitemap Generator. Replied to your email.