Only 45 of 150,000 Pages Are Crawled...
« on: May 13, 2011, 01:04:49 AM »
Hello,

My website had a 500 error listed and said "(121 seconds with no response)". So I contacted my hosting company. They said, "I have corrected the 500 Internal server error. Also I have increased the memory_limit and max_execution_time settings in php configuration." Well, it's back up but the xml-sitemaps program will only crawl 45 pages (of 150,000) every time I try (even after I clear the data files, upload a new blank sitemap.xml file, and change the sitemap.xml permissions to 666. Do you have any idea how to fix it?

Thanks!
I clicked on crawl and it just started working all of a sudden but now I am getting more "X seconds with no response" error messages. I do not know what to do next as the hosting company have already increased the 2 previously mentioned PHP parameters.

Therefore, please let me know what to do or try next. Here is what my webpage looks like:

Links depth: 2
Current page: quotes/mutual-funds/mutual-funds-vi.html?start=80
Pages added to sitemap: 754
Pages scanned: 1120 (40,365.6 KB)
Pages left: 16 (+ 7988 queued for the next depth level)
Time passed: 0:58:12
Time left: 0:00:49
Memory usage: 11,322.6 Kb
Auto-restart monitoring: Thu May 12 23:17:12 EDT 2011 (117 second(s) since last update)

Thu May 12 23:17:15 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:15:15 EDT 2011: resuming generator (121 seconds with no response)
Thu May 12 23:13:14 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:11:14 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:09:14 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:07:14 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:05:14 EDT 2011: resuming generator (121 seconds with no response)
Thu May 12 23:03:13 EDT 2011: resuming generator (120 seconds with no response)
Thu May 12 23:01:13 EDT 2011: resuming generator (121 seconds with no response)
Re: Only 45 of 150,000 Pages Are Crawled...
« Reply #2 on: May 13, 2011, 03:38:18 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.