Sitemap Generator Forum

I have a little bit of an interesting situation in that our website is hosted on a stand-alone server owned and maintained by our ERP system provider. I have access to the server only when I am RDPing from one of our SQL servers (in other words, I can only access the web server with a local 10.something ip address).

I should have well over 100,000 pages, although we don't know for sure. I am getting 500 Internal Server Error every time when I try to run the generator. We have upped the timeouts and everything else this site recommends. Please advise further.

Thank you

Hello,

in this case you can use option "Make delay for X seconds after each Y request" - that will slow down the crawling, reducing the server load, allowing to avoid server errors.

With a delay of 5 s after every 10 requests, I still get the 500 Internal Server Error. Here are my settings in my php file:

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

max_execution_time = 9000     ; Maximum execution time of each script, in seconds
max_input_time = 120   ; Maximum amount of time each script may spend parsing request data
;max_input_nesting_level = 64 ; Maximum input variable nesting level
memory_limit = 256M      ; Maximum amount of memory a script may consume (128MB)

When I click off of the crawling tab and then back to it, this is what I see:

Links depth: 2
Current page: default.aspx?page=item%20search%20results&SearchFieldName=VendorItemSearch&SearchSource=BrandsList&selVendorName=%22GearWrench%22&selVendorCode=GearWrench
Pages added to sitemap: 55
Pages scanned: 80 (3,690.3 KB)
Pages left: 273 (+ 50 queued for the next depth level)
Time passed: 0:01:13
Time left: 0:04:09
Memory usage: 1,307.7 Kb

I know that it's not maxing out after just 1.3Mb.

Hello,

please let me know your generator URL/login in private message to check this.