CGI Error
« on: November 08, 2006, 04:09:46 PM »
I am receiving the following error after crawling for a couple of minutes...

The specified CGI application misbehaved by not returning a complete set of HTTP headers.

Any ideas?
Re: CGI Error
« Reply #1 on: November 08, 2006, 08:41:11 PM »
I've worked through this issue, but now I'm stuck again.  I am on a shared server so the php.ini cannot be edited.  I am saving the progress and rerunning it after it times out  over and over, but this could take a very long time.  Is there anyway around this problem without access to php.ini?
Re: CGI Error
« Reply #2 on: November 09, 2006, 07:07:05 AM »
Some hosting environments (not all though) allow to change php settings per directory, so you can try to create php.ini file in generator/ folder:
Code: [Select]
max_execution_time=5000
memory_limit=128M
(you can change values to suit your instance)
OR create .htaccess file:
Code: [Select]
php_value memory_limit 128M
php_value max_execution_time 3000
Re: CGI Error
« Reply #3 on: June 21, 2012, 12:29:04 PM »
Im getting the same problem as the OP
The specified CGI application misbehaved by not returning a complete set of HTTP headers.

as he didnt post what he did to stop this, does anyone else know?
Im on a windows server and do have access to the php.ini file.

I also have this at the bottom of each page, not sure its related :
PHP Warning: PHP Startup: pdf: Unable to initialize module Module compiled with module API=20060613, debug=0, thread-safety=1 PHP compiled with module API=20060613, debug=0, thread-safety=0 These options need to match in Unknown on line 0
Re: CGI Error
« Reply #4 on: June 21, 2012, 12:50:03 PM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
http://forums.iis.net/p/1076662/1585343.aspx#1585343
http://www.iisadmin.co.uk/?p=7

The mentioned warning message is not related to generator (this is onyl related to php configuration).
Re: CGI Error
« Reply #5 on: June 21, 2012, 12:57:22 PM »
thanks, I will check the timeout for the timeout error but what about :
The specified CGI application misbehaved by not returning a complete set of HTTP headers.
is this the same cause?
Re: CGI Error
« Reply #6 on: June 21, 2012, 04:49:37 PM »
further to this does anyone have a good timeout setting for large sites? I have at least 100k urls to crawl and getting timeouts is really slowing down the process
Re: CGI Error
« Reply #7 on: June 22, 2012, 03:37:24 PM »
The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Re: CGI Error
« Reply #8 on: June 25, 2012, 10:09:58 AM »
I see the CGI headers error is getting ignored but I really need to solve it
"The specified CGI application misbehaved by not returning a complete set of HTTP headers."
is blocking every run and I have no idea whats causing it except that it appears to be your application.
Re: CGI Error
« Reply #10 on: June 25, 2012, 11:04:46 PM »
Are you 100% sure on that? as I get 1 error for time outs and this other one for something I dont know. From reading up on it, its something to do with the script inself sending data thats confusing the browser.
[ External links are visible to forum administrators only ]
is what I read today but not sure how relevant it is as I dont know the language. I would be looking to purchase another copy of this script if I can sort these issues out so its important they are resolved.
Re: CGI Error
« Reply #11 on: June 26, 2012, 11:24:41 AM »
From my experience, this message is shown because of timeouts.
Re: CGI Error
« Reply #12 on: June 26, 2012, 12:18:59 PM »
ok so from what I have read you can set the cgi timeout to a silly max value if 2 billion seconds. Or run it from a cron so it cant time out at all.
I will try running it again from the cron to see if it completes. I did attempt this yesterday but dont think it worked out, will try again.