Error when crawling HTTP headers: x_csize: 0
« on: May 26, 2016, 10:42:13 AM »
Hello I keep getting this error when trying to crawl my site for a sitemap. I have tried turning on Curl and also using the website IP address and still keep getting the same error.

Below is the error code I keep getting:

There was an error while retrieving the URL specified: http://www.clydesdale.net/
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:


Thanks Karen
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #1 on: May 26, 2016, 03:29:17 PM »
Hello,

Probably there is a configuration problem - it looks like your server doesn't allow local network connections via port 80 (http) - as a result sitemap generator is not able to crawl the site. This is usually related to firewall installed at the host - could you please contact your hosting support regarding this?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #2 on: August 20, 2016, 08:02:38 PM »
I am having the same issue as in the subject. Please advise.
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #3 on: August 21, 2016, 03:26:18 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #4 on: February 16, 2017, 03:45:54 PM »
Our websites on this server have had no problems until i tried it on https:// and I got
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:

do i have to adjust for https:
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #5 on: February 17, 2017, 07:28:40 AM »
Hello,

you need to make sure that PHP script is allowed to access your website. You can check this by using a test script like this:
<?php

$initurl = 'https://www.yourdomain.com';

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $initurl);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$fdata = curl_exec($ch);
curl_close($ch);
print_r($fdata);

Save it as PHP file and open in browser.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #6 on: February 17, 2017, 03:53:10 PM »
I added script and this was the response

HTTP/1.1 200 OK Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: keep-alive Keep-Alive: timeout=15 Date: Fri, 17 Feb 2017 15:50:13 GMT Server: Apache X-Powered-By: PHP/7.0.14 Link: ; rel="https://api.w.org/", ; rel=shortlink
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #7 on: February 18, 2017, 12:04:10 AM »
Please try to enable "Use curl" setting in generator configuration.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Error when crawling HTTP headers: x_csize: 0
« Reply #8 on: February 18, 2017, 11:45:41 AM »
Hi,

PHP cURL is not using an updated set of root certificates to verify server certs.

Around early September 2014, Mozilla removed the trust bits from the certs in their CA bundle that were still using RSA 1024 bit keys. This may lead to TLS libraries having a hard time to verify some sites if the library in question doesn't properly support "path discovery" as per RFC 4158. (That includes OpenSSL and GnuTLS.)

Resolution

Download the cacert.pem file from the main curl website [external links are visible to admins only] .

Add the following into php.ini or add into 'Additional directives' under Websites & Domains > PHP settings :

curl.cainfo = /var/www/vhosts/WebSite.com/httpdocs/cacert.pem