Crawl error
« on: October 13, 2017, 06:33:34 PM »
I cannot get a new site to crawl and even one of my sites that was working previously this year has stopped working for some reason. Checked with host and they did not have anything that would be stopping the generator from working. This is installed on my own cloud server. Can you help me get this fixed? Error is listed below:

There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:
Re: Crawl error
« Reply #1 on: October 14, 2017, 04:56:56 AM »

from what I see your website uses https protocol, make sure that Starting URL also has https:// instead of http://.
Re: Crawl error
« Reply #2 on: October 14, 2017, 05:03:31 PM »
I did that, many times. My hosting company went into the program and tried to get it to work but could not. They said I might have to turn on a 200 OK configuration to get it to work but no guarantees. I use the Joomla CMS and they had some recent updates. Would that be the issue?

Feel free to take a look at the generator if need be.
Re: Crawl error
« Reply #3 on: October 14, 2017, 05:47:35 PM »

your homepage currently returns 404 http code, it must return http code 200:

HTTP Headers found in response

Response Code   HTTP/1.1 404 Not Found
Date   Sat, 14 Oct 2017 16:46:51 GMT
Server   Apache
Cache-Control   no-cache
Pragma   no-cache
Set-Cookie   d36078e387d6969513256122ddf1bc0c=f071bf695e2325b441094f3fd46f93e3; path=/; secure; HttpOnly
Content-Length   22012
Connection   close
Content-Type   text/html; charset=UTF-8
Re: Crawl error
« Reply #4 on: October 14, 2017, 09:45:17 PM »
Figured out the problem so I'll post it here just in case anyone has the same issue.

With CMS sites like Joomla you have to make sure you have the main menu link to an article that has content on the frontpage or you'll receive that 404 error which makes other components and plugins return errors as well. This of course will in turn make the xml-sitemap generator return an error and not work. Hope this helps.
Re: Crawl error
« Reply #5 on: October 15, 2017, 06:30:11 AM »
I'm glad it works now and thank you for update!