https nginx + Apache 0 pages crawled
« on: April 07, 2017, 03:08:45 PM »
Hello,

I'm running Generator v.7.2. My server setup is nginx on the front end as a back proxy for Apache.  Nginx is serving static content only and redirecting all http requests to https (443).

In my configuration the starting URL is https://mydomain.com. However after starting crawling the process immediately stops saying 0 pages were crawled and no xml sitemap is created. I never had a problem earlier with the http site and Apache only on my dedicated server.

I have read some related topics but couldn't find out the reason. Could you please advise where to look?

Best Regards
Re: https nginx + Apache 0 pages crawled
« Reply #1 on: April 08, 2017, 06:48:11 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: https nginx + Apache 0 pages crawled
« Reply #2 on: April 08, 2017, 07:06:29 AM »
Done. If you need any other information pls let me know.

Thanks
Re: https nginx + Apache 0 pages crawled
« Reply #3 on: April 08, 2017, 03:50:09 PM »
Hello,

all pages are blocked with robots.txt
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: https nginx + Apache 0 pages crawled
« Reply #4 on: April 08, 2017, 11:03:36 PM »
Thanks,

XML sitemap is created now after fixing robots.txt.

But why links to HTML and Text sitemaps on a "View Sitemap" tab are starting from http and not from https? It is not a problem since they are redirected to https by the server, however I would think that they should start from https as the link to XML Sitemap. Any suggestions?

BR
Re: https nginx + Apache 0 pages crawled
« Reply #5 on: April 09, 2017, 05:08:01 AM »
Hello,

you need to access generator using https://domain.com/generator/ as well (not via http://).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: https nginx + Apache 0 pages crawled
« Reply #6 on: April 09, 2017, 06:07:47 AM »
I access generator by https://www.domain.com

After crawling is finished the links to sitemaps are:

HTML SiteMap
www.domain.com/generator/data/sitemap.html shown as http://www.domain.com/generator/data/sitemap.html

Text SiteMap
https://www.domain.com/generator/data/urllist.txt shown as http://www.domain.com/generator/data/urllist.txt

1. XML SiteMap File
https://www.domain.com/sitemap.xml shown as https://www.domain.com/sitemap.xml

XML sitemap link and its text is ok. Two others need to look the same in my opinion - to avoid possible confusion.
« Last Edit: April 09, 2017, 06:14:49 AM by capricorn »