XML Sitemaps Generator

Author Topic: How to configure the generator to crawl secure https pages?  (Read 11124 times)

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
How to configure the generator to crawl secure https pages?
« on: October 23, 2009, 12:40:38 PM »
I have just downloaded the latest version of  Unlimited PHP Sitemap Generator but cannot work out how to tell it to crawl the secure pages on my e-commerce website.
I have looked all over the documentation and the forums but seem to have missed the instructions.
Should I simply add 's' to the 'http' prefix in my site's URL?
Or is it something else?

« Last Edit: October 23, 2009, 12:45:53 PM by admin317 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: How to configure the generator to crawl secure https pages?
« Reply #1 on: October 23, 2009, 12:55:27 PM »
Hello,

yes, you should simple specify Starting URL as https://www.domain.com
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #2 on: October 23, 2009, 11:13:59 PM »
Thank you, Admin, for your quick reply.
I have tried that method, however the Sitemap Generator only crawls one single page and then gives me the following message -
Request date:
23 October 2009, 16:02
Processing time:
0:00:01s
Pages indexed:
1
Sitemap files:
1
Pages size:
0.00Mb

I must have configured something wrongly, but I cannot work out what it is.
BTW, OpenSSL *is* enabled on my server
« Last Edit: October 23, 2009, 11:18:20 PM by admin317 »

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #3 on: October 24, 2009, 09:59:47 AM »
Just some additional information that might help solve the problem:

1) ERROR MESSAGE
An error occured: There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP headers follow:
Sat, 24 Oct 2009 09:47:16 GMT
Apache/2.2.3 (CentOS)
[external links are visible to admins only]

2) ERROR LOG
My server's error log generated the following error lists the last couple of times I tried to get Sitemap Generator to crawl my site:

[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [external links are visible to admins only]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [external links are visible to admins only]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [external links are visible to admins only]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:45 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/robots.txt, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [external links are visible to admins only]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [external links are visible to admins only]
« Last Edit: October 24, 2009, 10:07:39 AM by admin317 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: How to configure the generator to crawl secure https pages?
« Reply #4 on: October 25, 2009, 07:34:55 AM »
Hello,

https://www.my-bookcafe.com/ shows hosting control panel login page for me, no site content there.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #5 on: October 26, 2009, 05:07:46 AM »
Thanks for your reply, Oleg!
My apologies, I overlooked that fact that I'm using a *shared* SSL certificate.
The URL should be  [external links are visible to admins only]
But this URL still gives me the same problem - only one page gets crawled:

Request date:
25 October 2009, 22:02
Processing time:
0:00:02s
Pages indexed:
1
Sitemap files:
1
Pages size:
0.02Mb

« Last Edit: October 26, 2009, 05:11:02 AM by admin317 »

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #6 on: October 26, 2009, 05:22:01 AM »
Server Error Log Messages when I try to run the generator:

[Sun Oct 25 22:02:08 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:11 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:12 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [external links are visible to admins only]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [external links are visible to admins only]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [external links are visible to admins only]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [external links are visible to admins only]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [external links are visible to admins only]
 ???

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: How to configure the generator to crawl secure https pages?
« Reply #7 on: October 27, 2009, 04:12:06 PM »
Hello,

it looks like all shop links from inner pages are pointing to your main domain anyway, so they are autoamtically excluded from sitemap (only pages from the same domain as Starting URL can be included in sitemap).

Entries from the error log do not seem to be related to sitemap generator.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

admin317

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #8 on: October 28, 2009, 01:23:58 AM »
Okay, thanks Oleg. I shall try to figure out how to fix this...

Best wishes.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2