XML Sitemaps Generator

Author Topic: Crawling only 6 pages  (Read 14034 times)

bernard

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Crawling only 6 pages
« on: September 24, 2008, 02:45:35 PM »
Hi

I've just installed the software and when run it will only index 6 pages before writing the necessary files.  The site ( [external links are visible to admins only] ) is crawlable which I confirmed by using your free online version where it indexed the 500 page limit.
Could you please advise me further.

Regards

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling only 6 pages
« Reply #1 on: September 24, 2008, 08:35:23 PM »
Please PM me your generator URL/login to check that.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bernard

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Crawling only 6 pages
« Reply #2 on: September 26, 2008, 02:31:31 PM »
Hi
I've overcome that problem by adding www into the site url.  However, I have a further problem as the server does not allow an HTTP request to be sent to the same server to avoid scripts potentially running in infinite loops.  This means that I can't index pages within the same site as the software is installed on.  I can index pages on another server but then I can't save the sitemap on the scanned site.  This situation of course does not allow me to run a cron job to automate updates.

Do you have any thoughts on how I can overcome this problem.

Regards

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling only 6 pages
« Reply #3 on: September 26, 2008, 08:10:35 PM »
Hello,

depending on server configuration, possibly there is a proxy service running on the server that allows to open local pages for scripts?
Also, you can setup sitemap generator on external site and then create another cron task on main site just to download resulting sitemap files.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2