XML Sitemaps Generator

Author Topic: online tool indexes more than stand-alone  (Read 10514 times)

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
online tool indexes more than stand-alone
« on: June 07, 2013, 08:31:23 AM »
Hello,

in our shop [external links are visible to admins only] the online tool indexes all articles like it should be, i.e. 61 pages with Xlayer products. But the stand-alone generator only indexes a fraction of those (10 pages with Xlayer products). How many pages the stand-alone indexes varies little and I can not see why one product page is indexed and the other is not.

I already tried to add a pause of 1 second between each request (under "fine tuning") to make sure that the stand-alone generator is not faster than the apache-server, but this made no difference (except the crawling time of course).

What could be the reason?

Best regards,
Axel

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: online tool indexes more than stand-alone
« Reply #1 on: June 07, 2013, 02:17:56 PM »
Meanwhile i have found that the pages are skipped because the server seems to respond  "301 moved permanently" although the page is available. I guess it is some timing problem with sitemap generator and the seo url rewrite of xt-commerce.

I just tried out: indeed if I disable the search engine friendly url of the webshop all pages are indexed. But I think the se-friendly url are useful because they contain the product name in the url, so I want to keep them.

Any hint for a solution? There should be one, because the online tool works also with se-friendly urls.

Best regards,
Axel Booltink

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: online tool indexes more than stand-alone
« Reply #2 on: June 09, 2013, 07:02:45 AM »
Hello,

 in case if server gets overloaded when running generator, you can use "Make delay for X seconds after each Y requests" setting in generator configuration - that will slow down the crawling, reducing the load.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: online tool indexes more than stand-alone
« Reply #3 on: June 09, 2013, 07:33:45 PM »
Dear Oleg,

I already tried this without any success (see my first post).

Best regards,
Axel Booltink

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: online tool indexes more than stand-alone
« Reply #4 on: June 11, 2013, 09:20:55 PM »
You could try to increase the delay in this case.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: online tool indexes more than stand-alone
« Reply #5 on: June 12, 2013, 07:00:19 AM »
Dear Oleg,

I already tried before with 5s after every 10 pages, but right now I am testing with 5s after every page and the result is the same. I am looking at debug.log (with tail -f) so I can easily see as soon as it goes wrong. Do you want me to send you the debug log?

I do not think that further increase will help so the reason seems to be something else.

Best regards,
Axel Booltink

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: online tool indexes more than stand-alone
« Reply #6 on: June 13, 2013, 09:01:39 AM »
Hello,

perhaps the site blocks access to it after certain number of requests.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: online tool indexes more than stand-alone
« Reply #7 on: June 13, 2013, 10:08:07 AM »
Hi,

No, it the server is not configured to block after a certain number of requests. By the way: if that would be the case it would also happen if the URL-rewrite is disabled. But if I disable SE-friendly URLs, so that the URL is not rewritten all pages are indexed.

And also: the online-tool indexes allway all pages.

Best regards,
Axel

ab

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: online tool indexes more than stand-alone
« Reply #8 on: June 25, 2013, 09:53:42 AM »
Hi,

too make sure it is not the fault of my outdated Debian, I just installed a brand-new server (I needed to migrate the shop anyway).
So, now I have a stand-alone server with only a fresh Debian installation and our newly installed webshop under IP 195.225.198.69.
If I do not enabled Search Engine Optimalization in the webshop everything is ok.
If I use SEO (which works with the rewrite engine of apache2) the online tool still indexes all pages but the stand-alone generator skips a lot of them.

What can I do?

Best regards,
Axel

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: online tool indexes more than stand-alone
« Reply #9 on: June 25, 2013, 08:04:13 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2