online tool indexes more than stand-alone

[ab]

Hello,

in our shop [ External links are visible to forum administrators only ] the online tool indexes all articles like it should be, i.e. 61 pages with Xlayer products. But the stand-alone generator only indexes a fraction of those (10 pages with Xlayer products). How many pages the stand-alone indexes varies little and I can not see why one product page is indexed and the other is not.

I already tried to add a pause of 1 second between each request (under "fine tuning") to make sure that the stand-alone generator is not faster than the apache-server, but this made no difference (except the crawling time of course).

What could be the reason?

Best regards,
Axel

[ab]

Meanwhile i have found that the pages are skipped because the server seems to respond  "301 moved permanently" although the page is available. I guess it is some timing problem with sitemap generator and the seo url rewrite of xt-commerce.

I just tried out: indeed if I disable the search engine friendly url of the webshop all pages are indexed. But I think the se-friendly url are useful because they contain the product name in the url, so I want to keep them.

Any hint for a solution? There should be one, because the online tool works also with se-friendly urls.

Best regards,
Axel Booltink

[XML-Sitemaps Support]

Hello,

in case if server gets overloaded when running generator, you can use "Make delay for X seconds after each Y requests" setting in generator configuration - that will slow down the crawling, reducing the load.

[ab]

Dear Oleg,

I already tried this without any success (see my first post).

Best regards,
Axel Booltink

[XML-Sitemaps Support]

You could try to increase the delay in this case.

[ab]

Dear Oleg,

I already tried before with 5s after every 10 pages, but right now I am testing with 5s after every page and the result is the same. I am looking at debug.log (with tail -f) so I can easily see as soon as it goes wrong. Do you want me to send you the debug log?

I do not think that further increase will help so the reason seems to be something else.

Best regards,
Axel Booltink

[XML-Sitemaps Support]

Hello,

perhaps the site blocks access to it after certain number of requests.

[ab]

Hi,

No, it the server is not configured to block after a certain number of requests. By the way: if that would be the case it would also happen if the URL-rewrite is disabled. But if I disable SE-friendly URLs, so that the URL is not rewritten all pages are indexed.

And also: the online-tool indexes allway all pages.

Best regards,
Axel

[ab]

Hi,

too make sure it is not the fault of my outdated Debian, I just installed a brand-new server (I needed to migrate the shop anyway).
So, now I have a stand-alone server with only a fresh Debian installation and our newly installed webshop under IP 195.225.198.69.
If I do not enabled Search Engine Optimalization in the webshop everything is ok.
If I use SEO (which works with the rewrite engine of apache2) the online tool still indexes all pages but the stand-alone generator skips a lot of them.

What can I do?

Best regards,
Axel

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.