ab

*
  • *
  • 19
online tool indexes more than stand-alone
« on: June 07, 2013, 09:31:23 AM »
Hello,

in our shop [ External links are visible to forum administrators only ] the online tool indexes all articles like it should be, i.e. 61 pages with Xlayer products. But the stand-alone generator only indexes a fraction of those (10 pages with Xlayer products). How many pages the stand-alone indexes varies little and I can not see why one product page is indexed and the other is not.

I already tried to add a pause of 1 second between each request (under "fine tuning") to make sure that the stand-alone generator is not faster than the apache-server, but this made no difference (except the crawling time of course).

What could be the reason?

Best regards,
Axel

ab

*
  • *
  • 19
Re: online tool indexes more than stand-alone
« Reply #1 on: June 07, 2013, 03:17:56 PM »
Meanwhile i have found that the pages are skipped because the server seems to respond  "301 moved permanently" although the page is available. I guess it is some timing problem with sitemap generator and the seo url rewrite of xt-commerce.

I just tried out: indeed if I disable the search engine friendly url of the webshop all pages are indexed. But I think the se-friendly url are useful because they contain the product name in the url, so I want to keep them.

Any hint for a solution? There should be one, because the online tool works also with se-friendly urls.

Best regards,
Axel Booltink
Re: online tool indexes more than stand-alone
« Reply #2 on: June 09, 2013, 08:02:45 AM »
Hello,

 in case if server gets overloaded when running generator, you can use "Make delay for X seconds after each Y requests" setting in generator configuration - that will slow down the crawling, reducing the load.

ab

*
  • *
  • 19
Re: online tool indexes more than stand-alone
« Reply #3 on: June 09, 2013, 08:33:45 PM »
Dear Oleg,

I already tried this without any success (see my first post).

Best regards,
Axel Booltink

ab

*
  • *
  • 19
Re: online tool indexes more than stand-alone
« Reply #5 on: June 12, 2013, 08:00:19 AM »
Dear Oleg,

I already tried before with 5s after every 10 pages, but right now I am testing with 5s after every page and the result is the same. I am looking at debug.log (with tail -f) so I can easily see as soon as it goes wrong. Do you want me to send you the debug log?

I do not think that further increase will help so the reason seems to be something else.

Best regards,
Axel Booltink

ab

*
  • *
  • 19
Re: online tool indexes more than stand-alone
« Reply #7 on: June 13, 2013, 11:08:07 AM »
Hi,

No, it the server is not configured to block after a certain number of requests. By the way: if that would be the case it would also happen if the URL-rewrite is disabled. But if I disable SE-friendly URLs, so that the URL is not rewritten all pages are indexed.

And also: the online-tool indexes allway all pages.

Best regards,
Axel

ab

*
  • *
  • 19
Re: online tool indexes more than stand-alone
« Reply #8 on: June 25, 2013, 10:53:42 AM »
Hi,

too make sure it is not the fault of my outdated Debian, I just installed a brand-new server (I needed to migrate the shop anyway).
So, now I have a stand-alone server with only a fresh Debian installation and our newly installed webshop under IP 195.225.198.69.
If I do not enabled Search Engine Optimalization in the webshop everything is ok.
If I use SEO (which works with the rewrite engine of apache2) the online tool still indexes all pages but the stand-alone generator skips a lot of them.

What can I do?

Best regards,
Axel
Re: online tool indexes more than stand-alone
« Reply #9 on: June 25, 2013, 09:04:13 PM »
Hello,

please let me know your generator URL/login in private message to check this.