Re: Number of pages indexed are less than crawled
« Reply #15 on: March 29, 2012, 08:17:44 AM »
I founded the mail with the reference for download but unfortunally it doesn't work.
Please, check it, here is part of letter:

Hello,
Thank you for purchasing the Standalone XML Sitemap Generator.
You can download it here:
...
PLEASE KEEP THIS EMAIL - YOU WILL USE THE SAME LINK TO DOWNLOAD UPDATES!


« Last Edit: March 29, 2012, 08:28:08 AM by XML-Sitemaps Support »
Re: Number of pages indexed are less than crawled
« Reply #16 on: March 29, 2012, 08:28:36 AM »
Hello,

the link works for me, what exactly is the error you get?
I've removed the link from your post since it should not be shared with anyone.
Re: Number of pages indexed are less than crawled
« Reply #17 on: March 29, 2012, 02:42:30 PM »
Seems it was problem of IE, I downloaded by the FireFox. But I can't copy the new version overright the last one - can it be the prohibition of generator?
Re: Number of pages indexed are less than crawled
« Reply #19 on: April 04, 2012, 02:55:43 PM »
After some problems (our site works under Bitrix soft) I reinstall the generator. Could You please, advise - where is the list of URLs that were crawled but not added in sitemap. Excuse me - I can't find it... Thanks in advance!
Re: Number of pages indexed are less than crawled
« Reply #20 on: April 05, 2012, 10:58:10 AM »
You should be able to see it on "Chanlog" page (click on specific log entry and it will show "added/removed/etc" pages.
Re: Number of pages indexed are less than crawled
« Reply #21 on: April 05, 2012, 01:07:49 PM »
Thanks! I will do the new XML-files after next DB updating on the weekend and check the skipped pages.
Re: Number of pages indexed are less than crawled
« Reply #22 on: April 18, 2012, 08:40:00 AM »
Dear sir! Excuse me for long time without information. It takes more than I think... I installed the new GENERATOR and received the long listing of skipped pages. It' rather big listing, please, start from these:

1. skipped page is "catalog/detail.php?ID=218560". To go for it address we can do following: [ External links are visible to forum administrators only ] (main page) -> [ External links are visible to forum administrators only ] (Rus: Все рубрики) -> and go to thise heading (Rus: АВТОЗАПЧАСТИ: РАЗБОРКИ АВТОМОБИЛЕЙ) as [ External links are visible to forum administrators only ]

2. next one: catalog/detail.php?ID=217856. Path is [ External links are visible to forum administrators only ] (main page) -> [ External links are visible to forum administrators only ] (Rus: Все рубрики) -> and go to thise heading (Rus: АВТОСЕРВИС: ГСТЕХОСМОТР,...) as [ External links are visible to forum administrators only ]

3. catalog/detail.php?ID=218079. [ External links are visible to forum administrators only ] (main page) -> [ External links are visible to forum administrators only ] (Rus: Все рубрики) -> and go to thise heading (Rus: КОВРЫ И НАПОЛЬНЫЕ ПОКРЫТИЯ: ЛИНОЛЕУМ) as [ External links are visible to forum administrators only ]

Thanks in advance!

Re: Number of pages indexed are less than crawled
« Reply #24 on: April 19, 2012, 07:47:46 AM »
Thanks for the suggestions. As far as I understood You change:
1. "Exclude URLs:" - "glossary.php?ID="
2. "Do not parse URLs:" - "print=
                                        catalog/\d*/\d+/"
I try to testing next weekend because new DB updating.
But I'm affraid a bit that these settings will decrease number of indexed pages with our customers, isn' it? Or these are don't do any decreasing?
Re: Number of pages indexed are less than crawled
« Reply #26 on: April 22, 2012, 09:57:58 AM »
Hello! Thanks for the new updated Generator version - it takes much less time, great!

At the same time - please, check following URLs which were skipped but it's our very important headings (parts of catalog) - there are some of them:

1. reference "catalog/detail.php?ID=218083" - to rich this: go to main page [ External links are visible to forum administrators only ] -> after (rus text) "Все рубрики" page ([ External links are visible to forum administrators only ]) -> next to heading named (in rus) "КОМПРЕССОРЫ, КОМПРЕССОРНАЯ ТЕХНИКА" -> just this page "[ External links are visible to forum administrators only ]"

2. catalog/detail.php?ID=218218: [ External links are visible to forum administrators only ] -> page [ External links are visible to forum administrators only ], (rus:) "ОДЕЖДА: АТЕЛЬЕ" - just this page "[ External links are visible to forum administrators only ]"

3. catalog/detail.php?ID=218312: [ External links are visible to forum administrators only ] -> page [ External links are visible to forum administrators only ], (rus:) "ПРОТИВОПОЖАРНАЯ БЕЗОПАСНОСТЬ: ОГНЕТУШИТЕЛИ И СРЕДСТВА ТУШЕНИЯ ОГНЯ" ->[ External links are visible to forum administrators only ]

Wait your information! Thanks in advance!
Re: Number of pages indexed are less than crawled
« Reply #27 on: April 22, 2012, 06:56:19 PM »
Hello,

I see these pages are in "skippped" list (you can see it in changelog). It is possible that website gets overloaded when generator crawls your site (a lot of requests coming in constantly), and as a rsult some pages were inaccessible.
You can try to use "Make delay for X seconds after each Y request" setting to slow down the crawler - that will make the process longer, but will reduce server load.
Re: Number of pages indexed are less than crawled
« Reply #28 on: April 23, 2012, 09:08:23 AM »
OK, next weekend I try to use your advise. Could You please, recommend the X, Y numbers?
Re: Number of pages indexed are less than crawled
« Reply #29 on: April 23, 2012, 10:07:23 AM »
It's more of "trial and error" process, you can try 1 second delay after each 5th request, for instance (to avoid making the process too long)