XML Sitemaps Generator

Author Topic: Number of pages indexed are less than crawled  (Read 18798 times)

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #15 on: March 29, 2012, 07:17:44 AM »
I founded the mail with the reference for download but unfortunally it doesn't work.
Please, check it, here is part of letter:

Hello,
Thank you for purchasing the Standalone XML Sitemap Generator.
You can download it here:
...
PLEASE KEEP THIS EMAIL - YOU WILL USE THE SAME LINK TO DOWNLOAD UPDATES!


« Last Edit: March 29, 2012, 07:28:08 AM by XML-Sitemaps Support »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #16 on: March 29, 2012, 07:28:36 AM »
Hello,

the link works for me, what exactly is the error you get?
I've removed the link from your post since it should not be shared with anyone.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #17 on: March 29, 2012, 01:42:30 PM »
Seems it was problem of IE, I downloaded by the FireFox. But I can't copy the new version overright the last one - can it be the prohibition of generator?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #18 on: March 29, 2012, 11:00:46 PM »
Try to remove generator folder and then reupload it.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #19 on: April 04, 2012, 01:55:43 PM »
After some problems (our site works under Bitrix soft) I reinstall the generator. Could You please, advise - where is the list of URLs that were crawled but not added in sitemap. Excuse me - I can't find it... Thanks in advance!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #20 on: April 05, 2012, 09:58:10 AM »
You should be able to see it on "Chanlog" page (click on specific log entry and it will show "added/removed/etc" pages.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #21 on: April 05, 2012, 12:07:49 PM »
Thanks! I will do the new XML-files after next DB updating on the weekend and check the skipped pages.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #22 on: April 18, 2012, 07:40:00 AM »
Dear sir! Excuse me for long time without information. It takes more than I think... I installed the new GENERATOR and received the long listing of skipped pages. It' rather big listing, please, start from these:

1. skipped page is "catalog/detail.php?ID=218560". To go for it address we can do following: [external links are visible to admins only] (main page) -> [external links are visible to admins only] (Rus: Все рубрики) -> and go to thise heading (Rus: АВТОЗАПЧАСТИ: РАЗБОРКИ АВТОМОБИЛЕЙ) as [external links are visible to admins only]

2. next one: catalog/detail.php?ID=217856. Path is [external links are visible to admins only] (main page) -> [external links are visible to admins only] (Rus: Все рубрики) -> and go to thise heading (Rus: АВТОСЕРВИС: ГСТЕХОСМОТР,...) as [external links are visible to admins only]

3. catalog/detail.php?ID=218079. [external links are visible to admins only] (main page) -> [external links are visible to admins only] (Rus: Все рубрики) -> and go to thise heading (Rus: КОВРЫ И НАПОЛЬНЫЕ ПОКРЫТИЯ: ЛИНОЛЕУМ) as [external links are visible to admins only]

Thanks in advance!


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #23 on: April 18, 2012, 09:42:18 PM »
I've updated your crawler settings, please check now.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #24 on: April 19, 2012, 06:47:46 AM »
Thanks for the suggestions. As far as I understood You change:
1. "Exclude URLs:" - "glossary.php?ID="
2. "Do not parse URLs:" - "print=
                                        catalog/\d*/\d+/"
I try to testing next weekend because new DB updating.
But I'm affraid a bit that these settings will decrease number of indexed pages with our customers, isn' it? Or these are don't do any decreasing?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #25 on: April 19, 2012, 04:18:52 PM »
It won't decrease the number of pages.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #26 on: April 22, 2012, 08:57:58 AM »
Hello! Thanks for the new updated Generator version - it takes much less time, great!

At the same time - please, check following URLs which were skipped but it's our very important headings (parts of catalog) - there are some of them:

1. reference "catalog/detail.php?ID=218083" - to rich this: go to main page [external links are visible to admins only] -> after (rus text) "Все рубрики" page ([external links are visible to admins only]) -> next to heading named (in rus) "КОМПРЕССОРЫ, КОМПРЕССОРНАЯ ТЕХНИКА" -> just this page "[external links are visible to admins only]"

2. catalog/detail.php?ID=218218: [external links are visible to admins only] -> page [external links are visible to admins only], (rus:) "ОДЕЖДА: АТЕЛЬЕ" - just this page "[external links are visible to admins only]"

3. catalog/detail.php?ID=218312: [external links are visible to admins only] -> page [external links are visible to admins only], (rus:) "ПРОТИВОПОЖАРНАЯ БЕЗОПАСНОСТЬ: ОГНЕТУШИТЕЛИ И СРЕДСТВА ТУШЕНИЯ ОГНЯ" ->[external links are visible to admins only]

Wait your information! Thanks in advance!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #27 on: April 22, 2012, 05:56:19 PM »
Hello,

I see these pages are in "skippped" list (you can see it in changelog). It is possible that website gets overloaded when generator crawls your site (a lot of requests coming in constantly), and as a rsult some pages were inaccessible.
You can try to use "Make delay for X seconds after each Y request" setting to slow down the crawler - that will make the process longer, but will reduce server load.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

igors.kuvaga

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 47
Re: Number of pages indexed are less than crawled
« Reply #28 on: April 23, 2012, 08:08:23 AM »
OK, next weekend I try to use your advise. Could You please, recommend the X, Y numbers?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Number of pages indexed are less than crawled
« Reply #29 on: April 23, 2012, 09:07:23 AM »
It's more of "trial and error" process, you can try 1 second delay after each 5th request, for instance (to avoid making the process too long)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2