Decrease number of indexed pages and some new mistakes
« on: April 17, 2013, 07:10:47 AM »
We moved the site for the new hosting parameters (the same company but new virtual site with more capacity). But number of pages indexed in XML-file extremely decreased (from 67 thsn to 4 thsnd, DB is nearlly the same, pls see - [ External links are visible to forum administrators only ], we use port 81 now instead ordinary address [ External links are visible to forum administrators only ]). At the same time some new mistakes (in broken links) which is really opened. Could You pls, advise - what we should do to correct work?
Re: Decrease number of indexed pages and some new mistakes
« Reply #1 on: April 17, 2013, 03:41:01 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Re: Decrease number of indexed pages and some new mistakes
« Reply #2 on: April 18, 2013, 06:35:15 AM »
Thanks for the answer. There are following:
1. Generator URL I used is [ External links are visible to forum administrators only ]
2. Sample of the URL not included in XML-file - [ External links are visible to forum administrators only ]
3. To reach this from homepage: [ External links are visible to forum administrators only ] -> [ External links are visible to forum administrators only ] (rus: "Все рубрики") -> list of headings.
4. Broken links like first "catalog/detail.php?ID=218221" looks strange (like some problems with Rus text coding - see [ External links are visible to forum administrators only ]) and really work
5. Number of indexed pages is around 4 thsnd but was 67 thsnd
Hope your help, it's very important for us for indexing our Inet-directory!!!
Re: Decrease number of indexed pages and some new mistakes
« Reply #3 on: April 20, 2013, 04:52:35 PM »
This link is included in sitemap from what I can see:
  <url>
       <loc>[ External links are visible to logged in users only ]</loc>
       <lastmod>2013-04-16T13:49:14+00:00</lastmod>
       <changefreq>daily</changefreq>
       <priority>0.6400</priority>
  </url>
Re: Decrease number of indexed pages and some new mistakes
« Reply #4 on: April 20, 2013, 07:22:44 PM »
I'm terribly sorry for this mistake.
Smth happend with search on my page.

Pls, check the following samples of the URL:
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

To reach this from homepage: [ External links are visible to forum administrators only ] -> [ External links are visible to forum administrators only ] (rus: "Все рубрики") -> list of headings and this heading which I send before: [ External links are visible to forum administrators only ] (rus: ЛАБОРАТОРНОЕ ОБОРУДОВАНИЕ...)

After go to company by company from the listing for the minisite of each
ones

Another problems are:
- Broken links like first "catalog/detail.php?ID=218221" looks strange (like some problems with Rus text coding - see [ External links are visible to forum administrators only ]) and really work
- Number of indexed pages is around 4 thsnd but was 67 thsnd

Sorry again, hope for your help...
Re: Decrease number of indexed pages and some new mistakes
« Reply #5 on: April 23, 2013, 11:13:43 AM »
I'm sorry but need to bother for reminding about our problems.I can't make the new XML-file (we have every week mostly new DB) and it looks not o good for indexing by search sytem. Do You have smth to advise us?
Re: Decrease number of indexed pages and some new mistakes
« Reply #7 on: April 26, 2013, 07:13:48 AM »
OK, I try to create again the XML-file this weekend. But before I did with the same result. The biggest problem for us - why number of pages decreased so many!!!
Re: Decrease number of indexed pages and some new mistakes
« Reply #8 on: April 29, 2013, 01:10:11 PM »
I'm terribly sorry but I need to bother You again because no result of XML generator. We started the crawl yesterday and it finished with only 5'000 pages indexed. And I can't to continue - see the attachmented file!!! Before we restart it a few times and had around 60000. Pls, check what's happend?
Re: Decrease number of indexed pages and some new mistakes
« Reply #9 on: April 30, 2013, 12:00:10 PM »
As mentioned above, the pages are not indexed since your server stops responding after a while when generator is crawling the site.
Re: Decrease number of indexed pages and some new mistakes
« Reply #10 on: April 30, 2013, 01:47:37 PM »
ОК, but why I can't continue the process? Before we had the same problems with the site. BUT I had possibility to continue... Now (see screen in previous post) I don't have it - only start new one, before was possibility to continue from the stopped place! That's my biggest question!
Re: Decrease number of indexed pages and some new mistakes
« Reply #11 on: May 06, 2013, 08:37:33 AM »
Possibly  crawl_dump.log file in generator/data/ folder is corrupted and generator  cannot read progress details.
Re: Decrease number of indexed pages and some new mistakes
« Reply #12 on: May 06, 2013, 09:13:36 AM »
Could You pls, advise what we can do in that way?
Re: Decrease number of indexed pages and some new mistakes
« Reply #13 on: May 07, 2013, 01:35:46 PM »
And what can we do?
Re: Decrease number of indexed pages and some new mistakes
« Reply #14 on: May 08, 2013, 01:29:16 PM »
I'm sorry but it's third time I ask the same question. What can we do in the situation You described? It's terrible for us - we need to update our XML-files, we bought your service specially fo it. May be we did smth wrong or no correct - pls, advise us!!!!