informer9
Registered Customer
Newbie
Posts: 7
|
 |
« on: May 09, 2007, 12:26:52 PM » |
|
Hi
Im using my generator for some time now and everything was working fine until may.
It has removed over 200 urls and dont want to crawl it back...
it looks like that
Pages scanned: 220 (5,181.4 Kb)
Pages left: 30 (+ 186 queued for the next depth level)
then it drops all the next level queued pages and finishes with 246 pages crawled for the sitemap!!
its not crawling new entries into the directory
it has removed the links to all the entries from the directory
it is still crawling all the other pages (categories, subcategories, search results)
[external links are visible to admins only]
Ive got few other pages on exactly the same script and generator is working fine there.
it is crawling all my links fine on [external links are visible to admins only]
Ive also tried other free generator from [external links are visible to admins only]
and it is working fine, crawling all the pages
ive updated the script to newest version
ive tried to reinstall the script
ive tried to use your free generator
still no joy
Please help
mike
|
|
|
|
|
Logged
|
|
|
|
t_a
Registered Customer
Newbie
Posts: 9
|
 |
« Reply #1 on: May 09, 2007, 11:26:15 PM » |
|
I am having the same issue :-(
|
|
|
|
|
Logged
|
|
|
|
|
admin
|
|
« Reply #2 on: May 09, 2007, 11:57:52 PM » |
|
Hello,
did you define any limitations? (depth level, "exclude URLs" or others)
|
|
|
|
|
Logged
|
|
|
|
t_a
Registered Customer
Newbie
Posts: 9
|
|
« Reply #3 on: May 10, 2007, 12:28:25 AM » |
|
I am using these excludes:
rss.php?c=
submit.php?c=
?s=
authors?Page=
?Page=
?ArticleId=
addfav
addread
print
As for dept level.. have all to use "0" unlimited there.
Here is an example of a removed page: (thousands of articles has been removed during the last week or so)
[external links are visible to admins only]
Any suggestions.
|
|
|
|
|
Logged
|
|
|
|
t_a
Registered Customer
Newbie
Posts: 9
|
|
« Reply #4 on: May 10, 2007, 07:11:25 PM » |
|
Here is a page with some 404's that should not be: [external links are visible to admins only]
The strange thing here is that some articles are added to the sitemap.
Example of added link:
[external links are visible to admins only]
Example of link returning 404:
[external links are visible to admins only]
I just cant seem figure this out.
I welcome any help I can get here.
|
|
|
|
« Last Edit: May 10, 2007, 07:18:45 PM by t_a »
|
Logged
|
|
|
|
|
admin
|
|
« Reply #5 on: May 11, 2007, 10:33:41 PM » |
|
Hello,
perhaps your site returned an error for some URLs because of crawling intensity. Try to define a delay between requests in sitemap generator configuration.
|
|
|
|
|
Logged
|
|
|
|
t_a
Registered Customer
Newbie
Posts: 9
|
|
« Reply #6 on: May 12, 2007, 08:19:56 AM » |
|
I tried a 3 second delay for each 10 requests, but that did not help. Could it be something else?
Do you want login information?
|
|
|
|
|
Logged
|
|
|
|
|
admin
|
|
« Reply #7 on: May 13, 2007, 08:06:04 AM » |
|
Please try 1 second delay after every request.
|
|
|
|
|
Logged
|
|
|
|
informer9
Registered Customer
Newbie
Posts: 7
|
|
« Reply #8 on: May 14, 2007, 11:05:34 PM » |
|
Hi
sorry for the delay
I dont have any limitations at all. I use 0 to have all web page crawled.
I have tried suggested 1 sec break between every request...
Still no joy !!! it has even removed my last entry in to the directory ( I lost 5 urls)
all the other pages are being crawled fine
Admin - please help... any other ideas?
regards
mike
|
|
|
|
|
Logged
|
|
|
|
|
admin
|
|
« Reply #9 on: May 15, 2007, 01:09:13 AM » |
|
Hello,
please send me a private message with your generator URL and example URL that is not included in sitemap.
|
|
|
|
|
Logged
|
|
|
|
t_a
Registered Customer
Newbie
Posts: 9
|
|
« Reply #10 on: May 23, 2007, 04:05:25 PM » |
|
I sent you a PM. I am missing appox 5000 pages in my sitemap.
Thanks.
|
|
|
|
|
Logged
|
|
|
|
|
admin
|
|
« Reply #11 on: May 25, 2007, 10:36:55 PM » |
|
Update: the problem has been resolved.
|
|
|
|
|
Logged
|
|
|
|
|
