XML Sitemaps Generator

Author Topic: Generator Removing URLs - Not Crawling New Pages  (Read 28680 times)

informer9

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 10
Generator Removing URLs - Not Crawling New Pages
« on: May 09, 2007, 11:26:52 AM »
Hi
Im using my generator for some time now and everything was working fine until may.

It has removed over 200 urls and dont want to crawl it back...

it looks like that

Pages scanned: 220 (5,181.4 Kb)
Pages left: 30 (+ 186 queued for the next depth level)

then it drops all the next level queued pages and finishes with 246 pages crawled for the sitemap!!

its not crawling new entries into the directory

it has removed the links to all the entries from the directory
it is still crawling all the other pages (categories, subcategories, search results)

[external links are visible to admins only]

Ive got few other pages on exactly the same script and generator is working fine there.
it is crawling all my links fine on [external links are visible to admins only]

Ive also tried  other free generator from  [external links are visible to admins only]
and it is working fine, crawling all the pages

ive updated the script to newest version
ive tried to reinstall the script
ive tried to use your free generator

still no joy

Please help

mike

t_a

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #1 on: May 09, 2007, 10:26:15 PM »
I am having the same issue :-(

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #2 on: May 09, 2007, 10:57:52 PM »
Hello,

did you define any limitations? (depth level, "exclude URLs" or others)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

t_a

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #3 on: May 09, 2007, 11:28:25 PM »
I am using these excludes:
rss.php?c=
submit.php?c=
?s=
authors?Page=
?Page=
?ArticleId=
addfav
addread
print

As for dept level.. have all to use "0" unlimited there.

Here is an example of a removed page: (thousands of articles has been removed during the last week or so)
[external links are visible to admins only]

Any suggestions.

t_a

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #4 on: May 10, 2007, 06:11:25 PM »
Here is a page with some 404's that should not be: [external links are visible to admins only]

The strange thing here is that some articles are added to the sitemap.

Example of added link:
[external links are visible to admins only]

Example of link returning 404:
[external links are visible to admins only]

I just cant seem figure this out.

I welcome any help I can get here.
« Last Edit: May 10, 2007, 06:18:45 PM by t_a »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #5 on: May 11, 2007, 09:33:41 PM »
Hello,

perhaps your site returned an error for some URLs because of crawling intensity. Try to define a delay between requests in sitemap generator configuration.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

t_a

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #6 on: May 12, 2007, 07:19:56 AM »
I tried a 3 second delay for each 10 requests, but that did not help. Could it be something else?

Do you want login information?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #7 on: May 13, 2007, 07:06:04 AM »
Please try 1 second delay after every request.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

informer9

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 10
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #8 on: May 14, 2007, 10:05:34 PM »
Hi
sorry for the delay

I dont have any limitations at all. I use 0 to have all web page crawled.

I have tried suggested 1 sec break between every request...


Still no joy !!! it has even removed my last entry in to the directory ( I lost 5 urls)

all the other pages are being crawled fine

Admin - please help... any other ideas?

regards

mike

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #9 on: May 15, 2007, 12:09:13 AM »
Hello,

please send me a private message with your generator URL and example URL that is not included in sitemap.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

t_a

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #10 on: May 23, 2007, 03:05:25 PM »
I sent you a PM. I am missing appox 5000 pages in my sitemap.

Thanks.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator Removing URLs - Not Crawling New Pages
« Reply #11 on: May 25, 2007, 09:36:55 PM »
Update: the problem has been resolved.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2