XML Sitemaps Generator

Author Topic: Generator crawls just a few links of our site... (Joomla + VM)  (Read 11011 times)

marcoboer1973

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Generator crawls just a few links of our site... (Joomla + VM)
« on: December 28, 2010, 11:41:35 AM »
Hi there,

I've been reading the forum already about this problem, and already found out that there have been more people having problems with crawling through Joomla + VM sites...

This is what I'm experiencing and I hope that one of you can have a look at it if possible, the generator works great with another site of mine so I'm pretty satisfied. But I really need this site to work also...

- I get 56 links crawled instead of around 300 that should be in there...
- It takes ages for the links to get crawled
- I have the feeling it takes random links, because not all the menu items end up in the XML, nor all the products end up there...

So because on the other site it works good I assume it could well be a problem that is server related, but I'd really like you opinion too before contacting the host...

Looking forward to here from you.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #1 on: December 28, 2010, 11:49:42 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

marcoboer1973

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #2 on: December 28, 2010, 12:16:15 PM »
PM is sent btw...

marcoboer1973

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #3 on: December 28, 2010, 07:28:06 PM »
Have you had the chance to check already?
« Last Edit: December 28, 2010, 07:30:43 PM by marcoboer1973 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #4 on: December 28, 2010, 07:51:58 PM »
Hello,

looks like your site is configured to block requests if they are sent too often (like "anti-flood mod for joomla"), as a result generator's crawler fails to index those pages. I've added 1 second delay after each request to slow down the crawling process, that should resolve the issue.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

marcoboer1973

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #5 on: December 28, 2010, 07:53:12 PM »
OK, cool, I'm gonna check it out, will let ya know...

marcoboer1973

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #6 on: December 28, 2010, 08:11:31 PM »
Hi Oleg,

It seems to work fine now, thanks a lot!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #7 on: December 28, 2010, 08:46:26 PM »
You are welcome!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2