Generator crawls just a few links of our site... (Joomla + VM)

[marcoboer1973]

Hi there,

I've been reading the forum already about this problem, and already found out that there have been more people having problems with crawling through Joomla + VM sites...

This is what I'm experiencing and I hope that one of you can have a look at it if possible, the generator works great with another site of mine so I'm pretty satisfied. But I really need this site to work also...

- I get 56 links crawled instead of around 300 that should be in there...
- It takes ages for the links to get crawled
- I have the feeling it takes random links, because not all the menu items end up in the XML, nor all the products end up there...

So because on the other site it works good I assume it could well be a problem that is server related, but I'd really like you opinion too before contacting the host...

Looking forward to here from you.

[XML-Sitemaps Support]

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?

[marcoboer1973]

PM is sent btw...

[marcoboer1973]

Have you had the chance to check already?

[XML-Sitemaps Support]

Hello,

looks like your site is configured to block requests if they are sent too often (like "anti-flood mod for joomla"), as a result generator's crawler fails to index those pages. I've added 1 second delay after each request to slow down the crawling process, that should resolve the issue.

[marcoboer1973]

OK, cool, I'm gonna check it out, will let ya know...

[marcoboer1973]

Hi Oleg,

It seems to work fine now, thanks a lot!

[XML-Sitemaps Support]

You are welcome!