Multi Directories CRAWL
« on: November 11, 2011, 08:50:40 PM »
Hello forum .
Here is my situation .
I run joomla site on shared hosting server , now have around 60.000+ articles . I used Xmap to generate sitemaps . Using Xmap component i created separate sitemap for each STATE (By state i mean actual USA States , Course articles goes be States and cities ) so i can submit separate sitemap to google and be indexed quickly (its 40.000+ indexed already) , but seance iam on shared hosting , i running to a problem of timeout or exeded memory , becourse each sitemap and article  category has growing to more then 10.000+ each , and when memory exeded that i can use on hosting its give me error and of course Google cant pull full sitemap . Thats the problem .

My qustions is :

Would i be able to create or point XML script/bot to crawl separate directories so script can create separate  simemaps ?(like: articles categories , exp: by state or city) ?
Exp:
1. sitemap : points to [ External links are visible to forum administrators only ].  XXX.com/Alabama/
Output : http:// www.  XXX.com/Alabama/sitemap.Alabama.xml

2. sitemap : points to http:// www. XXX.com/newyork/
Output : http:// www.  XXX.com/Alabama/sitemap.newyork.xml

and etc.....

Or the script runs through all link and output only 1 file ?

Also is is possible to create limit of links that scrips creating to sitemap for specific directory ?
Exp.

1. sitemap : points to [ External links are visible to forum administrators only ].  XXX.com/Alabama/    (limit set to 4000 links )
Output : http:// www.  XXX.com/Alabama/sitemap.Alabama.1page4000.xml
             http:// www.  XXX.com/Alabama/sitemap.Alabama.2page4000.xml
             http:// www.  XXX.com/Alabama/sitemap.Alabama.3page4000.xml
 

1. sitemap : points to [ External links are visible to forum administrators only ].  XXX.com/newyork/    (limit set to 4000 links )
Output : http:// www.  XXX.com/ Alabama/sitemap.newyork.1page4000.xml
             http:// www.  XXX.com /Alabama/sitemap.newyork.2page4000.xml
             http:// www.  XXX.com /Alabama/sitemap.newyork.3page4000.xml

I hope i explained my situation clearly , if not please point me ....

All this be course iam on the shared hosting and sitemaps too large for server to create output .

Ready to buy script , just need to clarify this things .

Thanks allot in advance , very hope this script can do what i need .
 Please PM me if so or post here ,\.



« Last Edit: November 12, 2011, 05:00:05 AM by antonpanin »
Re: Multi Directories CRAWL
« Reply #1 on: November 12, 2011, 08:03:25 AM »
Hello,

it's possible to create sitemaps for each directory, you would need to specify corresponding directory as Starting URL for sitemap generator.
Re: Multi Directories CRAWL
« Reply #2 on: November 12, 2011, 08:20:12 AM »
Hello,

it's possible to create sitemaps for each directory, you would need to specify corresponding directory as Starting URL for sitemap generator.



Thnk you for quick respond , but is it possible to follow this formula ?Below
So it will be multy sitemaps (with lets say 4000 links limit ) for only 1 specific category ?

1. sitemap : points to [external links are visible to admins only].  XXX.com/Alabama/    (limit set to 4000 links )
Output : http:// www.  XXX.com/Alabama/sitemap.Alabama.1page4000.xml
             http:// www.  XXX.com/Alabama/sitemap.Alabama.2page4000.xml
             http:// www.  XXX.com/Alabama/sitemap.Alabama.3page4000.xml
 

1. sitemap : points to [external links are visible to admins only].  XXX.com/newyork/    (limit set to 4000 links )
Output : http:// www.  XXX.com/ Alabama/sitemap.newyork.1page4000.xml
             http:// www.  XXX.com /Alabama/sitemap.newyork.2page4000.xml
             http:// www.  XXX.com /Alabama/sitemap.newyork.3page4000.xml
Re: Multi Directories CRAWL
« Reply #3 on: November 12, 2011, 11:27:13 AM »
Hello,

you can limit the number of URLs per sitemap file to 4000 and sitemap generator will create multiple sitemap files automatically.
Re: Multi Directories CRAWL
« Reply #4 on: November 12, 2011, 06:35:29 PM »
THNK ALLOT , Just what i need .
I more question . Is there option to avoid duplicates ?
« Last Edit: November 12, 2011, 06:39:26 PM by antonpanin »
Re: Multi Directories CRAWL
« Reply #5 on: November 17, 2011, 12:11:56 AM »
There won't be duplicate URLs in sitemap, it's auomtatically tracked.