Will not crawl subdirectories...
« on: December 17, 2013, 08:33:21 AM »
Hello,

I am using 6.1 unlimited sitemap generator. It is crawling [ External links are visible to forum administrators only ] but it will not crawl the URLs listed on that page and add to the sitemap. Any ideas?

Thanks!
Re: Will not crawl subdirectories...
« Reply #2 on: December 17, 2013, 01:04:56 PM »
Here is an example of one of the URLs that was not added: [ External links are visible to forum administrators only ] . Right now I have 3 URLs but soon (within 12 hours) I will have 10,000 + URLs like this one that will need to be added to the sitemap.

Thanks!
« Last Edit: December 17, 2013, 01:07:31 PM by cs24 »
Re: Will not crawl subdirectories...
« Reply #3 on: December 18, 2013, 06:43:11 PM »
Any ideas...? I need to resolve this as soon as possible, please!
Re: Will not crawl subdirectories...
« Reply #4 on: December 18, 2013, 08:46:24 PM »
Hello,

in case if starting URL is domain.org/table/centers/ then only URLs in that folder (table/centers/) are indexed. You need to set domain.org as starting URL.
Re: Will not crawl subdirectories...
« Reply #5 on: December 18, 2013, 08:49:09 PM »
Hello,

It is set like this: Starting URL: [ External links are visible to forum administrators only ]

Re: Will not crawl subdirectories...
« Reply #6 on: December 19, 2013, 01:57:28 PM »
Hello,

please let me know your generator URL/login in private message to check this.