More Pages Scanned then added to SiteMap?

[thebassman1]

I'm crawling one of my sites, and there are considerably more pages scanned then added to the sitemap... why?

[XML-Sitemaps Support]

Hello,

the number of "scanned" pages includes ALL links found on the pages, but some of them are NOT includes into sitemap depending on the configuration settings ("excludes pages", "exclude extensions").

[thebassman1]

Once the crawling is done, is there a way to add certain missed pages to the sitemap afterwards w/o recrawling?

[XML-Sitemaps Support]

Hello,

it can be done only by manual modification of generated xml file (in any text editor).

[thebassman1]

Oh... (yuck).

Still waiting for LyricsLane to finish crawling...


Has anyone else noticed that it stops every once and a while?  I leave it running, and occassionally, I have to go back in and start it up again...?

[XML-Sitemaps Support]

Yes, it can be stopped depending on your php configuration (memory_limit/max_execution_time settings).

[thebassman1]

Is there a way to fix that?

[XML-Sitemaps Support]

Hello,

yes, increase the values for the options mentioned above (in php.ini).

[thebassman1]

ok, that's something my host would have to do, I guess.

Now, when I go in to continue with the crawling, it won't even start... it goes to page cannot be displayed. 

[XML-Sitemaps Support]

Hello,

answered in email.

[thebassman1]

Ok, it has finally finished crawling the site.  But now it gives me the following error:

Sitemap file is not writable: /usr/home/public_html/sitemap1.xml
Sitemap file is not writable: /usr/home/public_html/sitemap2.xml
Sitemap file is not writable: /usr/home/public_html/sitemap3.xml
Sitemap file is not writable: /usr/home/public_html/sitemap4.xml
Sitemap file is not writable: /usr/home/public_html/sitemap5.xml

I have these filles created and they are writeable.

[XML-Sitemaps Support]

Great!

You should either have /usr/home/public_html/ folder writable with 0766 permissions (which is not recommended) or CREATE empty files sitemap1.xml...sitemap5.xml and set their permissions to 0666.

Now you can simply copy these files from the data/ folder to  /usr/home/public_html/ and set permissions.

[thebassman1]

I have already created those files, and set the permission @ 666.

So I copy the files from the data folder to public_html ?

[XML-Sitemaps Support]

Yes. And set 0666 permissions for them in public_html for further sitemap generations.