About htaccess, 301's, meta noindex pages
« on: December 17, 2011, 07:28:47 PM »
Unlimited Generator picked up on the site's htaccess, with the 301's listed and also the meta noindex pages, excluding both of these page types entirely from the sitemaps.

After a major redo on the site, which has only been this way for  two days now, I'm wondering if I should wait to link the sitemap to Google (webmaster tools) until Google can see the individual pages getting the 301's and meta noindex pages - even though Google was pinged    I assume  the sitemap must first be linked first for G to detect it? - But then again there is Yahoo and Bing.  Too, there are 3-4 pages having meta noindex within a large excluded folder.

From what I understand, rpc.pingomatic.com/ address provided is only for blogs, right?
What if the site has dated user comments/forms? Is this needed? Or updates on these be detected without it?

Also: Is there a way to delete a sitemap that is run or are they simply overwritten on each run?
Re: About htaccess, 301's, meta noindex pages
« Reply #1 on: December 17, 2011, 08:09:48 PM »
I think it's ok to submit sitemap now, providing Google with the actual structure of your website.

> From what I understand, rpc.pingomatic.com/ address provided is only for blogs, right?

Yes, it's not for search engines but I would keep it enabled.

> Also: Is there a way to delete a sitemap that is run or are they simply overwritten on each run?

Sitemap is overwritten each time.
Re: About htaccess, 301's, meta noindex pages
« Reply #2 on: December 17, 2011, 09:50:12 PM »
Quote
I think it's ok to submit sitemap now, providing Google with the actual structure of your website.
My main concern is that the values from the old 301's (not on the map) are transferred to the new pages.

Is it possible to exclude the htaccess file from the sitemap?

There are probable a little over three hundred 301 redirects.  As I said it is a major redo.
« Last Edit: December 17, 2011, 09:53:47 PM by rlife »
Re: About htaccess, 301's, meta noindex pages
« Reply #3 on: December 18, 2011, 08:30:48 PM »
Sitemap generator bot crawls your site similar to regular visitors, so all redirtects will be applied to it as well (you can temporarily remove them from  .htaccess while generator is creating sitemap and then re-enable them).
Re: About htaccess, 301's, meta noindex pages
« Reply #4 on: December 18, 2011, 09:17:04 PM »
Quote
Sitemap generator bot crawls your site similar to regular visitors, so all redirects will be applied to it as well
But, the old pages that are being redirected are no longer linked within the new, 'active site'. All the internal links were updated to the new pages being reidrected to.

How can I know if the Googlebot will recognize the 301's if the old pages are no longer connected within the site and they are not on the site map?

Will googlebot & others look for the old 'unconnected' yet established pages?
Does it know to  go by the htaccess alone which states the redirects?
What is standard recommended practice here?

Of course this exludes the meta noindex pages mentioned.
« Last Edit: December 18, 2011, 09:21:22 PM by rlife »
Re: About htaccess, 301's, meta noindex pages
« Reply #5 on: December 21, 2011, 10:38:45 AM »
If google has indexed those old pages before, its bot will re-visit them and detect the redirect.