why my new pages aren't being index
« on: March 27, 2006, 06:29:20 PM »
I'm new to sitemaps and need of help.

I have an old site I'm updating from asp to aspx.  I've made a sitemap and did notice google crawling my site but new pages haven't been index.  I was concerned about the old pages so in reading up I used the following code to redirect to new pages:

<%@ Language=VBScript %>
<%
response.status="301 moved permanently"
Response.AddHeader "Location", "[ External links are visible to forum administrators only ]"
%>

AGAIN MY PAGES DON'T SHOW UP, so I made this robot.txt file to ignore old pages but this robot.txt validator shows nothing but errors  [ External links are visible to forum administrators only ]  this is my robot.txt file, the pages I don't want google to index:

# robots.txt for [ External links are visible to forum administrators only ]
User-agent: *
Disallow:
Disallow: /cgi-bin/
Disallow: /Inspections/home_inspection.asp
Disallow: /Inspections/home_inspection_Air_Quality_Testing.asp
Disallow: /Inspections/home_inspection_Asbestos.asp
Disallow: /Inspections/home_inspection_Benefit_Gas_Fireplace.asp
Disallow: /Inspections/home_inspection_Buy_Selling.asp
Disallow: /Inspections/home_inspection_CheckList.asp
Disallow: /Inspections/home_inspection_Cooling_Alternatives.asp
Disallow: /Inspections/home_inspection_Franchise.asp
Disallow: /Inspections/home_inspection_Hiring_Inspector.asp
Disallow: /Inspections/home_inspection_Lead_Paint.asp
Disallow: /Inspections/home_inspection_NewHome.asp
Disallow: /Inspections/home_inspection_Oil_Tanks.asp
Disallow: /Inspections/home_inspection_Radon_Gas.asp
Disallow: /Inspections/home_inspection_reduce_basement_himidity.asp
Disallow: /Inspections/home_inspection_Roofing_Maintenance.asp
Disallow: /Inspections/home_inspection_Safety_security.asp
Disallow: /Inspections/home_inspection_Saving_Energy.asp
Disallow: /Inspections/home_inspection_Summer_Comfort.asp
Disallow: /solutions/asbestos.asp
Disallow: /solutions/become_home_inspector.asp
Disallow: /solutions/benefit_gas_fireplace.asp
Disallow: /solutions/cooling_alternatives.asp
Disallow: /solutions/gas_fireplace.asp
Disallow: /solutions/home_buying_selling.asp
Disallow: /solutions/home_buying_selling1.asp
Disallow: /solutions/home_inspections.asp
Disallow: /solutions/home_inspections_01.asp
Disallow: /solutions/home_inspections_02.asp
Disallow: /solutions/home_inspections_br.asp
Disallow: /solutions/indoor_air_quality_testing_toronto.asp
Disallow: /solutions/lead_paint.asp
Disallow: /solutions/mold_removal.asp
Disallow: /solutions/radon_gas.asp
Disallow: /solutions/reduce_basement_humidity.asp
Disallow: /solutions/roofing_maintenance.asp
Disallow: /solutions/safety_security.asp
Disallow: /solutions/saving_energy.asp
Disallow: /solutions/summer_comfort.asp
Disallow: /solutions/underground_oil_tanks.asp

WHAT ELSE SHOULD I BE DOING TO GET MY NEW ASPX PAGES TO BE INDEX?

Judy
Re: why my new pages aren't being index
« Reply #1 on: March 27, 2006, 06:42:15 PM »
Hello Judy,

you should remove this line from robots.txt:
Quote
Disallow:
because empty value will disallow ALL pages from being indexed.
Re: why my new pages aren't being index
« Reply #2 on: March 27, 2006, 06:50:08 PM »
thanks, THAT I SUPPOSE WAS HUGE.  the validator seems to have fixed all my errors except for line one, is there a problem with the first line of my robot.txt? 

any ideas as to why my new pages aren't being indexed?  I don't think google has used my robot.txt file yet, just worked on that this morning.  I was thinking it won't index because it's to similar to old pages. 

Could it possibly take a long time to index new pages even with a sitemap?
Re: why my new pages aren't being index
« Reply #3 on: March 28, 2006, 09:40:53 PM »
Hello,

do you mean indexing pages in Search Engines (google)? If so, of course it may take quite some time until they appear in the listings.