XML Sitemaps Generator

Author Topic: No pages indexed  (Read 14892 times)

robin1

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
No pages indexed
« on: December 16, 2008, 07:53:06 PM »
I am running the site map generator on our development server to test it, and no pages are being indexed.  It is my educated guess that it has to do with our robots.txt that disallows all robots.  I can fix this by adding the site map generator robot as User agent, but I don't know what name to use for the robot.  Can you help me with this?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: No pages indexed
« Reply #1 on: December 16, 2008, 10:22:13 PM »
Hello,

please check our search engine robot simulator to see if your site is "crawlable": http://www.xml-sitemaps.com/se-bot-simulator.html
In case if it's related to robots.txt, I'd recommend to update it to allow crawling.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

robin1

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: No pages indexed
« Reply #2 on: December 17, 2008, 07:37:03 PM »
Yes.  I know I need to update the robots.txt to allow the crawler, but I only want to allow the xml-sitemap generator crawler and no others.  In order to do that, I need the name of the crawler.

I already tested the simulator on our live site, and I know it works just fine.  This is a development site not available to the public or search engine crawlers.

Can someone give me the actual name of the crawler to reference in robots.txt?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: No pages indexed
« Reply #3 on: December 18, 2008, 12:39:30 AM »
You can manually disable (set to 0) 'xs_robotstxt' option in config.inc.php file for that.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

robin1

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: No pages indexed
« Reply #4 on: December 30, 2008, 08:59:21 PM »
Thank you.  That did the job.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2