XML Sitemaps Generator

Author Topic: SEF url - Crawl is not going deep enough  (Read 9848 times)

jay7

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
SEF url - Crawl is not going deep enough
« on: September 09, 2009, 02:53:28 PM »
Hello,

I recently purchased the standard alone version, but having an issue. The site that I'm running it on has basically 4 php files that hit a database of many records and is structuring the site with hierarchical navigation. The urls are rewritten as shown below. It seems that the crawler is not making it through all urls. For instance - if page [external links are visible to admins only] displays a list of links that pertains to blue widgets, the crawler isn't picking them up. Is there a known configuration/solution for this type of setup?

Also, on the links that it does pickup, on certain links, it's not displaying the page title in the html sitemap, rather it displays the query parameter or 'filename'

dynamic url format: /index\.php?product=widget&color=blue&size=small&texture=fuzzy&maker=widgetco
static url format: /product/widget/blue/small/fuzzy/widgetco

I've basically set my htaccess file to do the following:
# Enable mod_rewrite, start rewrite engine
Options +FollowSymLinks
rewriteengine on
#
# internally rewrite search engine friendly static url to dynamic filepath and query
rewriteRule ^product/([^/]+)/([^/]+)/([^/]+)/([^/]+)/([^/]+)/?$ /index.php?product=$1&color=$2&size=$3&texture=$4&maker=$5 [L]
#
# Externally redirect client requests for old dynamic urls to equivalent new static urls
rewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php\?product=([^&]+)&color=([^&]+)&size=([^&]+)&texture=([^&]+)&maker=([^\ ]+)\ HTTP/
rewriteRule ^index\.php$ [external links are visible to admins only] [R=301,L]

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: SEF url - Crawl is not going deep enough
« Reply #1 on: September 09, 2009, 10:26:18 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jay7

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: SEF url - Crawl is not going deep enough
« Reply #2 on: September 14, 2009, 04:41:17 PM »
I've sent you a PM, just checking to ensure you received it? Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: SEF url - Crawl is not going deep enough
« Reply #3 on: September 14, 2009, 08:49:03 PM »
Hello,

I replied you on the same day, please check your PM inbox.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2