SEF url - Crawl is not going deep enough
« on: September 09, 2009, 03:53:28 PM »
Hello,

I recently purchased the standard alone version, but having an issue. The site that I'm running it on has basically 4 php files that hit a database of many records and is structuring the site with hierarchical navigation. The urls are rewritten as shown below. It seems that the crawler is not making it through all urls. For instance - if page [ External links are visible to forum administrators only ] displays a list of links that pertains to blue widgets, the crawler isn't picking them up. Is there a known configuration/solution for this type of setup?

Also, on the links that it does pickup, on certain links, it's not displaying the page title in the html sitemap, rather it displays the query parameter or 'filename'

dynamic url format: /index\.php?product=widget&color=blue&size=small&texture=fuzzy&maker=widgetco
static url format: /product/widget/blue/small/fuzzy/widgetco

I've basically set my htaccess file to do the following:
# Enable mod_rewrite, start rewrite engine
Options +FollowSymLinks
rewriteengine on
#
# internally rewrite search engine friendly static url to dynamic filepath and query
rewriteRule ^product/([^/]+)/([^/]+)/([^/]+)/([^/]+)/([^/]+)/?$ /index.php?product=$1&color=$2&size=$3&texture=$4&maker=$5 [L]
#
# Externally redirect client requests for old dynamic urls to equivalent new static urls
rewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php\?product=([^&]+)&color=([^&]+)&size=([^&]+)&texture=([^&]+)&maker=([^\ ]+)\ HTTP/
rewriteRule ^index\.php$ [ External links are visible to forum administrators only ] [R=301,L]

Thanks
Re: SEF url - Crawl is not going deep enough
« Reply #1 on: September 09, 2009, 11:26:18 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?
Re: SEF url - Crawl is not going deep enough
« Reply #2 on: September 14, 2009, 05:41:17 PM »
I've sent you a PM, just checking to ensure you received it? Thanks