Our site has currently about 2000 products. The shopping cart (X-Cart) will create HTML only pages of the entire dbase and save them into a directory in the public area. Unforturnatly, the generator will not find the HTML pages even with a direct link on the HTML home page linking to index.htm (HTML catalog) in the public root. The generator will run thru the dynamic PHP pages in what seems to be an endless loop due to sorting many different ways which I don't want Google to see due to redundant information and keywordless addresses. The site is setup with static HTML front side with links to the dynamic PHP store.
Generator needs to pass over all dynamic pages, PHP, products.php?cat= and the like. It needs to include only the static HTML pages created by the shopping cart for which there is a direct link to. These HTML pages will send the user directly to the dynamic part of the store when they add something to their cart or login.
I have tried to limit using the "do not parse" pages with the php coding, and include only URL's. So far these static pages never get found. In order for the static catalog pages to work, they must be in the shopping cart directory, otherwise the images don't appear, so I can't exclude the shopping cart directory.
Just a question, when I installed the generator, we had about 175 products plus about 20 HTML pages. I set it to run on defaults and it found 3953 pages before I stopped it after an hour. Is this normal? And what would a search engine rank your pages at with each page duplicated 20 times?
Any help would be appreciated.