If I use the online generator my site, including my PHP pages are spidered.
When I use the stand alone version, only the HTML pages are spidered.
I along with others, have found parts of the instructions very difficult.
Especially permissions. I spent over 4 hrs to learn what permissions were as compared to attributes.
I have created a file called "permission.php"
These instructions were put inside and them I click run PHP.
I then re attempted a crawl, with the same results. The PHPs are not spidered in the stand alone program.
If I reload the files and start from the beginning and try to crawl my site, after about 2 minutes I get "page not found in the window. We I go to recrawl, it shows maybe 200 files found. So I press continue. Again after 2 minutes, I get the same message. If I recrawl again, the system shows another 150 files added. Again after about 2 min "page not found" appears"
The problem is that in my PHP files there are approx 30,000 entries. I would have to reload crawl for the next month to pick up all of the files.
My Site is [external links are visible to admins only]
There has got to be an easier way.