Only spiders a small amount of my pages

[support1]

Hi,

I don't know if this will help but in /var/log/messages I find a load of these. An entry per crawl or page found:

preg_match(): Unknown modifier '|' in /home/httpd/htdocs/sitemap/pages/class.grab.inc.php(2)

BTW the only config thing I can think of that would cause this sort of problem to your script would be that safe mode is on. But that said, that should just mean that your script cannot set a max exec time.

[admin]

Hi,

the script was tested in "safe_mode on" environment and works fine except of max time setting.
That preg_match() error looks strange (it was never reported before) and probably may be the reason for crawling problem. Didn't you modified the config.inc.php file manually or set some specific config values? (I can't access the generator instance that you sent me anymote)

[support1]

Hello

I have just tested this on another one of my servers spidering the same website and it fails again. This server has apache1.x PHP 4.4.0 with a bog standard unoptimised PHP config. I also made sure eaccelerator was not installed on there.

I will PM you the updated url to the sitemap install. I want to know what is different when you try this. I am running my crawls from the script in a screen'd shell.

[support1]

Can you tell me what needs to be compiled into PHP for this to work ?

I am grabbing at straws now ...

[admin]

Hi,

the generator script doesn't require any external php modules to be compiled-in. It uses standard php network functions (http://www.php.net/manual/en/ref.network.php) for crawling.
I will try to recreate the same environment (apache 1.x and php4.4.0) to test it.

[support1]

Thanks for your help.

This is driving me nuts :-)

Have tried it now on 2 servers. The only thing I have left to try is to install LAMP on one of my Windows PCs here and see if I can crawl it from here. Even if I did manage to do it it wouldn't get me very far other than to have the sitemap files to upload to my server.

[admin]

Hi,

please try win lamp case if you have a chance, that will help to narrow down the problem

[support1]

Takes a long time to run on a home broadband connection ...

Have just recompiled PHP and Apache on one of my servers. 1.3.33 4.4.0 and basically made it the most bloated PHP install possible by adding everything (it was as close to a Microsoft version of PHP as possible) and still your script wouldn't crawl properly.

Things I have tried:

Disabling mod_security
Disabling eaccelerator
Disabling mod_deflate
Recompiling PHP
Recompiling Apache
Resetting all permissions on Generator scripts
Redownloading from your site and reuploading the script to my server

[support1]

I get a huge amount of these in my error_logs when the script is running:

[Thu Aug 11 14:56:29 2005] [error] [client xxx.xxx.xxx.xxx.xxx] request failed: error reading the headers, referer: