Re: installation folder
« Reply #15 on: January 14, 2015, 10:01:09 AM »
Hello,

running from the command line normally has less restrictions.
The command to run is shown on the Crawling page of sitemap generator.

Reda

*
  • *
  • 20
Re: installation folder
« Reply #16 on: January 14, 2015, 10:51:23 AM »
1- I was more talking if I should put "php -a /path/runcrawl.php" or "php -f /path/runcrawl.php"

2- also, what is the recommended file permission for runcrawl.php?

3- Are you planing any fix for this issue?

4- I'll have one last question please: regarding the <lastmod> tag, it generates current date for dynamic pages, is there a way to be more accurate?
Re: installation folder
« Reply #17 on: January 15, 2015, 06:26:13 AM »
1. php -f /path/runcrawl.php

2. 644

3. In case if server configuration restricts scripts from running longer than certain amount of time, either resuming is required, or running in command line.

4. In case if server returns Lastmod time in http header, it will be also included in sitemap. Otherwise, current time is used.

Reda

*
  • *
  • 20
Re: installation folder
« Reply #18 on: January 16, 2015, 09:19:40 PM »
OK, thank you.

Reda

*
  • *
  • 20
Re: installation folder
« Reply #19 on: April 27, 2015, 01:13:09 PM »
Hi,

I'm back with one other question please.
I'm now running the crawl with command line and it works great.
But sometimes when my computer goes on saving mode, the crawl stops and doesn't resume. I quit the Terminal (MAC OSX) and when i run the commande again, it says <h4>Already in progress. Current process state is displayed:</h4>
so is there a way to completely interrupt the crawling?

Thanks a lot
Re: installation folder
« Reply #20 on: April 28, 2015, 05:53:13 AM »
Hello,

you need to  also remove generator/data/crawl_state.log file to reset it.

Reda

*
  • *
  • 20
Re: installation folder
« Reply #21 on: April 28, 2015, 11:09:20 AM »
OK, thank you very much :)

I have about 20 000 files to be crawled, is it normal that it takes about 35 minutes to complete the job?
Re: installation folder
« Reply #22 on: April 28, 2015, 06:38:35 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Reda

*
  • *
  • 20
Re: installation folder
« Reply #23 on: April 28, 2015, 07:59:54 PM »
Hello,

Good to know ;)
Thousand thanks for your support.

Cheers

Reda

*
  • *
  • 20
Re: installation folder
« Reply #24 on: June 28, 2015, 07:46:20 PM »
Hi again,

I have recently received an email informing me about a new version 7.1, i don't see the link to download it, where can I find it?

Thanks
Re: installation folder
« Reply #25 on: June 29, 2015, 07:44:27 AM »
You can see the red box at the top of this forum page, saying:
-----
Hello, you are a registered customer with the license assigned to email: ...
Your download link for the latest version of sitemap generator is: ...

Reda

*
  • *
  • 20
Re: installation folder
« Reply #26 on: June 29, 2015, 02:18:21 PM »
I don't see this red box! I even searched within the source code the word "assigned" and didn't find it!!  :-\
Re: installation folder
« Reply #27 on: June 30, 2015, 12:25:35 PM »
just updated to 7.1 and getting 500 error checked file permissions added htaccess with the one line
and memory_limit=512M
max_execution_time=50000

Re: installation folder
« Reply #28 on: June 30, 2015, 01:06:48 PM »
ok sorted there was a server hiccup at same time i installed new version just co-incidence