I did that and it seemed to fix that problem thanks, but now I have the problem where it just seems to hang at the very end of the process with a few seconds left and it never finishes:
"Links depth: 6
Current page: forum/living-ireland/work-visas-work-authorisation-164.html#post218
Pages added to sitemap: 2517
Pages scanned: 3620 (77,093.5 Kb)
Pages left: 7 (+ 361 queued for the next depth level)
Time passed: 32:38
Time left: 0:03
Memory usage: -
Total pages indexed: 2523"
It will stay like that for hours if I leave it.
I have ror.xml and sitemap.xml set to 0666 and I have a 5 second delay between each 20 requests.
Maximum execution time, seconds: 9000
Save the script state, every X seconds: 180
Any idea what might be causing this?