Sitemap Generator Forum

We have a problem where the pages added to the sitemap maxes out at 32 even though the crawler is crawling a lot more pages. Any idea why this happens? Thank you.

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

We increased
max_execution_time: 60
memory_limit: 2018M

to
max_execution_time: 1000
memory_limit: 4036M

but it doesn't seem any much better. The number of pages added to the sitemap increased from 32 to 48. If you take a look at the attached screenshot, runcrawl.php continues to run, but the number of pages, just stops increasing.

Any other ideas?

Hello,

what are the last lines of output? From what I see it keeps increasing (185 vs 183 on the last line)

I ran it again and this was what I got:
336 | 365 | 3,056.0 | 0:15:35 | 0:16:56 | 2 | 2,289.8 Kb | 32 | 641 | 2289
338 | 363 | 3,056.5 | 0:15:40 | 0:16:50 | 2 | 2,288.2 Kb | 32 | 641 | 2288
340 | 361 | 3,057.1 | 0:15:45 | 0:16:44 | 2 | 2,290.1 Kb | 32 | 641 | 2290
341 | 360 | 3,057.4 | 0:15:51 | 0:16:44 | 2 | 2,288.5 Kb | 32 | 641 | 2288
343 | 358 | 3,058.0 | 0:15:56 | 0:16:37 | 2 | 2,290.4 Kb | 32 | 641 | 2290
345 | 356 | 3,058.5 | 0:16:01 | 0:16:31 | 2 | 2,289.3 Kb | 32 | 641 | 2289
346 | 355 | 3,058.8 | 0:16:06 | 0:16:31 | 2 | 2,290.9 Kb | 32 | 641 | 2290
348 | 353 | 3,059.5 | 0:16:11 | 0:16:25 | 2 | 2,289.2 Kb | 32 | 641 | 2289
350 | 351 | 3,060.1 | 0:16:16 | 0:16:19 | 2 | 2,291.3 Kb | 32 | 641 | 2291
351 | 350 | 3,060.4 | 0:16:21 | 0:16:18 | 2 | 2,289.5 Kb | 32 | 641 | 2289
353 | 348 | 3,061.1 | 0:16:26 | 0:16:12 | 2 | 2,291.4 Kb | 32 | 641 | 2291
355 | 346 | 3,061.7 | 0:16:31 | 0:16:06 | 2 | 2,289.9 Kb | 32 | 641 | 2289
356 | 345 | 3,062.0 | 0:16:36 | 0:16:05 | 2 | 2,291.8 Kb | 32 | 641 | 2291
358 | 343 | 3,062.7 | 0:16:41 | 0:15:59 | 2 | 2,290.1 Kb | 32 | 641 | 2290
360 | 341 | 3,063.4 | 0:16:46 | 0:15:53 | 2 | 2,292.2 Kb | 32 | 641 | 2292
361 | 340 | 3,063.7 | 0:16:51 | 0:15:52 | 2 | 2,290.3 Kb | 32 | 641 | 2290
363 | 338 | 3,064.4 | 0:16:56 | 0:15:46 | 2 | 2,292.4 Kb | 32 | 641 | 2292
365 | 336 | 3,065.1 | 0:17:01 | 0:15:40 | 2 | 2,290.9 Kb | 32 | 641 | 2290
366 | 335 | 3,065.4 | 0:17:06 | 0:15:39 | 2 | 2,292.6 Kb | 32 | 641 | 2292
368 | 333 | 3,066.1 | 0:17:11 | 0:15:33 | 2 | 2,290.9 Kb | 32 | 641 | 2290
370 | 331 | 3,066.7 | 0:17:17 | 0:15:27 | 2 | 2,292.5 Kb | 32 | 641 | 2292
371 | 330 | 3,067.0 | 0:17:22 | 0:15:26 | 2 | 2,290.9 Kb | 32 | 641 | 2290
373 | 328 | 3,067.6 | 0:17:27 | 0:15:20 | 2 | 2,293.0 Kb | 32 | 641 | 2293
375 | 326 | 3,068.3 | 0:17:32 | 0:15:14 | 2 | 2,291.4 Kb | 32 | 641 | 2291
376 | 325 | 3,068.6 | 0:17:37 | 0:15:13 | 2 | 2,293.3 Kb | 32 | 641 | 2293
378 | 323 | 3,069.3 | 0:17:42 | 0:15:07 | 2 | 2,291.6 Kb | 32 | 641 | 2291
380 | 321 | 3,069.9 | 0:17:47 | 0:15:01 | 2 | 2,293.6 Kb | 32 | 641 | 2293
381 | 320 | 3,070.2 | 0:17:52 | 0:15:00 | 2 | 2,291.7 Kb | 32 | 641 | 2291
383 | 318 | 3,070.8 | 0:17:57 | 0:14:54 | 2 | 2,293.9 Kb | 32 | 641 | 2293
385 | 316 | 3,071.5 | 0:18:02 | 0:14:48 | 2 | 2,292.5 Kb | 32 | 641 | 2292
386 | 315 | 3,071.9 | 0:18:07 | 0:14:47 | 2 | 2,294.3 Kb | 32 | 641 | 2294
388 | 313 | 3,072.6 | 0:18:12 | 0:14:41 | 2 | 2,294.2 Kb | 32 | 641 | 2294
390 | 311 | 3,073.3 | 0:18:17 | 0:14:35 | 2 | 2,292.7 Kb | 32 | 641 | 2292
391 | 310 | 3,073.6 | 0:18:22 | 0:14:34 | 2 | 2,294.4 Kb | 32 | 641 | 2294
393 | 308 | 3,074.2 | 0:18:27 | 0:14:28 | 2 | 2,292.6 Kb | 32 | 641 | 2292
395 | 306 | 3,074.8 | 0:18:32 | 0:14:22 | 2 | 2,294.9 Kb | 32 | 641 | 2294
396 | 305 | 3,075.2 | 0:18:37 | 0:14:21 | 2 | 2,292.9 Kb | 32 | 641 | 2292
398 | 303 | 3,075.8 | 0:18:42 | 0:14:14 | 2 | 2,294.9 Kb | 32 | 641 | 2294
400 | 301 | 3,076.4 | 0:18:48 | 0:14:08 | 2 | 2,293.4 Kb | 32 | 641 | 2293
401 | 300 | 3,076.8 | 0:18:53 | 0:14:07 | 2 | 2,295.3 Kb | 32 | 641 | 2295
403 | 298 | 3,077.4 | 0:18:58 | 0:14:01 | 2 | 2,293.5 Kb | 32 | 641 | 2293
405 | 296 | 3,078.1 | 0:19:03 | 0:13:55 | 2 | 2,295.6 Kb | 32 | 641 | 2295
406 | 295 | 3,078.4 | 0:19:08 | 0:13:54 | 2 | 2,293.8 Kb | 32 | 641 | 2293
408 | 293 | 3,079.1 | 0:19:13 | 0:13:48 | 2 | 2,296.0 Kb | 32 | 641 | 2296
410 | 291 | 3,079.8 | 0:19:18 | 0:13:42 | 2 | 2,294.6 Kb | 32 | 641 | 2294
411 | 290 | 3,080.2 | 0:19:23 | 0:13:40 | 2 | 2,296.3 Kb | 32 | 641 | 2296
413 | 288 | 3,080.8 | 0:19:28 | 0:13:34 | 2 | 2,294.2 Kb | 32 | 641 | 2294
415 | 286 | 3,081.4 | 0:19:33 | 0:13:28 | 2 | 2,296.2 Kb | 32 | 641 | 2296
416 | 285 | 3,081.8 | 0:19:38 | 0:13:27 | 2 | 2,294.6 Kb | 32 | 641 | 2294
418 | 283 | 3,082.4 | 0:19:43 | 0:13:21 | 2 | 2,296.7 Kb | 32 | 641 | 2296
420 | 281 | 3,083.2 | 0:19:48 | 0:13:15 | 2 | 2,295.4 Kb | 32 | 641 | 2295
421 | 280 | 3,083.5 | 0:19:53 | 0:13:13 | 2 | 2,297.0 Kb | 32 | 641 | 2297
423 | 278 | 3,084.1 | 0:19:58 | 0:13:07 | 2 | 2,295.1 Kb | 32 | 641 | 2295
425 | 276 | 3,084.8 | 0:20:03 | 0:13:01 | 2 | 2,297.3 Kb | 32 | 641 | 2297
426 | 275 | 3,085.2 | 0:20:08 | 0:13:00 | 2 | 2,295.4 Kb | 32 | 641 | 2295
428 | 273 | 3,085.8 | 0:20:13 | 0:12:54 | 2 | 2,297.4 Kb | 32 | 641 | 2297
430 | 271 | 3,086.5 | 0:20:19 | 0:12:48 | 2 | 2,296.1 Kb | 32 | 641 | 2296
431 | 270 | 3,086.9 | 0:20:24 | 0:12:46 | 2 | 2,297.7 Kb | 32 | 641 | 2297
433 | 268 | 3,087.6 | 0:20:29 | 0:12:40 | 2 | 2,295.9 Kb | 32 | 641 | 2295
435 | 266 | 3,088.2 | 0:20:34 | 0:12:34 | 2 | 2,298.2 Kb | 32 | 641 | 2298
436 | 265 | 3,088.6 | 0:20:39 | 0:12:33 | 2 | 2,296.0 Kb | 32 | 641 | 2296
438 | 263 | 3,089.1 | 0:20:44 | 0:12:27 | 2 | 2,298.1 Kb | 32 | 641 | 2298
440 | 261 | 3,089.8 | 0:20:49 | 0:12:21 | 2 | 2,296.5 Kb | 32 | 641 | 2296
441 | 260 | 3,090.1 | 0:20:54 | 0:12:19 | 2 | 2,298.3 Kb | 32 | 641 | 2298
443 | 258 | 3,090.8 | 0:20:59 | 0:12:13 | 2 | 2,296.6 Kb | 32 | 641 | 2296
445 | 256 | 3,091.4 | 0:21:04 | 0:12:07 | 2 | 2,298.6 Kb | 32 | 641 | 2298
446 | 255 | 3,091.7 | 0:21:09 | 0:12:05 | 2 | 2,296.9 Kb | 32 | 641 | 2296
448 | 253 | 3,092.4 | 0:21:14 | 0:11:59 | 2 | 2,298.9 Kb | 32 | 641 | 2298
450 | 251 | 3,093.0 | 0:21:19 | 0:11:53 | 2 | 2,297.2 Kb | 32 | 641 | 2297
451 | 250 | 3,093.3 | 0:21:24 | 0:11:52 | 2 | 2,299.1 Kb | 32 | 641 | 2299
453 | 248 | 3,094.0 | 0:21:29 | 0:11:46 | 2 | 2,297.4 Kb | 32 | 641 | 2297
455 | 246 | 3,094.6 | 0:21:34 | 0:11:40 | 2 | 2,299.5 Kb | 32 | 641 | 2299
456 | 245 | 3,094.9 | 0:21:39 | 0:11:38 | 2 | 2,297.7 Kb | 32 | 641 | 2297
458 | 243 | 3,095.6 | 0:21:44 | 0:11:32 | 2 | 2,299.5 Kb | 32 | 641 | 2299
460 | 241 | 3,096.1 | 0:21:50 | 0:11:26 | 2 | 2,297.8 Kb | 32 | 641 | 2297
461 | 240 | 3,096.5 | 0:21:55 | 0:11:24 | 2 | 2,299.8 Kb | 32 | 641 | 2299
463 | 238 | 3,097.1 | 0:22:00 | 0:11:18 | 2 | 2,298.2 Kb | 32 | 641 | 2298
465 | 236 | 3,097.7 | 0:22:05 | 0:11:12 | 2 | 2,300.2 Kb | 32 | 641 | 2300
466 | 235 | 3,098.0 | 0:22:10 | 0:11:10 | 2 | 2,298.5 Kb | 32 | 641 | 2298
468 | 233 | 3,098.7 | 0:22:15 | 0:11:04 | 2 | 2,300.4 Kb | 32 | 641 | 2300
470 | 231 | 3,099.3 | 0:22:20 | 0:10:58 | 2 | 2,298.8 Kb | 32 | 641 | 2298
471 | 230 | 3,099.6 | 0:22:25 | 0:10:57 | 2 | 2,300.7 Kb | 32 | 641 | 2300
473 | 228 | 3,100.3 | 0:22:30 | 0:10:50 | 2 | 2,299.0 Kb | 32 | 641 | 2299
475 | 226 | 3,101.0 | 0:22:35 | 0:10:44 | 2 | 2,301.3 Kb | 32 | 641 | 2301
476 | 225 | 3,101.3 | 0:22:40 | 0:10:43 | 2 | 2,301.2 Kb | 32 | 641 | 2301
478 | 223 | 3,102.0 | 0:22:45 | 0:10:37 | 2 | 2,299.5 Kb | 32 | 641 | 2299
480 | 221 | 3,102.6 | 0:22:50 | 0:10:31 | 2 | 2,301.4 Kb | 32 | 641 | 2301
481 | 220 | 3,102.9 | 0:22:55 | 0:10:29 | 2 | 2,299.6 Kb | 32 | 641 | 2299
483 | 218 | 3,103.6 | 0:23:00 | 0:10:23 | 2 | 2,301.7 Kb | 32 | 641 | 2301
485 | 216 | 3,104.3 | 0:23:05 | 0:10:17 | 2 | 2,300.2 Kb | 32 | 641 | 2300
486 | 215 | 3,104.6 | 0:23:10 | 0:10:15 | 2 | 2,301.9 Kb | 32 | 641 | 2301
488 | 213 | 3,105.2 | 0:23:16 | 0:10:09 | 2 | 2,300.3 Kb | 32 | 641 | 2300
490 | 211 | 3,105.9 | 0:23:21 | 0:10:03 | 2 | 2,302.4 Kb | 32 | 641 | 2302
491 | 210 | 3,106.2 | 0:23:26 | 0:10:01 | 2 | 2,300.3 Kb | 32 | 641 | 2300
493 | 208 | 3,106.8 | 0:23:31 | 0:09:55 | 2 | 2,302.4 Kb | 32 | 641 | 2302
495 | 206 | 3,107.5 | 0:23:36 | 0:09:49 | 2 | 2,300.8 Kb | 32 | 641 | 2300
496 | 205 | 3,107.8 | 0:23:41 | 0:09:47 | 2 | 2,302.6 Kb | 32 | 641 | 2302
498 | 203 | 3,108.4 | 0:23:46 | 0:09:41 | 2 | 2,300.8 Kb | 32 | 641 | 2300
500 | 201 | 3,109.0 | 0:23:51 | 0:09:35 | 2 | 2,303.1 Kb | 32 | 641 | 2303
501 | 200 | 3,109.3 | 0:23:56 | 0:09:33 | 2 | 2,301.4 Kb | 32 | 641 | 2301
503 | 198 | 3,110.0 | 0:24:01 | 0:09:27 | 2 | 2,303.4 Kb | 32 | 641 | 2303
505 | 196 | 3,110.7 | 0:24:06 | 0:09:21 | 2 | 2,301.8 Kb | 32 | 641 | 2301
506 | 195 | 3,111.1 | 0:24:11 | 0:09:19 | 2 | 2,303.4 Kb | 32 | 641 | 2303
508 | 193 | 3,111.7 | 0:24:16 | 0:09:13 | 2 | 2,301.7 Kb | 32 | 641 | 2301
510 | 191 | 3,112.3 | 0:24:21 | 0:09:07 | 2 | 2,303.8 Kb | 32 | 641 | 2303
511 | 190 | 3,112.6 | 0:24:26 | 0:09:05 | 2 | 2,301.8 Kb | 32 | 641 | 2301
513 | 188 | 3,113.2 | 0:24:31 | 0:08:59 | 2 | 2,303.9 Kb | 32 | 641 | 2303
515 | 186 | 3,113.8 | 0:24:36 | 0:08:53 | 2 | 2,302.3 Kb | 32 | 641 | 2302
516 | 185 | 3,114.1 | 0:24:42 | 0:08:51 | 2 | 2,304.0 Kb | 32 | 641 | 2304
518 | 183 | 3,114.7 | 0:24:47 | 0:08:45 | 2 | 2,302.4 Kb | 32 | 641 | 2302
520 | 181 | 3,115.3 | 0:24:52 | 0:08:39 | 2 | 2,304.4 Kb | 32 | 641 | 2304
521 | 180 | 3,115.6 | 0:24:57 | 0:08:37 | 2 | 2,302.6 Kb | 32 | 641 | 2302
523 | 178 | 3,116.3 | 0:25:02 | 0:08:31 | 2 | 2,305.0 Kb | 32 | 641 | 2305
525 | 176 | 3,117.0 | 0:25:07 | 0:08:25 | 2 | 2,303.3 Kb | 32 | 641 | 2303
526 | 175 | 3,117.3 | 0:25:12 | 0:08:23 | 2 | 2,305.2 Kb | 32 | 641 | 2305
528 | 173 | 3,118.0 | 0:25:17 | 0:08:17 | 2 | 2,303.3 Kb | 32 | 641 | 2303
530 | 171 | 3,118.6 | 0:25:22 | 0:08:11 | 2 | 2,305.3 Kb | 32 | 641 | 2305
531 | 170 | 3,118.9 | 0:25:27 | 0:08:09 | 2 | 2,303.5 Kb | 32 | 641 | 2303
533 | 168 | 3,119.6 | 0:25:32 | 0:08:03 | 2 | 2,305.8 Kb | 32 | 641 | 2305
535 | 166 | 3,120.3 | 0:25:37 | 0:07:57 | 2 | 2,306.0 Kb | 32 | 641 | 2306
536 | 165 | 3,120.6 | 0:25:42 | 0:07:54 | 2 | 2,304.2 Kb | 32 | 641 | 2304
538 | 163 | 3,121.4 | 0:25:47 | 0:07:48 | 2 | 2,306.2 Kb | 32 | 641 | 2306
540 | 161 | 3,122.0 | 0:25:52 | 0:07:42 | 2 | 2,304.4 Kb | 32 | 641 | 2304
541 | 160 | 3,122.3 | 0:25:57 | 0:07:40 | 2 | 2,305.8 Kb | 32 | 641 | 2305
543 | 158 | 3,122.8 | 0:26:02 | 0:07:34 | 2 | 2,304.2 Kb | 32 | 641 | 2304
545 | 156 | 3,123.4 | 0:26:07 | 0:07:28 | 2 | 2,310.2 Kb | 32 | 641 | 2310
546 | 155 | 3,123.7 | 0:26:12 | 0:07:26 | 2 | 2,308.4 Kb | 32 | 641 | 2308
548 | 153 | 3,124.2 | 0:26:18 | 0:07:20 | 2 | 2,310.4 Kb | 32 | 641 | 2310
550 | 151 | 3,124.8 | 0:26:23 | 0:07:14 | 2 | 2,308.7 Kb | 32 | 641 | 2308
551 | 150 | 3,125.0 | 0:26:28 | 0:07:12 | 2 | 2,310.6 Kb | 32 | 641 | 2310
553 | 148 | 3,125.6 | 0:26:33 | 0:07:06 | 2 | 2,308.9 Kb | 32 | 641 | 2308
555 | 146 | 3,126.1 | 0:26:38 | 0:07:00 | 2 | 2,310.9 Kb | 32 | 641 | 2310
556 | 145 | 3,126.4 | 0:26:43 | 0:06:58 | 2 | 2,309.2 Kb | 32 | 641 | 2309
558 | 143 | 3,126.9 | 0:26:48 | 0:06:52 | 2 | 2,311.2 Kb | 32 | 641 | 2311
560 | 141 | 3,127.5 | 0:26:53 | 0:06:46 | 2 | 2,309.5 Kb | 32 | 641 | 2309
561 | 140 | 3,127.7 | 0:26:58 | 0:06:43 | 2 | 2,311.4 Kb | 32 | 641 | 2311
563 | 138 | 3,128.3 | 0:27:03 | 0:06:37 | 2 | 2,309.9 Kb | 32 | 641 | 2309
565 | 136 | 3,128.9 | 0:27:08 | 0:06:32 | 2 | 2,311.9 Kb | 32 | 641 | 2311
566 | 135 | 3,129.2 | 0:27:13 | 0:06:29 | 2 | 2,312.0 Kb | 32 | 641 | 2312
568 | 133 | 3,129.8 | 0:27:18 | 0:06:23 | 2 | 2,310.3 Kb | 32 | 641 | 2310
570 | 131 | 3,130.4 | 0:27:23 | 0:06:17 | 2 | 2,312.6 Kb | 32 | 641 | 2312
571 | 130 | 3,130.7 | 0:27:28 | 0:06:15 | 2 | 2,310.6 Kb | 32 | 641 | 2310
573 | 128 | 3,131.3 | 0:27:33 | 0:06:09 | 2 | 2,312.5 Kb | 32 | 641 | 2312
575 | 126 | 3,131.9 | 0:27:38 | 0:06:03 | 2 | 2,310.9 Kb | 32 | 641 | 2310
576 | 125 | 3,132.2 | 0:27:43 | 0:06:01 | 2 | 2,312.7 Kb | 32 | 641 | 2312
578 | 123 | 3,132.8 | 0:27:49 | 0:05:55 | 2 | 2,311.1 Kb | 32 | 641 | 2311
580 | 121 | 3,133.5 | 0:27:54 | 0:05:49 | 2 | 2,313.2 Kb | 32 | 641 | 2313
581 | 120 | 3,133.8 | 0:27:59 | 0:05:46 | 2 | 2,311.6 Kb | 32 | 641 | 2311
583 | 118 | 3,134.5 | 0:28:04 | 0:05:40 | 2 | 2,313.5 Kb | 32 | 641 | 2313
585 | 116 | 3,135.1 | 0:28:09 | 0:05:34 | 2 | 2,311.7 Kb | 32 | 641 | 2311
586 | 115 | 3,135.4 | 0:28:14 | 0:05:32 | 2 | 2,313.6 Kb | 32 | 641 | 2313
588 | 113 | 3,136.0 | 0:28:19 | 0:05:26 | 2 | 2,312.1 Kb | 32 | 641 | 2312
590 | 111 | 3,136.7 | 0:28:24 | 0:05:20 | 2 | 2,314.1 Kb | 32 | 641 | 2314
591 | 110 | 3,137.0 | 0:28:29 | 0:05:18 | 2 | 2,312.3 Kb | 32 | 641 | 2312
593 | 108 | 3,137.6 | 0:28:34 | 0:05:12 | 2 | 2,314.3 Kb | 32 | 641 | 2314
595 | 106 | 3,138.3 | 0:28:39 | 0:05:06 | 2 | 2,312.7 Kb | 32 | 641 | 2312
596 | 105 | 3,138.6 | 0:28:44 | 0:05:03 | 2 | 2,314.5 Kb | 32 | 641 | 2314
598 | 103 | 3,139.3 | 0:28:49 | 0:04:57 | 2 | 2,313.0 Kb | 32 | 641 | 2313
600 | 101 | 3,140.0 | 0:28:54 | 0:04:52 | 2 | 2,315.1 Kb | 32 | 641 | 2315
601 | 100 | 3,140.3 | 0:28:59 | 0:04:49 | 2 | 2,313.1 Kb | 32 | 641 | 2313
603 | 98 | 3,141.0 | 0:29:04 | 0:04:43 | 2 | 2,315.0 Kb | 32 | 641 | 2315
605 | 96 | 3,141.6 | 0:29:09 | 0:04:37 | 2 | 2,313.3 Kb | 32 | 641 | 2313
606 | 95 | 3,141.9 | 0:29:14 | 0:04:35 | 2 | 2,315.1 Kb | 32 | 641 | 2315
608 | 93 | 3,142.5 | 0:29:20 | 0:04:29 | 2 | 2,313.4 Kb | 32 | 641 | 2313
610 | 91 | 3,143.1 | 0:29:25 | 0:04:23 | 2 | 2,315.5 Kb | 32 | 641 | 2315
611 | 90 | 3,143.4 | 0:29:30 | 0:04:20 | 2 | 2,313.7 Kb | 32 | 641 | 2313
613 | 88 | 3,144.0 | 0:29:35 | 0:04:14 | 2 | 2,315.8 Kb | 32 | 641 | 2315
615 | 86 | 3,144.7 | 0:29:40 | 0:04:08 | 2 | 2,314.3 Kb | 32 | 641 | 2314
616 | 85 | 3,145.0 | 0:29:45 | 0:04:06 | 2 | 2,316.2 Kb | 32 | 641 | 2316
618 | 83 | 3,145.7 | 0:29:50 | 0:04:00 | 2 | 2,314.5 Kb | 32 | 641 | 2314
620 | 81 | 3,146.5 | 0:29:55 | 0:03:54 | 2 | 2,316.7 Kb | 32 | 641 | 2316
621 | 80 | 3,146.7 | 0:30:00 | 0:03:51 | 2 | 2,314.3 Kb | 32 | 641 | 2314
623 | 78 | 3,147.3 | 0:30:05 | 0:03:46 | 2 | 2,316.2 Kb | 32 | 641 | 2316
625 | 76 | 3,147.8 | 0:30:10 | 0:03:40 | 2 | 2,314.6 Kb | 32 | 641 | 2314
626 | 75 | 3,148.1 | 0:30:15 | 0:03:37 | 2 | 2,316.4 Kb | 32 | 641 | 2316
628 | 73 | 3,148.6 | 0:30:20 | 0:03:31 | 2 | 2,314.8 Kb | 32 | 641 | 2314
630 | 71 | 3,149.2 | 0:30:25 | 0:03:25 | 2 | 2,316.8 Kb | 32 | 641 | 2316
631 | 70 | 3,149.4 | 0:30:30 | 0:03:23 | 2 | 2,315.0 Kb | 32 | 641 | 2315
633 | 68 | 3,150.0 | 0:30:35 | 0:03:17 | 2 | 2,317.3 Kb | 32 | 641 | 2317
635 | 66 | 3,150.7 | 0:30:40 | 0:03:11 | 2 | 2,315.6 Kb | 32 | 641 | 2315
636 | 65 | 3,151.0 | 0:30:45 | 0:03:08 | 2 | 2,317.5 Kb | 32 | 641 | 2317
638 | 63 | 3,151.7 | 0:30:51 | 0:03:02 | 2 | 2,316.1 Kb | 32 | 641 | 2316
640 | 61 | 3,152.4 | 0:30:56 | 0:02:56 | 2 | 2,318.3 Kb | 32 | 641 | 2318
641 | 60 | 3,152.7 | 0:31:01 | 0:02:54 | 2 | 2,316.1 Kb | 32 | 641 | 2316
643 | 58 | 3,153.3 | 0:31:06 | 0:02:48 | 2 | 2,317.9 Kb | 32 | 641 | 2317
645 | 56 | 3,153.8 | 0:31:11 | 0:02:42 | 2 | 2,316.2 Kb | 32 | 641 | 2316
646 | 55 | 3,154.1 | 0:31:16 | 0:02:39 | 2 | 2,318.1 Kb | 32 | 641 | 2318
648 | 53 | 3,154.7 | 0:31:21 | 0:02:33 | 2 | 2,316.4 Kb | 32 | 641 | 2316
650 | 51 | 3,155.2 | 0:31:26 | 0:02:28 | 2 | 2,318.4 Kb | 32 | 641 | 2318
651 | 50 | 3,155.5 | 0:31:31 | 0:02:25 | 2 | 2,316.6 Kb | 32 | 641 | 2316
653 | 48 | 3,156.1 | 0:31:36 | 0:02:19 | 2 | 2,318.6 Kb | 32 | 641 | 2318
655 | 46 | 3,156.6 | 0:31:41 | 0:02:13 | 2 | 2,319.0 Kb | 32 | 641 | 2319
656 | 45 | 3,156.9 | 0:31:46 | 0:02:10 | 2 | 2,317.2 Kb | 32 | 641 | 2317
658 | 43 | 3,157.5 | 0:31:51 | 0:02:04 | 2 | 2,319.3 Kb | 32 | 641 | 2319
660 | 41 | 3,158.2 | 0:31:56 | 0:01:59 | 2 | 2,317.7 Kb | 32 | 641 | 2317
661 | 40 | 3,158.5 | 0:32:01 | 0:01:56 | 2 | 2,319.3 Kb | 32 | 641 | 2319
663 | 38 | 3,159.0 | 0:32:06 | 0:01:50 | 2 | 2,317.6 Kb | 32 | 641 | 2317
665 | 36 | 3,159.6 | 0:32:11 | 0:01:44 | 2 | 2,319.6 Kb | 32 | 641 | 2319
666 | 35 | 3,159.9 | 0:32:17 | 0:01:41 | 2 | 2,317.8 Kb | 32 | 641 | 2317
668 | 33 | 3,160.4 | 0:32:22 | 0:01:35 | 2 | 2,319.8 Kb | 32 | 641 | 2319
670 | 31 | 3,161.0 | 0:32:27 | 0:01:30 | 2 | 2,318.1 Kb | 32 | 641 | 2318
671 | 30 | 3,161.3 | 0:32:32 | 0:01:27 | 2 | 2,320.0 Kb | 32 | 641 | 2320
673 | 28 | 3,161.9 | 0:32:37 | 0:01:21 | 2 | 2,318.4 Kb | 32 | 641 | 2318
675 | 26 | 3,162.5 | 0:32:42 | 0:01:15 | 2 | 2,320.4 Kb | 32 | 641 | 2320
676 | 25 | 3,162.7 | 0:32:47 | 0:01:12 | 2 | 2,318.7 Kb | 32 | 641 | 2318
678 | 23 | 3,163.4 | 0:32:52 | 0:01:06 | 2 | 2,320.7 Kb | 32 | 641 | 2320
680 | 21 | 3,164.0 | 0:32:57 | 0:01:01 | 2 | 2,319.1 Kb | 32 | 641 | 2319
681 | 20 | 3,164.3 | 0:33:02 | 0:00:58 | 2 | 2,320.9 Kb | 32 | 641 | 2320
683 | 18 | 3,164.9 | 0:33:07 | 0:00:52 | 2 | 2,319.2 Kb | 32 | 641 | 2319
685 | 16 | 3,165.4 | 0:33:12 | 0:00:46 | 2 | 2,321.3 Kb | 32 | 641 | 2321
686 | 15 | 3,165.7 | 0:33:17 | 0:00:43 | 2 | 2,319.4 Kb | 32 | 641 | 2319
688 | 13 | 3,166.3 | 0:33:22 | 0:00:37 | 2 | 2,321.4 Kb | 32 | 641 | 2321
690 | 11 | 3,166.8 | 0:33:27 | 0:00:32 | 2 | 2,319.7 Kb | 32 | 641 | 2319
691 | 10 | 3,167.1 | 0:33:32 | 0:00:29 | 2 | 2,321.6 Kb | 32 | 641 | 2321
693 | 8 | 3,167.7 | 0:33:38 | 0:00:23 | 2 | 2,319.9 Kb | 32 | 641 | 2319
695 | 6 | 3,168.3 | 0:33:43 | 0:00:17 | 2 | 2,322.0 Kb | 32 | 641 | 2322
696 | 5 | 3,168.5 | 0:33:48 | 0:00:14 | 2 | 2,320.2 Kb | 32 | 641 | 2320
698 | 3 | 3,169.1 | 0:33:53 | 0:00:08 | 2 | 2,322.3 Kb | 32 | 641 | 2322
700 | 1 | 3,169.7 | 0:33:58 | 0:00:02 | 2 | 2,320.5 Kb | 32 | 641 | 2320
701 | 0 | 3,170.0 | 0:34:01 | 0:00:00 | 3 | 2,019.5 Kb | 32 | 641 | 2019
<h4>Crawling phase completed</h4>Total pages indexed: 32
<br>Creating sitemaps...
<div id="percprog"></div>
Creating HTML sitemap...<div id="percprog2"></div> |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
 |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
 |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
 |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
<br />Done, redirecting to sitemap view page. <script> top.location = 'index.php?op=view' </script>


The output looks good.

If the output looks good, any idea why only 32 pages are added to the sitemap?

What is your sitemap URL?

1. if you r using web cron try to reload page (Use Crone job instead)
 
2. Make all page internally connected to each other.
example :
website need to have href link to appllication 1,2,3 and application need to have link to pages in directory, that help to scrpit to add in sitemap.

I have the same problem here [ External links are visible to forum administrators only ] I have more 10,000 pages and PHP Script detected only 1300

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 

Hello, my host provider tell me, does not allow me to increase the time detection.

i have over 10,000 pages and php script detected only 1300, my host provider suggests me to convert sitemap.xml to sitemap.xml.gz and maximum 1000 pages for every file, to become multiple sitemaps. Example for 10,000 pages become 10 sitemaps.

This is a good solution or not?

Although I have chosen to create a sitemap.xml.gz but the php script not created this format sitemap. Do i need to create a blank folder sitemap.xml.gz in public html?

Hello, 

in this case I would recommend to run generator in command line if you have ssh access to your server.

>Do i need to create a blank folder sitemap.xml.gz in public html?

This is not required.

Yeah I have the same problem

My site has 2.5 million pages. SOmetimes it will build to 40k then 80k biggest I got was 400k. Jut cant get it to build the full run.

Have tried various settings etc. Frustrting as the last one ran for nearly 2 days then stopped