XML Sitemaps Generator

Free Online
Sitemap Generator
Unlimited
Sitemap Generator

About Sitemaps

Broken Links

Forum

Testimonials

Tools

Contact
    Advanced search
Sitemap Generator Forum
July 19, 2008, 04:53:48 AM
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
   Home   Help Search Login Register  
Sitemap software 2.9 released - Email notifications, html sitemap customizing and more
6804 Posts in 1674 Topics by Members
Latest Member: Question Mark
Pages: [1]
« previous next »
  Print  
Author Topic: Question  (Read 4920 times)
FlipediS
Jr. Member
**
Posts: 10


View Profile
« on: December 22, 2007, 03:08:13 AM »
Aloha,  I had a question,  I been trying to run the site map on my page but is taking forever.. been going at it for two days,  it goes for a few hours and stops... and the last two days so far it has scanned 2400 and has over 15,000 more to go.  and keeps finding more... is there any tricks to make this faster or any settings or anything? 
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #1 on: December 22, 2007, 07:47:25 AM »
Hello,

the most useful way to increase crawling speed is to configure "Do not parse" and "Exclude URLs" options. The first one defines which pages should not be *fetched* from website by the crawler (which is the most time consuming task), but they still *will* be included in sitemap. The second option will disallows including URLs in sitemap.
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #2 on: December 23, 2007, 08:20:42 PM »
Sorry im a but new so,  what do you mean fetched but still included?
I have also noticed when its running it adds a bunch to a shopping cart.  Is there a way to stop this?  and when i use exclude do i use the whole path?  lets say i have the starting patch at [external links are visible to admins only]  and i was to exclude  lets say [external links are visible to admins only]  would i place it in the exclude like that?
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #3 on: December 23, 2007, 09:05:06 PM »
1. Let me provide an example to illustrate that option.
Let's say the site structure is as following:

1. homepage:
domain.com/
2. homepage contains links to categories like:
domain.com/cat-1.html
domain.com/cat-2.html
etc.
3. every category page contains links to "entries" like:
domain.com/entry-1.html
domain.com/entry-2.html
domain.com/entry-3.html

Sitemap Generator will try to crawl ALL the pages it finds (all listed above). "Crawl" means request them from the server via http, similar to how the browser requests them.

However, you can add this to the "Do not parse" option:
Code:
entry-
That will tell sitemap generator NOT to request entry pages. The entry URLs will still be included in sitemap since "category" links are crawled and they contain links to entries.

Does it make sense?

2. You can exclude any SET of URLs by adding a *substring* that is common to those URLs, i.e. add this *both* to "Do not parse" and "Exclude URLs" options:
Code:
Shop/
That will exclude all urls in Shop/ folder.
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #4 on: December 23, 2007, 11:34:09 PM »
So what you mean by using the code entry in the do not phase would actually be the product name instead?  If so that would take even longer because there are like 16k products.  or am i miss understand? 
My root directory is [external links are visible to admins only]  ,  What im trying to do is build a good site map for google and other search engines
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #5 on: December 25, 2007, 11:31:14 PM »
No, you should add the string that is common to all product URLs, in your case this should work:
Code:
.html
also, you can exclude review pages with:
Code:
reviews/
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #6 on: December 27, 2007, 05:16:19 AM »
Thanks,  Now im running in to another problem.. it seems to run for a while then stop working... then i try to contnuie  and it seems to stop at the same place every time.. so then i ran in in PuTTy to try it there and it stoped at the same place and gave me this error
Fatal error: Out of memory (allocated 534773760) (tried to allocate 1228658 byte              s) in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2              ) : eval()'d code on line 6
  why am i getting this error?
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #7 on: December 27, 2007, 07:33:35 PM »
How many URLs do you have crawled at that point? How does the memory usage change when crawler is in progress?
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #8 on: December 28, 2007, 01:39:48 AM »
It seems to stop at diffreent amount of pages each time a start a fresh scan from the begining.  but i after it dose stop and i continue an previous scan that scan never makes it past what one it prevoulsy stoped on.  As for the memory.  How would i be abel to tell this?  sorry not famliar with that. 
this is what it said today when i ran it
<head>
<title>XML Sitemaps - Generation</title>
<meta http-equiv="Content-type" content="text/html;charset=iso-8859-15" />
<link rel=stylesheet type="text/css" href="pages/style.css">
</head>
<body>
Resuming the last session (last updated: 2007-12-26 21:46:24)200 | 60 | 7,288.6 | 9:25 | 2:49 | 2 | - | 163 | 797 | 0
220 | 40 | 8,547.3 | 10:39 | 1:56 | 2 | - | 183 | 1095 | 0
240 | 20 | 9,800.9 | 11:51 | 0:59 | 2 | - | 203 | 1414 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
260 | 1324 | 10,976.8 | 12:58 | 66:03 | 3 | - | 223 | 0 | 0
280 | 1304 | 11,536.2 | 13:38 | 63:33 | 3 | - | 237 | 279 | 0
300 | 1284 | 11,536.2 | 13:47 | 58:59 | 3 | - | 248 | 279 | 0
320 | 1264 | 11,536.2 | 13:53 | 54:52 | 3 | - | 259 | 279 | 0
340 | 1244 | 11,536.2 | 13:59 | 51:11 | 3 | - | 270 | 279 | 0
360 | 1224 | 11,536.2 | 14:04 | 47:51 | 3 | - | 280 | 279 | 0
380 | 1204 | 11,536.2 | 14:08 | 44:49 | 3 | - | 292 | 279 | 0
400 | 1184 | 11,536.2 | 14:13 | 42:06 | 3 | - | 302 | 279 | 0
420 | 1164 | 11,536.2 | 14:19 | 39:41 | 3 | - | 314 | 279 | 0
440 | 1144 | 11,536.2 | 14:23 | 37:24 | 3 | - | 326 | 279 | 0
460 | 1124 | 13,369.2 | 17:31 | 42:49 | 3 | - | 346 | 829 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
480 | 1104 | 15,255.8 | 20:46 | 47:46 | 3 | - | 366 | 1195 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
500 | 1084 | 17,246.0 | 24:20 | 52:47 | 3 | - | 386 | 1745 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
520 | 1064 | 19,141.7 | 27:48 | 56:53 | 3 | - | 406 | 2125 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
540 | 1044 | 21,033.1 | 31:20 | 60:36 | 3 | - | 426 | 2480 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
560 | 1024 | 23,003.5 | 35:25 | 64:47 | 3 | - | 446 | 2895 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
580 | 1004 | 24,942.7 | 38:59 | 67:30 | 3 | - | 466 | 3309 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
600 | 984 | 26,938.1 | 42:41 | 70:00 | 3 | - | 486 | 3913 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
620 | 964 | 28,871.6 | 46:01 | 71:33 | 3 | - | 506 | 4185 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
640 | 944 | 30,891.5 | 49:30 | 73:01 | 3 | - | 526 | 4606 | 0

Fatal error: Out of memory (allocated 533200896) (tried to allocate 1218213 bytes) in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6
[FlipediS@mgmt-01 ~]$
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #9 on: December 28, 2007, 03:14:45 AM »
Hello,

does it also happen via web interface?
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #10 on: December 28, 2007, 05:41:42 AM »
Yes it dose.  i don't get any error messages.  Just runs and seems to get stuck after a while...  thats why i decied to try it in PuTTy
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #11 on: December 29, 2007, 12:45:03 AM »
Can you provide access to your host for troubleshooting?
http://www.xml-sitemaps.com/contact.html
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
FlipediS
Jr. Member
**
Posts: 10


View Profile
« Reply #12 on: December 30, 2007, 05:56:22 AM »
did you get my e-mail?
Logged
admin
Administrator
Hero Member
*****
Posts: 2837


View Profile
« Reply #13 on: December 30, 2007, 11:40:44 PM »
No, I can't find it. Please use forums username/password that was sent to you in automated email with download link after purchase and send me details via private message here on the forum.

Thank you.
Logged
Oleg Ignatiuk
http://www.xml-sitemaps.com
Pages: [1]
  Print  
« previous next »
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.5 | SMF © 2006, Simple Machines LLC Valid XHTML 1.0! Valid CSS!
Total sitemaps: 2,570,142
Pages: 334,258,153
Bandwidth usage: 8,275.6Gb

The New PHP XML Sitemap creator script is available now with unlimited number of URLs per sitemap!
Copyright (c)2005-2007 XML Sitemaps | Contact | Privacy Policy | Affiliates | Web design by anholm.com