XML Sitemaps Generator

Author Topic: I have about 15,000 dynamic pages that need a place in my sitemap  (Read 11543 times)

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Hello

I have about 15,000 dynamic pages that need a place in my sitemap
I just bought this software today and having a hard time getting it to complete without some kind of error.

Any ideas on why and how I can set it up for faster and more reliable operation?
[external links are visible to admins only]

All dynamic pages start with /idx/ ... An example is
[external links are visible to admins only]

Typical error is:

Sat Mar 12 2011 20:27:14 GMT-0500 (Eastern Standard Time): resuming generator (120 seconds with no response)

Or some kind of browser crash error like this in Firefox:
"Internal Server Error. The server encountered an internal error or misconfiguration and was unable to complete your request."

Many thanks
« Last Edit: March 13, 2011, 01:53:33 AM by grahamginsberg »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #1 on: March 13, 2011, 01:55:25 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

Also, in your case I'd add "ids/mls-" in "Do not parse" setting to add those URLs in sitemap directly, for better performance.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #2 on: March 13, 2011, 05:38:46 PM »
Hello

As far as I know, the hosting company, who are very flexible in their configuration, has maximums set, so that shouldn't be a problme.

I notice you added the exclusion but it should be idX not ids, but it didn't make a difference and still bombed out

Any other ideas?

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #3 on: March 13, 2011, 07:41:54 PM »
I have kept all variables except the 'max depth'
I started at 2-completed sitemap

I ended at 5 and it too completed the sitemmap, but it bombs out after 5 (ie 6,7,8...)

Now 5 has 5656 pages on the sitemap, so I'm still missing 10,000 pages  8)

Any ideas of adjusting any more parameters?

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #4 on: March 13, 2011, 11:26:59 PM »
 ;) ok I have it up to 6700 pages now, but still a way to go

I'm still looking for advise on how to reduce the load on this search.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #5 on: March 14, 2011, 10:48:53 AM »
Replied to your email.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #6 on: March 14, 2011, 02:11:36 PM »
http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html

In the refund page:
This product appears to be "not-as-described".

Why?

4.It will work for all sites including dynamic websites such as portals, blogs, forums, and online stores.
6.Support of LARGE websites, dividing the sitemap on the parts per 50,000 URLs each and creates a Sitemap Index file according to the Google sitemap protocol.

NO MENTION IS MADE OF EXCESIVE SCRIPT LOAD TIMES WHICH WILL NOT WORK ON 90% OF WEBSITES

HOSTGATOR LIMITS 30SEC SCRIPT RUN TIME

THIS PROBLEM, AND IT IS A PROBLEM, SHOULD BE STATED ON THE PAGE
http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html
_____________________________________________________________
11. SITES WITH MANY PAGES MAY CAUSE OUR SCRIPT TO TAKE A LONGER TO COMPLETE LOADING, IN WHICH CASE VPS OR DEDICATED SERVERS ARE SUGGESTED.
_____________________________________________________________

This needs to be added, because you're mass marketing a product that has very limited use and a poor refund policy and even worse tech support!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #7 on: March 14, 2011, 02:56:15 PM »
Hello,

mm.. a sitemap with 8,000+ URLs has been created by sitemap generator on your website already.
Example URLs that were not included in sitemap and mentioned in your email were not added because you have incorrect canonical meta tags on those pages. You can disable "canonical meta tags support" feature in sitemap generator, but it's better to correct them on your site since that might affect indexing in search engines.
If there are any other URLs that are not added in sitemap, please let me know.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

djohnson2323

  • Approved member
  • *
  • Posts: 5
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #8 on: March 15, 2011, 09:04:30 PM »
I have a very similar question, but I must admit I enjoy your product and am thankful for the support you provide.  Here is my issue.  It appears as if many pages are scanned, but why would they all not be added to the sitemap?

Pages added to sitemap: 5708
Pages scanned: 14880 (291,287.9 KB)


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #9 on: March 15, 2011, 11:36:37 PM »
Some pages are excluded if you have "Exclude URLs" setting applied, some excluded according to your robots.txt file, robots meta tag, canonical meta tag etc.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

djohnson2323

  • Approved member
  • *
  • Posts: 5
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #10 on: March 15, 2011, 11:40:39 PM »
Would you be able to guide me regarding what is creating these pages from not being indexed.  As a heads up I do not believe in wasting your time as I can see how precise you are in helping people  and that is the reason I have come to your product.  For about 15 years I have gained about 60K clients that need this type of product and am simply trying to perfect it so I can send off recommendations (by the way not a sales pitch to get you to help me :) )

but... knowing why these links do not get included will be by biggest resolve.  thank you for the consideration in helping.
David

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #11 on: March 15, 2011, 11:43:07 PM »
Hello,

please let me know your generator URL/login to check this: http://www.xml-sitemaps.com/contact.html
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

djohnson2323

  • Approved member
  • *
  • Posts: 5
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #12 on: March 16, 2011, 12:07:57 AM »
thank you I have sent an email.
David

grahamginsberg

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #13 on: March 17, 2011, 02:13:21 AM »
 :)
thanks for your help thus far Oleg. You have amazing software. It just takes some getting used to. I changed hoting companies just for this script to run well, now all 19,000 pages are in one sitemap, first time and very happy about thses dynamic pages in a sitemap.

thanks again  :D

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: I have about 15,000 dynamic pages that need a place in my sitemap
« Reply #14 on: March 17, 2011, 11:59:19 AM »
Thank you for your kind words, I'm glad it works fine for you now!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2