Optimizing Dynamic Sites

by Rick Archer

Dynamic content is delivered to the Web browser in a different form than it exists on the server, while static content is stored on the Web server in the same format that is delivered to the Web browser. Dynamic site pages are generated from a database ³on the fly² as users request them.

You can often tell when you are looking at a dynamically generated page, because dynamic URLs contain one or more "query strings," or question marks (?) while static URLs do not, but there are exceptions to this rule, which we shall discuss below.

Search engines have a hard time with dynamic URLs. Dynamic URLs may cause search engines to mistake a small site for a very large one because an unlimited number of URLs can be used to provide essentially the same content. This can cause search engine spiders to avoid such sites for fear of falling into "dynamic spider traps," crawling through thousands of URLs when only a few are needed to represent the available content. Here's how three popular search engines handle this:

* The FAST search engine will crawl and index dynamic URLs as quickly and easily as static ones (at this time). * AltaVista doesn't crawl dynamic URLs at all, but it will index each dynamic URL that you take the time to submit individually. * But these two search engines are relatively insignificant. Google will crawl dynamic URL's at about a third the speed and depth at which it indexes static pages. It will barely crawl at all if there are session IDs in the query strings, because it will soon discover that multiple URLs lead to the same page and regard the site as being full of duplicate content.

Another challenge dynamic sites throw at search engines is serving up different core content at the same URL. This might result when a site has content that may be viewed at the same URL in multiple languages, depending on the browser settings, or content, such as on a news site, which changes every few minutes.

Search engines want to be accurate ­ they want visitors to a particular URL to see the same content the spider saw. They also want to be comprehensive. They vie with each other to have the largest database. Thus, they have billions of pages to index and typically can only visit each URL once every few weeks or so (although Google is pretty good at recognizing content that changes frequently, and spidering it more often). So if a search engine indexes your English content at a given URL, it will probably not index your Spanish content at the same URL during the same indexing period.

The solution is to give each search engine unique core content at a unique URL, and ensure that all visitors see the same core content. There are three main ways of achieving this.

1) Use static URLs to reference dynamic content. If a search engine sees a static URL, it is more likely to index the content at that URL than if it found the same content at a dynamic URL. There are several ways of turning dynamic URLs into static URLs, despite the fact that you are serving dynamic content. Your method will depend upon your server and other factors. A friend of mine had the following experience after implementing this solution for a client: > "For the last year, since rewriting the dynamic URLs, my client's site has > been riding high in the rankings for thousands of search terms. Before the URL > rewriting, Google had indexed just about 3,000 pages in the course of 18 > months, on the first week of using URL rewriting, Google was grabbing 3,000 > pages per day from the 500,000-item database it had previously barely touched. > By the end of the first 2 months of using URL rewriting, Google had indexed > over 200,000 pages from the site."

The following sites offer instructions for two popular servers: * Apache: * ASP:

A good step-by-step tutorial can be found at fantomaster.com. The article links are on the right hand side. There are four articles in the series.

Here are some examples of sites that have implemented one of these approaches: * Yahoo.com (yes, Yahoo!) * Epinions.com * Dooyoo.co.uk * Pricerunner.com

URL rewriting is a very common practice. Not only is it exceptionally powerful in terms of search engine optimization, but it is also superb for usability and marketing in general. A shorter, more logical-seeming URL is far easier for people to pass on in an email, link to from their homepage, or spell out to a friend on the telephone. Shorter URLs are good business.

2) Link to dynamic URLs from static URL pages. The above solutions are elegant, but may be difficult for some sites to implement. Fortunately, there is a simple work around for smaller sites.

One method search engines use to crawl dynamic content while avoiding dynamic spider traps is to follow links to dynamic URLs from static URLs. If your site isn't too large, you could build a static site map page consisting of links to dynamic URLs. The search engines should crawl those links, but will probably go no further.

An even more effective technique would be to get other sites to link to your dynamic pages. If these sites have good Google PageRank, your dynamic pages will not only be indexed, but the likelihood of their achieving a high ranking for the key words on them will increase significantly.

3) Pay for inclusion? AltaVista, Ask Jeeves/TEOMA, FAST and Inktomi offer Pay-per-inclusion (PPI) programs. You pay $25/page (or so) to ensure that that page is spidered frequently (Inktomi spiders every 48 hours for that price). This will garner some traffic, but since Google now accounts for over 70% of all search engine traffic and continues to grow stronger all the time, don't throw too much money into this solution unless you have deep pockets. If your site is huge, the cost could be prohibitive. Paying to have your pages spidered does not guarantee that they will rank well, so they must be optimized properly. Frequent spidering enables you to experiment with optimization and see your results within a day or two. Search engines, including those with PPI options, want their databases to be as large as possible. So if you don't pay for inclusion, and instead implement one of the solutions discussed above, your pages will probably be indexed anyway. On the other hand, if you pay for some of your pages to be spidered, there's a good chance the ones you don't pay for won't be.

To summarize: 1. Search engines have problems indexing dynamic content. 2. If possible, use static URLs to reference dynamic content. 3. Otherwise, try to link to your dynamic URLs from static pages. 4. If your budget allows, consider using paid-inclusion programs.

© Rick Archer 2004. Rick Archer owns and operates SearchSummit, a one-man SEO company. His clients include many site owners who used to employ large SEO firms, but found that a personal, direct approach was preferable. Visit his website at http://www.searchsummit.com

More SEO articles: