what is this?

this article is intended for beginners to SEO to gain an understanding of some of the ways search engines work, and how they have changed, in a very short descriptive way. it will give you some indication of how to affect your position in their listings and shortly explain the methodology behind search engine positioning. a lot of this article was written in note form, and although ive tried to go into more depth with most of the information, it may still appear briefer than you might like, therefore ive added a further reading section towards the bottom that will let you read further into each point ive opended up in this article.

• robot

  this is a program that automatically traverses a pages structure and content on the internet. also known as crawlers, spiders and bots, it simply visits a website and requests documents from them, referenced within the first page it comes to. WebAnts are distrubuted co-operating robots.

• crawl/crawled

  a term used in this article to simplify the detail in which a robot visits a website, indexes all referenced pages and archives it into their search engine database.

• backlink

  a backlink is best explained with an example: when SiteA references SiteB with a anchor tag in its html, SiteB has one backlink.

• PR

  an acronym for PageRank, a number out of 10 that Google assigns to a page it has 'crawled'. it refers to the importance of the page based on their number and quality of 'backlinks'.

• SEO

  an acronym for Search Engine Optimisation, simply its the term for the methodology in which we change our structure and content of pages to appear higher in a search engines results for specific search terms.

1. Specific search engine information

   • search.msn
   • Google
   • Yahoo!
   • directories

2. SEO techniques

   • content
   • structure
   • other sources
   • submissions
   • robots

3. Further reading

   • resources

Search.MSN

MSN will eventually use their brand new MSN directory which they have been starting from scratch. MSN follows Overture & as such they like title tags & meta keywords (the same way Yahoo does).

MSN used to use the human edited directory, Zeal (owned by LookSmart) for results. It is now powered by Overture's (previously Inktomi) results and doesn't use their prototype MSNBot's indexes in any of their results. Yet.

Microsoft's new MSNBot is currently indexing the internet at an impressive rate.

Google

Google has its own GoogleBot to provide results for their SE, More importantly; Pagerank affects Google's relevancy listings.

For a new site, PR can take time to appear, with a few backlinks in the first couple of months for a new page, it's PR should be around 2 but its not uncommon for many pages to stay at 0. PR updates are infrequent and inconsistent because of the number of name servers dishing out the numbers.

Pagerank (PR) is roughly calculated by backlinks, quality of backlinks and also links from the page to outside sources. If a high PR page contains hundreds of links to PRzero pages off the domain, the pagerank of the page is likely to decrease and the linked pages to increase because of the way pagerank is calculated.

Ranking in Google is primarily based on back links and the relevancy of the text its linking with. The higher the PR the higher you are going to appear, but this must also tie in with title and meta keywords tags to target specific search terms.

Yahoo!

Yahoo uses Inktomi's old robot, slurp. It is common belief that Slurp requires robots.txt to crawl a site, buts its not the case. Infact no robot requires robots.txt in your server root to crawl your site, its merely a way to gain some control and where and when your pages are indexed. You can also get a free listing in the Yahoo directory. The search index and the directory still operate differently.

Yahoo likes keywords and titles, specifically keywords in text, and repetition in alt and title tags. It also takes into account the page optimization and weight a lot more than other major search engines. Backlinked anchor text is important for listings and the number and quality of the backlinks. Currently the Yahoo bot is not as aggressive as googlebot so changes in results will take time.

Yahoo also has morgue bot which checks your site for dead pages.

Directories

ODP/DMOZ.org - Getting listed in the open directory project isn't as important as it was a couple of years ago, since google's use of its own directory and the fact that more major search engines are better marketed, it will, for the majority of the time, increase your traffic as any other backlink would. ODP has become just one of the more important directories, as opposed to the major listing source it used to be

The Yahoo directory is a payed inclusion but it can literally double a pages traffic overnight. For many sites, particularly in the US, the Yahoo directory accounts for almost half their traffic. MSN used to use the Looksmart directory for listings but since Overtures taking of Inktomi (and partnership with Yahoo!), its been using overture as a source for their sponsored listings and search results.

Looksmart recently closed its UK operations and UKDirectory acquired the listings.

Content

content is definetly king for search engines. your content should contain your targeted keywords also but try to write it for the user and not the search engine. good content will always equal good specific rankings and relevant traffic. keyword density is hard to judge within content, too high and it may be classed as a spam page and banned from the listings, too little and the page's competitors will beat it in ranking. 6% is a good benchmark for keyword density, or about 1 in every 15 words. bolding and italising keywords within content adds weight to the keyword.

header tags let the bot know that there is probably an important page title in the text in between, and in the same way with bold tags and anchors. Your should also use alt text and titles on your links to increase keyword density on your pages

Its also a good idea to repeat the description of your site to be the first peice of text read by the robot, making sure it has a few of your most important keywords in there aswell. this is because a lot of search engines pick out parts of your content containing the keywords rather than the description in the meta description tag.

Structure

while building your page you should think about how a spider will read the page. If you remove all images and all tags except for italic and bold styling, anchors, titles, headers and meta tags you'll get exactly what most major search bots index when coming across your page. The tags aren't necessarily followed through to the search results but give the robot a good impression of what weight to give the text in between them.

the title tag should be descriptive of the website and the current page, but kept short. Below 60 chars is recommend. And keep in mind that this is the piece of text that will attract the users mouse to click once your result shows up. The keywords and description should also be relevant to the site as a whole, and the current page.

Recently more websites are also giving importance to pages with semantic markup and style seperation, specifically Yahoo!, plus this method also makes it easier to get the page's content indexed in a better structured way.

Other Sources

anchor tags can have quite a good effect on your position within listings. relavant wording in backlinks can help the pages position stay high because the keywods in the anchor text give the backlink more weight and therefore give the page more importance. backlinks are no doubt the most effective way of gaining position on a competitor, but without good content, specific titles and relevant keywords, the pages importance would be useless.

Submissions

submission to search engines are usually only nessecary for new sites or search engines with a smaller database and less frequent crawls. over submission doesn't do any harm, but probably doesn't do much good either. for a new site, once a fortnight is adequate. submissions to directories are a good way to build up a page's backlinks without having the return the favour of a reciprocal link.

Overture uses MSN and Yahoo! to do pay-per-click results, this is a payed inclusion where you can specify your own title, description and URL to be displayed above every other result for your specified keyword. the higher the bid per click, the higher the page will rank within the sponsored lisitings. There are many other PPC (pay per click) search engines and sponsored links (such as Google's, AdWords). there are also payed inclusion services by overture and many others which guarantee's a URL to be crawled, but these are just as effective as free submission except for the time time it takes to be inlcuding.

Robots

Like any meta tag, the meta robots tag should be placed in the head section of an HTML page. The content of the meta robots tag contains directives separated by commas. The currently defined directives are [NO]INDEX and [NO]FOLLOW.



You can also use a robots.txt file in the server root to allow/disallow certain robots and set delays for how often it should retrieve a referenced page. for example...

User-agent: msnbot
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
Crawl-Delay: 20

the text above, placed in the robots text file, will not allow msnbot to enter the directories stated, and will retrieve a page at the rate of one every 20 seconds. To exclude all files except one is a bit awkward as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "docs", and leave the one file in the level above this directory.

Resource

www.searchguild.com - the best SEO forum on the internet, ask your question and get an answer.
www.robotstxt.org - in depth information in the use of robots.txt file and the robots meta tag.
www.webmaster-toolkit.com - an excellent resource for search engine tools and information.
substrukt - home, and where this article is updated more often.