Latent Semantic Indexing
Search Engines are beginning to use more sophisticated systems for measuring the relevance of pages to keyword terms. Up until recently a page/document either contained a given word, or it didn't with no middle ground - the page was then ranked using a system that looked at the frequency of the use of the word (too low penalised, too high penalised) and the position of the word. each page / document would be looked at individually
Latent semantic indexing adds an important step to the page/document indexing process. In addition to recording which keywords are on a page, or in a document, the method examines a collection of pages as a whole, to see which other pages contain some of those same words. LSI considers pages that have many words in common to be semantically close, and ones with few words in common to be semantically distant.
This simple method correlates surprisingly well with how a human being, looking at content, might classify a collection of pages. Although the LSI algorithm doesn't understand anything about what the words mean, the patterns it notices can make it seem astonishingly intelligent.
As well as putting the relevant keywords onto web pages we can now build a "Theme" for the page using Antonyms and Synonyms of thes words, without being penalised for over use of the keywords