Search engines 'struggling' to index the Internet

The article, Search Engines: Evolution and Diffusion, says that the market is far from limitless and that firms relying on seaches are unlikely to generate revenues above $200 million unless they introduce pay schemes, such as Overture's.

That's important because the cost of indexing is sky-rocketing, the authors claim.

The paper also warns that natural language processing is still a way away, especially where huge content is involved.

The article, written by Harry Collier and Stephen E. Arnold, reckons that content is growing much quicker than any machine or human hybrid can cope with.

It's hard to make software behave like humans and rule based systems fall over when there's a mixture of media, including PDFs, Powerpoint files, text, radio and video.

One single index entry can cost between $3 and $50, the authors claim.

More in this PDF, here. µ