HAVING BEEN ignominiously turned down by Yahoo! in its attempt to gain ground in the search arena, Microsoft has tried a different approach and stumped up a rumoured $100 million for start-up semantic search company Powerset.

The purchase, which is $44.5 billion less than what Microsoft was prepared to pay for Yahoo!, might signal a long overdue change in the Vole’s search strategy. By investing in semantics, a technology notoriously difficult to master and apply successfully, but which holds the promise of revolutionising search, the firm appears to be supporting fresh and innovative ideas, rather than playing the usual game of ramming ever increasing quantities of ads down the throats of users who want nothing more than relevant results. Could Microsoft actually be listening to the wishes of wibblers everywhere?

Backing semantics and natural language processing is a risky business, as there is no guarantee that it will work on a massive scale as effectively as existing keyword matching engines or that users will be prepared to re-evaluate their search habits. Google, for one, see little benefit in the concept, as Peter Norvig, head of research testifies. Similarly, critics argue that the Resource Description Framework (RDF) developed by the World Wide Web Consortium, which allows the compilation of semantic information on the net, is just too tough for most webmasters to implement. Powerset’s current foray into semantic search has been limited to the closed and predictable environment of Wikipedia, where the resource intensive tools required to index information using natural language processing aren’t stretched too much: semantic scans of web pages take a lot longer and use far more computational power than traditional techniques.

The benefits can be impressive though, correcting what Microsoft sees as the major issue with traditional key word searches, that a huge number of searches don’t deliver meaningful or relevant results on the first few attempts. The problem, according to the Live Search blog, is that search engines do not understand and deal with language in the same way as humans, meaning that associated words like ‘chair’ and ‘bench’ are entirely autonomous in search terms. Likewise, operators like ‘who’, ‘what’, ‘where’, ‘how’ ‘why’ and ‘when’ are basically useless. Using semantics, users can ask questions that can be answered directly, and have relevant information displayed even if it doesn’t contain the exact terms searched for in the first place. Context, intent, and abstract meaning are analysed to deliver results offering greater accuracy and richer associations, which, if properly managed, could make searching the web a quicker, easier and more intuitive process.

Before Microsoft’s offer, Powerset had managed to raise around $12million worth of investment to get its pioneering technology off the ground. The money allowed the company to develop their engine and demonstrate that the semantic technique worked, but it was clear that far larger investments were needed to expand indexing beyond the confines of Wikipedia. Enter Microsoft, desperate to improve its paltry share in the search market, with lots of cash at its disposal, and perhaps a little bullish after the embarrassing mess that became of the Yahoo! bid. Initiatives such as the Live Search cashback scheme would never deliver particularly impressive or long-term gains in market share, and in fact only resulted in widespread criticism and ridicule for an inability to improve search in any remotely imaginative way.

Partnering with a young outfit like Powerset is a step in the right direction that’s mutually beneficial to both parties. Microsoft can finally appear to be investing some inventive thinking into their search strategy, while hopefully improving search performance too. Powerset get the money they so sorely need to research, develop and expand into the wider web, overcoming the gargantuan engineering and computing tasks they face. Results won’t materialise overnight, but at least the groundwork has already been laid for real progress to begin on the task of making Microsoft search into a real competitor to the likes of Google, who remain light years ahead in terms of market share. It is probable that semantic searches will be limited at first to specific areas of information, such as on the microsoft.com website itself, as an Office extension or, as one bogger suggests, on the Microsoft knowledge base. Doing so would of course be much quicker to roll out, and allow users to get used to the concept and nuances of semantic search while PowerVole irons out the kinks. Great for letting people get a taste of what’s to come, and potentially providing tangible improvements to existing services too, provided it works well.

It’s heartening to see Microsoft taking a novel direction in an area it has lagged so far behind in for so long, one which smacks less of desperation and misjudgement of the user base than some recent efforts. Truly it would be a rare sight indeed if they can deliver a service that actually shakes things up and comes out ahead of the competition. In a market dominated so completely by a single company, competition would be welcome to keep things progressing, and usher in further innovations. Let’s just hope that Powerset isn’t completely subsumed and stifled what with the hundreds of millions of dollars flying around, and that the innovations driven by being a small and underfunded start-up can continue in the heady world of the mass market. µ