Practical Chip Design

I had the privilege yesterday evening, courtesy of angel-investor group Silicom Ventures, of listening to a panel of very eminent professionals from the search engine world talking about the future of search technology. They had many cogent things to say about where search was going, including these important points. First, search results are enormously improved by reducing the range of the search to a specific topic area—say, software development or invertebrate biology or Indian restaurants in San Francisco. Second, search results are improved enormously by recording the post-search behavior of prior users who have made similar searches. This of course is the basis of the famous (or infamous) Amazon “customers who bought this book also bought …” feature. In fact Steve Larsen, CEO of specialized search start-up Krugle, said that Amazon sometimes has to dial back their so-called collaborative filtering algorithm because it becomes so good at predicting that customers find it uncomfortable.

Third, the user interface on search engines is hopelessly archaic. Text entry of keywords, for heaven’s sake. A single text list of results, with only priority ranking and no other form of indexing. Only very modest text previews of the result pages. All of that is 1960s stuff, of the same generation as the text-based adventure games we used to play on teletype machines.

A more controversial point was that the whole concept of text-matching, even with page ranking thrown in, is inadequate. Search needs to extract meaning, not keywords, from target pages and from queries, and match the meanings, at least in the view of Powerset, Inc. CEO Barney Pell, who has bet a good deal of money on this assertion.

All of this was quite interesting. But the thing that interested me most was the remarkable bias of all the speakers. Two were venture capitalists, and hence can be excused for a rather myopic view of systems design. But two were computer science PhDs, and this concerns me.

The myopia is this. All the panelists clearly think of search as a software problem, not a systems problem. When they discuss the future of the technology, they talk in terms of algorithms, always implicitly assuming that the algorithms are coded in something C-ish and run on increasingly preposterous giant server farms.