Search engines, software developers and systems engineers

I had the privilege yesterday evening, courtesy of angel-investor group Silicom Ventures, of listening to a panel of very eminent professionals from the search engine world talking about the future of search technology. They had many cogent things to say about where search was going, including these important points. First, search results are enormously improved by reducing the range of the search to a specific topic area—say, software development or invertebrate biology or Indian restaurants in San Francisco. Second, search results are improved enormously by recording the post-search behavior of prior users who have made similar searches. This of course is the basis of the famous (or infamous) Amazon “customers who bought this book also bought …” feature. In fact Steve Larsen, CEO of specialized search start-up Krugle, said that Amazon sometimes has to dial back their so-called collaborative filtering algorithm because it becomes so good at predicting that customers find it uncomfortable.

Third, the user interface on search engines is hopelessly archaic. Text entry of keywords, for heaven’s sake. A single text list of results, with only priority ranking and no other form of indexing. Only very modest text previews of the result pages. All of that is 1960s stuff, of the same generation as the text-based adventure games we used to play on teletype machines.

A more controversial point was that the whole concept of text-matching, even with page ranking thrown in, is inadequate. Search needs to extract meaning, not keywords, from target pages and from queries, and match the meanings, at least in the view of Powerset, Inc. CEO Barney Pell, who has bet a good deal of money on this assertion.

All of this was quite interesting. But the thing that interested me most was the remarkable bias of all the speakers. Two were venture capitalists, and hence can be excused for a rather myopic view of systems design. But two were computer science PhDs, and this concerns me.

The myopia is this. All the panelists clearly think of search as a software problem, not a systems problem. When they discuss the future of the technology, they talk in terms of algorithms, always implicitly assuming that the algorithms are coded in something C-ish and run on increasingly preposterous giant server farms.

My conclusion, and I hope I’m being vastly unjust here, is that even in very fine institutions, computer science still means programming technology, neither really about computers nor really science. There doesn’t seem to be the mindset, even among fine CS graduates, to address complex problems as systems issues—to assume that the system hardware, topology and software will have to be developed concurrently. The hardware is a given—multicore CPUs from Intel or AMD. The topology is a given—whatever the server vendors make available to lash blades and racks together. The guiding lights of search technology address the problem as if the only independent variable were algorithms. At that rate, they will be searching for a long time.