NM: What do you attribute that to?

DC: Well, the reason for that is that the [INAO] website is marked "don't crawl me" with a file that says "search engines: go away."  So, you cannot find that content on Google.  And there's a lot of sites like that; there's a lot of sites whose structure is set up so that they're not search engine friendly.  Some of the best wine sites in the world are that way.

NM: So, early on you identified problems with the way Google and Yahoo were — or were not, in many cases — indexing wine websites.  How did that evolve into this full-scale endeavor?

DC: Well, it started as a bookmark list.  I wasn't originally thinking I was going to make a search engine.  I was making a list of really authoritative, useful, in-depth wine sites for myself, as I was digging around and studying for my diploma exam.  By the end, I think I had about 500 really great wine websites, and I thought, "Oh, what the heck, maybe I'll make a little website of [links from] my list of good wine sites."  And unfortunately, or fortunately, I thought maybe I would just download an open-source search engine and see what it would be like if I tried to make a little search engine to search these [wine] sites.  So, I did.  I downloaded this open-source search engine called Nutch, started playing with it, and bit by bit, I started thinking, "Well, you know what?  If I just rewrite this bit here, I could make it work better for wine-related queries; and if I just change that bit there, I could make it even better in this other way."  So, I started tinkering.  I really wanted to make something that was uniquely useful, so I began to think about how I could completely rebuild this search-engine that I'd started with, make it wine-specific, and make it do things that you simply couldn't do with Google.  I spent a lot of time thinking about how people search for wine information, and a lot of time thinking how search engines work and how a general-purpose search engine doesn't always give you the best results for wine-related queries.  Eventually, I'd more or less re-written huge chunks of this search engine to do wine-specific things.  At the same time, I took the list of 500 sites that I started with and kept growing it, and started building software to go out and find every producer in the world, every organization, every publication (consumer, trade, scientific), every academic resource, and the best wine blogs — all in every language in the world of wine.  At this point, there's about 40,000 websites in the [Able Grape] index, and about 15 million pages of information.

NM: This is all quite complex and comprehensive.  But even though it's entirely focused on wine, aren't you in essence reinventing the wheel?  What is it that Able Grape offers that a larger, all-encompassing search engine in widespread use, like Google, does not?

DC: It really comes down to this: if you do one thing, and you really focus on it and specialize, you can do it really well.  Google is a general-purpose search engine.  Their policy is one of inclusion; they include pretty much everything.  There's all kinds of retail content in there — if you look up any wine, 99% of the content is retailers trying to sell you that wine.  And the retailers are much more savvy about search-engine optimization, more so than the producers who have the best information about the wines they make.  Sometimes finding a producer's own information can be like finding a needle in a haystack.  Similarly, there's a lot of people who are trying to make money by gathering web traffic, and other people who are putting up websites with not necessarily high-quality information, in some cases downright garbage.  Then there's this sort of this gray area of sites that may have 'okay' information, but maybe it's not very deep or trustworthy.  You never know what you're going to get with these sites.  So, one of the things I wanted to do is select everything editorially, so that I knew I'd only be getting high-quality wine-related sites — no retail, no commercial garbage.  And also, every single site is categorized, so that you know which sites are producers, which sites are importers, which are organizations, and which are publications and what sort of publications they are.  And then, when you're looking for information, you can instantly filter for the kind of information want.

[One stark reality that renders in high relief the difference between Able Grape and other large-scale, general purpose search engines is that a considerable number of wine-related websites which contain high-quality information are actually in the language native to their country of origin and lack an English version.  These pages would more than likely be completely overlooked by a search engine like Google.  Doug identified the necessity of including these non-English sites in Able Grape's master index, in the interest of maintaining the search engine's high standard of quality and comprehensiveness in the gathering and referencing of online wine resources.]