Search Schizophrenia And The Doomsday Machine That Just Won't Start

Microsoft is mad because Google is trying to finalize a deal that would give it a monopoly over the right to make digital copies of orphan books. But Microsoft is apparently in talks with News Corp. to obtain a monopoly over the right to make digital copies of News Corp's websites. News Corp. is mad because its content isn't making as much money as Rupert Murdoch wants it to. So his plan is to charge money for something nobody pays for -- the right to search and index websites. Murdoch hopes Microsoft is mad enough at Google to write a giant check for something everyone gets for free.

So where does all the madness lead? Nowhere.

The reason nobody pays for the right to index websites is because the law lets you do it for free. Yes, indexing typically involves making entire copies of websites, and yes that's something that might ordinarily create a copyright problem. But if you're making a copy of something for a good reason -- one that has a new and important purpose that adds value and functionality to the world -- and you're not undermining the underlying value of the thing you're copying, then copyright law will usually let you make that copy. It's called fair use. Here in the Ninth Circuit, we've thrashed out the fair use issue as it regards search engines, albeit primarily in the context of image search. The punchline is that copying content to facilitate search functionality is protected by fair use. That rule makes good sense. It creates immense social value by making it easier to sort through incomprehensibly vast quantities of information to find what you need in seconds.

Sometimes people get distracted, and think search engines can only crawl websites by permission, and that permission is provided (or withheld) by the robots.txt file. That's the file that tells search engines they can (or can't) index this site. Search engines have typically respected this "flag" but that doesn't mean they have to. But then again the fact they don't have to doesn't mean they won't. In fact, they'll probably continue to do so. That's because nobody is going to be crazy enough to start the doomsday machine Rupert Murdoch says he's ready to start.

Murdoch might be mad enough to tell Google it can't crawl his sites, even if that means a drastic loss in traffic. But he can only hope to recoup that loss if Microsoft is mad enough to pay for something it appears to have the right to get for free. Is that going to happen? I doubt it. Microsoft isn't so much mad at Google as it is jealous of its immensely profitable search business. But what happens to the search business if every website becomes convinced its entitled to charge every search engine for the right to crawl and index it? Well, lots of things. Most are bad for everybody. But they're particularly bad for the search engines. They will end up paying fees every time they crawl. (Or paying even bigger fees to lawyers to fight about this prospect.) And this will add up. Big time, site by site.

It's hard to imagine Microsoft will help destroy the profitability of the search business it covets. But even if Microsoft is mad enough at Google to take that road, where will it lead?

There might be a big fight. Google and other search engines might crawl anyway, on the strength of its fair use rights. The case would undoubtedly take years to sort out. In the meantime, Microsoft would wonder if it's paying truckloads of cash for something that's really free.

Or maybe there wouldn't be a big fight. Google and other search engines might respect the flag. Then what? Revolt, I imagine. Users will hate a system where they have to use one search engine to access one source of content, a difference search engine to access another source, and have to keep track of which is which along the way. And they will hate Microsoft for helping to create that system. News Corp. might not like the fallout, either. I can see all sorts of boycotts.

So Microsoft can start paying for things everybody gets for free, and take the risk of either paying a huge price that turn out to be free or alienating a huge swath of customers. Sweet deal. Is anyone mad enough to do that? I guess we'll see.


I can only speak for small business owners like myself. First small business owners search is a form of free advertising. So, I say let the big boys fight it out and while they're fighting it out we the small business owners will continue to do business as usual. What I don't understand is No matter if you're a big player or small player on the Internet you need to be found by potential clients, and those clients find you through search Engines. Speaking from experience their favorite search engines. If you're not listed by their favorite search engines you're probably not going to be found. People are lazy and not likely to change quickly So let's see how this plays out for News Corp.

Well I guess Google can set the agenda and will always remain at the top and will hold everything.
I wonder what's the dollar amount for all the good will that Google will generate in the US and Europe by turning it's lost in China to a stand on "moral" value.

Google wil probably beat every other search engine - it's just the best. It has the most relevant search, and that's scary because it's also the most powerful...

I think Google is ahead of everything. As long as it is the largest search enginge and have the funds, they can set the agenda. I dont mind, I think they develop features taht are a huge advantage for everyone.

in my opinion, i believe that google is dominating and monopolizing everything. i find it hard as a content writer to compete with google, as clearly, the search engine giant gobbles everything up and list them on the search engine, as if it's free.
what do you think?

Add new comment