We've written before a time or two about how Google was testing "Caffeine," a new indexing system for their engine. And now they're done testing. The Google search engine as you knew it is done… it is now officially powered by Caffeine… for everyone. The Google index as you know it is done, replaced by its younger, faster little brother.
It's not easy explaining the complicated workings of a search engine, but Google's blog gives it a shot. They say the old index was built on layers, and that some layers were refreshed at faster rates than others—but every time a layer was refreshed they had to crawl the entire web. This method created a delay between a page being discovered by Google's spiders and that page being made available in the results for users.
The new index is apparently a swirling vortex of information (that may or may not also be nuclear in nature). Here's Google's little animation to show the difference between the two index formats:
Caffeine doesn't use layers. Instead, it crawls the web in "small portions," bits at a time, with the index being perpetually updated on a continuous basis. Instead of updating the whole darn layer all at one time, the index can be updated in pieces and parts, allowing for more up-to-date results. (Can I also just say that this image is one of the worst visual aids I've ever seen, and actually does very little to help explain the "layers versus portions" distinction.)
Google makes some pretty bold statements about Caffeine's power. For example:
"Caffeine lets us index web pages on an enormous scale. In fact, every second Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.”
Which is all just a very fancy way of saying "our index crunches more numbers in faster time than your index," which itself is just the search engine variation on "my dad can beat up your dad.” But I'm willing to allow them some bragging time. They're proud of Caffeine. And heck, it's not every day the world's unquestioned leader in search completely overhauls how its engine is built.
It sounds like Caffeine is here to stay for a while too, since the blog post is careful to point out that they've built it with the future in mind, making it much more scalable. But it will be improved moving forward, they say. And that's par for the course with Google—most any service they think is useful is regularly enhanced and upgraded.
It's very interesting to me that in previous announcements about Caffeine, made during its testing phase, Google went out of their way to mention that users weren't likely to see any real change in the actual results served up by the new index. However, in this official announcement yesterday, there's no similar statement. That's because—I think—the results could end up being very, very different. Everything about their blog post screams "relevancy" and "timeliness." They want to find more content the moment it's published, and they want to show you results that are at the maximum level of relevancy for you and your life.
So… what does all this mean for video? Well, in theory, it should mean that your videos will be found and indexed by Google more rapidly than ever before. Additionally, it's pretty clear that Caffeine is all about getting these emerging content varieties (video, news, blog posts, tweets, etc.) to a place of prominence similar to traditional websites. That probably means even more video rolled into the results. And hopefully, it means that Google just took a huge step toward being more "real time" than any previous search engine.
When Google first hit the search market, it was the index that helped them achieve dominance, because it was larger in scope and served up faster results. It's no small thing for a company in Google's position to completely overhaul the foundation of their core product. Sure… I'm guessing they tested the heck out of it, and that they're pretty darn positive it's ready for prime time… or they wouldn't be rolling it out. I'm sure it's solid.
But this is no small tweak. This is a completely new indexing system, one that has different abilities and different goals. It would be easy for Google to rest on their previous successes. It would be easy to say, "Let's stay the same instead of changing, because the current model already works.” Whatever your personal opinion on Google, you have to admire the kind of courage it takes to say, "Yeah, we're on top, and we have been for a long time, but that's not good enough… we still want to improve.”
Okay. That's it for me. I'm off to go play with the "new" Google and see what I can see. It's going to be fun over the next few weeks as various search marketers blog about their personal findings on Caffeine. Be sure to let us know in the comments if you spot any major differences in your results.