Google's New Index, Caffeine, Is Now Fully Awake

Googles New Index, Caffeine, Is Now Fully Awake

We've written before a time or two about how Google was testing "Caffeine," a new indexing system for their engine.  And now they're done testing.  The Google search engine as you knew it is done… it is now officially powered by Caffeine… for everyone.  The Google index as you know it is done, replaced by its younger, faster little brother.

It's not easy explaining the complicated workings of a search engine, but Google's blog gives it a shot.  They say the old index was built on layers, and that some layers were refreshed at faster rates than others—but every time a layer was refreshed they had to crawl the entire web.  This method created a delay between a page being discovered by Google's spiders and that page being made available in the results for users.

The new index is apparently a swirling vortex of information (that may or may not also be nuclear in nature).  Here's Google's little animation to show the difference between the two index formats:

Googles New Index, Caffeine, Is Now Fully Awake

Caffeine doesn't use layers.  Instead, it crawls the web in "small portions," bits at a time, with the index being perpetually updated on a continuous basis.  Instead of updating the whole darn layer all at one time, the index can be updated in pieces and parts, allowing for more up-to-date results.  (Can I also just say that this image is one of the worst visual aids I've ever seen, and actually does very little to help explain the "layers versus portions" distinction.)

Google makes some pretty bold statements about Caffeine's power.  For example:

"Caffeine lets us index web pages on an enormous scale. In fact, every second Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.”

Which is all just a very fancy way of saying "our index crunches more numbers in faster time than your index," which itself is just the search engine variation on "my dad can beat up your dad.”  But I'm willing to allow them some bragging time.  They're proud of Caffeine.  And heck, it's not every day the world's unquestioned leader in search completely overhauls how its engine is built.

It sounds like Caffeine is here to stay for a while too, since the blog post is careful to point out that they've built it with the future in mind, making it much more scalable.  But it will be improved moving forward, they say.  And that's par for the course with Google—most any service they think is useful is regularly enhanced and upgraded.

It's very interesting to me that in previous announcements about Caffeine, made during its testing phase, Google went out of their way to mention that users weren't likely to see any real change in the actual results served up by the new index.  However, in this official announcement yesterday, there's no similar statement.  That's because—I think—the results could end up being very, very different.  Everything about their blog post screams "relevancy" and "timeliness."  They want to find more content the moment it's published, and they want to show you results that are at the maximum level of relevancy for you and your life.

So… what does all this mean for video? Well, in theory, it should mean that your videos will be found and indexed by Google more rapidly than ever before.  Additionally, it's pretty clear that Caffeine is all about getting these emerging content varieties (video, news, blog posts, tweets, etc.) to a place of prominence similar to traditional websites.  That probably means even more video rolled into the results.  And hopefully, it means that Google just took a huge step toward being more "real time" than any previous search engine.

When Google first hit the search market, it was the index that helped them achieve dominance, because it was larger in scope and served up faster results.  It's no small thing for a company in Google's position to completely overhaul the foundation of their core product.  Sure… I'm guessing they tested the heck out of it, and that they're pretty darn positive it's ready for prime time… or they wouldn't be rolling it out.  I'm sure it's solid.

But this is no small tweak.  This is a completely new indexing system, one that has different abilities and different goals. It would be easy for Google to rest on their previous successes.  It would be easy to say, "Let's stay the same instead of changing, because the current model already works.”  Whatever your personal opinion on Google, you have to admire the kind of courage it takes to say, "Yeah, we're on top, and we have been for a long time, but that's not good enough… we still want to improve.”

Okay.  That's it for me.  I'm off to go play with the "new" Google and see what I can see.   It's going to be fun over the next few weeks as various search marketers blog about their personal findings on Caffeine.  Be sure to let us know in the comments if you spot any major differences in your results.


Don't Miss Out - Join Our VIP Video Marketing Community!
Get daily online video tips and trends via email!


Posted in Video Search, Video SEO
About the Author -
Jeremy Scott is the founder of The Viral Orchard, an Internet marketing firm offering content writing and development services, viral marketing consulting, and SEO services. Jeremy writes constantly, loves online video, and enjoys helping small businesses succeed in any way he can. View All Posts By -

What do you think? ▼
  • Matth

    great summary, thanks. It will be interesting to see how this all pans out over the next few weeks/months. I'm holding onto my hat!

  • https://twitter.com/angelramayan Angel Ramayan

    Well I think this new caffeinated filters could spell disaster for some businesses that thrive as a result of their organic position.

  • http://www.VideoLeadsOnline.com/ Ronnie Bincer

    From what my SEO mentor tells me this has been brewing for quite some time. We most likely have been being "cafinated" for a while to one degree or another.My best take this early on in the "now its live" announcement is to say (like you did) that it is important to be relevant and fresh. Making changes that are relevant would be a winning combo. I think that the video space is posed to take a big gulp of traffic boosting buzz from this if we can keep the videos relevant and fresh! I wonder what flavor will be the first to be recognized since there seems to be a lot of flavors in the mix!

  • Justin

    I think we will notice a lot of people keeping a close eye on their results, and in turn larger amounts of people testing and more queries for Google. Oh joy, the frantic top 3 on Hoogle, how things will ever change.

  • http://www.example.com.ru Example

    Congratulations Google! It is very fast and much more intelligent than it used to be. Sure, there is a long way to go. They need to find way to clean up their index. They need to do something with the data provided by "The Internet Archive". I could not find a single book there which was properly scanned. They need to fix snippets in "books" (90% of them do not display the information they are supposed to display) and so on.
    Congratulations anyway. It is very exciting to watch.