The New York Times and Taboola teamed up to do some A/B testing and find out which system for intelligent video content recommendations worked best. Taboola pitted its "secret sauce" algorithms for choosing the best possible videos for viewers versus a standard "other popular in this category" variety. The data might be hard to believe.
Taboola is reporting that there was a 250% uplift in click through rate. Like I said, it might be hard to believe. I'm sort of skeptical myself, by nature journalists are supposed to be, that makes us ask better questions. I managed to get some time with Adam Singolda, the Founder and CEO of Taboola, to ask him about the results.
How the uplift was determined
Since it's just a large number I wanted to first look in on the methodology. Now some people would just say clicks don't lie, but that in itself is a lie right there. There are all sorts of well documented ways to both achieve and combat click fraud. I am in no way suggesting that there was anything of the sort here, I'm merely filling in background information.
So Adam told me that A/B testing was done with two buckets of course. Somewhere around 5%, but potentially as low as 1%, of viewers saw the Bucket A recommendations while the rest saw the bucket B recommendations. Bucket A was the "popular videos" setup while B was Taboola's personalized system. Daily the average CTR rates were totaled and compared to find that the uplift on Bucket B was 250% the CTR of Bucket A over the 8 month test period. That means a user was 2.5 times more likely to click through to another video when exposed to Taboola's recommendations over the standard ones. UI changes and inventory growth had no effect on the average CTR results.
How the videos were recommended
Of course Taboola isn't going to give us their exact method of recommending videos, that would jeopardize their business. But I did get a bit of info on the subject. The New York Times groups content through categories and tags on the content, like many of us do. That means the most popular videos were of course from the same category or tag pool as the viewed video. That will work in many places and certainly has proven useful over time.
Taboola, took three years and developed a unique mathematical solution to recommending other content. It tries to understand how people got to the videos, how long they are engaged with said videos, what recommendation choices they normally make, how they got there, whether from a search or an article or other link, and some stored data in browser cookies. That means they might know you never click on a video over five minutes but love to watch quick one minute clips. Right there, they can more readily target you with videos that you are more likely to click on.
They can also take other information into account. Like time of day, maybe you watch quick videos at work but longer ones on your commute or while at home. Perhaps you like more hard hitting news instead of the fluff pieces and human interest stories. Stuff like that can be quite useful, even I can see that, when recommending videos to users. Offering up a screen of choices that fit into their behavior profile must certainly increase CTR over standard recommendation that don't take that into account.
Other Things Not Taken Into Account
I asked if certain demographics were found to be more likely to click through via one system or the other as I thought that might be some useful information. Unfortunately, that sort of data wasn't tracked in this test. Nor was the frequency of videos showing up in both buckets via recommendations and the specific uplift on those videos which I also would have liked to see. If a video shows up X times in Bucket A and X times in Bucket B and suddenly has a 200% increase in click through, well, you can see how that would be valuable information, right?
The Bucket List
We can't give you specific numbers on the sizes of the buckets, but considering it's the New York Times one might imagine that it's millions. Even I have been known to slide over to their site every now and again and take in some news or a quick news video. I was assured that the data sets were large enough to lower the margin of error to a very tolerable rate. I'd guess millions of views from at least tens of thousands of viewers if you raked me over the coals about it. Adam agreed that it would need to be at least tens of thousands to get some quality data out of a test like this.
Taboola themselves recommend to over 51 million users a month (that's an pretty large stat there) and it seems that they're still growing, especially now that the New York Times looks like they might be convinced of the efficacy of the system.
Looking at some comScore numbers, which I dug up from June 2010, the New York Times Digital property was #13 on the Top 50 Web Properties, hauling in 68,912,000 users for the month. Now that's total audience not uniques. That makes them bigger than Apple. If only 10% of them viewed one video it's about 6.9 million videos. Now extrapolate that out to 8 months and you get some 56 million videos, hypothetically. Imagine a 250% increase in click through on those numbers and you can see why Taboola is reticent when asked to discuss their methods.
Still, Adam Singolda was more than happy to write massive emails back and forth with me. And why shouldn't he? After all, if their system works as well as it looks like, everyone should want it and want to utilize it if they can. I know I could certainly use some 250% increase in video viewing over at GDN and I'm sure all of you would be interested in that type of increase in your numbers.
Taboola was founded in 2007 by a team of veteran Israel National Security Agency (I-NSA) researchers with years of advanced practical mathematical research experience. That's sort of scary. But at least they went into online video and not say, freelance bounty hunting. Come to think of it, I might have gone into the bounty hunting myself.
Taboola offers several major products all aimed at helping you maximize your video views in a realistic and logical way.
- Text2Video - Ties your text content into your video content by offering video recommendations based on what your users are reading. This then turns them into video viewers as well.
- Video2Video - This is the system that they used on this testing I'm guessing. Recommending videos based on video viewing habits.
- Affiliate2Video - Allows your video content to show up on affiliate sites and blogs to help drive traffic to your videos.
Of course on top of just driving views it also drives revenue as more video views means for ad impressions and if you've got some good CPM already on those video ads you'll certainly see a jump in revenue to offset the potential cost of implementing Taboola's offerings. I asked Adam about pricing and he said there's not a price attached to the first two. They're actually widgets that you embed into your site and through sponsored recommendations they earn revenue. Also, both Bloomberg.com and eHow.com are using the service and soon, maybe Gamers Daily News.