Confluent's whiz kids: Rulers of the stream
Three data whizzes from inside LinkedIn have spun out into one of the hottest little companies in tech
To find one of technology’s next big ideas, you have to drive a couple miles down the road from the headquarters of Google and LinkedIn to a one-storey bungalow next to a dentist’s office in Mountain View, California. Behind an unmarked door you’ll find Confluent, a year-old startup keeping the data flowing at some of the biggest and most information-rich firms in Silicon Valley, including Uber and LinkedIn.
Confluent’s founders and 17 employees can still get in the company’s single conference room, where CEO Jay Kreps’s laughter echoes off the walls. “We’re one step up from a garage,” he says, a giggle tumbling out mid-phrase.
Not for long. Soon Confluent will move into a shiny new office in Palo Alto more befitting a company that invented the way Twitter manages its tweet analytics, the movie recommendations at Netflix and the surge pricing at Uber all keep running.
Confluent is building a business on top of open-source data-processing software called Apache Kafka that its founding team created in 2010 while at LinkedIn.
Apache Kafka, the free version, is now used by thousands of companies and tens of thousands of users, with downloads up 400 percent in the first six months of 2015. Confluent intends to build a business selling management tools and services that make it easier to run Kafka.
Fresh off raising $24 million from Index Ventures and early backer Benchmark, Confluent plans to double head count and roll out a major new software release in October, bringing the gospel of real-time data flow to more nontech companies.
Improving the use of and access to data has become a priority for most businesses, in and out of the tech sector. Better data can tell a company whom to advertise to, whom to target in sales calls and when a demand spike will occur. But the amounts of data under review are reaching unmanageable levels. LinkedIn, for example, manages more than 300 billion—with a “b”— user-related events every day. From website visits to mobile phone usage and sensors of all kinds, companies are struggling to access this flow in real time. Typically data are retrieved and analysed in batches, but batch runs can take hours to compute.
Confluent’s software solves the immediacy issue by ensuring that all of the incoming data flow in a continuous stream, like a chocolate river in each customer’s Wonka factory. Right away companies can change prices (eg, Uber surge fares) or pick the best product to show a customer (such as Netflix suggested titles) by plucking such data as they go by. Those insights get fed back into the river, making the whole system a little bit smarter the next time. “You currently find out what happens with your data once a day, at midnight,” says Neha Narkhede, a co-founder who manages Confluent’s engineers. “We turn data into a central nervous system, so you can react to events faster.”
The idea of using data in streams predates Confluent, but the practice was limited to internal use at the big tech companies that already employ engineers steeped in the arcana of real-time processing—much like Confluent’s founding trio. Kreps, 35, is a longtime Californian who rose through LinkedIn’s engineering ranks to serve as technical lead of its relevancy algorithms and data systems. Originally from India, Narkhede got a master’s degree at the Georgia Institute of Technology before working on internal projects at LinkedIn, before settling on Kafka. A Beijing native and a PhD, Jun Rao spent a decade as an IBM researcher before joining LinkedIn’s data team in 2010. He now focuses on open-source Kafka and early customer adoption. The three of them, along with a handful of other LinkedIn employees, created the basic Kafka software and released it publicly for other companies to use and improve. “It’s our passion to do this,” Rao says, “because infrastructure is a means to an end for so many things.”
Kafka quickly picked up early users at Airbnb, Box, Cisco, PayPal, Square and Yahoo. When the three decided early last year they could build a separate business around Kafka, they got strong support from the higher-ups at LinkedIn, which decided to make an investment in Confluent, its first in a startup. Confluent’s revenue today is minimal, but there’s strong evidence it can turn on the money spigot quickly.
Its version of Kafka is downloaded 50,000 times a month, and without a real sales team or any big marketing push it already has paying customers, including a leading traditional retailer, one of the world’s biggest advertising agencies, a huge credit card issuer and a media holding company.
Confluent will spend the next few months adding a range of security features and tutorials to make it easier for more types of developers. Says Mike Volpi, who led Index Ventures’s investment in the startup: “The more on-ramps you have, the more valuable the freeway becomes.”
Experts suggest Confluent’s revenue could approach $10 million next year and pass $50 million in 2017. The company could echo the recent success of another open-source darling, Docker, which has turned record adoption of its computing tools called “containers” into a growing enterprise suite and a $1 billion valuation. Confluent is likely worth about one-sixth that today but not for long.
“Every person we hire uncovers millions of dollars in sales,” says early investor Eric Vishria of Benchmark. “There’s real potential [for Confluent] to be an enterprise phenomenon.”
(This story appears in the 18 September, 2015 issue of Forbes India. To visit our Archives, click here.)
