* * *
September 22, 2009 A $1 Million Research Bargain for Netflix, and Maybe a Model for Others By STEVE LOHR
Even the near-miss losers in the Netflix million-dollar-prize competition seemed to have few regrets.
Netflix, the movie rental company, announced on Monday that a seven-man team was the winner of its closely watched three-year contest to improve its Web site’s movie recommendation system. That was expected, but the surprise was in the nail-biter finish.
The losing team, as it turned out, precisely matched the performance of the winner, but submitted its entry 20 minutes later, just before the final deadline expired.
Under contest rules, in the event of a tie, the first team past the post was the winner. “That 20 minutes was worth a million dollars,” Reed Hastings, chief executive of Netflix, said at a news conference in New York.
Yet the scientists and engineers on the second-place team, and the employers who gave many of them the time and freedom to compete in the contest, were hardly despairing.
Arnab Gupta, chief executive of Opera Solutions, a data analytics company based in New York, took a small group of his leading researchers off other work for two years. “We’ve already had a $10 million payoff internally from what we’ve learned,” Mr. Gupta said.
Working on the contest helped the researchers come up with improved statistical analysis and predictive modeling techniques that his firm has used with clients in fields like marketing, retailing and finance, he said. “So for us, the $1 million prize was secondary, almost trivial.”
Indeed, since it began in October 2006, the Netflix contest was significant less for the prize money than as a test case for new ideas about how to efficiently foster innovation in the Internet era — notably, offering prizes as an incentive and encouraging online collaboration to tap minds worldwide.
The lessons of the Netflix contest could extend well beyond improving movie picks. The researchers from around the world were grappling with a huge data set — 100 million movie ratings — and the challenges of large-scale modeling, which can be applied across the fields of science, commerce and politics.
The prize model is increasingly being tried on work like new science and freelance projects in design and advertising. The X Prize Foundation, for example, is offering multimillion-dollar prizes for path-breaking advances in genomics, alternative energy cars and private space exploration.
InnoCentive is a marketplace for business projects, where companies post challenges — often in areas like product development or applied science — and workers or teams compete for cash payments or prizes offered by the companies. A start-up, Genius Rocket, runs a similar online marketplace mainly for marketing, advertising and design projects.
“The great advantage of the prize model is that it moves work away from the realm of the beauty contest to being performance-oriented,” said Michael Schrage, research fellow at the Center for Digital Business at the Sloan School of Management at the Massachusetts Institute of Technology. “It’s the results produced that matters.”
The emerging prize economy, according to some labor market analysts, does carry the danger of being a further shift in the balance of power toward the buyers — corporations — and away from most workers.
Thousands of teams from more than 100 nations competed in the Netflix prize contest. And it was a good deal for Netflix. “You look at the cumulative hours and you’re getting Ph.D.’s for a dollar an hour,” Mr. Hastings said in an interview.
Netflix, Mr. Hastings said, did not do a crisp cost-benefit analysis of its investment in the contest. But several crucial techniques garnered from the contest have been folded into the company’s in-house movie recommendation software, Cinematch, and customer retention rates have improved slightly. Better recommendations, Netflix says, enhance customer satisfaction.
“We strongly believe this has been a big winner for Netflix,” Mr. Hastings said.
The prize winner was a team of statisticians, machine-learning experts and computer engineers from the United States, Austria, Canada and Israel, calling itself BellKor’s Pragmatic Chaos. The group was actually a merger of teams that came together late in the contest.
In late June, the team finally surpassed the threshold to qualify for the prize by doing at least 10 percent better than Cinematch in accurately predicting the movies customers would like, as measured against actual ratings. Under the contest rules, that set off a 30-day period in which other teams could try to beat them.
That, in turn, prompted a wave of mergers among competing teams, who joined forces at the last minute to try to top the leader. In late July, Netflix declared the contest over, and its online leader board showed two teams had passed the 10 percent threshold: BellKor and the Ensemble, a global alliance with some 30 members.
Netflix said the contest was too close to call, and the leader board showed a slight edge to the Ensemble. However, the teams’ software had to go through two data sets — one public, which was the basis for the leader board, and another hidden one, which determined the outcome of the contest.
The second data set was there to ensure that the winning solution really was the best at making better movie recommendations in general, and was not just tailored to get the best score from the public data set.
Win or lose, researchers agreed that they entered the contest in good part to get access to the Netflix data. “It was incredibly alluring to work on such a large, high-quality data set,” said Joe Sill, an independent consultant and machine-learning expert who was a member of the Ensemble.
Chris Volinsky, a member of BellKor, who is a scientist at AT&T Research, said Netflix “made a brilliant move by realizing that there was a research community out there that worked on these kinds of models and was starving for data.
“Netflix had the data, but only a handful of people working on the problem.”
Netflix was so pleased with the results of its first contest that it announced a second one on Monday. The new contest will present contestants with demographic and behavioral data, including renters’ ages, gender, ZIP codes, genre ratings and previously chosen movies — but not ratings. Contestants will then have to predict which movies those people will like.
Unlike the first challenge, the contest will have no specific accuracy target. Instead, $500,000 will be awarded to the team in the lead after the first six months, and $500,000 to the leader after 18 months.
The winners of the first contest said the money would be split seven ways, according to a formula they declined to disclose. The amounts each received, they said, would certainly help with a car, house payments or children’s college educations — but were not life-changing.
When asked if he planned to take on the second Netflix prize, Bob Bell, a scientist at AT&T Research, said, “I like the notion, but I think I’m too tired.”
http://www.nytimes.com/2009/09/22/technology/internet/22netflix.html?partner=rss&emc=rss
* * *
A market for ideas Sep 17th 2009 The Economist
A pioneering “innovation marketplace” is making steady progress
A PROBLEM shared is a problem solved: that is the belief that inspired InnoCentive, a firm that describes itself as the “world’s first open innovation marketplace”. Conceived in 1998 by three scientists working for Eli Lilly, a big drug company, InnoCentive was spun off as an independent start-up three years later. It is based on a simple idea: if a firm cannot solve a problem on its own, why not use the reach of the internet to see if someone else can come up with the answer?
Companies, which InnoCentive calls “seekers”, post their challenges on the firm’s website. “Solvers”, who number almost 180,000, compete to win cash “prizes” offered by the seekers. Around 900 challenges have been posted so far by some 150 firms including big multinationals such as Procter & Gamble and Dow Chemicals. More than 400 have been solved. InnoCentive reckons the approach can work for innovations in all sorts of fields, from chemistry to business processes and even economic development. It has formed a partnership with the Rockefeller Foundation, a charity, to help solve problems posted by non-profits working in poor countries, with some initial success.
Forrester, a consultancy, studied a pilot partnership between InnoCentive and SCA, a Swedish maker of personal-hygiene products with over 50,000 employees and annual sales of €10 billion ($14.6 billion). It found that challenges posted by the firm generated an average return on investment of 74%, with a payback period of less than three months. Forrester also found cost savings, a faster research process and a more innovative culture.
SCA was attracted by the fact that InnoCentive took great care in managing the intellectual-property issues that result from its approach, and that it paid for results rather than mere effort. According to one SCA executive, the firm piloted the relationship by posting several chemistry challenges, “some of which worked and some didn’t.” Phrasing the challenge turned out to be key. Rather than posting a problem specific to the firm, SCA got much better results when it posed a general problem (how to make a material more absorbent, say). That way the potential network of experts was wider. So far the challenges have been small: the largest prize paid was $25,000. But SCA now expects to expand its use of InnoCentive into the mainstream of its innovation process.
InnoCentive recently raised a second round of venture capital to pay for what it hopes will be rapid growth. Sceptics worry that its success on incremental, scientific challenges may not extend to broader, more substantial innovation because the research culture in most firms is incapable of posing the right questions or knowing what to do with the sort of answers produced by InnoCentive’s solvers.
A recent innovation may help. Called InnoCentive at Work, this replicates the solver network inside a firm, so that challenges are first offered to “seeker” companies’ own employees. Only if they cannot help is the outside network brought into play. “Companies often don’t know how much they already know,” says Dwayne Spradlin, InnoCentive’s chief. An early challenge at one firm was to find a source of some data, which, it turned out, had already been acquired by another division.
One of the first firms to test the @Work model was InnoCentive’s mother ship, Eli Lilly. That firm is struggling: on September 14th it announced plans to shed 5,500 jobs. The fact that Eli Lilly is making ever more use of InnoCentive to drive innovation even as it is making cuts elsewhere proves that it pays, says Mr Spradlin.
http://www.economist.com/businessfinance/displayStory.cfm?story_id=14460185