James Surowiecki, in his book “Wisdom of Crowds”, builds upon an idea that under certain conditions, groups of people make better decisions than any single individual could expect to make. According to Surowiecki, crowds are collectively smarter than any single person under four conditions: 1) diversity of opinion: when each person has his/her own view and the interpretation of the event; 2) independence: when the opinions of individuals are not influenced by other individuals; 3) decentralization: when people are able to specialize and draw on local knowledge; 4) aggregation: when it is possible to turn individual judgments into collective decisions.
Surowiecki proposes that a large group of amateurs can make good guesses based on the fact that every individual judgement may be wrong, but the sum of all wrong judgements collectively can produce a correct outcome because all wrong answers effectively cancel out. However, not all problems can be solved by amateurs, some challenges require a pool of well-trained experts.
Jeff Howe defines crowdsourcing as “the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” As the cost of collaboration is falling dramatically due to low communication costs, businesses explore opportunities to source ideas externally.
Recently, turning to crowds to improve existing algorithms has become a trend. One of the famous examples is Netflix Prize. To make their recommendation system better, Netflix launched a crowdsourcing effort. The goal was to improve the existing recommendation system accuracy by 10%. The prize was $1 million dollars. 51051 contestants on 41305 teams from 186 different countries responded to the challenge. As a result, a team “BellKor’s Pragmatic Chaos” was able to improve the existing algorithm by 10.6%.
Inspired by Netflix’s success Overstock.com offers $1 million to anyone who can improve their product recommendation system. To get $1 million, a team is required to improve the existing algorithm by 10%.
Heritage Health Prize offers $3 million to anyone who will help resolve the problem of predicting how many days a patient will spend in a hospital. According to Heritage Provider Network, so far $30 billion has been spent on unnecessary hospital admissions. The new algorithm will be instrumental in designing new healthcare plans. As of today, 1342 players entered the competition.
Searching for a better algorithm requires enormous computing power. The most powerful computer nowadays is the K-computer manufactured by a Japanese company Fujitsu. It employs 68,544 CPUs, has 548,352 cores and performs 8 quadrillion calculations per second (8 petaflops). Not many researchers and even large research organizations can afford it.
Grid computing can help solve a problem of limited computer power. This technology allows creating a super computer by combining the power of multiple, geographically dispersed computers in the network. Grid computing has been successfully used by major universities to crunch large amounts of data. For example, a project of the University of California, Berkeley uses grids to search for extra-terrestrial intelligence. The Scripps Research Institute uses grids to get assistance in the development of new medicines, and Stanford University for fighting Alzheimer and Parkinson diseases.
If crowdsourcing and the combined power of millions of computers are used together, the opportunities can be unlimited for both commercial and academic use, and hopefully more practical solutions will be found to address existing challenges.