Crowd intelligence in AI 2.0 era

LI Wei; Wen-jun WU; Huai-min WANG; Xue-qi CHENG; Hua-jun CHEN; Zhi-hua ZHOU; Rong DING

doi:10.1631/FITEE.1601859

Your Location：

Home >

Browse articles >

Crowd intelligence in AI 2.0 era

Special Issue on Artificial Intelligence 2.0 | Updated：2022-05-19

- Crowd intelligence in AI 2.0 era
- AI2.0时代的群体智能
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 1, Pages: 15-43(2017)
- Affiliations：
  
  1State Key Laboratory of Software Development, Beihang University, Beijing 100191, China
  2National Laboratory for Parallel and Distributed Processing, College of Computer, National University of Defense Technology, Changsha 410073, China
  3Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
  4College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  5National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
- Author bio：
  
  E-mail:wwj@nlsde.buaa.edu.cn
- Funds：
- DOI：10.1631/FITEE.1601859
  CLC： TP18
- Received：23 December 2016，
  
  Accepted：03 January 2017，
  
  Published：2017-01
- Accepted：
Scan QR Code
LI Wei, Wen-jun WU, Huai-min WANG, et al. Crowd intelligence in AI 2.0 era[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(1): 15-43.
DOI：

LI Wei, Wen-jun WU, Huai-min WANG, et al. Crowd intelligence in AI 2.0 era[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(1): 15-43. DOI： 10.1631/FITEE.1601859.

摘要

基于互联网的信息物理世界深刻地改变了人工智能（artificial intelligence

AI）发展的信息环境，将人工智能研究的新浪潮推进到人工智能2.0新纪元。作为AI2.0时代最突出的研究特点之一，群体智能引起了产业界和学术界的广泛关注。具体来说，为应对挑战，群体智能提供了一种通过聚集群体的智慧解决问题的新模式。特别是由于共享经济的快速发展，群体智能不仅成为了解决科学难题的新途径，而且也已融入日常生活的各个方面，例如线上到线下（online-to-offline

O2O）应用、实时交通监控、以及物流管理。本文对现有群体智能研究成果进行总结和综述：首先，论述了群体智能的基本概念，并对其与现有相关概念（如众包和人本计算）的关系进行了解释。然后，介绍了四类具有代表性的群体智能平台，总结了三项核心问题以及最新的群体智能技术。最后，讨论了群体智能研究的未来发展方向。

Abstract

The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence (AI)

bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era

crowd intelligence has attracted much attention from both industry and research communities. Specifically

crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular

due to the rapid development of the sharing economy

crowd intelligence not only becomes a new approach to solving scientific challenges

but has also been integrated into all kinds of application scenarios in daily life

e.g.

online-tooffline (O2O) application

real-time traffic monitoring

and logistics management. In this paper

we survey existing studies of crowd intelligence. First

we describe the concept of crowd intelligence

and explain its relationship to the existing related concepts

e.g.

crowdsourcing and human computation. Then

we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally

we discuss promising future research directions of crowd intelligence.

关键词

Keywords

references

I Abraham , , , O Alonso , , , V Kandylas , , , 等 . . Adaptive crowdsourcing algorithms for the bandit survey problem . . 2013 . . Proc. 26th Conf. on Computational Learning Theory . . 882 - - 910 . . . .

J Ballesteros , , , B Carbunar , , , M Rahman , , , 等 . . Towards safe cities: a mobile and social networking approach . . IEEE Trans. Parall. Distr. Syst. , , 2014 . . 25 ( ( 9 ): ): 2451 - - 2462 . . DOI: 10.1109/TPDS.2013.190 http://doi.org/10.1109/TPDS.2013.190 . .

VR Basili , , , LC Briand , , , WL Melo . . A validation of object-oriented design metrics as quality indicators . . IEEE Trans. Softw. Eng. , , 1996 . . 22 ( ( 10 ): ): 751 - - 761 . . DOI: 10.1109/32.544352 http://doi.org/10.1109/32.544352 . .

P Bhattacharya , , , I Neamtiu . . Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging . . 2010 . . IEEE Int. Conf. on Software Maintenance . . 1 - - 10 . . DOI: 10.1109/ICSM.2010.5609736 http://doi.org/10.1109/ICSM.2010.5609736 . .

C Bird , , , A Gourley , , , P Devanbu , , , 等 . . Mining email social networks . . 2006 . . Proc. Int. Workshop on Mining Software Repositories . . 137 - - 143 . . DOI: 10.1145/1137983.1138016 http://doi.org/10.1145/1137983.1138016 . .

C Bird , , , D Pattison , , , R de Souza , , , 等 . . Latent social structure in open source projects . . 2008 . . Proc. 16th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering . . 24 - - 35 . . DOI: 10.1145/1453101.1453107 http://doi.org/10.1145/1453101.1453107 . .

C Bird , , , N Nagappan , , , B Murphy , , , 等 . . Don't touch my code!: examining the effects of ownership on software quality . . 2011 . . Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering . . 4 - - 14 . . DOI: 10.1145/2025113.2025119 http://doi.org/10.1145/2025113.2025119 . .

J Bollen , , , HN Mao , , , XJ Zeng . . Twitter mood predicts the stock market . . J. Comput. Sci. , , 2011 . . 2 ( ( 1 ): ): 1 - - 8 . . DOI: 10.1016/j.jocs.2010.12.007 http://doi.org/10.1016/j.jocs.2010.12.007 . .

E Bonabeau . . Decisions 2.0: the power of collective intelligence . . MIT Sloan Manag. Rev. , , 2009 . . 50 ( ( 2 ): ): 45 - - 52 . . . .

KD Borne , , , Team Zooniverse . . The Zooniverse: a framework for knowledge discovery from citizen science data . . American Geophysical Union Fall Meeting , , 2011 . . .

JA Burke , , , D Estrin , , , M Hansen , , , 等 . . Participatory sensing . . Workshop on World-Sensor-Web: Mobile Device Centric Sensor Networks and Applications , , 2006 . . 117 - - 134 . . . .

CC Cao , , , JY She , , , YX Tong , , , 等 . . Whom to ask? Jury selection for decision making tasks on micro-blog services . . Proc. VLDB Endow. , , 2012 . . 5 ( ( 11 ): ): 1495 - - 1506 . . DOI: 10.14778/2350229.2350264 http://doi.org/10.14778/2350229.2350264 . .

CC Cao , , , YX Tong , , , L Chen , , , 等 . . Wisemarket: a new paradigm for managing wisdom of online social users . . 2013 . . Proc. 19th ACM Int. Conf. on Knowledge Discovery and Data Mining . . 455 - - 463 . . DOI: 10.1145/2487575.2487642 http://doi.org/10.1145/2487575.2487642 . .

OF Castaneda . . Hierarchy in Meritocracy: Community Building and Code Production in the Apache Software Foundation . . MS Thesis, Delft University of Technology, Delft, Netherlands , , 2010 . . .

X Chen , , , QH Lin , , , DY Zhou . . Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing . . 2013 . . Proc. 30th Int. Conf. on Machine Learning . . 64 - - 72 . . . .

X Chen , , , QH Lin , , , DY Zhou . . Statistical decision making for optimal budget allocation in crowd labeling . . J. Mach. Learn. Res. , , 2015 . . 16 1 - - 46 . . . .

CAL Dantec , , , M Asad , , , A Misra , , , 等 . . Planning with crowdsourced data: rhetoric and representation in transportation planning . . 2015 . . Proc. 18th ACM Conf. on Computer Supported Cooperative Work . . 1717 - - 1727 . . DOI: 10.1145/2675133.2675212 http://doi.org/10.1145/2675133.2675212 . .

AP Dawid , , , AM Skene . . Maximum likelihood estimation of observer error-rates using the EM algorithm . . Appl. Statist. , , 1979 . . 28 ( ( 1 ): ): 20 - - 28 . . DOI: 10.2307/2346806 http://doi.org/10.2307/2346806 . .

B de Alwis , , , J Sillito . . Why are software projects moving from centralized to decentralized version control systems? . . 2009 . . Proc. ICSE Workshop on Cooperative and Human Aspects on Software Engineering . . 36 - - 39 . . DOI: 10.1109/CHASE.2009.5071408 http://doi.org/10.1109/CHASE.2009.5071408 . .

O Dekel , , , O Shamir . . Vox Populi: collecting highquality labels from a crowd . . 2009 . . Proc. 22nd Conf. on Learning Theory . . . .

AP Dempster , , , NM Laird , , , DB Rubin . . Maximum likelihood from incomplete data via the EM algorithm . . J. R. Stat. Soc. Ser. B , , 1977 . . 39 ( ( 1 ): ): 1 - - 38 . . . .

DE Difallah , , , G Demartini , , , GP Cudr-Mauroux . . Pick-a-crowd: tell me what you like, and I'll tell you what to do . . 2013 . . Proc. 22nd Int. Conf. on World Wide Web . . 367 - - 374 . . DOI: 10.1145/2488388.2488421 http://doi.org/10.1145/2488388.2488421 . .

DE Difallah , , , G Demartini , , , GP Cudr-Mauroux . . Scheduling human intelligence tasks in multi-tenant crowd-powered systems . . 2016 . . Proc. 25th Int. Conf. on World Wide Web . . 855 - - 865 . . DOI: 10.1145/2872427.2883030 http://doi.org/10.1145/2872427.2883030 . .

XL Dong , , , B Saha , , , D Srivastava . . Less is more: selecting sources wisely for integration . . Proc. VLDB Endow. , , 2012 . . 6 ( ( 2 ): ): 37 - - 48 . . DOI: 10.14778/2535568.2448938 http://doi.org/10.14778/2535568.2448938 . .

JR Erenkrantz , , , RN Taylor . . Supporting Distributed and Decentralized Projects: Drawing Lessons From the Open Source Community , , 2003 . . : ISR Technical Report No. UCI-ISR-03-4 : Institute for Software Research, University of California , , Irvine, USA . .

K Farkas , , , AZ Nagy , , , T Toms , , , 等 . . Participatory sensing based real-time public transport information service . . 2014 . . IEEE Int. Conf. on Pervasive Computing and Communications Workshops . . 141 - - 144 . . DOI: 10.1109/PerComW.2014.6815181 http://doi.org/10.1109/PerComW.2014.6815181 . .

ZN Feng , , , YM Zhu , , , Q Zhang , , , 等 . . Trac: truthful auction for location-aware collaborative sensing in mobile crowdsourcing . . 2014 . . Proc. IEEE Conf. on Computer Communications . . 1231 - - 1239 . . DOI: 10.1109/INFOCOM.2014.6848055 http://doi.org/10.1109/INFOCOM.2014.6848055 . .

G Fowler , , , J Schectman . . Citizen surveillance helps officials put pieces together , , 2013 . . The Wall Street Journal, April 17. http://www.wsj.com/articles/SB10001424127887324763404578429220091342796 http://www.wsj.com/articles/SB10001424127887324763404578429220091342796 . .

C Gao , , , DY Zhou . . Minimax optimal convergence rates for estimating ground truth from crowdsourced labels . . ePrint Archive, arXiv:1310.5764 , , 2013 . . .

DW Gao , , , YX Tong , , , JY She , , , 等 . . Top-k team recommendation in spatial crowdsourcing . . LNCS , , 2016 . . 9658 191 - - 204 . . DOI: 10.1007/978-3-319-39937-9_15 http://doi.org/10.1007/978-3-319-39937-9_15 . .

L Gao , , , F Hou , , , JW Huang . . Providing long-term participation incentive in participatory sensing . . 2015 . . IEEE Conf. on Computer Communications . . 2803 - - 2811 . . DOI: 10.1109/INFOCOM.2015.7218673 http://doi.org/10.1109/INFOCOM.2015.7218673 . .

RA Ghosh . . Understanding Free Software Developers: Findings from the Floss Study , , 2005 . . : Cambrige, USA : MIT Press , , . .

G Gousios , , , M Pinzger , , , A Deursen . . An exploratory study of the pull-based software development model . . 2014 . . Proc. 36th Int. Conf. on Software Engineering . . 345 - - 355 . . DOI: 10.1145/2568225.2568260 http://doi.org/10.1145/2568225.2568260 . .

G Gousios , , , A Zaidman , , , MA Storey , , , 等 . . Work practices and challenges in pull-based development: the integrator's perspective . . 2015 . . Proc. 37th Int. Conf. on Software Engineering . . 358 - - 368 . . DOI: 10.1109/ICSE.2015.55 http://doi.org/10.1109/ICSE.2015.55 . .

K Han , , , C Zhang , , , J Luo , , , 等 . . Truthful scheduling mechanisms for powering mobile crowdsensing . . IEEE Trans. Comput. , , 2016 . . 65 ( ( 1 ): ): 294 - - 307 . . DOI: 10.1109/TC.2015.2419658 http://doi.org/10.1109/TC.2015.2419658 . .

A Hars , , , S Ou . . Working for free? Motivations of participating in open source projects . . 2001 . . Proc. 34th Annual Hawaii Int. Conf. on System Sciences . . 1 - - 9 . . DOI: 10.1109/HICSS.2001.927045 http://doi.org/10.1109/HICSS.2001.927045 . .

AE Hassan . . Predicting faults using the complexity of code changes . . 2009 . . Proc. 31st Int. Conf. on Software Engineering . . 78 - - 88 . . DOI: 10.1109/ICSE.2009.5070510 http://doi.org/10.1109/ICSE.2009.5070510 . .

G Hertel , , , S Niedner , , , S Herrmann . . Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel . . Res. Polic. , , 2003 . . 32 ( ( 7 ): ): 1159 - - 1177 . . DOI: 10.1016/S0048-7333(03)00047-7 http://doi.org/10.1016/S0048-7333(03)00047-7 . .

CJ Ho , , , JW Vaughan . . Online task assignment in crowdsourcing markets . . 2012 . . Proc. 26th AAAI Conf. on Artificial Intelligence . . 45 - - 51 . . . .

CJ Ho , , , S Jabbari , , , JW Vaughan . . Adaptive task assignment for crowdsourced classification . . 2013 . . Proc. 30th Int. Conf. on Machine Learning . . 534 - - 542 . . . .

ML Hoffman . . Is altruism part of human nature? J . . Personal. Soc. Psychol. , , 1981 . . 40 ( ( 1 ): ): 121 - - 137 . . DOI: 10.1037/0022-3514.40.1.121 http://doi.org/10.1037/0022-3514.40.1.121 . .

LG Jaimes , , , I Vergara-Laurens , , , MA Labrador . . A location-based incentive mechanism for participatory sensing systems with budget constraints . . 2012 . . Proc. 10th Annual IEEE Int. Conf. on Pervasive Computing and Communications . . 103 - - 108 . . DOI: 10.1109/PerCom.2012.6199855 http://doi.org/10.1109/PerCom.2012.6199855 . .

S Jain , , , S Gujar , , , S Bhat , , , 等 . . An incentive compatible multi-armed-bandit crowdsourcing mechanism with quality assurance . . ePrint Archive, arXiv:1406.7157 , , 2014 . . .

G Jeong , , , S Kim , , , T Zimmermann . . Improving bug triage with bug tossing graphs . . 2009 . . Proc. 7th Joint Meeting of the European Software Engineering Conf. and the ACM SIGSOFT Symp. on the Foundations of Software Engineering . . 111 - - 120 . . DOI: 10.1145/1595696.1595715 http://doi.org/10.1145/1595696.1595715 . .

DR Karger , , , S Oh , , , D Shah . . Iterative learning for reliable crowdsourcing systems . . 2011 . . Advances in Neural Information Processing Systems . . 1953 - - 1961 . . . .

A Khetan , , , S Oh . . Achieving budget-optimality with adaptive schemes in crowdsourcing . . 2016 . . Advances in Neural Information Processing Systems . . 4844 - - 4852 . . . .

A Kittur , , , B Smus , , , S Khamkar , , , 等 . . Crowdforge: crowdsourcing complex work . . 2011 . . Proc. 24th Annual ACM Symp. on User Interface Software and Technology . . 43 - - 52 . . DOI: 10.1145/2047196.2047202 http://doi.org/10.1145/2047196.2047202 . .

V Krishna . . Auction Theory , , 2009 . . : New York, USA : Academic Press , , . .

I Krontiris , , , A Albers . . Monetary incentives in participatory sensing using multi-attributive auctions . . Int. J. Parall. Emerg. Distr. Syst. , , 2012 . . 27 ( ( 4 ): ): 317 - - 336 . . DOI: 10.1080/17445760.2012.686170 http://doi.org/10.1080/17445760.2012.686170 . .

E Law , , , L Ahn . . Human computation . . Synth. Lect. Artif. Intell. Mach. Learn. , , 2011 . . 5 ( ( 3 ): ): 1 - - 121 . . . .

D Lazer , , , R Kennedy , , , G King , , , 等 . . The parable of Google flu: traps in big data analysis . . Science , , 2014 . . 343 ( ( 6167 ): ): 1203 - - 1205 . . DOI: 10.1126/science.1248506 http://doi.org/10.1126/science.1248506 . .

JS Lee , , , B Hoh . . Sell your experiences: a market mechanism based incentive for participatory sensing . . 2010 . . IEEE Int. Conf. on Pervasive Computing and Communications . . 60 - - 68 . . DOI: 10.1109/PERCOM.2010.5466993 http://doi.org/10.1109/PERCOM.2010.5466993 . .

GL Li , , , JN Wang , , , YD Zheng , , , 等 . . Crowdsourced data management: a survey . . IEEE Trans. Knowl. Data Eng. , , 2016 . . 28 ( ( 9 ): ): 2296 - - 2319 . . DOI: 10.1109/TKDE.2016.2535242 http://doi.org/10.1109/TKDE.2016.2535242 . .

HW Li , , , B Yu . . Error rate bounds and iterative weighted majority voting for crowdsourcing . . ePrint Archive, arXiv:1411.4086 , , 2014 . . .

X Li , , , XL Dong , , , K Lyons , , , 等 . . Truth finding on the deep Web: is the problem solved? Proc . . VLDB Endow. , , 2012 . . 6 ( ( 2 ): ): 97 - - 108 . . DOI: 10.14778/2535568.2448943 http://doi.org/10.14778/2535568.2448943 . .

CJ Lintott , , , K Schawinski , , , A Slosar , , , 等 . . Galaxy Zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey . . Month. Not. R. Astronom. Soc. , , 2008 . . 389 ( ( 3 ): ): 1179 - - 1189 . . DOI: 10.1111/j.1365-2966.2008.13689.x http://doi.org/10.1111/j.1365-2966.2008.13689.x . .

Q Liu , , , J Peng , , , AT Ihler . . Variational inference for crowdsourcing . . Advances in Neural Information Processing Systems , , 2012 . . 692 - - 700 . . . .

T Luo , , , HP Tan , , , LR Xia . . Profit-maximizing incentive for participatory sensing . . 2014 . . Proc. IEEE Conf. on Computer Communications . . 127 - - 135 . . DOI: 10.1109/INFOCOM.2014.6847932 http://doi.org/10.1109/INFOCOM.2014.6847932 . .

TW Malone , , , R Laubacher , , , C Dellarocas . . Harnessing Crowds: Mapping the Genome of Collective Intelligence . . MIT Sloan Research Paper No. 4732-09, Sloan School of Management, Massachusetts Institute of Technology, MA, USA , , 2009 . . DOI: 10.2139/ssrn.1381502 http://doi.org/10.2139/ssrn.1381502 . .

L Mamykina , , , B Manoim , , , M Mittal , , , 等 . . Design lessons from the fastest Q&A site in the west . . 2011 . . Proc. SIGCHI Conf. on Human Factors in Computing Systems . . 2857 - - 2866 . . DOI: 10.1145/1978942.1979366 http://doi.org/10.1145/1978942.1979366 . .

AH Maslow , , , R Frager , , , J Fadiman , , , 等 . . Motivation and Personality , , 1970 . . : New York, USA : Harper & Row , , . .

P Mavridis , , , D Gross-Amblard , , , Z Mikls . . Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing . . 2016 . . Proc. 25th Int. Conf. on World Wide Web . . 843 - - 853 . . . .

R Meng , , , YX Tong , , , L Chen , , , 等 . . CrowdTC: crowdsourced taxonomy construction . . 2015 . . Proc. IEEE Int. Conf. on Data Mining . . 913 - - 918 . . DOI: 10.1109/ICDM.2015.77 http://doi.org/10.1109/ICDM.2015.77 . .

A Mockus , , , RT Fielding , , , JD Herbsleb . . Two case studies of open source software development: Apache and Mozilla . . ACM Trans. Softw. Eng. Meth. , , 2002 . . 11 ( ( 3 ): ): 309 - - 346 . . DOI: 10.1145/567793.567795 http://doi.org/10.1145/567793.567795 . .

R Moser , , , W Pedrycz , , , G Succi . . A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction . . 2008 . . Proc. 30th Int. Conf. on Software Engineering . . 181 - - 190 . . DOI: 10.1145/1368088.1368114 http://doi.org/10.1145/1368088.1368114 . .

N Nagappan , , , T Ball . . Use of relative code churn measures to predict system defect density . . 2005 . . Proc. 27th Int. Conf. on Software Engineering . . 284 - - 292 . . DOI: 10.1109/ICSE.2005.1553571 http://doi.org/10.1109/ICSE.2005.1553571 . .

K Nakakoji , , , Y Yamamoto , , , Y Nishinaka , , , 等 . . Evolution patterns of open-source software systems and communities . . 2002 . . Proc. Int. Workshop on Principles of Software Evolution . . 76 - - 85 . . DOI: 10.1145/512035.512055 http://doi.org/10.1145/512035.512055 . .

J Ok , , , S Oh , , , J Shin , , , 等 . . Optimality of belief propagation for crowdsourced classification . . ePrint Archive, arXiv:1602.03619 , , 2016 . . .

WR Ouyang , , , LM Kaplan , , , P Martin , , , 等 . . Debiasing crowdsourced quantitative characteristics in local businesses and services . . 2015 . . Proc. 14th Int. Conf. on Information Processing in Sensor Networks . . 190 - - 201 . . DOI: 10.1145/2737095.2737116 http://doi.org/10.1145/2737095.2737116 . .

YH Pan . . Heading toward artificial intelligence 2.0 . . Engineering , , 2016 . . 2 ( ( 4 ): ): 409 - - 413 . . DOI: 10.1016/J.ENG.2016.04.018 http://doi.org/10.1016/J.ENG.2016.04.018 . .

L Pierre . . Collective intelligence: mankind's emerging world in cyberspace . . 1997 . . Bononno, R., translator. Perseus Books, Cambridge, USA . . . .

AJ Quinn , , , BB Bederson . . Human computation: a survey and taxonomy of a growing field . . 2011 . . Proc. SIGCHI Conf. on Human Factors in Computing Systems . . 1403 - - 1412 . . DOI: 10.1145/1978942.1979148 http://doi.org/10.1145/1978942.1979148 . .

DR Raban , , , FM Harper . . Motivations for Answering Questions Online . . 2008 . . http://www.researchgate.net/publication/241053908 http://www.researchgate.net/publication/241053908 . .

F Rahman , , , PT Devanbu . . Ownership, experience and defects: a fine-grained study of authorship . . 2011 . . Proc. 33rd Int. Conf. on Software Engineering . . 491 - - 500 . . DOI: 10.1145/1985793.1985860 http://doi.org/10.1145/1985793.1985860 . .

F Rahman , , , PT Devanbu . . How, and why, process metrics are better . . 2013 . . Proc. 35th Int. Conf. on Software Engineering . . 432 - - 441 . . . .

RK Rana , , , CT Chou , , , SS Kanhere , , , 等 . . Earphone: an end-to-end participatory urban noise mapping system . . 2010 . . Proc. 9th ACM/IEEE Int. Conf. on Information Processing in Sensor Networks . . 105 - - 116 . . DOI: 10.1145/1791212.1791226 http://doi.org/10.1145/1791212.1791226 . .

VC Raykar , , , S Yu . . Eliminating spammers and ranking annotators for crowdsourced labeling tasks . . J. Mach. Learn. Res. , , 2012 . . 13 ( ( Feb ): ): 491 - - 518 . . . .

VC Raykar , , , S Yu , , , LH Zhao , , , 等 . . Supervised learning from multiple experts: whom to trust when everyone lies a bit . . 2009 . . Proc. 26th Int. Conf. on Machine Learning . . 889 - - 896 . . DOI: 10.1145/1553374.1553488 http://doi.org/10.1145/1553374.1553488 . .

VC Raykar , , , S Yu , , , LH Zhao , , , 等 . . Learning from crowds . . J. Mach. Learn. Res. , , 2010 . . 11 ( ( Apr ): ): 1297 - - 1322 . . . .

E Raymond . . The cathedral and the bazaar . . Knowl. Technol. Polic. , , 1999 . . 12 ( ( 3 ): ): 23 - - 49 . . DOI: 10.1007/s12130-999-1026-0 http://doi.org/10.1007/s12130-999-1026-0 . .

PC Rigby , , , DM German , , , L Cowen , , , 等 . . Peer review on open-source software projects: parameters, statistical models, and theory . . ACM Trans. Softw. Eng. Meth. , , 2014 . . 23 ( ( 4 ): ): No.35 DOI: 10.1145/2594458 http://doi.org/10.1145/2594458 . .

EM Rogers . . Diffusion of Innovations , , 2010 . . : New York, USA : Simon and Schuster , , . .

T Sakaki , , , M Okazaki , , , Y Matsuo . . Earthquake shakes Twitter users: real-time event detection by social sensors . . 2010 . . Proc. 19th Int. Conf. on World Wide Web . . 851 - - 860 . . DOI: 10.1145/1772690.1772777 http://doi.org/10.1145/1772690.1772777 . .

NB Shah , , , D Zhou . . Double or nothing: multiplicative incentive mechanisms for crowdsourcing . . Advances in Neural Information Processing Systems , , 2015 . . 1 - - 9 . . . .

NB Shah , , , D Zhou . . No oops, you won't do it again: mechanisms for self-correction in crowdsourcing . . 2016 . . Proc. 33rd Int. Conf. on Machine Learning . . 1 - - 10 . . . .

NB Shah , , , D Zhou , , , Y Peres . . Approval voting and incentives in crowdsourcing . . ePrint Archive, arXiv:1502.05696 , , 2015 . . .

JY She , , , YX Tong , , , L Chen . . Utility-aware social event-participant planning . . 2015a . . Proc. ACM Int. Conf. on Management of Data . . 1629 - - 1643 . . DOI: 10.1145/2723372.2749446 http://doi.org/10.1145/2723372.2749446 . .

JY She , , , YX Tong , , , L Chen , , , 等 . . Conflict-aware event-participant arrangement . . 2015b . . Proc. 31st IEEE Int. Conf. on Data Engineering . . 735 - - 746 . . DOI: 10.1109/ICDE.2015.7113329 http://doi.org/10.1109/ICDE.2015.7113329 . .

JY She , , , YX Tong , , , L Chen , , , 等 . . Conflict-aware event-participant arrangement and its variant for online setting . . IEEE Trans. Knowl. Data Eng. , , 2016 . . 28 ( ( 9 ): ): 2281 - - 2295 . . DOI: 10.1109/TKDE.2016.2565468 http://doi.org/10.1109/TKDE.2016.2565468 . .

HW Shen , , , AL Barabsi . . Collective credit allocation in science . . PNAS , , 2014 . . 111 ( ( 34 ): ): 12325 - - 12330 . . DOI: 10.1073/pnas.1401992111 http://doi.org/10.1073/pnas.1401992111 . .

JB Smith . . Collective Intelligence in Computer-Based Collaboration , , 1994 . . : USA : CRC Press, Boca Raton , , . .

A Subramanian , , , GS Kanth , , , R Vaze . . Offline and online incentive mechanism design for smart-phone crowd-sourcing . . ePrint Archive, arXiv:1310.1746 , , 2013 . . .

R Subramanyam , , , MS Krishnan . . Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects . . ACM Trans. Softw. Eng. Meth. , , 2003 . . 29 ( ( 4 ): ): 297 - - 310 . . DOI: 10.1109/TSE.2003.1191795 http://doi.org/10.1109/TSE.2003.1191795 . .

BL Sullivan , , , CL Wood , , , MJ Iliff , , , 等 . . eBird: a citizen-based bird observation network in the biological sciences . . Biol. Consev. , , 2009 . . 142 ( ( 10 ): ): 2282 - - 2292 . . DOI: 10.1016/j.biocon.2009.05.006 http://doi.org/10.1016/j.biocon.2009.05.006 . .

A Tamrawi , , , TT Nguyen , , , JM Al-Kofahi , , , 等 . . Fuzzy set and cache-based approach for bug triaging . . 2011 . . Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering . . 365 - - 375 . . DOI: 10.1145/2025113.2025163 http://doi.org/10.1145/2025113.2025163 . .

JC Tang , , , M Cebrian , , , NA Giacobe , , , 等 . . Reflecting on the DARPA red balloon challenge . . Commun. ACM , , 2011 . . 54 ( ( 4 ): ): 78 - - 85 . . DOI: 10.1145/1924421.1924441 http://doi.org/10.1145/1924421.1924441 . .

R Teodoro , , , P Ozturk , , , M Naaman , , , 等 . . The motivations and experiences of the on-demand mobile workforce . . 2014 . . Proc. 17th ACM Conf. on Computer Supported Cooperative Work & Social Computing . . 236 - - 247 . . DOI: 10.1145/2531602.2531680 http://doi.org/10.1145/2531602.2531680 . .

J Thebault-Spieker , , , LG Terveen , , , B Hecht . . Avoiding the south side and the suburbs: the geography of mobile crowdsourcing markets . . 2015 . . Proc. 18th ACM Conf. on Computer Supported Cooperative Work & Social Computing . . 265 - - 275 . . DOI: 10.1145/2675133.2675278 http://doi.org/10.1145/2675133.2675278 . .

P Thongtanunam , , , C Tantithamthavorn , , , RG Kula , , , 等 . . Who should review my code? A file locationbased code-reviewer recommendation approach for modern code review . . 2015 . . IEEE 22nd Int. Conf. on Software Analysis, Evolution, and Reengineering . . 141 - - 150 . . DOI: 10.1109/SANER.2015.7081824 http://doi.org/10.1109/SANER.2015.7081824 . .

T Tian , , , J Zhu . . Max-margin majority voting for learning from crowds . . 2015 . . Advances in Neural Information Processing Systems . . 1621 - - 1629 . . . .

YX Tong , , , L Chen , , , BL Ding . . Discovering threshold-based frequent closed itemsets over probabilistic data . . 2012a . . Proc. IEEE 28th Int. Conf. on Data Engineering . . 270 - - 281 . . DOI: 10.1109/ICDE.2012.51 http://doi.org/10.1109/ICDE.2012.51 . .

YX Tong , , , L Chen , , , YR Cheng , , , 等 . . Mining frequent itemsets over uncertain databases . . Proc. VLDB Endow. , , 2012b . . 5 ( ( 11 ): ): 1650 - - 1661 . . . .

YX Tong , , , CC Cao , , , CJ Zhang , , , 等 . . CrowdCleaner: data cleaning for multi-version data on the web via crowdsourcing . . 2014a . . Proc. IEEE 28th Int. Conf. on Data Engineering . . 1182 - - 1185 . . DOI: 10.1109/ICDE.2014.6816736 http://doi.org/10.1109/ICDE.2014.6816736 . .

YX Tong , , , CC Cao , , , L Chen . . TCS: efficient topic discovery over crowd-oriented service data . . 2014b . . Proc. 20th ACM Int. Conf. on Knowledge Discovery and Data Mining . . 861 - - 870 . . DOI: 10.1145/2623330.2623647 http://doi.org/10.1145/2623330.2623647 . .

YX Tong , , , L Chen , , , JY She . . Mining frequent itemsets in correlated uncertain databases . . J. Comput. Sci. Technol. , , 2015a . . 30 ( ( 4 ): ): 696 - - 712 . . DOI: 10.1007/s11390-015-1555-9 http://doi.org/10.1007/s11390-015-1555-9 . .

YX Tong , , , R Meng , , , JY She . . On bottleneck-aware arrangement for event-based social networks . . 2015b . . Proc. 31st IEEE Int. Conf. on Data Engineering Workshops . . 216 - - 223 . . DOI: 10.1109/ICDEW.2015.7129579 http://doi.org/10.1109/ICDEW.2015.7129579 . .

YX Tong , , , JY She , , , R Meng . . Bottleneck-aware arrangement over event-based social networks: the maxmin approach . . World Wide Web , , 2016a . . 19 ( ( 6 ): ): 1151 - - 1177 . . DOI: 10.1007/s11280-015-0377-6 http://doi.org/10.1007/s11280-015-0377-6 . .

YX Tong , , , JY She , , , BL Ding , , , 等 . . Online mobile micro-task allocation in spatial crowdsourcing . . 2016b . . Proc. 32nd IEEE Int. Conf. on Data Engineering . . 49 - - 60 . . DOI: 10.1109/ICDE.2016.7498228 http://doi.org/10.1109/ICDE.2016.7498228 . .

YX Tong , , , JY She , , , BL Ding , , , 等 . . Online minimum matching in real-time spatial data: experiments and analysis . . Proc. VLDB Endow. , , 2016c . . 9 ( ( 12 ): ): 1053 - - 1064 . . DOI: 10.14778/2994509.2994523 http://doi.org/10.14778/2994509.2994523 . .

YX Tong , , , XF Zhang , , , L Chen . . Tracking frequent items over distributed probabilistic data . . World Wide Web , , 2016d . . 19 ( ( 4 ): ): 579 - - 604 . . DOI: 10.1007/s11280-015-0341-5 http://doi.org/10.1007/s11280-015-0341-5 . .

YX Tong , , , Y Yuan , , , YR Cheng , , , 等 . . A survey of spatiotemporal crowdsourced data management techniques . . J. Softw. , , 2017 . . 28 ( ( 1 ): ): 35 - - 58 . . . .

L Tran-Thanh , , , S Stein , , , A Rogers , , , 等 . . Efficient crowdsourcing of unknown experts using multi-armed bandits . . 2012 . . Proc. European Conf. on Artificial Intelligence . . 768 - - 773 . . DOI: 10.3233/978-1-61499-098-7-768 http://doi.org/10.3233/978-1-61499-098-7-768 . .

J Tsay , , , L Dabbish , , , JD Herbsleb . . Influence of social and technical factors for evaluating contribution in GitHub . . 2014a . . 36th Int. Conf. on Software Engineering . . 356 - - 366 . . DOI: 10.1145/2568225.2568315 http://doi.org/10.1145/2568225.2568315 . .

J Tsay , , , L Dabbish , , , JD Herbsleb . . Let's talk about it: evaluating contributions through discussion in GitHub . . 2014b . . Proc. 22nd ACM Int. Symp. on Foundations of Software Engineering . . 144 - - 154 . . DOI: 10.1145/2635868.2635882 http://doi.org/10.1145/2635868.2635882 . .

B Vasilescu , , , Y Yu , , , H Wang , , , 等 . . Quality and productivity outcomes relating to continuous integration in GitHub . . 2015 . . Proc. 10th Joint Meeting on Foundations of Software Engineering . . 805 - - 816 . . DOI: 10.1145/2786805.2786850 http://doi.org/10.1145/2786805.2786850 . .

W Vickrey . . Counterspeculation, auctions, and competitive sealed tenders . . J. Finan. , , 1961 . . 16 ( ( 1 ): ): 8 - - 37 . . DOI: 10.1111/j.1540-6261.1961.tb02789.x http://doi.org/10.1111/j.1540-6261.1961.tb02789.x . .

L von Ahn , , , B Maurer , , , C McMillen , , , 等 . . reCAPTCHA: human-based character recognition via web security measures . . Science , , 2008 . . 321 ( ( 5895 ): ): 1465 - - 1468 . . DOI: 10.1126/science.1160379 http://doi.org/10.1126/science.1160379 . .

D Wang , , , TF Abdelzaher , , , LM Kaplan , , , 等 . . Recursive fact-finding: a streaming approach to truth estimation in crowdsourcing applications . . 2013 . . IEEE 33rd Int. Conf. on Distributed Computing Systems . . 530 - - 539 . . DOI: 10.1109/ICDCS.2013.54 http://doi.org/10.1109/ICDCS.2013.54 . .

D Wang , , , MTA Amin , , , S Li , , , 等 . . Using humans as sensors: an estimation-theoretic perspective . . 2014 . . Proc. 13th Int. Conf. on Information Processing in Sensor Networks . . 35 - - 46 . . . .

HM Wang , , , G Yin , , , X Li , , , 等 . . TRUSTIE: a software development platform for crowdsourcing . . 2015 . . In: Li W, Huhns MN, Tsai WT, et al. (Eds.), Crowdsourcing. Springer Berlin Heidelberg, Berlin, Germany . . 165 - - 190 . . . .

JN Wang , , , GL Li , , , T Kraska , , , 等 . . Leveraging transitive relations for crowdsourced joins . . 2013 . . Proc. ACM Int. Conf. on Management of Data . . 229 - - 240 . . DOI: 10.1145/2463676.2465280 http://doi.org/10.1145/2463676.2465280 . .

L Wang , , , ZH Zhou . . Cost-saving effect of crowdsourcing learning . . 2016 . . Proc. 25th Int. Joint Conf. on Artificial Intelligence . . 2111 - - 2117 . . . .

W Wang , , , ZH Zhou . . Crowdsourcing label quality: a theoretical study . . Sci. China Inform. Sci. , , 2015 . . 58 ( ( 11 ): ): 1 - - 12 . . DOI: 10.1007/s11432-015-5391-x http://doi.org/10.1007/s11432-015-5391-x . .

FL Wauthier , , , MI Jordan . . Bayesian bias mitigation for crowdsourcing , , 2011 . . Advances in Neural Information Processing Systems , , 1800 - - 1808 . . . .

P Welinder , , , S Branson , , , P Perona , , , 等 . . The multidimensional wisdom of crowds . . 2010 . . Advances in Neural Information Processing Systems . . 2424 - - 2432 . . . .

J Whitehill , , , T Wu , , , J Bergsma , , , 等 . . Whose vote should count more: optimal integration of labels from labelers of unknown expertise . . Advances in Neural Information Processing Systems , , 2009 . . 2035 - - 2043 . . . .

WJ Wu , , , WT Tsai , , , W Li . . An evaluation framework for software crowdsourcing . . Front. Comput. Sci. , , 2013 . . 7 ( ( 5 ): ): 694 - - 709 . . DOI: 10.1007/s11704-013-2320-2 http://doi.org/10.1007/s11704-013-2320-2 . .

Y Yan , , , GM Fung , , , R Rosales , , , 等 . . Active learning from crowds . . 2011 . . Proc. 28th Int. Conf. on Machine Learning . . 1161 - - 1168 . . . .

DJ Yang , , , X Fang , , , GL Xue . . Truthful incentive mechanisms for k-anonymity location privacy . . 2013 . . Proc. IEEE Conf. on Computer Communications . . 2994 - - 3002 . . DOI: 10.1109/INFCOM.2013.6567111 http://doi.org/10.1109/INFCOM.2013.6567111 . .

DJ Yang , , , GL Xue , , , X Fang , , , 等 . . Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing . . 2012 . . Proc. 18th Annual Int. Conf. on Mobile Computing and Networking . . 173 - - 184 . . DOI: 10.1145/2348543.2348567 http://doi.org/10.1145/2348543.2348567 . .

YW Ye , , , K Kishida . . Toward an understanding of the motivation of open source software developers . . 2003 . . Proc. 25th Int. Conf. on Software Engineering . . 419 - - 429 . . DOI: 10.1109/ICSE.2003.1201220 http://doi.org/10.1109/ICSE.2003.1201220 . .

Y Yu , , , G Yin , , , H Wang , , , 等 . . Exploring the patterns of social behavior in GitHub . . 2014 . . Proc. 1st Int. Workshop on Crowd-Based Software Development Methods and Technologies . . 31 - - 36 . . DOI: 10.1145/2666539.2666571 http://doi.org/10.1145/2666539.2666571 . .

Y Yu , , , G Yin , , , T Wang , , , 等 . . Determinants of pull-based development in the context of continuous integration . . Sci. China Inform. Sci. , , 2016a . . 59 ( ( 8 ): ): 080104 DOI: 10.1007/s11432-016-5595-8 http://doi.org/10.1007/s11432-016-5595-8 . .

Y Yu , , , HM Wang , , , G Yin , , , 等 . . Reviewer recommendation for pull-requests in GitHub: what can we learn from code review and bug assignment? Imform . . Softw. Technol. , , 2016b . . 74 204 - - 218 . . DOI: 10.1016/j.infsof.2016.01.004 http://doi.org/10.1016/j.infsof.2016.01.004 . .

CJ Zhang , , , L Chen , , , YX Tong . . MaC: a probabilistic framework for query answering with machinecrowd collaboration . . 2014a . . Proc. 23rd ACM Int. Conf. on Information and Knowledge Management . . 11 - - 20 . . DOI: 10.1145/2661829.2661880 http://doi.org/10.1145/2661829.2661880 . .

CJ Zhang , , , YX Tong , , , L Chen . . Where to: crowdaided path selection . . Proc. VLDB Endow. , , 2014b . . 7 ( ( 11 ): ): 2005 - - 2016 . . DOI: 10.14778/2733085.2733105 http://doi.org/10.14778/2733085.2733105 . .

CJ Zhang , , , L Chen , , , Y Tong , , , 等 . . Cleaning uncertain data with a noisy crowd . . 2015 . . Proc. 31st IEEE Int. Conf. on Data Engineering . . 6 - - 17 . . DOI: 10.1109/ICDE.2015.7113268 http://doi.org/10.1109/ICDE.2015.7113268 . .

Y Zhang , , , X Chen , , , D Zhou , , , 等 . . Spectral methods meet EM: a provably optimal algorithm for crowdsourcing . . Advances in Neural Information Processing Systems , , 2014 . . 1260 - - 1268 . . . .

D Zhao , , , XY Li , , , HD Ma . . How to crowdsource tasks truthfully without sacrificing utility: online incentive mechanisms with budget constraint . . 2014 . . Proc. IEEE Conf. on Computer Communications . . 1213 - - 1221 . . DOI: 10.1109/INFOCOM.2014.6848053 http://doi.org/10.1109/INFOCOM.2014.6848053 . .

JH Zhong , , , K Tang , , , ZH Zhou . . Active learning from crowds with unsure option . . 2015 . . Proc. 24th Int. Joint Conf. on Artificial Intelligence . . 1061 - - 1067 . . . .

D Zhou , , , S Basu , , , Y Mao , , , 等 . . Learning from the wisdom of crowds by minimax entropy . . Advances in Neural Information Processing Systems , , 2012 . . 2195 - - 2203 . . . .

Y Zhou , , , X Chen , , , J Li . . Optimal PAC multiple arm identification with applications to crowdsourcing . . 2014 . . Proc. 31st Int. Conf. on Machine Learning . . 217 - - 225 . . . .

Y Zhu , , , Q Zhang , , , H Zhu , , , 等 . . Towards truthful mechanisms for mobile crowdsourcing with dynamic smartphones . . 2014 . . Proc. 34th Int. Conf. on Distributed Computing Systems . . 11 - - 20 . . DOI: 10.1109/ICDCS.2014.10 http://doi.org/10.1109/ICDCS.2014.10 . .

Views

255

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Crowdsourcing intelligent design

Related Author

Wei XIANG

Ling-yun SUN

Wei-tao YOU

Chang-yuan YANG

Related Institution

State Key Lab of CAD & CG, Zhejiang University

Modern Industrial Design Institute, Zhejiang University

International User Experience Business Unit, Alibaba Group

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰