Crowd intelligence in AI 2.0 era

LI Wei; Wen-jun WU; Huai-min WANG; Xue-qi CHENG; Hua-jun CHEN; Zhi-hua ZHOU; Rong DING

doi:10.1631/FITEE.1601859

Your Location：

Home >

Browse articles >

Crowd intelligence in AI 2.0 era

Special Issue on Artificial Intelligence 2.0 | Updated：2022-05-19

- Crowd intelligence in AI 2.0 era
- AI2.0时代的群体智能
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 1, Pages: 15-43(2017)
- Affiliations：
  
  1State Key Laboratory of Software Development, Beihang University, Beijing 100191, China
  2National Laboratory for Parallel and Distributed Processing, College of Computer, National University of Defense Technology, Changsha 410073, China
  3Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
  4College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  5National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
- Author bio：
  
  E-mail:wwj@nlsde.buaa.edu.cn
- Funds：
- DOI：10.1631/FITEE.1601859
  CLC：
Scan for full text
LI Wei, Wen-jun WU, Huai-min WANG, et al. Crowd intelligence in AI 2.0 era. [J]. Frontiers of Information Technology & Electronic Engineering 18(1):15-43(2017)
DOI：

LI Wei, Wen-jun WU, Huai-min WANG, et al. Crowd intelligence in AI 2.0 era. [J]. Frontiers of Information Technology & Electronic Engineering 18(1):15-43(2017) DOI： 10.1631/FITEE.1601859.

摘要

基于互联网的信息物理世界深刻地改变了人工智能（artificial intelligence

AI）发展的信息环境，将人工智能研究的新浪潮推进到人工智能2.0新纪元。作为AI2.0时代最突出的研究特点之一，群体智能引起了产业界和学术界的广泛关注。具体来说，为应对挑战，群体智能提供了一种通过聚集群体的智慧解决问题的新模式。特别是由于共享经济的快速发展，群体智能不仅成为了解决科学难题的新途径，而且也已融入日常生活的各个方面，例如线上到线下（online-to-offline

O2O）应用、实时交通监控、以及物流管理。本文对现有群体智能研究成果进行总结和综述：首先，论述了群体智能的基本概念，并对其与现有相关概念（如众包和人本计算）的关系进行了解释。然后，介绍了四类具有代表性的群体智能平台，总结了三项核心问题以及最新的群体智能技术。最后，讨论了群体智能研究的未来发展方向。

Abstract

The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence (AI)

bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era

crowd intelligence has attracted much attention from both industry and research communities. Specifically

crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular

due to the rapid development of the sharing economy

crowd intelligence not only becomes a new approach to solving scientific challenges

but has also been integrated into all kinds of application scenarios in daily life

e.g.

online-tooffline (O2O) application

real-time traffic monitoring

and logistics management. In this paper

we survey existing studies of crowd intelligence. First

we describe the concept of crowd intelligence

and explain its relationship to the existing related concepts

e.g.

crowdsourcing and human computation. Then

we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally

we discuss promising future research directions of crowd intelligence.

关键词

群体智能人工智能2.0（AI2.0）众包人本计算

Keywords

Crowd intelligenceArtificial intelligence 2.0CrowdsourcingHuman computation

references

I Abraham,,,O Alonso,,,V Kandylas,,,等..Adaptive crowdsourcing algorithms for the bandit survey problem..2013..Proc. 26th Conf. on Computational Learning Theory..882--910....

J Ballesteros,,,B Carbunar,,,M Rahman,,,等..Towards safe cities: a mobile and social networking approach..IEEE Trans. Parall. Distr. Syst.,,2014..25((9):):2451--2462..DOI:10.1109/TPDS.2013.190http://doi.org/10.1109/TPDS.2013.190..

VR Basili,,,LC Briand,,,WL Melo..A validation of object-oriented design metrics as quality indicators..IEEE Trans. Softw. Eng.,,1996..22((10):):751--761..DOI:10.1109/32.544352http://doi.org/10.1109/32.544352..

P Bhattacharya,,,I Neamtiu..Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging..2010..IEEE Int. Conf. on Software Maintenance..1--10..DOI:10.1109/ICSM.2010.5609736http://doi.org/10.1109/ICSM.2010.5609736..

C Bird,,,A Gourley,,,P Devanbu,,,等..Mining email social networks..2006..Proc. Int. Workshop on Mining Software Repositories..137--143..DOI:10.1145/1137983.1138016http://doi.org/10.1145/1137983.1138016..

C Bird,,,D Pattison,,,R de Souza,,,等..Latent social structure in open source projects..2008..Proc. 16th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering..24--35..DOI:10.1145/1453101.1453107http://doi.org/10.1145/1453101.1453107..

C Bird,,,N Nagappan,,,B Murphy,,,等..Don't touch my code!: examining the effects of ownership on software quality..2011..Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering..4--14..DOI:10.1145/2025113.2025119http://doi.org/10.1145/2025113.2025119..

J Bollen,,,HN Mao,,,XJ Zeng..Twitter mood predicts the stock market..J. Comput. Sci.,,2011..2((1):):1--8..DOI:10.1016/j.jocs.2010.12.007http://doi.org/10.1016/j.jocs.2010.12.007..

E Bonabeau..Decisions 2.0: the power of collective intelligence..MIT Sloan Manag. Rev.,,2009..50((2):):45--52....

KD Borne,,,Team Zooniverse..The Zooniverse: a framework for knowledge discovery from citizen science data..American Geophysical Union Fall Meeting,,2011...

JA Burke,,,D Estrin,,,M Hansen,,,等..Participatory sensing..Workshop on World-Sensor-Web: Mobile Device Centric Sensor Networks and Applications,,2006..117--134....

CC Cao,,,JY She,,,YX Tong,,,等..Whom to ask? Jury selection for decision making tasks on micro-blog services..Proc. VLDB Endow.,,2012..5((11):):1495--1506..DOI:10.14778/2350229.2350264http://doi.org/10.14778/2350229.2350264..

CC Cao,,,YX Tong,,,L Chen,,,等..Wisemarket: a new paradigm for managing wisdom of online social users..2013..Proc. 19th ACM Int. Conf. on Knowledge Discovery and Data Mining..455--463..DOI:10.1145/2487575.2487642http://doi.org/10.1145/2487575.2487642..

OF Castaneda..Hierarchy in Meritocracy: Community Building and Code Production in the Apache Software Foundation..MS Thesis, Delft University of Technology, Delft, Netherlands,,2010...

X Chen,,,QH Lin,,,DY Zhou..Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing..2013..Proc. 30th Int. Conf. on Machine Learning..64--72....

X Chen,,,QH Lin,,,DY Zhou..Statistical decision making for optimal budget allocation in crowd labeling..J. Mach. Learn. Res.,,2015..161--46....

CAL Dantec,,,M Asad,,,A Misra,,,等..Planning with crowdsourced data: rhetoric and representation in transportation planning..2015..Proc. 18th ACM Conf. on Computer Supported Cooperative Work..1717--1727..DOI:10.1145/2675133.2675212http://doi.org/10.1145/2675133.2675212..

AP Dawid,,,AM Skene..Maximum likelihood estimation of observer error-rates using the EM algorithm..Appl. Statist.,,1979..28((1):):20--28..DOI:10.2307/2346806http://doi.org/10.2307/2346806..

B de Alwis,,,J Sillito..Why are software projects moving from centralized to decentralized version control systems?..2009..Proc. ICSE Workshop on Cooperative and Human Aspects on Software Engineering..36--39..DOI:10.1109/CHASE.2009.5071408http://doi.org/10.1109/CHASE.2009.5071408..

O Dekel,,,O Shamir..Vox Populi: collecting highquality labels from a crowd..2009..Proc. 22nd Conf. on Learning Theory....

AP Dempster,,,NM Laird,,,DB Rubin..Maximum likelihood from incomplete data via the EM algorithm..J. R. Stat. Soc. Ser. B,,1977..39((1):):1--38....

DE Difallah,,,G Demartini,,,GP Cudr-Mauroux..Pick-a-crowd: tell me what you like, and I'll tell you what to do..2013..Proc. 22nd Int. Conf. on World Wide Web..367--374..DOI:10.1145/2488388.2488421http://doi.org/10.1145/2488388.2488421..

DE Difallah,,,G Demartini,,,GP Cudr-Mauroux..Scheduling human intelligence tasks in multi-tenant crowd-powered systems..2016..Proc. 25th Int. Conf. on World Wide Web..855--865..DOI:10.1145/2872427.2883030http://doi.org/10.1145/2872427.2883030..

XL Dong,,,B Saha,,,D Srivastava..Less is more: selecting sources wisely for integration..Proc. VLDB Endow.,,2012..6((2):):37--48..DOI:10.14778/2535568.2448938http://doi.org/10.14778/2535568.2448938..

JR Erenkrantz,,,RN Taylor..Supporting Distributed and Decentralized Projects: Drawing Lessons From the Open Source Community,,2003..:ISR Technical Report No. UCI-ISR-03-4:Institute for Software Research, University of California,,Irvine, USA..

K Farkas,,,AZ Nagy,,,T Toms,,,等..Participatory sensing based real-time public transport information service..2014..IEEE Int. Conf. on Pervasive Computing and Communications Workshops..141--144..DOI:10.1109/PerComW.2014.6815181http://doi.org/10.1109/PerComW.2014.6815181..

ZN Feng,,,YM Zhu,,,Q Zhang,,,等..Trac: truthful auction for location-aware collaborative sensing in mobile crowdsourcing..2014..Proc. IEEE Conf. on Computer Communications..1231--1239..DOI:10.1109/INFOCOM.2014.6848055http://doi.org/10.1109/INFOCOM.2014.6848055..

G Fowler,,,J Schectman..Citizen surveillance helps officials put pieces together,,2013..The Wall Street Journal, April 17.http://www.wsj.com/articles/SB10001424127887324763404578429220091342796http://www.wsj.com/articles/SB10001424127887324763404578429220091342796..

C Gao,,,DY Zhou..Minimax optimal convergence rates for estimating ground truth from crowdsourced labels..ePrint Archive, arXiv:1310.5764,,2013...

DW Gao,,,YX Tong,,,JY She,,,等..Top-k team recommendation in spatial crowdsourcing..LNCS,,2016..9658191--204..DOI:10.1007/978-3-319-39937-9_15http://doi.org/10.1007/978-3-319-39937-9_15..

L Gao,,,F Hou,,,JW Huang..Providing long-term participation incentive in participatory sensing..2015..IEEE Conf. on Computer Communications..2803--2811..DOI:10.1109/INFOCOM.2015.7218673http://doi.org/10.1109/INFOCOM.2015.7218673..

RA Ghosh..Understanding Free Software Developers: Findings from the Floss Study,,2005..:Cambrige, USA:MIT Press,,..

G Gousios,,,M Pinzger,,,A Deursen..An exploratory study of the pull-based software development model..2014..Proc. 36th Int. Conf. on Software Engineering..345--355..DOI:10.1145/2568225.2568260http://doi.org/10.1145/2568225.2568260..

G Gousios,,,A Zaidman,,,MA Storey,,,等..Work practices and challenges in pull-based development: the integrator's perspective..2015..Proc. 37th Int. Conf. on Software Engineering..358--368..DOI:10.1109/ICSE.2015.55http://doi.org/10.1109/ICSE.2015.55..

K Han,,,C Zhang,,,J Luo,,,等..Truthful scheduling mechanisms for powering mobile crowdsensing..IEEE Trans. Comput.,,2016..65((1):):294--307..DOI:10.1109/TC.2015.2419658http://doi.org/10.1109/TC.2015.2419658..

A Hars,,,S Ou..Working for free? Motivations of participating in open source projects..2001..Proc. 34th Annual Hawaii Int. Conf. on System Sciences..1--9..DOI:10.1109/HICSS.2001.927045http://doi.org/10.1109/HICSS.2001.927045..

AE Hassan..Predicting faults using the complexity of code changes..2009..Proc. 31st Int. Conf. on Software Engineering..78--88..DOI:10.1109/ICSE.2009.5070510http://doi.org/10.1109/ICSE.2009.5070510..

G Hertel,,,S Niedner,,,S Herrmann..Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel..Res. Polic.,,2003..32((7):):1159--1177..DOI:10.1016/S0048-7333(03)00047-7http://doi.org/10.1016/S0048-7333(03)00047-7..

CJ Ho,,,JW Vaughan..Online task assignment in crowdsourcing markets..2012..Proc. 26th AAAI Conf. on Artificial Intelligence..45--51....

CJ Ho,,,S Jabbari,,,JW Vaughan..Adaptive task assignment for crowdsourced classification..2013..Proc. 30th Int. Conf. on Machine Learning..534--542....

ML Hoffman..Is altruism part of human nature? J..Personal. Soc. Psychol.,,1981..40((1):):121--137..DOI:10.1037/0022-3514.40.1.121http://doi.org/10.1037/0022-3514.40.1.121..

LG Jaimes,,,I Vergara-Laurens,,,MA Labrador..A location-based incentive mechanism for participatory sensing systems with budget constraints..2012..Proc. 10th Annual IEEE Int. Conf. on Pervasive Computing and Communications..103--108..DOI:10.1109/PerCom.2012.6199855http://doi.org/10.1109/PerCom.2012.6199855..

S Jain,,,S Gujar,,,S Bhat,,,等..An incentive compatible multi-armed-bandit crowdsourcing mechanism with quality assurance..ePrint Archive, arXiv:1406.7157,,2014...

G Jeong,,,S Kim,,,T Zimmermann..Improving bug triage with bug tossing graphs..2009..Proc. 7th Joint Meeting of the European Software Engineering Conf. and the ACM SIGSOFT Symp. on the Foundations of Software Engineering..111--120..DOI:10.1145/1595696.1595715http://doi.org/10.1145/1595696.1595715..

DR Karger,,,S Oh,,,D Shah..Iterative learning for reliable crowdsourcing systems..2011..Advances in Neural Information Processing Systems..1953--1961....

A Khetan,,,S Oh..Achieving budget-optimality with adaptive schemes in crowdsourcing..2016..Advances in Neural Information Processing Systems..4844--4852....

A Kittur,,,B Smus,,,S Khamkar,,,等..Crowdforge: crowdsourcing complex work..2011..Proc. 24th Annual ACM Symp. on User Interface Software and Technology..43--52..DOI:10.1145/2047196.2047202http://doi.org/10.1145/2047196.2047202..

V Krishna..Auction Theory,,2009..:New York, USA:Academic Press,,..

I Krontiris,,,A Albers..Monetary incentives in participatory sensing using multi-attributive auctions..Int. J. Parall. Emerg. Distr. Syst.,,2012..27((4):):317--336..DOI:10.1080/17445760.2012.686170http://doi.org/10.1080/17445760.2012.686170..

E Law,,,L Ahn..Human computation..Synth. Lect. Artif. Intell. Mach. Learn.,,2011..5((3):):1--121....

D Lazer,,,R Kennedy,,,G King,,,等..The parable of Google flu: traps in big data analysis..Science,,2014..343((6167):):1203--1205..DOI:10.1126/science.1248506http://doi.org/10.1126/science.1248506..

JS Lee,,,B Hoh..Sell your experiences: a market mechanism based incentive for participatory sensing..2010..IEEE Int. Conf. on Pervasive Computing and Communications..60--68..DOI:10.1109/PERCOM.2010.5466993http://doi.org/10.1109/PERCOM.2010.5466993..

GL Li,,,JN Wang,,,YD Zheng,,,等..Crowdsourced data management: a survey..IEEE Trans. Knowl. Data Eng.,,2016..28((9):):2296--2319..DOI:10.1109/TKDE.2016.2535242http://doi.org/10.1109/TKDE.2016.2535242..

HW Li,,,B Yu..Error rate bounds and iterative weighted majority voting for crowdsourcing..ePrint Archive, arXiv:1411.4086,,2014...

X Li,,,XL Dong,,,K Lyons,,,等..Truth finding on the deep Web: is the problem solved? Proc..VLDB Endow.,,2012..6((2):):97--108..DOI:10.14778/2535568.2448943http://doi.org/10.14778/2535568.2448943..

CJ Lintott,,,K Schawinski,,,A Slosar,,,等..Galaxy Zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey..Month. Not. R. Astronom. Soc.,,2008..389((3):):1179--1189..DOI:10.1111/j.1365-2966.2008.13689.xhttp://doi.org/10.1111/j.1365-2966.2008.13689.x..

Q Liu,,,J Peng,,,AT Ihler..Variational inference for crowdsourcing..Advances in Neural Information Processing Systems,,2012..692--700....

T Luo,,,HP Tan,,,LR Xia..Profit-maximizing incentive for participatory sensing..2014..Proc. IEEE Conf. on Computer Communications..127--135..DOI:10.1109/INFOCOM.2014.6847932http://doi.org/10.1109/INFOCOM.2014.6847932..

TW Malone,,,R Laubacher,,,C Dellarocas..Harnessing Crowds: Mapping the Genome of Collective Intelligence..MIT Sloan Research Paper No. 4732-09, Sloan School of Management, Massachusetts Institute of Technology, MA, USA,,2009..DOI:10.2139/ssrn.1381502http://doi.org/10.2139/ssrn.1381502..

L Mamykina,,,B Manoim,,,M Mittal,,,等..Design lessons from the fastest Q&A site in the west..2011..Proc. SIGCHI Conf. on Human Factors in Computing Systems..2857--2866..DOI:10.1145/1978942.1979366http://doi.org/10.1145/1978942.1979366..

AH Maslow,,,R Frager,,,J Fadiman,,,等..Motivation and Personality,,1970..:New York, USA:Harper & Row,,..

P Mavridis,,,D Gross-Amblard,,,Z Mikls..Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing..2016..Proc. 25th Int. Conf. on World Wide Web..843--853....

R Meng,,,YX Tong,,,L Chen,,,等..CrowdTC: crowdsourced taxonomy construction..2015..Proc. IEEE Int. Conf. on Data Mining..913--918..DOI:10.1109/ICDM.2015.77http://doi.org/10.1109/ICDM.2015.77..

A Mockus,,,RT Fielding,,,JD Herbsleb..Two case studies of open source software development: Apache and Mozilla..ACM Trans. Softw. Eng. Meth.,,2002..11((3):):309--346..DOI:10.1145/567793.567795http://doi.org/10.1145/567793.567795..

R Moser,,,W Pedrycz,,,G Succi..A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction..2008..Proc. 30th Int. Conf. on Software Engineering..181--190..DOI:10.1145/1368088.1368114http://doi.org/10.1145/1368088.1368114..

N Nagappan,,,T Ball..Use of relative code churn measures to predict system defect density..2005..Proc. 27th Int. Conf. on Software Engineering..284--292..DOI:10.1109/ICSE.2005.1553571http://doi.org/10.1109/ICSE.2005.1553571..

K Nakakoji,,,Y Yamamoto,,,Y Nishinaka,,,等..Evolution patterns of open-source software systems and communities..2002..Proc. Int. Workshop on Principles of Software Evolution..76--85..DOI:10.1145/512035.512055http://doi.org/10.1145/512035.512055..

J Ok,,,S Oh,,,J Shin,,,等..Optimality of belief propagation for crowdsourced classification..ePrint Archive, arXiv:1602.03619,,2016...

WR Ouyang,,,LM Kaplan,,,P Martin,,,等..Debiasing crowdsourced quantitative characteristics in local businesses and services..2015..Proc. 14th Int. Conf. on Information Processing in Sensor Networks..190--201..DOI:10.1145/2737095.2737116http://doi.org/10.1145/2737095.2737116..

YH Pan..Heading toward artificial intelligence 2.0..Engineering,,2016..2((4):):409--413..DOI:10.1016/J.ENG.2016.04.018http://doi.org/10.1016/J.ENG.2016.04.018..

L Pierre..Collective intelligence: mankind's emerging world in cyberspace..1997..Bononno, R., translator. Perseus Books, Cambridge, USA....

AJ Quinn,,,BB Bederson..Human computation: a survey and taxonomy of a growing field..2011..Proc. SIGCHI Conf. on Human Factors in Computing Systems..1403--1412..DOI:10.1145/1978942.1979148http://doi.org/10.1145/1978942.1979148..

DR Raban,,,FM Harper..Motivations for Answering Questions Online..2008..http://www.researchgate.net/publication/241053908http://www.researchgate.net/publication/241053908..

F Rahman,,,PT Devanbu..Ownership, experience and defects: a fine-grained study of authorship..2011..Proc. 33rd Int. Conf. on Software Engineering..491--500..DOI:10.1145/1985793.1985860http://doi.org/10.1145/1985793.1985860..

F Rahman,,,PT Devanbu..How, and why, process metrics are better..2013..Proc. 35th Int. Conf. on Software Engineering..432--441....

RK Rana,,,CT Chou,,,SS Kanhere,,,等..Earphone: an end-to-end participatory urban noise mapping system..2010..Proc. 9th ACM/IEEE Int. Conf. on Information Processing in Sensor Networks..105--116..DOI:10.1145/1791212.1791226http://doi.org/10.1145/1791212.1791226..

VC Raykar,,,S Yu..Eliminating spammers and ranking annotators for crowdsourced labeling tasks..J. Mach. Learn. Res.,,2012..13((Feb):):491--518....

VC Raykar,,,S Yu,,,LH Zhao,,,等..Supervised learning from multiple experts: whom to trust when everyone lies a bit..2009..Proc. 26th Int. Conf. on Machine Learning..889--896..DOI:10.1145/1553374.1553488http://doi.org/10.1145/1553374.1553488..

VC Raykar,,,S Yu,,,LH Zhao,,,等..Learning from crowds..J. Mach. Learn. Res.,,2010..11((Apr):):1297--1322....

E Raymond..The cathedral and the bazaar..Knowl. Technol. Polic.,,1999..12((3):):23--49..DOI:10.1007/s12130-999-1026-0http://doi.org/10.1007/s12130-999-1026-0..

PC Rigby,,,DM German,,,L Cowen,,,等..Peer review on open-source software projects: parameters, statistical models, and theory..ACM Trans. Softw. Eng. Meth.,,2014..23((4):):No.35DOI:10.1145/2594458http://doi.org/10.1145/2594458..

EM Rogers..Diffusion of Innovations,,2010..:New York, USA:Simon and Schuster,,..

T Sakaki,,,M Okazaki,,,Y Matsuo..Earthquake shakes Twitter users: real-time event detection by social sensors..2010..Proc. 19th Int. Conf. on World Wide Web..851--860..DOI:10.1145/1772690.1772777http://doi.org/10.1145/1772690.1772777..

NB Shah,,,D Zhou..Double or nothing: multiplicative incentive mechanisms for crowdsourcing..Advances in Neural Information Processing Systems,,2015..1--9....

NB Shah,,,D Zhou..No oops, you won't do it again: mechanisms for self-correction in crowdsourcing..2016..Proc. 33rd Int. Conf. on Machine Learning..1--10....

NB Shah,,,D Zhou,,,Y Peres..Approval voting and incentives in crowdsourcing..ePrint Archive, arXiv:1502.05696,,2015...

JY She,,,YX Tong,,,L Chen..Utility-aware social event-participant planning..2015a..Proc. ACM Int. Conf. on Management of Data..1629--1643..DOI:10.1145/2723372.2749446http://doi.org/10.1145/2723372.2749446..

JY She,,,YX Tong,,,L Chen,,,等..Conflict-aware event-participant arrangement..2015b..Proc. 31st IEEE Int. Conf. on Data Engineering..735--746..DOI:10.1109/ICDE.2015.7113329http://doi.org/10.1109/ICDE.2015.7113329..

JY She,,,YX Tong,,,L Chen,,,等..Conflict-aware event-participant arrangement and its variant for online setting..IEEE Trans. Knowl. Data Eng.,,2016..28((9):):2281--2295..DOI:10.1109/TKDE.2016.2565468http://doi.org/10.1109/TKDE.2016.2565468..

HW Shen,,,AL Barabsi..Collective credit allocation in science..PNAS,,2014..111((34):):12325--12330..DOI:10.1073/pnas.1401992111http://doi.org/10.1073/pnas.1401992111..

JB Smith..Collective Intelligence in Computer-Based Collaboration,,1994..:USA:CRC Press, Boca Raton,,..

A Subramanian,,,GS Kanth,,,R Vaze..Offline and online incentive mechanism design for smart-phone crowd-sourcing..ePrint Archive, arXiv:1310.1746,,2013...

R Subramanyam,,,MS Krishnan..Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects..ACM Trans. Softw. Eng. Meth.,,2003..29((4):):297--310..DOI:10.1109/TSE.2003.1191795http://doi.org/10.1109/TSE.2003.1191795..

BL Sullivan,,,CL Wood,,,MJ Iliff,,,等..eBird: a citizen-based bird observation network in the biological sciences..Biol. Consev.,,2009..142((10):):2282--2292..DOI:10.1016/j.biocon.2009.05.006http://doi.org/10.1016/j.biocon.2009.05.006..

A Tamrawi,,,TT Nguyen,,,JM Al-Kofahi,,,等..Fuzzy set and cache-based approach for bug triaging..2011..Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering..365--375..DOI:10.1145/2025113.2025163http://doi.org/10.1145/2025113.2025163..

JC Tang,,,M Cebrian,,,NA Giacobe,,,等..Reflecting on the DARPA red balloon challenge..Commun. ACM,,2011..54((4):):78--85..DOI:10.1145/1924421.1924441http://doi.org/10.1145/1924421.1924441..

R Teodoro,,,P Ozturk,,,M Naaman,,,等..The motivations and experiences of the on-demand mobile workforce..2014..Proc. 17th ACM Conf. on Computer Supported Cooperative Work & Social Computing..236--247..DOI:10.1145/2531602.2531680http://doi.org/10.1145/2531602.2531680..

J Thebault-Spieker,,,LG Terveen,,,B Hecht..Avoiding the south side and the suburbs: the geography of mobile crowdsourcing markets..2015..Proc. 18th ACM Conf. on Computer Supported Cooperative Work & Social Computing..265--275..DOI:10.1145/2675133.2675278http://doi.org/10.1145/2675133.2675278..

P Thongtanunam,,,C Tantithamthavorn,,,RG Kula,,,等..Who should review my code? A file locationbased code-reviewer recommendation approach for modern code review..2015..IEEE 22nd Int. Conf. on Software Analysis, Evolution, and Reengineering..141--150..DOI:10.1109/SANER.2015.7081824http://doi.org/10.1109/SANER.2015.7081824..

T Tian,,,J Zhu..Max-margin majority voting for learning from crowds..2015..Advances in Neural Information Processing Systems..1621--1629....

YX Tong,,,L Chen,,,BL Ding..Discovering threshold-based frequent closed itemsets over probabilistic data..2012a..Proc. IEEE 28th Int. Conf. on Data Engineering..270--281..DOI:10.1109/ICDE.2012.51http://doi.org/10.1109/ICDE.2012.51..

YX Tong,,,L Chen,,,YR Cheng,,,等..Mining frequent itemsets over uncertain databases..Proc. VLDB Endow.,,2012b..5((11):):1650--1661....

YX Tong,,,CC Cao,,,CJ Zhang,,,等..CrowdCleaner: data cleaning for multi-version data on the web via crowdsourcing..2014a..Proc. IEEE 28th Int. Conf. on Data Engineering..1182--1185..DOI:10.1109/ICDE.2014.6816736http://doi.org/10.1109/ICDE.2014.6816736..

YX Tong,,,CC Cao,,,L Chen..TCS: efficient topic discovery over crowd-oriented service data..2014b..Proc. 20th ACM Int. Conf. on Knowledge Discovery and Data Mining..861--870..DOI:10.1145/2623330.2623647http://doi.org/10.1145/2623330.2623647..

YX Tong,,,L Chen,,,JY She..Mining frequent itemsets in correlated uncertain databases..J. Comput. Sci. Technol.,,2015a..30((4):):696--712..DOI:10.1007/s11390-015-1555-9http://doi.org/10.1007/s11390-015-1555-9..

YX Tong,,,R Meng,,,JY She..On bottleneck-aware arrangement for event-based social networks..2015b..Proc. 31st IEEE Int. Conf. on Data Engineering Workshops..216--223..DOI:10.1109/ICDEW.2015.7129579http://doi.org/10.1109/ICDEW.2015.7129579..

YX Tong,,,JY She,,,R Meng..Bottleneck-aware arrangement over event-based social networks: the maxmin approach..World Wide Web,,2016a..19((6):):1151--1177..DOI:10.1007/s11280-015-0377-6http://doi.org/10.1007/s11280-015-0377-6..

YX Tong,,,JY She,,,BL Ding,,,等..Online mobile micro-task allocation in spatial crowdsourcing..2016b..Proc. 32nd IEEE Int. Conf. on Data Engineering..49--60..DOI:10.1109/ICDE.2016.7498228http://doi.org/10.1109/ICDE.2016.7498228..

YX Tong,,,JY She,,,BL Ding,,,等..Online minimum matching in real-time spatial data: experiments and analysis..Proc. VLDB Endow.,,2016c..9((12):):1053--1064..DOI:10.14778/2994509.2994523http://doi.org/10.14778/2994509.2994523..

YX Tong,,,XF Zhang,,,L Chen..Tracking frequent items over distributed probabilistic data..World Wide Web,,2016d..19((4):):579--604..DOI:10.1007/s11280-015-0341-5http://doi.org/10.1007/s11280-015-0341-5..

YX Tong,,,Y Yuan,,,YR Cheng,,,等..A survey of spatiotemporal crowdsourced data management techniques..J. Softw.,,2017..28((1):):35--58....

L Tran-Thanh,,,S Stein,,,A Rogers,,,等..Efficient crowdsourcing of unknown experts using multi-armed bandits..2012..Proc. European Conf. on Artificial Intelligence..768--773..DOI:10.3233/978-1-61499-098-7-768http://doi.org/10.3233/978-1-61499-098-7-768..

J Tsay,,,L Dabbish,,,JD Herbsleb..Influence of social and technical factors for evaluating contribution in GitHub..2014a..36th Int. Conf. on Software Engineering..356--366..DOI:10.1145/2568225.2568315http://doi.org/10.1145/2568225.2568315..

J Tsay,,,L Dabbish,,,JD Herbsleb..Let's talk about it: evaluating contributions through discussion in GitHub..2014b..Proc. 22nd ACM Int. Symp. on Foundations of Software Engineering..144--154..DOI:10.1145/2635868.2635882http://doi.org/10.1145/2635868.2635882..

B Vasilescu,,,Y Yu,,,H Wang,,,等..Quality and productivity outcomes relating to continuous integration in GitHub..2015..Proc. 10th Joint Meeting on Foundations of Software Engineering..805--816..DOI:10.1145/2786805.2786850http://doi.org/10.1145/2786805.2786850..

W Vickrey..Counterspeculation, auctions, and competitive sealed tenders..J. Finan.,,1961..16((1):):8--37..DOI:10.1111/j.1540-6261.1961.tb02789.xhttp://doi.org/10.1111/j.1540-6261.1961.tb02789.x..

L von Ahn,,,B Maurer,,,C McMillen,,,等..reCAPTCHA: human-based character recognition via web security measures..Science,,2008..321((5895):):1465--1468..DOI:10.1126/science.1160379http://doi.org/10.1126/science.1160379..

D Wang,,,TF Abdelzaher,,,LM Kaplan,,,等..Recursive fact-finding: a streaming approach to truth estimation in crowdsourcing applications..2013..IEEE 33rd Int. Conf. on Distributed Computing Systems..530--539..DOI:10.1109/ICDCS.2013.54http://doi.org/10.1109/ICDCS.2013.54..

D Wang,,,MTA Amin,,,S Li,,,等..Using humans as sensors: an estimation-theoretic perspective..2014..Proc. 13th Int. Conf. on Information Processing in Sensor Networks..35--46....

HM Wang,,,G Yin,,,X Li,,,等..TRUSTIE: a software development platform for crowdsourcing..2015..In: Li W, Huhns MN, Tsai WT, et al. (Eds.), Crowdsourcing. Springer Berlin Heidelberg, Berlin, Germany..165--190....

JN Wang,,,GL Li,,,T Kraska,,,等..Leveraging transitive relations for crowdsourced joins..2013..Proc. ACM Int. Conf. on Management of Data..229--240..DOI:10.1145/2463676.2465280http://doi.org/10.1145/2463676.2465280..

L Wang,,,ZH Zhou..Cost-saving effect of crowdsourcing learning..2016..Proc. 25th Int. Joint Conf. on Artificial Intelligence..2111--2117....

W Wang,,,ZH Zhou..Crowdsourcing label quality: a theoretical study..Sci. China Inform. Sci.,,2015..58((11):):1--12..DOI:10.1007/s11432-015-5391-xhttp://doi.org/10.1007/s11432-015-5391-x..

FL Wauthier,,,MI Jordan..Bayesian bias mitigation for crowdsourcing,,2011..Advances in Neural Information Processing Systems,,1800--1808....

P Welinder,,,S Branson,,,P Perona,,,等..The multidimensional wisdom of crowds..2010..Advances in Neural Information Processing Systems..2424--2432....

J Whitehill,,,T Wu,,,J Bergsma,,,等..Whose vote should count more: optimal integration of labels from labelers of unknown expertise..Advances in Neural Information Processing Systems,,2009..2035--2043....

WJ Wu,,,WT Tsai,,,W Li..An evaluation framework for software crowdsourcing..Front. Comput. Sci.,,2013..7((5):):694--709..DOI:10.1007/s11704-013-2320-2http://doi.org/10.1007/s11704-013-2320-2..

Y Yan,,,GM Fung,,,R Rosales,,,等..Active learning from crowds..2011..Proc. 28th Int. Conf. on Machine Learning..1161--1168....

DJ Yang,,,X Fang,,,GL Xue..Truthful incentive mechanisms for k-anonymity location privacy..2013..Proc. IEEE Conf. on Computer Communications..2994--3002..DOI:10.1109/INFCOM.2013.6567111http://doi.org/10.1109/INFCOM.2013.6567111..

DJ Yang,,,GL Xue,,,X Fang,,,等..Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing..2012..Proc. 18th Annual Int. Conf. on Mobile Computing and Networking..173--184..DOI:10.1145/2348543.2348567http://doi.org/10.1145/2348543.2348567..

YW Ye,,,K Kishida..Toward an understanding of the motivation of open source software developers..2003..Proc. 25th Int. Conf. on Software Engineering..419--429..DOI:10.1109/ICSE.2003.1201220http://doi.org/10.1109/ICSE.2003.1201220..

Y Yu,,,G Yin,,,H Wang,,,等..Exploring the patterns of social behavior in GitHub..2014..Proc. 1st Int. Workshop on Crowd-Based Software Development Methods and Technologies..31--36..DOI:10.1145/2666539.2666571http://doi.org/10.1145/2666539.2666571..

Y Yu,,,G Yin,,,T Wang,,,等..Determinants of pull-based development in the context of continuous integration..Sci. China Inform. Sci.,,2016a..59((8):):080104DOI:10.1007/s11432-016-5595-8http://doi.org/10.1007/s11432-016-5595-8..

Y Yu,,,HM Wang,,,G Yin,,,等..Reviewer recommendation for pull-requests in GitHub: what can we learn from code review and bug assignment? Imform..Softw. Technol.,,2016b..74204--218..DOI:10.1016/j.infsof.2016.01.004http://doi.org/10.1016/j.infsof.2016.01.004..

CJ Zhang,,,L Chen,,,YX Tong..MaC: a probabilistic framework for query answering with machinecrowd collaboration..2014a..Proc. 23rd ACM Int. Conf. on Information and Knowledge Management..11--20..DOI:10.1145/2661829.2661880http://doi.org/10.1145/2661829.2661880..

CJ Zhang,,,YX Tong,,,L Chen..Where to: crowdaided path selection..Proc. VLDB Endow.,,2014b..7((11):):2005--2016..DOI:10.14778/2733085.2733105http://doi.org/10.14778/2733085.2733105..

CJ Zhang,,,L Chen,,,Y Tong,,,等..Cleaning uncertain data with a noisy crowd..2015..Proc. 31st IEEE Int. Conf. on Data Engineering..6--17..DOI:10.1109/ICDE.2015.7113268http://doi.org/10.1109/ICDE.2015.7113268..

Y Zhang,,,X Chen,,,D Zhou,,,等..Spectral methods meet EM: a provably optimal algorithm for crowdsourcing..Advances in Neural Information Processing Systems,,2014..1260--1268....

D Zhao,,,XY Li,,,HD Ma..How to crowdsource tasks truthfully without sacrificing utility: online incentive mechanisms with budget constraint..2014..Proc. IEEE Conf. on Computer Communications..1213--1221..DOI:10.1109/INFOCOM.2014.6848053http://doi.org/10.1109/INFOCOM.2014.6848053..

JH Zhong,,,K Tang,,,ZH Zhou..Active learning from crowds with unsure option..2015..Proc. 24th Int. Joint Conf. on Artificial Intelligence..1061--1067....

D Zhou,,,S Basu,,,Y Mao,,,等..Learning from the wisdom of crowds by minimax entropy..Advances in Neural Information Processing Systems,,2012..2195--2203....

Y Zhou,,,X Chen,,,J Li..Optimal PAC multiple arm identification with applications to crowdsourcing..2014..Proc. 31st Int. Conf. on Machine Learning..217--225....

Y Zhu,,,Q Zhang,,,H Zhu,,,等..Towards truthful mechanisms for mobile crowdsourcing with dynamic smartphones..2014..Proc. 34th Int. Conf. on Distributed Computing Systems..11--20..DOI:10.1109/ICDCS.2014.10http://doi.org/10.1109/ICDCS.2014.10..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Crowdsourcing intelligent design

Related Author

No data

Related Institution

State Key Lab of CAD & CG, Zhejiang University

Modern Industrial Design Institute, Zhejiang University

International User Experience Business Unit, Alibaba Group

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰