Selection of Crowd in Crowdsourcing for Smart Intelligent Applications: A Systematic Mapping Study

Shang, Rui; Ma, YongMei; Ali, Farhad; Hu, ChuanShuang; Nazir, Shah; Wei, Huafei; Khan, Abdullah

doi:https://doi.org/10.1155/2021/9368128

Scientific Programming

On this page

Abstract Introduction Results and Discussion Discussion Conclusion Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Special Issue

Scientific Programming for Smart Internet of Things

View this Special Issue

Review Article | Open Access

Volume 2021 | Article ID 9368128 | https://doi.org/10.1155/2021/9368128

Selection of Crowd in Crowdsourcing for Smart Intelligent Applications: A Systematic Mapping Study

Rui Shang,¹YongMei Ma,²Farhad Ali,³ChuanShuang Hu,^4,5Shah Nazir ,³Huafei Wei,⁶and Abdullah Khan³

Academic Editor: Mian Ahmad Jan

Received24 Apr 2021

Accepted08 Jun 2021

Published08 Jul 2021

Abstract

Crowdsourcing is a task-solving model in which human crowd is hired to solve a particular task. During the crowdsourcing process, the crowd selection is performed in order to select appropriate crowd workers for a specific task; without appropriate selection of crowd workers, the process of crowdsourcing is aimless. The main goal of this paper was to identify the features of crowd in crowdsourcing activity, reasons behind crowd participation in the activity of crowdsourcing, and the existing techniques that were utilized for crowd selection in crowdsourcing. Search strings with corresponding keywords were used to capture relevant studies related to crowdsourcing, and crowd selection was classified under conference papers, journal articles, proceedings, and book chapters. 81 relevant studies are selected from 7 digital data repositories using a search strategy. In crowdsourcing practices, crowd selection was considerably addressed. Nonetheless, it has been noticed that the selection is based only on a single crowd worker attribute such as confidence, past success, efficiency, and experience. For the efficiency and effectiveness of the crowdsourcing operation, crowd selection on multicriteria features is essential.

1. Introduction

Crowdsourcing is a task-solving model in which human crowd is employed for solving complex tasks [1]. It is an online activity that is used for gathering the collective knowledge, skills of people in order to complete an ordinary task [2]. Using crowdsourcing, an organization outsources different types of tasks to a huge crowd using an open call. The term crowdsourcing was first coined by Jeff Howe; in the magazine, Wired, he defined it as “the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” Crowdsourcing is a popular approach that is used for performing various activities. The process of crowdsourcing consists of different entities and their interactions such as requester/crowdsourcer/clients who plan, organize, and manage crowdsourcing projects and are capable of submitting the task request, managing the crowd which consists of an online workforce that participates in crowdsourcing tasks or projects, and providing a platform that provides a communication link between the crowd and the requester, i.e., to cope with the task and the crowd [3–7]. Different tasks are performed with the help of crowdsourcing, such as spell checking, creating contents, simple coding, machine learning, distributed solution of a problem like in the development of software and its testing. The tasks are posted on a platform and the crowd participates for performing various types of tasks [6, 8–10]. In crowdsourcing, participants with various backgrounds, levels of qualification, different expertise in various areas, and having various skills and experience work together to perform a specific task or to solve diverse problems [11–14]. Organizations use crowdsourcing applications as the participation of a huge crowd will enable large challenging tasks to be performed or solved in parallel by the crowd. Crowdsourcing is as an effective means for finding a solution toward a corporate problem [15–17]. Crowdsourcing allows organizations (clients) to recruit global, cheap, and skilled work force through online platforms [18, 19].

Nowadays, with the help of the Internet, companies are capable of hiring a large group of people with a little effort, as the Internet provides a communication link through which people and organizations can collaborate with each other using different devices such as mobile phones, tablets, iPad, and wearable devices [1, 9, 20–23], and platforms like Amazon Mechanical Turk (AMT) are used by organizations to hire a huge crowd from the global community for accomplishing different types of tasks [24]. In crowdsourcing, an organization can obtain high-quality solutions in less time with a little investment, as it allows a lot of people to complete various tasks with negligible employment and management expenses [25, 26]. As crowdsourcing is an Internet-enabled activity, the online platforms give rise to certain challenges such as the issue of how to announce tasks or jobs, how to select an appropriate platform for submitting task requests, and how to recruit competent people [27]. Recruitment of workers becomes a challenging phase with the increase in the popularity of crowdsourcing. In a large crowd of humans, there might be an untrustworthy participant that often makes mistakes in solving different types of tasks. The identification of the right worker makes the crowdsourcing activity successful [14, 28, 29] for the enhancement of developmental process units, if right skill crowd is hired [30].

In a crowdsourcing task, workers’ behaviors and their attitudes are significant features for engaging workers. The crowd must perform the task in the best possible way as the engagement of the crowd depends highly on valuable outcomes. The engagement process will make recruiters able to engage the right worker from a large pool of the crowd. Worker engagement might be influenced by work and worker characteristics. Work characteristic includes task characteristics in which the engagement of the crowd depends highly on the autonomy of task, and it will increase the level of participation with the flexibility and mobility in tasks. The task dissimilarity may influence the participation of workers as irresponsible workers might violate rules of a task. The performance of crowd workers will be increased with the feedback related to their task and they will carefully perform various tasks. Crowd worker’s visibility is the second worker characteristic that has a positive impact on the engagement of the crowd as participants will be visible to organizations. By an increase in the visibility level, workers might be identified among a large crowd pool. The third characteristic of work is Work Setting, on engaging workers for performing a task the worker setting and environmental characteristics play an important role. Organizations have provided a number of options for workers to participate. Crowds can participate virtually from homes, shops, and even from other places of employment. With the increase in potential work setting, the availability, flexibility, and independence of workers is increased, and it will probably increase the performance and satisfaction of workers. The fourth work characteristic is worker recognition, which has practical influence on participant engagement. There are certain factors that drive worker engagement in a job. The unknown nature of crowd work brings some of the challenges that are concerned with the best identification and recognition of crowd workers and how to reward high performers. Personal recognition might be affected by anonymity in crowd settings. Recognition and rewarding participants has shown that the achievements directly impact attitudes and behaviors of workers. Recognition may increase worker engagement by revealing to the crowd that they are valuable assets of the organization. Individuals enroll for complex tasks not because of reward but they think that they will be valued by the organization. By receiving recognition from the organization, individual involvement will be improved and one will put in his or her best efforts and concentration in performing tasks.

Crowd worker characteristics are also responsible for worker engagement. It includes Worker Expectations, Extrinsic Motivation, Satisfying Motivation, and Meaningfulness. Worker expectations represent the expectation of crowd workers in an activity from client organizations for their participation in performing a task. The workers who register themselves for performing tasks have certain expectations regarding requirement of the task, period of completion, complexity, and cognitive load that is mandatory for completing the task. When these expectations are violated, they impact comfort, motivation, and satisfaction of workers. Before participation, workers are provided the relevant information about the task such as requirement of the task, the period of completion, and the exact remuneration to participants. With this, the crowd can participate and select the tasks in which they are interested and which matches their skills level. A worker will participate in a task if the task is simple, only less effort is required for thinking, and can be completed in less time. If workers are not provided with task information, they might participate in complex tasks, might become frustrated with the increase in mental effort, and will waste much time. This will affect the behavior and attitudes of workers, thereby lowering the engagement of workers. Crowd workers may be engaged due to the extrinsic motivation factor in which the crowd participates in activities and they are provided with money or reward. There is a liner relation between crowd engagement and extrinsic motivation. Another characteristic is Satisfying Motivation in which the crowd workers expend their cognition, physical, and emotional energies for performing various tasks. Individuals contribute with a goal for achieving the best probable outcomes (example, task performance level). For attaining this, participants use their efforts and concentrate on tasks, and perform well beyond the requirements of satisfiers. A crowd worker depends on resources for achieving high performance level and this will increase the level of engagement of workers. Contrary to satisfying motivation, participants consume their energy, which is required to generate a “good enough” result, and, as a result, decrease the complete engagement of participants in a task. Meaningfulness is another crowd worker characteristic for engaging a crowd; it reflects the degree to which the participants analyze that their contribution is important. Their work is appreciated, valued by clients. This can impact working outcomes such as performance, motivation, and satisfaction. By observing that the tasks performed by workers are valued and accepted, i.e., their efforts are not countless, their interest in engagement will be increased and they will spend more energy in the accomplishment of tasks [31].

Analyzing crowdsourcing and its importance in today’s business world, a systematic literature review was carried out for the last ten years, i.e., 2010–2020. The proposed research study is composed of three-fold contributions that are enumerated as follows:(i)To identify the significant features of the crowd(ii)To pinpoint the reasons behind the crowd for active participation in crowdsourcing activities(iii)To explore various existing methods utilized for crowd selection in crowdsourcing

The presented systematic literature review (SLR) mainly focuses on crowd selection that will be used by various organizations to capture the appropriate crowd from a global community to perform various tasks.

The paper is structured in a linear fashion and is categorized into 9 sections. Section 2 briefly describes and discusses the overall research process carried out for this review. Section 3 explains the validation and threats for this review. Section 4 presents results and discussions of analyzed papers. Section 5 presents the research findings of the review. Section 6 briefly discusses the review topic. Section 7 concludes the review. Section 8 presents the limitation of the proposed review. Section 9 presents future research suggestions.

2. Research Methodology

With the interest to analyze the feature, motives of the crowd, and strategies used for the selection of a crowd in a crowdsourcing activity, a systematic literature review was carried out. SLR is a systematic practice for identifying and analyzing related studies as a regard to a specific interest area [32]. For ensuring validity and the value of SLR, a number of standards and protocols are applied by researchers. We adopted the protocol suggested by [33] in carrying out this research. The review protocol and its stages are represented in Figure 1. The review protocol comprised seven stages, namely, analysis of research topic, extraction of research questions, designing a search strategy, extraction of results from data, scrutiny, criteria for quality assessment, and synthesis of data. Research questions are extracted from various studies in the second stage; in the third stage, the searching strategies are designed in accordance with research questions that consist of search term identification and selection of literature resources; collection of extracted data takes place in the fourth stage of the protocol; the fifth stage is concerned with scrutinizing, i.e., refinement of extracted data; scrutinized studies are then evaluated based on a criteria of quality assessment in the six stage. In stage seven, selections of final prospective studies for analysis are carried out.

The following sections show the details of the research method.

2.1. Research Topic Analysis

Research papers from various digital data repositories were thoroughly reviewed to know the crowdsourcing concept, to discover the issue related to the field, and to find out what researchers have done so far to solve many problems over time.

2.2. Research Question

For getting a better perspective on crowdsourcing conception and to identify areas for researchers, a thorough assessment was carried out on the topic of crowdsourcing. Relevant papers, conference proceedings, book chapters, and journals were studied at the initial stage that clearly defined crowdsourcing activity. Our preliminary study identified that crowd selection is a problem-solving technique that is utilized by many organizations for the accomplishment of various tasks by harnessing the collective energy of humans. In order to carry out the research, the proposed study focuses on addressing some research questions given in Table 1 that are extracted based on the assessment of various articles/papers for the purpose of making this review an effective one.

2.3. Searching Strategy

Search terms, literature resources, and search processes make up the search strategy. The descriptions of each are discussed in subsequent steps.

2.3.1. Search String

For creating search terms, the following steps are carried out:(i)From formulated research questions, major terms were derived(ii)Synonyms are identified for main terms(iii)Keywords were identified from relevant books or papers(iv)Boolean OR is used to link synonyms(v)Main terms are linked with Boolean AND

The strings that are formed as a result are presented as follows:(i)(Multicriteria OR crowd OR crowd selection OR worker selection) AND (crowdsourcing) AND (software OR software engineering)(ii)(“Multicriteria” OR “crowd selection” OR “worker selection”) AND (crowdsourcing) AND (software engineering)

2.3.2. Literature Resources

For synchronization, a thorough assessment of the proposed study was carried out on seven digital libraries to extract data from various scholars’ works on the topic of crowdsourcing. These digital libraries include: ScienceDirect, IEEE Xplore, ACM Digital Library, Springer, Taylor & Francis, Wiley, and Hindawi. For carrying out this research title, abstracts, index terms, contents of conference proceedings, published research papers, journal articles, book chapters, and conference papers were analyzed.

2.3.3. The Search Process

For a comprehensive search, a systematic literature review of relevant sources is significant to be carried out. Yet, the search process that is utilized in the proposed study consists of some stages that are mentioned below. These steps are also represented in Figure 2:(i)Stage 1: seven digital libraries are thoroughly searched to extract papers related to the proposed research. The search result has been assembled to be a collection of prospective papers.(ii)Stage 2: on the basis of titles, papers were filtered from total papers.(iii)Stage 3: relevant papers were extracted by applying quality assessment criteria.

2.4. Prospective Study Selection

The first stage of searching retrieved 2098 metadata studies (abstract, title, contents) from seven digital databases. After this stage, the titles of perspective studies were used for extracting relevant research work. This process is mandatory for the exclusion of irrelevant and duplicate research studies. After applying title base filtering, 146 related studies were identified. Finally, we applied quality assessment criteria (QoA) on the filtered papers, i.e., each paper was studied and assessed in detail to check whether these studies answer the questions that were formulated. At the end of this process, 81 relevant studies capable of answering the questions were selected.

2.4.1. Scrutinization Process

Scrutiny was necessary to extract the relevant work as we have obtained 2098 prospective studies in the initial search process. At first, title-wise paper selection was used to filter relevant papers. Next, the relevant studies are analyzed on the basis of paper contents. Hence, all articles that were unable to address our research question and do not reveal the discussion topic were excluded. Studies that are published in conferences, proceedings, journal publications, and written only in English were included in the relevant studies. Duplicate research studies were also eliminated in this process. A systematic literature review was carried out for a period of ten years, i.e., 2010–2020, in order to select recent studies on the topic. Our initial search process on seven digital databases was conducted between January 24, 2020 and January 29, 2020. The criteria adopted for the process of scrutiny is presented in Table 2.

Figure 3 shows the type of papers along with the year of publication.

Table 3 shows the library-wise search based on title and contents.

Figure 4 shows the reference of paper, types of paper with the year of publication.

2.5. Quality Assessment (QoA) of the Selected Papers

The quality of assessment was obtained for selected studies by scoring or evaluating these studies in order to check whether these studies are capable of answering formulated research questions or not. Such questions are dealt with in Table 4. There are only 2 answers (options) to each question, i.e., “yes” = 1 and “No” = 0. The quality score of each study is calculated as the sum of answering the formulated questions. The reliability of the proposed study was obtained by considering studies that are relevant and whose quality score is two or three, i.e., papers that are capable of answering at least two research questions. Based on quality scoring, 65 studies were excluded and 81 papers were selected as they were capable of answering at least two formulated questions. The quality scores of selected studies are shown in Table 5 and the graphical representation is represented in Figure 5.

2.6. Synthesizing Data

The significance of this step is to synthesize and summarize the proofs in order to answer formulated questions. Eighty-one relevant studies have been synthesized using thematic analysis, which is a powerful technique used to rigorously generate themes and patterns for data [84]. A theme extract is something significant about the RQ data, and it can be derived inductively or deductively from the data based on themes derived in previous studies. The process will help in synchronization of the relevant study in order to intensify clarity. This will also assist in identification of specific answers to formulated questions. A summary of data synthesis for the research question is explained below. The data associated with all RQs is represented in a table form and it is also discussed in detail. In RQ1, the features of the crowd are extracted from selected studies. RQ2 explains in detail the reasons for crowd participation in an activity. RQ3 focuses on various techniques used for crowd selection.

3. Results and Discussion

This segment summarizes the findings of the study. First, we present an overview of selected articles. Next, we present in separate subsections a description of the results of the review process, in accordance with the questions formulated.

3.1. Overview of Selected Papers

For the proposed study, 81 Research studies are selected on the basis of QoA criteria. Among these studies, 55 papers were published in journal articles, 1 paper in book section, 20 are extracted from conference proceedings, and 5 studies were retrieved from conference papers. The total number of selected papers and their percentages are presented in Figure 6; the selected papers based on publication year are shown in Figure 7.

3.2. Significant Features of Crowd in Crowdsourcing (RQ1)

Organizations are interested in leveraging and gaining the knowledge of people. Superior techniques are applied for collecting this knowledge from external experts for enhancing the performance of products. Managers of an organization who are interested in solving challenges must select specialized professionals or subject experts from the external world who have the knowledge for solving a new organizational challenge. These specialized persons represent a solid approach to increase the absorptive capacity or the company’s capability to identify, assimilate, and exploit external knowledge. Various approaches are used by specialized professionals for increasing the storage capacity, or they may use the potential of the company for identifying, selecting, assimilating, and utilizing the knowledge of experts [71]. In crowdsourcing, the entire project or a part of it is outsourced to a large pool [39], of distributive and cost effective’s labors [72]. Crowd workers possess some characteristics, i.e., they can be identified with certain rules that direct their behaviors, they do not depend on others, they control their actions, and they may be flexible [34, 81] in changing their behavior with the changing environment [34]. Crowd workers may be newcomers (novices) [12, 20, 41, 47]; anonymous participants [3, 6, 7, 11, 19, 73, 83]; inexperienced [11], fraudulent [1], incompetent [21] workers; heterogeneous workers [38, 52, 56, 64] in working styles; unknown workers [73]; transient solvers (it greatly influences the quality of a task) [83]; undefined workers [3]; nonprofessionals [49, 56, 83]; nonexperts [49, 68]; untrained participants [3, 35]; untrustworthy workers only interested in gaining the rewards that are associated with tasks and thus do not work sincerely, and this is a negative factor of the crowd [29]. They act selfishly (negative feature) for utility utilization [18]. Workers may act maliciously [35, 80, 83],which limits the quality of tasks, or may be “eager beavers” [35] who outperform appropriate workers [5]. Trainees may accomplish tasks [3, 41] in the crowdsourcing activity. The crowd may be autonomous [34], fast [2] unique [69], appropriate [3], right [28, 79], reliable/efficient [2, 25, 72, 73, 80], loyal [70], truthful [36], trustworthy [29, 42, 44, 50, 63] who complete assigned tasks sincerely. Workers are coordinative, adaptable [16] (as they can change themselves with the change in the work environment), “Energetic” having energy [10, 17], capable [1, 42, 60] of performing tasks, and are “creative” as they creatively [67, 68, 78] perform different types of work. They are competent workers [16, 39, 63] who utilize common sense [68] in various tasks. Crowds are smart [25, 27], educated/qualified [1, 13, 42], professionals [71, 75], having skills (skillful workers) [2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59–61, 67, 76–78], expertise [58, 59, 81, 82], experience [14, 17, 19, 24, 43, 46, 47, 53, 76] who possess knowledge [2, 8, 11, 13–15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76] for getting a task done [35]. Workers may be cooperative [12], collaborative [21, 39, 76]. Crowds may work as volunteers [10, 15, 17, 74–76] for solving large problems [5, 7, 21, 54, 60, 64, 68, 75, 78], i.e., they are solution providers [5]. Workers can make decisions [15].The various features of the crowd are represented in Table 6 and Table 7.

3.3. Reasons behind Crowd Participation (RQ2)

Various incentive measures are adopted to motivate participant for contribution like for teams in organizations, pointing system is adopted and participants are provided with different points depending on their performance level. These points are then converted into prizes and gifts. External participants are provided with honoraria and likelihood of career prospects/advancement [14, 19, 20, 41] and being employed in future work [71]. Various workers complete different types of tasks such as sentiment analysis and tagging of image, solving microprogramming tasks in order to receive benefits of remuneration [2, 19, 41, 56] financial [6], monetary [41, 47, 83], reward [37, 78], prize [36, 37, 49, 71, 78], payment [45, 65, 66] like immediate payoffs (making money) [5], cash [50, 75], fee [57], direct compensation [16, 31, 61, 66] and money [8, 55], extra bonuses [12, 35], extra income [43], non-monetary awards including delayed payoffs (the potential to take freelance work) [5], status [3], pleasure personal enjoyment/fun [5, 62], and for learning purpose [6, 45] to increase their knowledge [15–17], increase reputation [65, 77] in public display by increase in point and badges. The motivation may be an introjected one in which crowd participates for recognition [6, 62] among peers, fame [37]. People also compete with each other to develop their skills [6, 45]. Crowds are rewarded based on the quality of the completed task. A worker who produces quality results is awarded with incentives [35, 37] and access for future work [8, 43]. Worker observes developers [15] work, raise, and collecting funds [12, 49, 53, 75] for projects. Crowd also participates for inspiration [49] from other solutions and revision of solutions for generating better alternative [3, 17] for existing solutions. Communication [39, 53] with each other to manage their time [39] is also a reason for participation. Crowd works in cooperation [15, 20] and in collaboration [12, 53] for improving products. Crowd participants may also participate to provide ideas [12, 53] for various developmental projects. A participant participates as he/she perceives curiosity [12, 60, 77], self-affirmation [60], self-estimation [6, 78], and appreciation [49] of requester organizations. Workers are motivated for self-marketability [41, 77], visibility [7]. Self-efficacy [20, 41, 54] which constitutes four types of incentives, namely, accomplishment [12, 40, 41] (workers feel that they are accomplishing something while participating in tasks), importance (workers observe that their doing is important to others), competence [41] (working on crowdsourcing tasks provide a wisdom of competency), and doing best job (workers desire to know that they are doing best), and to contribute. There are four incentives regarding contribution, namely, effectiveness (how effective the tasks are to others), helping others (participant completes the task to help other people), trust (workers are trusted by organizations), and supporting others (participants support each other in the accomplishment of tasks) [41]. The crowd also participates to find an appropriate answer for a question [3, 35, 49]. An individual can be motivated to gain or share experience [12, 62] and increase his or her expertise level [59]. Other reasons for motivating the crowd to participate in tasks are volunteering (free contribution) [6, 10, 74], reciprocity and expectancy in which users participate for intellectual stimulation [2]. To help a community as they expect that others will also contribute in society [2, 41, 77], contribution of each every one will help in solving a community problem. Crowd contributes altruistically [9] without greediness/expectations for something; they do it out of sympathy, which is known as altruism [62]. There are also some factors that motivate the crowd for participation like making new friends [16] and for socialization [54, 70] purposes; people also contribute in crowdsourcing as they have interest in [16, 60, 65] a field. Developers for task are motivated by ideology, self-need, self-development, and for preserving authorship of their own work to get paid to provide best solutions to problems [39]. Crowds may provide a service [2, 40], acquire information [40, 62], or may accomplish a task like booking a table in a restaurant [40]. Contestants also participate so that organizations provide them feedback [47, 55] on their completed work. The crowds are motivated by means of intrinsic and extrinsic motivation [77] Intrinsic motivation is concerned with participation that increases interest and joy due to which participant feels competency, fulfillment, and satisfaction [12, 61]. Extrinsic motivation is working on something that provides some distinguishable results. It involves engagement because of external incentives such as compensation [66]. The participation may be due to identifiable motivation, which is a reaction of freedom and violation since the deeds are concerned with identity and personal goal achievements [61]. Integrative motives [60] may be a cause of participation in which participant activities are considered significant and meaningful. The crowd is also motivated by networking motives for building professional or personal relation [17, 60]. This will increase self-belonging [60] and increase engagement. Self-marketing [41, 60, 77] or self-advertisement [60, 61] is a major motivation for participation in projects for individuals looking for a job. Workers perceive that competition provides workers with a number of benefits, and they have to show eagerness to engage in competitions [15, 16, 61]. Workers are willing to participate as they are provided with task autonomy [61]; participants are offered a high level of control on their actions in competition. Task variety is another motive for participation where a participant is allowed to provide solutions from different viewpoints. Players are allowed to use their abilities and skills for developing solutions. Task complexity is another motive for participation as task is complex; its completion requires high competency, skills. Workers are interested in performing complex tasks to gain self-expression and a sense of competency. The crowd also participates for sponsoring an organization [61]. The crowd also shares and acquires information [40, 62], recommends [62, 75] some things, shares life experiences, provides suggestion, and increases the level of ranking [62]. Crowd teams participate in crowd source task for receiving tangible (monetary) and intangible (reputation) rewards, for social comparison (self-evaluation with others) [54]. Co-creation, or working as mediators for crowd acquisition, task specification, and evaluating outcomes, advocating, and co-development [70] are also motives for the crowd to participate in activities. Expectations [78] of earning profit such as “Cambros,” credit point, selling digital pictures online, voting things, scoring, commenting, decision-making, sharing reviews about a product [75] are also the reasons for crowd participation. Virtual teams complete tasks for bringing social capital [16, 17] (relationship) to their teams. The crowd also participates to brainstorm ideas and to agree about certain activities for alternative [3, 17] solutions [17, 39, 53, 60]; crowd workers are encouraged to participate as they are considered meaningful or relatedness as compared with others in the community [12]. The crowd contributes for detecting and correcting errors, and they desire to satisfy standards [21]. Membership, efficacy, reciprocity, etc., may be the reason for participation [20]. The reasons for crowd participation are represented in Table 8.

3.4. Existing Techniques Used for Crowd Selection (RQ3)

3.4.1. Profile-Based Searching

Profile [40] is an identity of crowd workers. In the registration process, crowd workers have to express themselves by posting personal profiles. Workers are selected for different projects [7] using profile data. Information about age, sex, and ability of workers to perform tasks is present in the profile [40]. Profile captures properties, sets, and describes participant name, location, etc., using ontology-defined vocabulary [82]. Workers’ attributes [68] and trust factor are present in the profile [34]. Personality-trait-based tool on various profiles is used to select workers [71]. Profiling will help in personalization of the intelligent decision and enhances the quality and selection of appropriate workers (crowd) for tasks. Personalization of crowd members would improve the decision quality [57].

3.4.2. Selection on the Basis of Skill Assessment

Skills assessment is used for evaluation and certification of worker reliability to assist in the job matching process. Organization offers certification, which certifies that a worker has relevant skills [3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]; these certifications are then used for recruiting purpose [51]. Workers who possess required skills are assigned tasks [3]. Skill evolution mechanism may be adopted for giving a chance to prove skills [52] Varieties of skills are needed from general, to specific, to situational [60]. Initial screening of the crowd is carried out based on skills [49] that are required for completing a task [25].

3.4.3. Expertise Filtering

Expertise [3, 9, 13, 31, 46, 49, 58, 68, 71, 79] filtering is carried out to check the expertise level of workers. Expertise-estimation approaches estimate the expertise of workers [58]. The right crowd is selected based on their expertise level [68, 79]. Workers are assigned to a task if they have expertise in the required field [3]. As an individual has expertise in the task, he or she will try to complete it with full attention [31]. It depends on task relevance and diverse expertise, and the level of participant selected [46].

3.4.4. Selection on the Basis of Crowd Attributes, Behaviors, and Attitudes

There are various attributes of crowd workers [68] which are significant for value match [31]. Attributes may be social [36], which shows crowd online or offline status, and social activities of workers assist in understanding routine behaviors [53]. Workers’ behavior is noticed when they complete tasks and bear a definite behavioral pattern [26]. Agent-based (AB) technique is applied to identify behaviors of crowd workers individually and to analyze the diverse attribute of participants. Workers are represented as agents and they have various characteristics. At platform, these agents arrive in nonhomogeneous passion distribution which are assigned with unique IDs based on their characteristics specified with utility factor which show their behaviors. An agent has a decision-making state that consists of two components, i.e., registration and submission of tasks. Agent decision is related to information that is received from agent’s community and social environment, and from other competing agents. The agent is responsible for submitting the task, and upon submission his or her attributes for reliability factor are updated [34]. The positivity of the crowd worker attitude increases their interest in solving a task [57].

3.4.5. Selection on the Basis of Trustworthiness

Trust is the main factor for selecting workers for a task [3, 9, 29, 48]. A requestor may recruit trustworthy crowd workers [9]. For identifying trustworthy workers, SSC (Strong Social Component) and C-AWSA (Context-Aware Worker Selection Approach) is used, which is an efficient and valuable algorithm for selecting trustworthy workers. For optimization purposes, quality of trust as well as path utility is used. Forward searching Algorithm is used to calculate the trustworthiness of a worker [29]. Trust-Based Access Control (TBAC) strategy is utilized for computing trust value. Fuzzy Inference System evaluates the trust values and based on these values, access for a task is granted to crowd workers. The discrete model is used to decide whether an entity is trustworthy or not [48]. Trust factor of agent is updated when he or she submits a task [34]. “Career ladders” may be generated for trusted workers by organization for participating in high-level assignments [31]. Trustworthy workers are placed at a higher level [47].

3.4.6. Selection on the Basis of Performance

Crowd workers are selected according to past performance [42] in various tasks [6, 68]. Worker previous task registration and winner records [49] are necessary for assigning tasks. The workers’ performances may increase with training [31]. For dealing with unknown worker performance in multiarmed bandit (MAB), initial exploration phase consistently samples the performance of workers using budget, and in the exploitation phase best workers are selected [56]. Affinity values can be calculated by participants’ social attributes and their previous task execution record [36]. Crowdsourcing platform is concerned with monitoring workers for evaluating and updating their skills depending on the quality of the tasks completed in the past [47]. December uses previous records of members of a crowd [81] from previous activities or social networks [82] for selection.

3.4.7. Selection on the Basis of Feedback

The crowdsourcing platform gives feedback (temporary ranking) to crowd workers and based on these feedback, workers decides to quit or to compete for performing a task [3]. Feedback is increased or decreased while working [68].

3.4.8. Role-Wise Selection

Roles assigned to crowd workers change with the change in the crowdsourcing environment [48]. Flash organization selects a crowd automatically, as they are structured in hierarchical form according to their roles and responsibilities [3].

3.4.9. Selection on the Basis of Bid

Crowd workers are selected on the basis of winning bids. These workers make a bid based on their calculated effort and cost for getting a task. Only workers with winning bids are selected to submit a solution of the task [3]. Workers are selected statically, and after bidding winners are selected dynamically [50]. Workers with lower bids are hired first. Each new bid must satisfy the affinity and ability constraints. In the process, the winning bid gets the payment from the platform. For carrying out decision by platform, the users wait and stay online, and they may report dishonest time of entrance or exit in order to exploit their service. They may postpone their bid time and confer a prior exit time. During the time new challenges arise bid-independence, a user's payment cannot be affected by its bid, and the real appearance/exit time of the participant should not be known and is submitted truthfully. For the subsequent challenge, every active bid will contribute in the auction and their payment will be updated [36]. Classical bid auction was extended for ensuring quality [52].

3.4.10. Selection Based on Job/Qualification Tests

Requesters specify requirements for workers to participate in activity. These requirements may consist of various tests that a participant has to complete in order to qualify for a job [68]. Workers are judged through qualification [3]or through job test [31]. Workers must pass qualification test before participating in tasks/projects [6]. Qualification test will ensure that worker has the knowledge for designing the user interface. Workers were selected based on the score achieved in the qualification test [4]. Requestor selects workers based on fulfilling some qualification requirement [66].

3.4.11. Time- and Constraints-Wise Crowd Selection

Workers are encouraged to appear on time [50]. Crowds are registered, requirement of requestors are analyzed, time and duration is decided, and an appropriate crowd is selected [3]. Task agent announces an open call with instruction to available workers who will fulfill precise conditions. The task agent receives a reply from participants that are willing. A favorable set of workers is selected based on time constraints. Subsequently, it transfers a confirmation to the selected worker [40]. Requestors divide a complex task into subtasks, which increases the task probability of workers to accept the task in a specified time [1]. Participants are selected on the basis of timeliness of workers in prior accomplished tasks [9].

3.4.12. Selection on the Basis of Belt Level

Workers are grouped in the registration order into five different belts, i.e., red, green, yellow, blue, and gray, which represent the skill level of workers. Reliability of workers is measured by registration and completion of the submitted task [44].

3.4.13. Crowd Selection on the Basis of Experience Level

Crowd from a large pool of people are selected on the basis of experience [7, 45] of at least two years [24]. Experience strategy is developed for selecting experienced workers (testers) who have reported many bugs in the past [46]. Leader of a team is selected according to his or her previous experience [22].

3.4.14. Ranking System Used for Crowd Selection

A ranking system was proposed, which dynamically modifies the worker skills. In ranking systems, trustworthy workers are placed at a higher level and others at a lower level. Workers are ranked in three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find an appropriate worker for a specific task [47].

3.4.15. Task Assignment Technique for Crowd Selection

For selection of qualified workers for sensing task, various task assignment techniques are applied. In MCS, worker selection is a challenging issue and it affects the sensing efficiency and quality. Various criteria are adopted for filtering the unsuitable worker. The task assignments framework consists of participants who use sensors for obtaining or measuring required data about his or her interested subject. Application/end users request with data through task and utilization of information required by participant, tasking entities distribute task to workers [62].

3.4.16. Crowd Selection on the Basis of Capability/Ability

Organizations select employees who have the capability to produce great and diverse ideas for solving technical problems [71]. The organization makes a decision to pre-asses the ability of a crowd to perform complex tasks [66]. A worker is selected on the basis of ability [1]. The Borda ranking algorithm is utilized to check capability for selecting participants for the accomplishment of a dynamic task [23].

3.4.17. Demographic Filtering for Crowd Selection

Crowd workers have different demographics [42, 46]. Demographic filtering selects specific country/location people [6, 68]; workers are selected according to their demography [22]. Workers provide information about demography if they are willing to participate in tasks [4].

3.4.18. Crowd Selection on the Basis of Availability of Workers

Workers transfer messages to ensure their availability [40] or online status [36] on platform. Tasks are successively posted to a pool of workers [80]. Workers’ characteristics are identified for analyzing the availability of workers to perform and complete tasks [44]. Workers are assigned with tasks based on their availability [13].

3.4.19. Recommendation-Based Crowd Selection

For a project, appropriate workers are recommended [10, 71]. The recommendation system is used to assess the crowd [28]. Workers are ranked according to recommendation levels [48]. For guiding workers to perform suitable tasks, recommendation techniques are used [49]. The recommendation system helps crowd workers in finding related tasks [39].

3.4.20. Screening-Based Crowd Selection

Initial screening of the crowd is carried out based on skills that are required for completing a task [25]. Workers must pass an initial prescreening exam that consists of gold tasks. Prescreening assists requesters in inferring worker’s reliability [80].

3.4.21. Relevant Background Filtering for Crowd Selection

A crowd is selected from diverse sources and with various backgrounds [60]. In some crowdsourcing activities, crowd workers’ recruitment is restricted to only workers who have declared that they have a relevant background [24].

3.4.22. Crowd Selection on the Basis of Reputation

For checking the suitability of workers, their reputations are checked [9]. The reputation system calculates the reputation score of online workers based on collective ratings of employers that had hired them for tasks in the past [51]. A truthful online reputation updating algorithm was proposed for updating workers’ reputation [50]. Reputations of registered workers are checked before their participation in tasks. After the submission of tasks, quality is determined as it is used to control the reputation [65].

3.4.23. Selection of Crowd on the Basis of Rating

Rating was used as important measure for generation of participant profiles. Workers are assigned only those tasks which were suitable for them [52]. Ratings assist in selecting an appropriate candidate for tasks [25].

3.4.24. Crowd Selection on the Basis of Incentives/Reward Mechanisms

Incentive mechanism was proposed for worker selection. Workers are selected statically, and after bidding, winners are selected dynamically [50]. Participant incentives’ requirements are considered for selection [23]. Tasks are assigned to crowd based on associated reward [13].

3.4.25. Crowd Selection on the Basis of Voting Scores

Task requester selects workers based on some criteria such as majority voting [26, 71].

3.4.26. Crowd Selection According to Social Relation of Workers

A worker having high social context values is selected [29]. For distributing tasks in friends without getting their information, social relationship is utilized in the process. “A Task-Distributing System of Crowdsourcing Based on Social Relation Cognition, TDSRC”only accommodates interaction information of requestor and friends. Using social relationships, the model builds a trust chain among requestors and solvers, and thus increases the reliability of task distribution [83].

3.4.27. Crowd Selection on the Basis of Auction

Organizations utilize auction-based model for the selection of crowd workers based on their skills, qualification, and trustworthiness [3]. Classical bid auction was extended for ensuring quality. Appropriate workers are invited and the bids are ranked for specific rewards [52].

3.4.28. Crowd Selection on the Basis of Matching Mechanism

Matching of task to appropriate workers should help the crowd to spend their strength in tasks to benefit the crowd as well as organization [31]. Task and tester matching technique is used to select a crowd (testers) from a large pool of people [45]. Project managers use a matching method to select the leader of a team [22]. The existing techniques for selecting an appropriate crowd for tasks are represented in Table 9.

4. Research Findings

The crowdsourcing activity involves allocation of tasks to the Internet crowd [2, 39]; with an open call, the crowd is involved in solving tasks that are complex. These crowds are employed from web-enabled communities. Crowd workers represent some attributes, such as qualification, age, gender, language, worker location, skills, past service, and experience. Workers are selected on the basis of these attributes [13, 17, 40, 45]. Crowdsourcing assists workers or an organization to seek services, contents, ideas, or services from a huge group of people online (crowd) [80]. The crowd workers are dynamic and creative, with a variety of motivations and experiences. Unique crowd workers’ participation is required in crowdsourcing. Global workers have to represent themselves as experts by posting personalized profiles created during the process of registration [7, 69]. Information of workers, such as sex, age, and ability, is present in the worker profile [40]. The crowd is evaluated on the basis of the work performed [22]. An appropriate set of workers is selected for performing tasks [46]. For achieving quality outcomes, an employer should carefully select workers [56]. The quality provided by workers depends on the distribution of tasks to the members of the crowd, which requires appropriate mechanisms to be controlled such as by screening workers [13, 42]. Initial screening of the crowd is carried out on the basis of skills required to complete a task. Care must be taken while selecting the crowd, as it is noticed that the quality improves with a diverse crowd selection [25]. Quality outcomes produced by crowd workers depends on certain aspects such as worker skills, their experience regarding tasks, and their commitment for performing tasks [47].

5. Discussion

The analysis highlights the point of view behind research in the area of crowdsourcing and provides a synopsis of current crowd selection techniques based on certain aspects. Crowd selection is an important phase in crowdsourcing. With the help of selection methodologies, the appropriate participant for a task is selected. The review has a number of implications in the crowdsourcing activity. In the crowdsourcing activity, the task is performed by crowd workers. The study highlights the multi features of an online crowd in order to choose the right candidate for a task from Internet communities. The review also underscores as to why crowds participate in crowdsourcing tasks? Various techniques were suggested in literature for crowd selection. They are represented in this review for the purpose of understanding how the crowd was selected previously and to identify the measures for this selection process.

6. Validation Threats

The biased nature of publications and extraction of inaccurate data are considered as major threats regarding review protocol. The studies are selected on the basis of search string, described in the search strategy, which includes literature resources, selection on the basis of criteria, and quality assessment criteria. Search string relevant to specific formulated questions was used for the extraction of relevant studies. However, there might also be some irrelevant studies that are extracted using search strings. A selection criterion was adopted in compliance with formulated questions to extract relevant studies. Finally, the studies are evaluated on the basis of the quality of assessment, and these papers are analyzed thoroughly to minimize the threats linked with the extraction of inaccurate data.

7. Conclusion

In the proposed study, we addressed the problem of crowd selection in crowdsourcing. One of the key contributions of our research is to illustrate the various multiple features of the crowd. Another contribution of the review is to identify the various reasons of crowd participation, and the third contribution of this study is to capture various selection methods used for crowd selection. Research aims are extracted from various studies in the literature. The aims are extracted from online data repositories such as IEEE, ACM, ScienceDirect, Taylor & Francis, Wiley, Hindawi, and Springer. They were searched on the basis of search strings to extract relevant studies.

Our preliminary objective was to capture the current state of crowd selection in crowdsourcing research and to assist developers and researchers to highlight what has been done in the literature. The review points out the multi features of the crowd, reasons for participation of the crowd for engagement in various tasks, and the different techniques used for the selection of an appropriate candidate for tasks. A systematic review was conducted, and a total of 81 relevant studies are selected based on QoA. Subsequent to study selection, on the basis of title or abstract, the studies were properly reviewed for the purpose of acknowledging that the study answers minimally two research questions or not.

Our finding suggests that crowd selection is an important phase in the crowdsourcing activity; without proper selection of the crowd, the crowdsourcing activity will not be productive. This review will be a base study for future crowdsourcing researchers, as it highlights and explains the entity “crowd,” their feature, and their effectiveness in crowdsourcing activities. The study will also be useful to the organization if they want to hire/select appropriate workers for a task. The survey can also benefit platforms, as it provides them with insight features and the motivation of crowd workers.

8. Research Limitations

The review was conducted rigorously within the limits of specified questions. As a result, 81 related studies have been listed and chosen to be able to answer at least two formulated questions effectively. The related studies are chosen from 2010 to 2020. With the paradigm shift and the complex nature of the crowd and crowd selection, however, we cannot fully guarantee that all possible studies in the proposed research have been captured. Another point of concern is that, because only papers published in English were included in this study, significant or appropriate research published in non-English journals was overlooked. Assessment was carried out for research biasness due to the strong tendency of reporting biases of the precise strength and weakness of current techniques. However, in this review, the selected studies did not include gray literature, technical reports, work in progress, and unpublished work, which may be capable of answering any research question. Title and content of various studies were analyzed in order to ensure that these studies are capable of answering the formulated research questions. The assessment criteria used in this analysis cannot guarantee that those criteria are sufficient to identify the appropriate studies.

9. Future Research Suggestions

Crowd selection is an important phase in the crowdsourcing activity; without appropriate selection of crowd workers, the crowdsourcing activity is aimless. In the literature, the crowd workers are selected on the basis of only a single feature, such as trust, performance, reliability, accurateness, experience, knowledge, skills, expertise, etc. As no multi-criteria features have been used in the past for crowd selection, multi-criteria-based crowd selection is mandatory for the success of the crowdsourcing activity.

Data Availability

No data were used to support this study.

Conflicts of Interest

The authors declare no conflicts of interest.

Acknowledgments

This paper is funded by the University Patent Development Project of Heilongjiang Intellectual Property Office in 2019.

References

L. Zhai, H. Wang, and X. Li, “Optimal task partition with delay requirement in mobile crowdsourcing,” Wireless Communications and Mobile Computing, vol. 2019, Article ID 5216495, 12 pages, 2019.
View at: Publisher Site | Google Scholar
A. Ellero, P. Ferretti, and G. Furlanetto, “Realtime crowdsourcing with payment of idle workers in the retainer model,” Procedia Economics and Finance, vol. 32, pp. 20–26, 2015.
View at: Publisher Site | Google Scholar
A. Sarı, A. Tosun, and G. I. Alptekin, “A systematic literature review on crowdsourcing in software engineering,” Journal of Systems and Software, vol. 153, pp. 200–219, 2019.
View at: Publisher Site | Google Scholar
E. R. Q. Weidema, C. López, S. Nayebaziz, F. Spanghero, and A. v. d. Hoek, “Toward microtask crowdsourcing software design work,” in Proceedings of the 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE), pp. 41–44, Institute of Electrical and Electronics Engineers, Austin, TX, USA, May 2016.
View at: Google Scholar
X. Wang, H. J. Khasraghi, and H. Schneider, “Towards an understanding of participants’ sustained participation in crowdsourcing contests,” Information Systems Management, vol. 37, pp. 1–14, 2019.
View at: Publisher Site | Google Scholar
F. R. Assis Neto and C. A. S. Santos, “Understanding crowdsourcing projects: a systematic review of tendencies, workflow, and quality management,” Information Processing & Management, vol. 54, no. 4, pp. 490–506, 2018.
View at: Publisher Site | Google Scholar
H. J. Pongratz, “Of crowds and talents: discursive constructions of global online labour,” New Technology, Work and Employment, vol. 33, no. 1, pp. 58–73, 2018.
View at: Publisher Site | Google Scholar
L. G. Pee, E. Koh, and M. Goh, “Trait motivations of crowdsourcing and task choice: a distal-proximal perspective,” International Journal of Information Management, vol. 40, pp. 28–41, 2018.
View at: Publisher Site | Google Scholar
H. Amintoosi, S. S. Kanhere, and M. Allahbakhsh, “Trust-based privacy-aware participant selection in social participatory sensing,” Journal of Information Security and Applications, vol. 20, pp. 11–25, 2015.
View at: Publisher Site | Google Scholar
N. Mazlan, S. S. Syed Ahmad, and M. Kamalrudin, “Volunteer selection based on crowdsourcing approach,” Journal of Ambient Intelligence and Humanized Computing, vol. 9, no. 3, pp. 743–753, 2018.
View at: Publisher Site | Google Scholar
M. Hosseini, A. Shahri, K. Phalp, J. Taylor, and R. Ali, “Crowdsourcing: a taxonomy and systematic mapping study,” Computer Science Review, vol. 17, pp. 43–69, 2015.
View at: Publisher Site | Google Scholar
B. Morschheuser, J. Hamari, and A. Maedche, “Cooperation or competition - when do people contribute more? A field experiment on gamification of crowdsourcing,” International Journal of Human-Computer Studies, vol. 127, pp. 7–24, 2019.
View at: Publisher Site | Google Scholar
L. Machado, R. Prikladnicki, F. Meneguzzi, C. R. B. d. Souza, and E. Carmel, “Task allocation for crowdsourcing using AI planning,” in Proceedings of the 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE), pp. 36–40, Institute of Electrical and Electronics Engineers, Austin, TX, USA, May , 2016.
View at: Google Scholar
M. Garcia Martinez and B. Walton, “The wisdom of crowds: the potential of online communities as a tool for data analysis,” Technovation, vol. 34, no. 4, pp. 203–214, 2014.
View at: Publisher Site | Google Scholar
T. D. LaToza, W. B. Towne, A. v. d. Hoek, and J. D. Herbsleb, “Crowd development,” in Proceedings of the 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 85–88, Institute of Electrical and Electronics Engineers, San Francisco, CA, USA, May 2013.
View at: Google Scholar
O. Tokarchuk, R. Cuel, and M. Zamarian, “Analyzing crowd labor and designing incentives for humans in the loop,” IEEE Internet Computing, vol. 16, no. 5, pp. 45–51, 2012.
View at: Publisher Site | Google Scholar
I. Dissanayake, J. Zhang, and B. Gu, “Virtual team performance in crowdsourcing contest: a social network perspective,” in Proceedings of the 2015 48th Hawaii International Conference on System Sciences, pp. 4894–4897, Institute of Electrical and Electronics Engineers, Kauai, HI, USA, January 2015.
View at: Google Scholar
P. Shi, M. Zhao, W. Wang et al., “Best of both worlds: mitigating imbalance of crowd worker strategic choices without a budget,” Knowledge-Based Systems, vol. 163, pp. 1020–1031, 2019.
View at: Publisher Site | Google Scholar
K.-J. Stol, B. Caglayan, and B. Fitzgerald, “Competition-based crowdsourcing software development: a multi-method study from a customer perspective,” IEEE Transactions on Software Engineering, vol. 45, no. 3, pp. 237–260, 2019.
View at: Publisher Site | Google Scholar
A. Ghezzi, D. Gabelloni, A. Martini, and A. Natalicchio, “Crowdsourcing: a review and suggestions for future research,” International Journal of Management Reviews, vol. 20, no. 2, pp. 343–363, 2018.
View at: Publisher Site | Google Scholar
J. Lee and D. Seo, “Crowdsourcing not all sourced by the crowd: an observation on the behavior of Wikipedia participants,” Technovation, vol. 55-56, pp. 14–21, 2016.
View at: Publisher Site | Google Scholar
M. Alsayyari and S. Alyahya, “Supporting coordination in crowdsourced software testing services,” in Proceedings of the 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE), pp. 69–75, Institute of Electrical and Electronics Engineers, Bamberg, Germany, March 2018.
View at: Google Scholar
B. Zhang, C. H. Liu, J. Lu et al., “Privacy-preserving QoI-aware participant coordination for mobile crowdsourcing,” Computer Networks, vol. 101, pp. 29–41, 2016.
View at: Publisher Site | Google Scholar
M. Alhamed and T. Storer, “Estimating software task effort in crowds,” in Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 281–285, Institute of Electrical and Electronics Engineers, Cleveland, OH, USA, September 2019.
View at: Google Scholar
A. Dwarakanath and U. Chintala, “CrowdBuild: a methodology for enterprise software development using crowdsourcing,” in Proceedings of the Second International Workshop on CrowdSourcing in Software Engineering, Institute of Electrical and Electronics Engineers, May 2015.
View at: Google Scholar
C. Qiu, A. C. Squicciarini, B. Carminati, J. Caverlee, and D. R. Khare, “CrowdSelect: increasing accuracy of crowdsourcing tasks through behavior prediction and user selection,” in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Institute of Electrical and Electronics Engineers, Indianapolis, IN, USA, October 2016.
View at: Google Scholar
E. Mourelatos and M. Tzagarakis, “An investigation of factors affecting the visits of online crowdsourcing and labor platforms,” Netnomics: Economic Research and Electronic Networking, vol. 19, no. 3, pp. 95–130, 2018.
View at: Publisher Site | Google Scholar
A. Dubey, G. Virdi, M. S. Kuriakose, and V. Arora, “Towards adopting alternative workforce for software engineering,” in Proceedings of the 2016 IEEE 11th International Conference on Global Software Engineering (ICGSE), pp. 16–23, Institute of Electrical and Electronics Engineers, Irvine, CA, USA, August 2016.
View at: Publisher Site | Google Scholar
Y. Zhao, G. Liu, K. Zheng, A. Liu, Z. Li, and X. Zhou, “A context-aware approach for trustworthy worker selection in social crowd,” World Wide Web, vol. 20, no. 6, pp. 1211–1235, 2017.
View at: Publisher Site | Google Scholar
B. Satzger, R. Zabolotnyi, S. Dustdar et al., “Toward collaborative software engineering leveraging the crowd,” in Economics-Driven Software Architecture, I. Mistrik, R. Bahsoon, R. Kazman, and Y. Zhang, Eds., pp. 159–182, Morgan Kaufmann, Boston, MA, USA, 2014.
View at: Publisher Site | Google Scholar
J. T. Bush and R. M. Balven, “Catering to the crowd: an HRM perspective on crowd worker engagement,” Human Resource Management Review, vol. 31, no. 1, Article ID 100670, 2018.
View at: Google Scholar
B. Kitchenham, Procedures for Performing Systematic Reviews, vol. 33, Keele University, Keele, UK, 2004.
B. Kitchenham and S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering, School of Computer Science and Mathematics, Keele University, Keele, UK, 2007, Technical Report EBSE-2007-01.
R. Saremi, “A hybrid simulation model for crowdsourced software development,” in Proceedings of the 2018 IEEE/ACM 5th International Workshop on Crowd Sourcing in Software Engineering (CSI-SE), pp. 28-29, Institute of Electrical and Electronics Engineers, Gothenburg, Sweden, June 2018.
View at: Google Scholar
T. D. LaToza and A. v. d. Hoek, “A vision of crowd development,” in Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 563–566, Institute of Electrical and Electronics Engineers, Florence, Italy, May 2015.
View at: Google Scholar
M. Li, M. Wang, X. Jin, and C. Guo, “Affinity-Aware online selection mechanisms in mobile crowdsourcing sensing,” in Proceedings of the 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), pp. 1–6, Institute of Electrical and Electronics Engineers, Beijing, China, November 2018.
View at: Google Scholar
S. Sharma, N. Hasteer, and J. P. Van-Belle, “An exploratory study on perception of Indian crowd towards crowdsourcing software development,” in Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), pp. 901–905, Institute of Electrical and Electronics Engineers, Noida, India, May 2017.
View at: Google Scholar
A. L. Zanatta, I. Steinmacher, L. S. Machado, C. R. B. de Souza, and R. Prikladnicki, “Barriers faced by newcomers to software-crowdsourcing projects,” IEEE Software, vol. 34, no. 2, pp. 37–43, 2017.
View at: Publisher Site | Google Scholar
A. L. Zanatta, L. Machado, and I. Steinmacher, “Competence, collaboration, and time management: barriers and recommendations for crowdworkers,” in Proceedings of the 2018 IEEE/ACM 5th International Workshop on Crowd Sourcing in Software Engineering (CSI-SE), pp. 9–16, Institute of Electrical and Electronics Engineers, Gothenburg, Sweden, May 2018.
View at: Google Scholar
X. Peng, J. Gu, T. H. Tan et al., “CrowdService: serving the individuals through mobile crowdsourcing and service composition,” in Proceedings of the 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 214–219, Institute of Electrical and Electronics Engineers, Singapore, Singapore, September 2016.
View at: Google Scholar
F. Moodley, J. V. Belle, and N. Hasteer, “Crowdsourced software development: exploring the motivational and inhibiting factors of the South African crowd,” in Proceedings of the 2017 7th International Conference on Cloud Computing, Data Science & Engineering - Confluence, pp. 656–661, Institute of Electrical and Electronics Engineers, Noida, India, January 2017.
View at: Google Scholar
T. Awwad, N. Bennani, K. Ziegler, V. Sonigo, L. Brunie, and H. Kosch, “Efficient worker selection through history-based learning in crowdsourcing,” in Proceedings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), pp. 923–928, Institute of Electrical and Electronics Engineers, Turin, Italy, July 2017.
View at: Google Scholar
X. Zhang, Z. Chen, C. Fang, and Z. Liu, “Guiding the crowds for android testing,” in Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 752-753, Institute of Electrical and Electronics Engineers, Austin, TX, USA, May 2016.
View at: Google Scholar
R. L. Saremi, Y. Ye, G. Ruhe, and D. Messinger, “Leveraging crowdsourcing for team elasticity: an empirical evaluation at TopCoder,” in Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 103–112, Institute of Electrical and Electronics Engineers, Buenos Aires, AR, USA, May 2017.
View at: Google Scholar
N. Leicht, I. Blohm, and J. M. Leimeister, “Leveraging the power of the crowd for software testing,” IEEE Software, vol. 34, no. 2, pp. 62–69, 2017.
View at: Publisher Site | Google Scholar
Q. Cui, J. Wang, G. Yang, M. Xie, Q. Wang, and M. Li, “Who should Be selected to perform a task in crowdsourced testing?” in Proceedings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), pp. 75–84, Institute of Electrical and Electronics Engineers, Turin, Italy, July 2017.
View at: Google Scholar
D. Sánchez-Charles, J. Nin, M. Solé, and V. Muntés-Mulero, “Worker ranking determination in crowdsourcing platforms using aggregation functions,” in Proceedings of the 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1801–1808, Institute of Electrical and Electronics Engineers, Beijing, China, July 2014.
View at: Google Scholar
O. Folorunso and O. A. Mustapha, “A fuzzy expert system to Trust-Based Access Control in crowdsourcing environments,” Applied Computing and Informatics, vol. 11, pp. 116–129, 2015.
View at: Publisher Site | Google Scholar
K. Mao, L. Capra, M. Harman, and Y. Jia, “A survey of the use of crowdsourcing in software engineering,” Journal of Systems and Software, vol. 126, pp. 57–84, 2017.
View at: Publisher Site | Google Scholar
Y. Wang, Z. Cai, G. Yin, Y. Gao, X. Tong, and G. Wu, “An incentive mechanism with privacy protection in mobile crowdsourcing systems,” Computer Networks, vol. 102, pp. 157–171, 2016.
View at: Publisher Site | Google Scholar
M. Christoforaki and P. G. Ipeirotis, “A system for scalable and reliable technical-skill testing in online labor markets,” Computer Networks, vol. 90, pp. 110–120, 2015.
View at: Publisher Site | Google Scholar
B. Satzger, H. Psaier, D. Schall, and S. Dustdar, “Auction-based crowdsourcing supporting skill management,” Information Systems, vol. 38, no. 4, pp. 547–560, 2013.
View at: Publisher Site | Google Scholar
S. Franken, S. Kolvenbach, W. Prinz, I. Alvertis, and S. Koussouris, “CloudTeams: bridging the gap between developers and customers during software development processes,” Procedia Computer Science, vol. 68, pp. 188–195, 2015.
View at: Publisher Site | Google Scholar
I. Dissanayake, N. Mehta, P. Palvia, V. Taras, and K. Amoako-Gyampah, “Competition matters! Self-efficacy, effort, and performance in crowdsourcing teams,” Information & Management, vol. 56, Article ID 103158, 2019.
View at: Publisher Site | Google Scholar
E. Segev, “Crowdsourcing contests,” European Journal of Operational Research, vol. 281, no. 2, pp. 241–255, 2020.
View at: Publisher Site | Google Scholar
L. Tran-Thanh, S. Stein, A. Rogers, and N. R. Jennings, “Efficient crowdsourcing of unknown experts using bounded multi-armed bandits,” Artificial Intelligence, vol. 214, pp. 89–111, 2014.
View at: Publisher Site | Google Scholar
A. Smirnov, A. Ponomarev, and N. Shilov, “Hybrid crowd-based decision support in business processes: the approach and reference model,” Procedia Technology, vol. 16, pp. 376–384, 2014.
View at: Publisher Site | Google Scholar
A. Moayedikia, W. Yeoh, K.-L. Ong, and Y. L. Boo, “Improving accuracy and lowering cost in crowdsourcing through an unsupervised expertise estimation approach,” Decision Support Systems, vol. 122, Article ID 113065, 2019.
View at: Publisher Site | Google Scholar
P. Micholia, M. Karaliopoulos, I. Koutsopoulos, L. M. Aiello, G. D. F. Morales, and D. Quercia, “Incentivizing social media users for mobile crowdsourcing,” International Journal of Human-Computer Studies, vol. 102, pp. 4–13, 2017.
View at: Publisher Site | Google Scholar
M. Modaresnezhad, L. Iyer, P. Palvia, and V. Taras, “Information Technology (IT) enabled crowdsourcing: a conceptual framework,” Information Processing & Management, vol. 57, Article ID 102135, 2020.
View at: Publisher Site | Google Scholar
M. Garcia Martinez, “Inspiring crowdsourcing communities to create novel solutions: competition design and the mediating role of trust,” Technological Forecasting and Social Change, vol. 117, pp. 296–304, 2017.
View at: Publisher Site | Google Scholar
D. E. Boubiche, M. Imran, A. Maqsood, and M. Shoaib, “Mobile crowd sensing - taxonomy, applications, challenges, and solutions,” Computers in Human Behavior, vol. 101, pp. 352–370, 2019.
View at: Publisher Site | Google Scholar
F. Saab, I. H. Elhajj, A. Kayssi, and A. Chehab, “Modelling cognitive bias in crowdsourcing systems,” Cognitive Systems Research, vol. 58, pp. 1–18, 2019.
View at: Publisher Site | Google Scholar
E. Schenk, C. Guittard, and J. Pénin, “Open or proprietary? Choosing the right crowdsourcing platform for innovation,” Technological Forecasting and Social Change, vol. 144, pp. 303–310, 2019.
View at: Publisher Site | Google Scholar
M. Kadadha, H. Otrok, R. Mizouni, S. Singh, and A. Ouali, “SenseChain: a blockchain-based crowdsensing framework for multiple requesters and multiple workers,” Future Generation Computer Systems, vol. 105, pp. 650–664, 2020.
View at: Publisher Site | Google Scholar
F. Kamoun, D. Alhadidi, and Z. Maamar, “Weaving risk identification into crowdsourcing lifecycle,” Procedia Computer Science, vol. 56, pp. 41–48, 2015.
View at: Publisher Site | Google Scholar
T. Mack and C. Landau, “Winners, losers, and deniers: self-selection in crowd innovation contests and the roles of motivation, creativity, and skills,” Journal of Engineering and Technology Management, vol. 37, pp. 52–64, 2015.
View at: Publisher Site | Google Scholar
N. Luz, N. Silva, and P. Novais, “A survey of task-oriented crowdsourcing,” Artificial Intelligence Review, vol. 44, no. 2, pp. 187–213, 2015.
View at: Publisher Site | Google Scholar
U. Gadiraju and R. Kawase, “Improving reliability of crowdsourced results by detecting crowd workers with multiple identities,” Lecture Notes in Computer Science, vol. 2015, pp. 190–205, 2017.
View at: Publisher Site | Google Scholar
J. Troll, I. Blohm, and J. M. Leimeister, “Why incorporating a platform-intermediary can increase crowdsourcees’ engagement,” Business & Information Systems Engineering, vol. 61, no. 4, pp. 433–450, 2019.
View at: Publisher Site | Google Scholar
K. L. Jeffcoat, T. J. Eveleigh, and B. Tanju, “A conceptual framework for increasing innovation through improved selection of specialized professionals,” Engineering Management Journal, vol. 31, no. 1, pp. 22–34, 2019.
View at: Publisher Site | Google Scholar
S. Hosio, J. Goncalves, N. van Berkel, S. Klakegg, S. I. Konomi, and V. Kostakos, “Facilitating collocated crowdsourcing on situated displays,” Human-Computer Interaction, vol. 33, no. 5-6, pp. 335–371, 2018.
View at: Publisher Site | Google Scholar
A. Moayedikia, W. Yeoh, K.-L. Ong, and Y.-L. Boo, “Framework and literature analysis for crowdsourcing’s answer aggregation,” Journal of Computer Information Systems, vol. 60, no. 1, pp. 49–60, 2020.
View at: Publisher Site | Google Scholar
P. Shi, W. Wang, Y. Zhou et al., “Practical POMDP-based test mechanism for quality assurance in volunteer crowdsourcing,” Enterprise Information Systems, vol. 13, no. 7-8, pp. 979–1001, 2019.
View at: Publisher Site | Google Scholar
G. D. Saxton, O. Oh, and R. Kishore, “Rules of crowdsourcing: models, issues, and systems of control,” Information Systems Management, vol. 30, no. 1, pp. 2–20, 2013.
View at: Publisher Site | Google Scholar
G. Montelisciani, D. Gabelloni, G. Tazzini, and G. Fantoni, “Skills and wills: the keys to identify the right team in collaborative innovation platforms,” Technology Analysis & Strategic Management, vol. 26, no. 6, pp. 687–702, 2014.
View at: Publisher Site | Google Scholar
H. Zheng, D. Li, and W. Hou, “Task design, motivation, and participation in crowdsourcing contests,” International Journal of Electronic Commerce, vol. 15, no. 4, pp. 57–88, 2011.
View at: Publisher Site | Google Scholar
S. Standing and C. Standing, “The ethical use of crowdsourcing,” Business Ethics: A European Review, vol. 27, no. 1, pp. 72–80, 2018.
View at: Publisher Site | Google Scholar
Z. Zhao, D. Yan, W. Ng, and S. Gao, “A transfer learning based framework of crowd-selection on twitter,” in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, Institute of Electrical and Electronics Engineers, Chicago, IL, USA, August 2013.
View at: Google Scholar
C. Qiu, A. Squicciarini, D. R. Khare, B. Carminati, and J. Caverlee, “CrowdEval: a cost-efficient strategy to evaluate crowdsourced worker’s reliability,” in Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Institute of Electrical and Electronics Engineers, Stockholm, Sweden, July 2018.
View at: Google Scholar
Y. Amsterdamer, T. Milo, A. Somech, and B. Youngmann, “December: a declarative tool for crowd member selection,” Proceedings of the VLDB Endowment, vol. 9, no. 13, pp. 1485–1488, 2016.
View at: Publisher Site | Google Scholar
Y. Amsterdamer, T. Milo, A. Somech, and B. Youngmann, “Declarative user selection with soft constraints,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Institute of Electrical and Electronics Engineers, Beijing, China, November 2019.
View at: Google Scholar
Z. Peng, X. Gui, J. An, R. Gui, and Y. Ji, “TDSRC: a task-distributing system of crowdsourcing based on social relation cognition,” Mobile Information Systems, vol. 2019, Article ID 7413460, 2019.
View at: Publisher Site | Google Scholar
V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative Research in Psychology, vol. 3, no. 2, pp. 77–101, 2006.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2021 Rui Shang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies