Abstract
The quality of tasks performed by a crowd worker is questionable. Most of the workers are lost if their first work is of low quality, which may influence them to acquire future work in crowdsourcing. Research has highlighted that workers either lack motivation or capability. However, the integrative perspective of capability and motivation in current crowdsourcing research is scarce. There is a need to investigate the relationship of capability and motivation of crowd worker to understand the phenomenon of getting better performance, which ultimately produces a quality outcome. This research aims toward understanding such relationship with mathematical perspective. The traditional renowned and well-accepted theories related to job performance are used for the quantification of motivation, capability, and performance for crowd workers to investigate the impact of capability in relation to the motivation on the performance of crowd worker. Experimental results suggest that formulae will benefit the requester to evaluate the performance of a crowd worker before providing him/her the task and benefit in reducing unemployment in the situation of COVID-19 pandemic.
1. Introduction
Recent advances in web and communication technologies have made organizations consider crowdsourcing as an attractive business model for today’s world that can provide a promising solution for institutions across industries who want to utilize the power of the masses, especially in the current situation of the pandemic. However, the quality of work produced by the crowd remains the key concern in crowdsourcing [1–11]. There are only about half of the available crowdsourcing platforms offering some kind of quality control [12]. It was reported by [13] that on one of the major crowdsourcing platforms, more than 30 percent of the work submitted was of low quality. Crowdsourcing is frequently reported as being “plagued by low-quality results” [11]. Though measures are taken to address the issue, it is still quite a challenge to ensure quality outcomes from workers [14].
One of the major challenges which contribute to low quality of outcome is that workers, in general, opt for the task(s) in which their capabilities do not match. Even most crowd workers, with sufficient knowledge and skill, could not get the second task if they underperform in the first task [15]. Worker’s motivation is another challenge, as reported by [16–18].
Motivation and capability of workers are related, but this relationship is given limited value in crowdsourcing research, especially in terms of how to quantify the two constructs. Motivation may be extrinsic and intrinsic [19]. Workers mostly motivated by financial gains; however, financial incentives may be counterproductive [20]. Therefore, financial incentives are not always the better choice. With low monetary rewards in microtask crowdsourcing, little is known about the motivation of a crowd worker to participate in crowdsourcing. In case of insufficient monetary rewards, low performance may be observed [21, 22]. Even an increase in monetary rewards may result in a lack of quality despite an increase in the quantity of work [20, 23]. Reputation is reported as extrinsic motivation and it seems appropriate as workers in the crowdsourcing domain try to maintain their reputation to access future work. Moreover, challenging task plays its role to entice the worker intrinsically and is used to attract more workers to perform on a particular task, as challenging yet achievable tasks have the potential to keep the motivation of an individual [24].
Some of the possible reasons reported in the literature for the low quality of submissions from the crowd worker include lack of knowledge among the crowd [25], reduced capability [15, 26], biased attitude toward rewards [13], malicious activities such as fake reviews to fulfill foul interests [27], lesser control on crowd workers due to their temporary connectedness with the requesters/solution seekers [28], and less knowledge about the capable crowd [29]. However, the integrative perspective of capability and motivation in current crowdsourcing research is scarce. There is a need to investigate the relationship of capability and motivation of crowd worker to understand the phenomenon of getting better performance, which ultimately produces quality outcome.
This research paper aims to contribute to the crowdsourcing domain by modeling the relationship between motivation, capability, and performance. Since crowdsourcing is considered as a nontraditional business model as there is no contract bounding [26], traditional theories to develop understanding is advised and emphasized in research [30]. The traditional renowned and well-accepted theories related to job performance are used for the quantification of motivation, capability, and performance for crowd workers to investigate the impact of capability in relation to the motivation on the performance of crowd worker. To the best of our knowledge, this work is the first attempt to model and investigate the integrative relationship of capability, motivation, and performance and formulate to quantify these constructs for crowd workers.
Performance is considered a predictor of quality and is a multidimensional construct [31]. Traditionally, employees (or workers) are recruited from a pool of capable applicants to get better performance [32], and the level of individual’s knowledge and skill (in the traditional workplace) is considered as a job performance criterion [33]. The motivation of workers is also essential in order to warrant better performance [34, 35]. Ability and motivation are considered as resources offered by an individual in the traditional workplace to provide better performance [31, 36, 37]. This suggests that to acquire quality work from crowd workers, their motivation, in addition to the required knowledge and skill, is significant. In this study, the research questions are as follows: Q1: Do less capable crowd workers perform better when provided with intrinsic and/or extrinsic motivation (with a constant financial reward)? Q2: Do capable crowd workers underperform when no intrinsic and/or extrinsic motivation is provided (with a constant financial reward)?
Through an experimental approach, the performance of crowd workers is formulated and evaluated on the basis of their capability and motivation. To quantify capability, the viewpoint of the human capability framework [38] is used. According to [38], capability is the result of the capacity (knowledge and skill) of an individual regarding a task that suits or matches that individual’s capacity. Motivation is measured using the expectancy theory of motivation [39]. The motivation and capability scores are then used to evaluate the overall performance of crowd workers using the theoretical lenses of the Campbell job performance model (1990) and AMO model [36].
Inspiring by the task uploaded on Google’s Crowdsource application, translation validation is used in the experiment as the task to be provided to the crowd worker. Using a language comprehension test, the capacity of crowd workers was tested. Two groups were formed randomly, namely, the informed group and uninformed group after the capacity test. The informed group was provided with the information regarding motivation criteria, whereas it was not provided to the uninformed group. However, a fixed compensation amount was offered to participants of each group to avoid the effect of financial reward. Each group was given validation tasks for English and Urdu. English Validation test consists of English sentences with multiple Urdu translation options. Crowd workers (participants from now on) were required to select one best option. Similarly, the Urdu Validation test consists of Urdu sentences with multiple English translation options. Each validation test (Urdu and English) was offered with two levels (level 1which is mandatory and level 2 which is optional). Using mandatory level, the task matching of each participant was identified, i.e., whether the crowd worker’s capacity matches with English or Urdu validation. This helped in identifying capable and less-capable crowd workers among the participants. The optional level (level 2) consists of the challenge as intrinsic motivation. Participants were given a choice to either attempt level 2 (a challenging task) or quit. To extrinsically motivate, stars were awarded to each participant on the basis of correct responses to maintain the reputation of that participant. The informed group was notified about this motivational criterion, whereas it was not shared with the uniformed group. The overall performance was measured on the basis of correct responses out of the questions attempted.
Findings suggest that in the informed group, less capable crowd workers attempted the challenging task and performed better as compared to the capable crowd workers belonging to the uninformed group; despite that, both groups were informed of the same financial scheme. The only difference was in terms of the challenge (as intrinsic motivation) and reputation (as extrinsic motivation).
The contribution of this research is as follows:(i)Relationship of capability and motivation of crowd worker to get better performance, thereby ensuring the quality of outcome produced.(ii)Formula to quantify capability, motivation, and performance of a crowd worker for translation task.(iii)Integrative effect of capability and motivation to acquire better performance.(iv)Understanding that financial incentives are not the only motive to attract crowd workers and get better performance even from a less capable individual who has a limited capability for a given task.
In the rest of the research paper, the related work is discussed in Section 2. In Section 3, the theoretical perspective is discussed using the theories of performance, human capability model, and theory of motivation to develop hypotheses. Further, in Section 4, the formulation of performance is provided on the basis of theoretical perspective capability and motivation. Research design is presented in Section 5, which follows the results and discussion section. Finally, the conclusion is presented in Section 6.
2. Related Work
Kaufmann et al. [40] proposed a model to explore the worker’s motivation to participate in crowdsourcing, but the relationship between motivational aspect and performance remains unexplored. In another research, [25] proposed the model based on a probabilistic approach for performance prediction of crowd workers based on their capabilities. They provided the taxonomy regarding the capabilities (in terms of human ability to perform a task) of the crowd worker who provides the services to accomplish microtasks available on crowdsourcing platforms. The authors explored the mapping of the capabilities to the task that can help to predict the performance of workers in crowdsourcing platforms using the probabilistic approach. The ability to perform the given task was considered to predict performance. However, the influence of worker’s knowledge, skill, and motivation remained unexplored. In another research, [41] using survey approach, considering self-determination and motivational affordance theory, provided a model to study motivation of crowd workers to participate in crowdsourcing. However, the quality of work produced by the workers was not investigated. Sun et al. [27] use the expectancy theory to develop a model to explore the effect of workers’ motivation on the effort expended on a task by the crowd worker. The research proposed the nonlinear relationship of self-efficacy and effort with moderating effect of task complexity for paid crowdsourcing platform in China. However, the effect of motivation on the worker’s performance has not been investigated. In another research to understand the motivation behind the involvement of a crowd worker in crowdsourcing, [42] emphasize the need to explore the relationship of motivation and performance.
In recent research, Nouri and et al. [14] have focused on clarity issues at the end of requesters to describe the task descriptions so that the crowd worker may attempt the task without any ambiguity. A similar effort was made by Lim and et al. [43] to examine the effects of clarity in goals to achieve and feedback on the quality of the outcome of crowdsourced tasks.
Although the clarity of task description and goals to achieve are necessary for producing a quality outcome, however, no matter how clear a task (to be performed) is explained or the goal is set, if the individual, to whom the task (or job) is (meant to be) assigned, does not have the necessary knowledge and skill to perform that task, the desired outcome cannot be achieved. Moreover, what if an individual does possess both the ability and motivation, but there is no (or less) opportunity provided such as resources, available time, working environment? This will lower the overall performance of that individual [44]. There is a need to focus on theoretical development and empirical investigation to understand the aspects which influence the quality outcome in crowdsourcing on the basis of renowned theories of traditional work [30].
This warrants a need to investigate the integrative relationship of capability and motivation of crowd worker to understand the phenomenon of getting better performance, which ultimately produces a quality outcome. This research is aimed toward understanding such a relationship with a mathematical perspective. The mathematical approach will help in quantifying the concepts of capability and motivation of a crowd worker, which are vital to compute the respective performance hence helps in producing a quality outcome.
3. Theoretical Perspective
In this section, the key concepts of knowledge, skill, and motivation are reviewed to understand and formulate their relationship with (job/task) performance. The theoretical perspective is based on several research theories, namely the Job Performance model [31], Ability-Motivation-Opportunity (AMO) theory [36], the Human Capability Framework (HCF) [38], and Expectancy theory of motivation [39].
3.1. Understanding the Dimensions of (Worker’s) Performance
Performance is a multidimensional concept [31]. To explore its primary dimensions, two theories are discussed in this section, namely, Job Performance Model and AMO model.
3.1.1. Job Performance Model
Campbell et al. [31] and view performance as a multidimensional construct. According to their model, performance can be considered as a function of basic components, job-specific components, and motivational components. These components were further explored in another research by McCloy et al. [45] to propose a model for job performance. According to this model, performance “P” of an individual is considered to be a function of declarative knowledge “DK,” procedural knowledge “PK,” and motivation “M,” as expressed in the equation below:
As per the model, declarative knowledge answers “what (task) to do” questions, whereas procedural knowledge deals with “how to do (a task)” questions. In other words, DK assesses the task understanding of an individual through job knowledge tests. Motivation and Procedural Knowledge are not necessary elements in the assessment of an individual’s job knowledge initially [45]. On the contrary, to assess the work performance of an individual, it is necessary to test procedural knowledge and motivation. The model suggested a latent structure of performance determinants. As far as the operations for the function is concerned, researchers suggested not to have the additive operator; instead, they considered multiplicative function as a better option. The model has helped in bridging management practices with an individual’s job performance [46].
The reason for using [31] job performance model as one of the theoretical lenses in this study is its prominence in the literature and the explanation it provides to bridge the link between an individual’s perspective for job performance and management practices. As crowdsourcing is a business process that relies on human involvement, it is necessary to study the individual’s perspective to perform tasks available on crowdsourcing platforms. The above discussion suggests that it requires a crowd worker to have the necessary knowledge, skills, and motivation to perform a task to achieve the desired objectives of the requester.
3.1.2. Ability-Motivation-Opportunity (AMO) Model
Appelbaum [36] explores the effect of the high-performance work practices at the firm (or organization) level. High-performance work practices consist of three elements (or dimensions), namely Ability, Opportunity, and Motivation [36]. According to this understanding, a work environment should provide the opportunity (O) to the workers (or employees) to contribute, leading to an increase in the level of worker’s ability (A) to perform and thereby creating required motivation (M) for effective participation. AMO model is a framework that provides a structured way to understand the relationship between human resource management and (employee’s) performance [47].
Liao et al. [46] explored the role of workers (or employees) of an organization to bridge the managerial efforts and the overall performance. Liao [46] suggested that the performance of an individual depends primarily upon her human capital and motivation, in addition to the provided opportunity, as expressed in the following equation:
Human capital is explained as the knowledge, skills, and abilities of an individual that will benefit the organization [48].
The focus on employees’ human capital and motivation provided by Liao et al. [46] is in line with Campbell’s job performance model [31]. This suggests that the worker’s ability, skills and knowledge are necessary elements that are utilized by the organization to perform better provided with the motivational environment. Sonnentag et al. [49] highlighted the significance of knowledge, skills, and motivation of an individual along with the clarity of job description.
The theories discussed above to understand the dimensions of performance reveal that an individual’s ability, knowledge, skill, and motivation are the major components of her performance along with the provided opportunity.
To proceed further, it is necessary to understand the terms ability, knowledge, skills, and motivation. The lens of the human capability framework [38] and expectancy theory of motivation [39] will help in developing this understanding.
3.2. Human Capability Framework (HCF)
According to [38], HCF identifies capacity, opportunity, and (task) matching as three components of the capability of an individual.
Capacity of an individual defines the level of knowledge and skill with which the work is performed (e.g., average, good, and best). Opportunities are the options available that can help individuals to achieve financial (or personal) rewards by performing the task. The work environment provides opportunities to an individual that may include variables such as allocated tasks, working conditions, tools, materials, and time. Matching is the component that helps to find suitability between an individual (worker) and task s/he may perform.
HCF helps to clarify the concepts of procedural and declarative knowledge of Campbell et al.’s [31] perspective and ability of Appelbaum’s [36] perspective and defines these concepts as a single term, i.e., capacity. The concept of ability discussed in the AMO theory and the concept of the declarative and procedural knowledge discussed by Campbell et al. [31] in the job performance model are congruent to the concept of capacity described in HCF [38]. According to AMO theory [36], ability is defined as the skills and knowledge to perform and Campbell’s declarative and procedural knowledge [31] also refers to the skill and knowledge of an individual.
Therefore, the current study considers “ability” and “declarative as well as procedural knowledge” congruent to “capacity.”
Using HCF, in this research, the capability (Cap) of crowd worker is defined as “A function of capacity (C) possess by the crowd worker to perform a task, the task matching (TM), and the provided Opportunity (O) to perform that task”.
It is expressed in the following equation:
3.3. Expectancy Theory of Motivation
The expectancy theory is considered as one of the renowned theories among several process theories of motivation in view of organizational and industrial psychologists [50]. Expectancy theory was originally suggested by Victor [39]. In expectancy theory, the motivation of an individual worker is based on three motivational elements, namely expectancy (E), instrumentality (I), and valence (V), as shown in the following equation:
Expectancy is the self-expectation of an individual that putting in (a certain amount of) effort on a given task will help to achieve (a certain level of) performance. Instrumentality is the belief that if an individual fulfills the performance criterion, then s/he will get rewarded. Instrumentality can be thought of as the carrot hung in front of the rabbit cart to keep the rabbit running. Valence is the significance (or worth) associated with the reward (expected by an individual). Examples of valence may include self-recognition, skill development, incentives earned, and reputation.
According to the expectancy theory, motivation is the multiplicative function of elements E, I, and V as expressed in the following equation:
4. Formulation for Crowd Worker’s Performance
In this section, the formula to calculate performance incorporating both capability and motivation is derived using the concepts discussed in the section of theoretical perspective.
4.1. Incorporating Capability
Referring to equation (3), capability is considered as the combination of capacity, opportunity, and task matching. That is, a capable crowd worker is an individual with the necessary capacity score (C), the appropriate task matching (TM) with the assigned task. Then capability (Cap) as an additive function can be computed as shown in the following equation:
4.2. Incorporating Motivation
Motivation can be measured by using the perspective of multiplicative function explained by Vroom’s Expectancy theory. To quantify motivation (M), considering compensation (Cmp), reputation score (Rep), and challenge (Ch), using equation (5) we can express M as follows:
4.3. Quantifying Performance
Now that the capability and motivation have been formulized, the crowd worker’s performance is quantified.
Referring to the concept of capability discussed in Section 4.1, and using equation (1) and equation (2), the performance can be expressed as the function of capability and motivation as shown in the following equation:
The original formula proposed in AMO model was multiplicative. So, equation (8) can be expressed as follows:
On the contrary, in view of some other scholars, the performance should better be explained by additive function [51, 52] as expressed in the following equation:
The reason for this is the capability of an individual is an indispensable requirement, and without ensuring the capability, motivation alone cannot influence the performance. However, according to equation (9), the performance is zero in case of zero motivation. Therefore, to reflect the importance of capability and motivation, in this research paper, the formula used to quantify performance is expressed in equation (11), which shows that in the absence of motivation, the performance will depend on the capability of an individual.
5. Hypotheses Development
In this paper, Challenge, Compensation, and Reputation are used as motivators. Challenge is considered as an important motivational element expected by workers in achieving the outcomes [53]. It is the intrinsic type of motivation. Even in easy-to-solve scenarios, there is some sort of challenge involved to keep up the motivation level. The extent of the challenge is indicated by the complexity and difficulty of the task connected with human needs for competence [19]. However, it is observed that the complexity and difficulty in the task should not make the desired solution unachievable (for an individual); otherwise, the effect of motivation will sharply decrease [54].
A good reward/compensation scheme is a key motivation element to keep an individual motivated [34, 55]. According to Lakhani and Wolf [56], monetary reward is a type of compensation and is also defined as an extrinsic motivation in terms of immediate payoff [3, 25].
Moreover, reputation is a kind of extrinsic motivation. For online tasks, as provided in crowdsourcing, the system relies on networks of people with connected computers in their workplaces [57]. People participating in such networks are mostly anonymous at the peer level and to the requester as well [58]. In such an anonymous scenario, some kind of reputation tracking associated with individual workers is necessary to help the requester to assess the worker's reputation [2, 3, 11].
It is stated in crowdsourcing research that clarity of task description is required to get a quality outcome. However, no matter how clearly a task is described, if the worker does not have the capability that is required to perform that task, it will be very difficult to achieve the desired outcome and will lower the overall performance of that individual [44].
Considering expectancy theory, a crowd worker would consider attempting a task if s/he believes that s/he can perform a challenging task (expectancy) (despite s/he has the low capability of attempting it), for which the rewards are available within the crowdsourcing environment (instrumentality) and s/he will be motivated to perform a task to achieve (or maintain) the desired valued outcome (valence/worth of reward), which is considered as reputation. This leads to the following hypotheses: H1: Intrinsic motivation (Challenge) helps a less capable crowd worker to perform better. H2: Extrinsic motivation (Reputation) helps a less capable crowd worker to perform better.
6. Research Design
To investigate the hypothesis proposed in the previous section, a true experiment is conducted.
Inspiring by the task uploaded on the Google Crowdsource (https://crowdsource.google.com/) application, translation validation is used in the experiment as the task to be provided to the crowd. The crowd is the set of undefined people [58, 59]. For the experiment, crowdsourcing platform “Microworkers” was approached. Initially, a call for translation validation contest among the crowd workers is made with the knowledge of both English and Urdu languages. This call is necessary to make crowd workers feel that they are participating in the contest and in this way it will help to avoid the reactivity threat–a factor affecting the external validity. The crowd workers registered for the contest will be termed as participants hereafter. The variables used, for calculation purposes, with their definitions are given in Table 1.
6.1. Testing for the Participant’s Capability
Capability, as formulated in the section of theoretical perspective, is the additive result of capacity and task matching. It is assumed that a participant is capable if she has the capacity and the matching with the given task, else she will be considered less capable. To calculate the capability, participants are tested for their capacity and task matching.
6.1.1. Capacity Test and Capacity Score Measurement
For the capacity test, all the participants were tested for translation in two languages, i.e., English and Urdu. This included a translation from English to Urdu as well as from Urdu to English.
Translation from one language to another language requires comprehension and bilingualism to be tested [60]. For this purpose, the testing instrument was designed by the language experts, consisting of comprehension for each language. The test task was assigned to all participants.
The capacity score of participant “x” (CSx) was computed by averaging the scores of the English capacity test and Urdu capacity test for the total number of questions asked. If m number of questions asked in the English capacity test and n number of questions asked in the Urdu capacity test, then the capacity score was computed by using the following equation:
Here, CSxE, as expressed in equation (13), is the capacity score of participant x in the English capacity test that can be computed by calculating the number of correct answers provided by the participant x in the English capacity test. CSxU, as expressed in equation (14), is the capacity score of participant “x” in the Urdu capacity test was computed by calculating the number of correct answers provided by the participant “x” in the Urdu capacity test.
where variables AE and AU are used to hold the score of each correct answer for questions asked in English and Urdu tests, respectively. The number of questions asked in a capacity test is represented by the subscript “i.”
6.1.2. Group Formation and Validation Test Task
After the capacity test, two groups were formed randomly, the informed group and the uninformed group. The informed group had the knowledge about the motivational criteria, whereas the uniformed group was not informed about any motivational criteria. Each group was given validation tasks for English and Urdu. The purpose of the validation task was to help in computing the task matching with respect to tasks performed by participants. English Validation test consisted of English sentences with multiple Urdu translation options. Participants were required to select one best option. Urdu Validation test consisted of Urdu sentences with multiple English translation options. Participants were required to select one best option.
The capability and motivation scores for each participant were calculated after the validation test. All participants were tested for validation tests (Urdu and English) with two levels. The first level (level 1) was mandatory and the second level (level 2) was optional. Each participant was given a total of 30 questions (15 questions each for English and Urdu validation). Among these 15 questions (each for English and Urdu Validation), 10 questions were mandatory, whereas other 5 questions were optional.
6.1.3. Task Matching Test and Task Matching Score Measurement
The first 10 questions, each from English and Urdu Validation tests, were used for this purpose. Task matching was assessed in a way such that if a participant scored better in the capacity test of English and also scored better in the validation test of English; then it was considered as task matching. Otherwise, it was considered as not matching. A similar assessment for the Urdu capacity test was conducted. That is, the score of each participant for the capacity test was compared with the score of the validation test and a binary value “1” was assigned for task matching and “0” otherwise.
TMx is considered as the status of task matching for a participant x. Task matching occurs if the following condition holds:
The italicized letters CEx, CUx, CBx are the paticipant’s capacity statuses and VEx, VUx, VBx are the statuses of a participant’s validation statuses, as defined in Table 1.
For each participant, it was required to assess the status of participant’s capacity, i.e., whether participant “x” scored better in the capacity test for English or Urdu or both. The capacity status is a Boolean variable (italicized) and its assessment criteria are given in Table 2. Further, the validation scores were compared for each participant, i.e., whether participant “x” scored better in the validation test for English or Urdu or both. The validation status is a Boolean variable (italicized) and its assessment criteria are given in Table 3.
6.1.4. Capability Score Measurement
Since all the participants were given the opportunity to participate; therefore, the opportunity was ignored (i.e., O is considered to be equal to 1). Hence, the capability of participant “x” (Capx) was measured by adding the capacity score (CSx) and task matching (TMx). Using equation (5), we can express capability (Capx) as follows:
6.2. Motivation Test
Table 4 shows the criterion regarding motivation introduced to both informed and uninformed groups. The informed group had the knowledge about the motivational criteria, whereas the uniformed group was not informed about any motivational criteria.
There was a fixed compensation amount offered to participants of each group initially. Among 15 questions, the first 10 questions were mandatory to every participant in level 1, whereas last 5 questions were optional (termed as level (2) to attempt as it offered a challenge.
In the form of negative marking, participants were given a choice to either attempt a challenging task or quit. Also, participants were awarded stars, as ad form of reputation, for correct answers.
Reputation score and challenge attempt were measured for each participant, irrespective of informed or uninformed group, as discussed in the following subsections.
6.2.1. Reputation Score Measurement
A participant x will be awarded a number of stars as reputation. For 15 questions in the validation test, a participant x can earn 9 stars, and for 30 questions, participant x can earn 18 stars at the.
Most similarly, for 5 optional questions with negative markings (for each English and Urdu), stars will be awarded for the correct answer. Further, the number of stars will be deducted in case of wrong answers for optional questions. A reputation score is based on the number of stars a participant can earn. The reputation score will be calculated for the participants of the uninformed group and informed group, though, uninformed group will not be aware of reputation.
6.2.2. Challenge Attempt Measurement
Challenge attempt status for participant x is considered as Chx. If x attempts the last five questions for (each English and Urdu) validation tests, then Chx holds 1, else 0.
6.2.3. Calculation of Motivation
Motivation will be measured by using the perspective of the multiplicative effect of Vroom’s Expectancy theory. To quantify motivation, three components, compensation, reputation score, and challenge attempt, are used.
Initially, a fixed compensation (Cmpx) will be available to all participants irrespective of the group. The initial motivation is considered as Mx_initial and is expressed in equation (16).
Further, reputation will be introduced in level 1 to motivate the participants. This is considered as Mx_L1 and expressed in equation (17) as the product of compensation and reputation score.
Attempting challenging tasks to earn more reputation and compensation, introduced in level 2, is considered as Mx_L2 and expressed in equation (18).
It is to be notified here that both the groups remain unaware of this relative increment of compensation to avoid the dependence of their performance on the monetary reward. Both the groups were informed that compensation will be given on the overall performance.
If a participant does not attempt the challenge, Mx_L2 will be zero. The overall motivation is considered as an additive function of Mx_intial, Mx_L1, Mx_L2. Hence, the overall motivation of a participant “x” is considered as Mx and is calculated using the following equation:
6.3. Calculation of Performance
Now that the measurements of capability and motivation have been discussed, the measurement for performance will be detailed. As discussed earlier in the theoretical perspective, that job performance is the function of motivation and capability. If Px is considered as the performance of a participant “x,” then using equation (11), it can be formalized as given in the following equation:
7. Results and Discussion
On the call for registration, total of 53 participants were registered for the experiment approaching the crowdsourcing platform “Microworkers”. Out of 53 participants, 40 were enrolled. The overall capability scores of participants range between 0.4 and 1.66 for uninformed and informed groups, as depicted in Figures 1 and 2 respectively. The corresponding author may provide the data, which is used to support the findings of this research, on request. Due to ethical restrictions, the data are not publicly available.
Findings suggest that less capable crowd workers attempted the challenging task and performed better in the informed group as compared to the capable crowd workers belonging to the uninformed group, despite that both groups were informed of the same financial scheme. The only difference was in terms of the challenge (as intrinsic motivation) and reputation (as extrinsic motivation). The observations regarding performance and motivation of participants belonging to the uninformed and informed group before level 2 are depicted in Figures 3 and 4, whereas observations regarding performance and motivation of participants belonging to uninformed and informed group after attempting level 2 are depicted in Figures 5 and 6 respectively. Figures 5 and 6 reveal that after attempting level 2, the performance of the majority of capable participants of uninformed is decreased compared to the participants of the informed group with similar capability scores. Hence, it is evident that the participants have better performance due to motivation. Even participants who scored lesser in capability performed better when motivated. The capability scores of participants belonging to the informed group with low scores in the capability test and their respective increase in performance, when motivated, are presented in Table 5.
Figures 7 and 8 reflect the performance of each group for both levels that show the clear increase in performance of participants who are even less-capable but provided with motivation in the form of challenge (intrinsic) and reputation (extrinsic). These analyses are in favor of hypotheses, that is, less capable crowd worker perform better provided with intrinsic and extrinsic motivation. On the other hand, a decrease in performance is observed in the case of the capable crowd in the uninformed group.
8. Conclusion
This paper explores the relationship of motivation, capability, and performance, from the perspective of crowd workers. In crowdsourcing, the quality of tasks performed by a crowd worker is questionable and there is a lack of understanding about how to (define or) measure capable crowd worker? Moreover, in microtask, it is not clear that which type of motivation (intrinsic or extrinsic) is required to get better performance from a crowd worker in case of low compensation (monetary reward)? That is, how to assess the capability of a crowd worker and whether a less capable crowd worker be motivated, intrinsically or extrinsically (except monetary rewards), to perform better? Theories from job performance, human capability framework, and expectancy theory of motivation are incorporated to understand the construct of capability, motivation, and performance. Using this understanding, formulae to evaluate the capability and motivation of crowd worker are derived, which are then used to evaluate the performance of crowd worker. Capacity as human capital (i.e., knowledge and skill) and task matching are used to evaluate an individual’s capability, which helps to identify less capable and capable crowd worker. Reputation and challenge are used as extrinsic and intrinsic motivation predictors, respectively, while keeping compensation constant.
For the experiment, the crowdsourcing platform “Microworkers” was approached. Participants were informed of the link for the experiment. It was instructed that no collaboration and communication was allowed during the experiment among the participants. This is the reason not to include the motivators of collaboration and communication.
Findings suggest that less capable crowd workers performed better when motivated intrinsically and extrinsically while capable crowd workers underperformed when no intrinsic and extrinsic motivation was provided, keeping the constant monetary reward. The primary contribution of this research is the proposed formula to compute the performance of crowd workers. To the best of our knowledge, this is the first time a formulization is proposed to compute the combined effect of capability and motivation on the performance of crowd workers. It is anticipated that this will benefit the requester to evaluate the performance of a crowd worker before providing him/her the task and also benefit the crowd worker for self-assessment.
The research will benefit the practitioners to design tasks where financial rewards may not be of any significance and identify the capacity crowd by generating simple tests. It will not only help in identifying the capable individual but also provides an insight into her motivation level, which will definitely have a positive influence on the overall quality of the submission. Recently, crowdsourcing has been used as a data collection instrument for the purpose of machine learning and deep learning in the field, such as language translation, audio transcription, and hand-writing analysis. If the data provided by the crowd worker is not valid or has low quality, then it may not be that useful for such avenues. The approach used in this research also provides the measures for capability, motivation in addition to the overall performance of a crowd worker and may contribute to building the model using machine learning and/or deep learning to predict the performance of crowd workers. Secondly, the performance measurement will definitely help solution-seekers/requesters to predict the performance of crowd workers when recruiting them. Another contribution of this research is the quantification of the capability in the crowdsourcing domain. This also will help to identify the capable crowd worker, thereby, providing solution-seekers/requesters to generate a pool of capable crowd workers as was suggested by [29].
Further, in the current situation of the COVID-19 pandemic, where most of the work is performed online, our approach can also be applied to identify capable online workers as well as online learners and help in reducing unemployment.
Data Availability
Access to data is restricted due to confidentiality and ethical issues.
Conflicts of Interest
The authors declare that they have no conflicts of interest.