Abstract
Public services have increasingly sought to use lay assessors (often known as “experts by experience”) as members of inspection teams in health, social care, and education settings. This involvement has been credited as giving more influence to users over how services ought to run. Yet, little is known about the process or outcome of engaging with lay assessors. We conducted a systematic review to understand the benefits and challenges of involving lay assessors in the inspection of public services. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we searched the literature in English using five bibliographic databases with the date of publication limited to 2000 onward. Across 27 eligible studies, results suggest that including diverse views and perspectives is a strength, yet little consideration is given to issues of “representativeness” of lay assessors or the power differentials within mixed groups that can shape which perspectives gain dominance. Despite a frequent rehearsal of the many benefits of involving lay assessors in inspections as a potential force to drive up the quality of inspections and inspected services, the impact of including lay assessors in inspections in terms of improving services was hard to determine. When designing training for lay assessors, services needed to carefully consider the tension between maintaining the assessor’s “naive eye” versus becoming “professionalised.” It was also apparent that expectations are often not clearly shared over how lay perspectives could be included in final inspection reports, thus risking disengagement. Involving lay assessors is still a fairly novel, yet rich and meaningful way to improve services, yet a lack of clear expectations, and typical exclusion of lay assessors in setting standards for regulation, can still act as barriers to meaningful involvement, preventing lay views from being heard and acted upon.
1. Introduction
In 2001, in this journal, Kerrison and Pollock bemoaned the absence of service users’ voices in the inspection and regulation of public services [1]. Over twenty years later, citizens and people who share key characteristics with actual beneficiaries of services are increasingly invited into processes of quality control undertaken by service providers themselves or external regulatory bodies. This development is part of what has been called the participatory turn in public administration [2] and a wider trend in policy and practice towards coproduction in the delivery and evaluation of services [3]. Coproduction is an umbrella term for involving service users in different roles and processes in the provision and governance of public services, which Loeffler and Bovaird [4, 5] have identified as cocommissioning, codesign, codelivery, and coassessment of services. Of these processes, coassessment has received the least attention from researchers interested in coproduction.
Simply accessing the service user voice—inviting service users to comment on the services they receive—is only one aspect of coassessment. Coassessment can involve citizens in key elements of inspection and regulation of services, working alongside paid assessors to evaluate the delivery of public services. McKenna [6] arranges these elements in a “wheel of coassessment,” an iterative cycle of cofocusing (identifying what needs to be assessed), codirecting (setting quality standards), codetecting (gathering data on providers’ performance), cojudging (deciding how well provision meets agreed benchmarks), and coeffecting (bringing about positive change in services) [6].
These are the sorts of activities, with or without the coelement, that are undertaken by bodies engaged in national and local regulation and inspection of health, education, and social care services across the Economic North, such as the Office for Standards in Education (Ofsted), the Care Quality Commission (CQC) in the UK, the Australian Commission on Safety and Quality in Health Care (ACSQHC), and the Centres for Medicare and Medicaid (CMS) in the US. The expansion of the regulatory landscape has been driven by a range of intersecting factors including increased scrutiny and publicity of adverse events in public services, the proliferation and diversification of private-sector providers, and diminishing trust in professional expertise [7].
The participatory turn in public administration and expanding regulatory regimes is the background for the expansion of roles for nonprofessional citizens in coassessment, working together with service providers involved in self-evaluation, or with established inspection and regulation agencies. Citizen inspectors have been given different names and titles over the years: “lay assessors,” “service users,” “mystery shoppers,” “quality checkers,” and more commonly these days “experts by experience.” These terms suggest different subject positions for citizens to occupy which shape particular relationships between those who commission and provide public services and those who use them.
The layperson exists in binary opposition to the expert and ideally is brought in as a corrective to entrenched professional assumptions. They might or might not have experience of service use but are assumed to possess attributes of common sense and a sense of civic duty, in contrast to the implied vested interests of professionals [8]. The term “service user” was adopted as a neutral label by patient activists, though can be critiqued as reductionist, rendering all diverse and multiple identities that might be claimed by the individual as inconsequential, apart from their status as recipients of care and support with connotations of dependency [9]. Mystery shoppers go “undercover,” sampling the service being evaluated, with the surprise element presumably capturing genuine and unrehearsed service delivery [10]. Originating in the private retail sector, this term suggests a consumerist model of public service delivery [11] where the “customer is always right.”
The role of quality checker has been created in the UK for individuals from service user groups who often experience inequities and discrimination in their use of services including disabled people, people with mental health needs, and carers. Quality checkers are paid by service providers, commissioners, and managers of health and social care services and often focus on how services ensure the inclusion of marginalized groups [12]. Experts by experience can be seen as occupying a complementary position with professionals, having access to specialist knowledge derived from their lived experience of using services. However, the nature of this knowledge, who can lay claim to it and its place in decision-making, is often only vaguely specified [13]. In contrast to professional knowledge which is assumed to be derived from theory and professional practice, the knowledge brought by experts by experience is necessary context-bound and embodied [14], so it can be hard to translate into official discourse, which tends to claim universality and neutrality [15].
The foregoing discussion makes it clear that different terms for citizens engaged in the inspection and regulation of services and the identities these suggest have been shaped by specific institutional contexts. Nevertheless, for the purposes of this review, after some discussion, we have chosen the term “lay assessor.” This term has perhaps fallen somewhat out of fashion since Mordaunt explored it in 1998 [8]. However, the term covers a range of citizens who might or might not have personally experienced conditions or services and who have in common a “lay”/nonprofessional identity.
Our aim, therefore, was to systematically search for and synthesize findings from published accounts of involving lay assessors in the inspection of public services to address the following research question: What are the benefits and challenges in involving lay assessors in the inspection and regulation of public services? This is the first review of its kind and will have the potential to inform future policy and practice in the fields of service inspection and regulation.
2. Methods
2.1. Study Selection
The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [16] drawing on information sources written in English. The review was registered with the PROSPERO international register of systematic reviews of health-related literature (ID CRD42021268134). The review aimed to gain insight into how nonprofessionals are involved in the inspection or checking of health, social care, and education services anywhere in the world. To reflect more contemporary practices in inspection and regulation, we included information sources from 2000 onward, involving adults, children, and young people. We excluded information sources that involved citizens inspecting or regulating any other services (e.g., retail, hospitality, and IT) or taking part in other activities, for example, policy formation, research, service codesign, and general reflections or feedback on services. We included peer-reviewed articles, “grey literature” and reports, empirical articles, think pieces, and editorials, and both qualitative and quantitative studies relevant to the research questions.
2.2. Information Sources
We searched the following bibliographic databases covering research and practice in health, social care, and education: Medline, PsychINFO, Social Policy and Practice, Social Services Abstracts, and ERIC. The following grey literature sources were also searched using the same key words: Social Care Institute for Excellence (SCIE), Community Care Journal, and the King’s Fund, with all results being reviewed. The process of searching was iterative, with a final update run in February 2023. We also included hand-searched articles meeting the inclusion criteria identified, carrying out forward and backward hand searching using reference lists and citations from relevant sources. In eight instances, we were unable to access the literature, in some cases because of websites being removed. Although we contacted the authors, we were unsuccessful in obtaining copies.
2.3. Search Strategy
To support the identification of search terms, we used the PICO [17] framework (population, intervention, comparison, and outcome) designed for healthcare research (Table 1). We chose to omit the comparison element as not relevant to this review (for discussion see [18]).
2.4. Selection Process
We used Covidence [19] to manage all information sources to ensure ease of communication between research team members. One reviewer (KB) screened imported information sources’ titles and abstracts for eligibility. The full text of all included information sources was then assessed independently by DC. Any conflicts of opinion over the inclusion of information sources were discussed and resolved by reference to the research questions. The selection process is represented in Figure 1.
2.5. Data Extraction and Analysis
We jointly devised a data extraction form and independently extracted key data (authors, dates, terminology, aims, design, participants, design, findings, recommendations, and limitations) from each of the information sources (see Table 2). We used thematic synthesis [46] to synthesize findings, involving line-by-line coding using qualitative data analysis software NVivo 1.6. (QSR, 2022) and generation of descriptive and analytic themes.
2.6. Quality Appraisal
The CASP [47] for qualitative studies was used to evaluate the quality of the information sources with an empirical research component that had been peer reviewed (see Table 3). Overall, there was great variation in the clarity, detail, and adherence to research quality standards , particularly as some of the sources were not written for academic audiences. Generally, detail was lacking regarding methods of recruitment, methods of data analysis and synthesis, and consideration of the positioning of the authors, and researcher reflexivity was largely absent. Nevertheless, no sources were rejected because of poor quality as all contained insights relevant to the review research question from a valuable range of perspectives. In mixed methods studies, we evaluated only the qualitative elements due to our focus on the rich detail of involving lay assessors in inspection and regulation, and the goal of understanding different participants’ perspectives.
3. Results
3.1. Description of Papers Included in the Review
Twenty-seven papers were included in the final data extraction and analysis.
The majority (19) of papers were from the UK, 4 were from the Netherlands, and 3 were from Ireland. Two studies had contributors from many countries.
Most of the papers reported or commented on national, centrally managed inspection programmes, but we also included reports of single services undergoing self-evaluation or quality improvement where lay assessors helped to conduct these processes (see Table 2).
There were two overarching themes suggested by our research question, namely, the benefits and challenges of involving lay assessors in inspection and regulation. We added a third theme generated through inductive coding—Who are the Lay Assessors?
3.2. Theme 1
Who are the Lay assessors?
Here, we consider and critically review findings relating to the reported identities of the lay assessors and the methods used for their recruitment. The findings in this section reflect the ambiguity and lack of clarity regarding the key attributes of lay assessors, discussed at some length by Scourfield [38]. We found considerable variation in our reviewed documents relating to the approaches adopted for the recruitment of lay assessors and lack of detail about who exactly was included in the inspection programmes described. We suggest a framework for understanding assessors’ identities across intersecting dimension of “layness” and “familiarity” with compromises evident where those with a convincingly “lay” or nonprofessional/community member identity might present as less convincingly “familiar” with the particular setting being evaluated and vice versa. A cross-cutting dimension relates to how “representative” the lay assessor might be of the wider group with whom they share an identity (Figure 2).
In some studies, both nonprofessional identity and lived experience as someone who has drawn on the service under scrutiny were prioritized. Hibbert’s [30] report of local authority inspections of services for children and young people involved young people with care experiences themselves. Similarly, in Robertson [44], social housing lay inspections were conducted by tenants. Lay assessors saw their shared identity with service users as a strength, though Hibbert [30] mentioned that inspecting can be emotionally challenging for lay inspectors who have had bad experiences of care themselves. However, insisting on lived experience can very much narrow the field of potential lay assessors. It can be a challenge for instance to find individuals with lived experience of high support environments who are also able to take part in the inspection process [42]. Using the “mystery shopper” approach extends opportunities for evaluation to people who appear identical to “real” service users, though not necessarily having previous personal experience of service use [10, 20].
In a number of other programmes, the lay assessor group was heterogenous in terms of familiarity with the setting under review, including those who, because of personal characteristics, might be more likely to access those services in the future (for instance, older adults in Unwin and Leverett [36] or young LGBT people in Baraitser et al. [20]). Mixed groups might also include relatives of those who call on services and members of the “general public.” Individuals whose familiarity with a service setting derives from their professional experience within these settings might also join with lay assessors in mixed groups [28].
Unwin and Leverett [36] celebrate this heterogeneity as a strength, in that diverse views and perspectives are included. However, little consideration is given to the power differentials that can arise within mixed groups that can result in some perspectives gaining dominance. This may be a particular concern when those from a lived experience background are in a disempowered minority [36, 40, 42, 45]. However, the respective numbers of lay assessors from different backgrounds were not always specified in the programmes reviewed here, suggesting a tendency to overlook important dynamics within the lay assessor groups.
The degree to which the lay assessors reflected key characteristics of the wider group they were meant to represent was linked to the recruitment methods used. In general, detail on recruitment methods was not always given and could appear quite opaque, even to the inspection teams involved [23]. In some cases, an open recruitment call was implied [45]. In other inspection programmes, lay assessors were recruited through contacts with existing networks or organisations providing services to user groups [30, 31, 36, 40]. In the school self-evaluation projects described by O’Brien et al. [28], students were able to self-nominate. These sorts of recruitment methods can mean that the more confident and articulate individuals take up lay assessor roles, or those with a particular “axe to grind” [23], or whose experience of services has been shaped through contact with a specific organisation [31, 43].
There were examples of recruitment methods that offered more equitable access to lay assessor roles. Weinstein [34] described how service user representatives were elected by others using the service. The tenant inspectors in Robertson [44] and the young inspectors described by Hibbert [30] underwent a process of shortlisting and interviews.
As some lay assessor roles are voluntary and unpaid [28, 35, 36, 42, 45, 31], this may exclude those with lower incomes. Simmill-Binning et al. [35] noted that the majority of the lay assessors in their study were white British men with professional backgrounds over the age of 60, and the National Care Standards Commission (NCSC) [31] report identified that the majority of their lay inspectors were from those same social and age groups, though most in their survey were women. Information on the ethnic and cultural composition of lay assessors and the extent to which this reflects local populations was scant.
Wright [32] reflects that the make-up of lay assessor teams raised questions in their report about how representative lay assessors might be of the service users who use the services under inspection, leading to situations where lay assessors’ background and experience may not be relevant to the service they are inspecting [43]. However, this reservation was not widely shared within the documents reviewed, and some respondents in the NCSC report [31] argued that lay inspectors’ lack of experience in the care setting and statutory and legislative restrictions was actually a plus. There is perhaps an assumption that the defining attribute of lay assessors—namely that they are not regular staff members of inspection bodies—is enough to ensure that they are fit for the job.
3.3. Theme 2
Genuine benefits or “window dressing”?
This theme addressed the kinds of justifications given for involving lay assessors and the added value that they brought to inspection and regulation and the mechanisms through which they achieved this. We found that potential tangible benefits to end users themselves were rarely evidenced, leading to concerns that the presence of lay assessors was more a form of “window dressing” for the activities of inspectors and regulatory bodies.
3.3.1. Instrumental and Democratic Benefits
We synthesized the benefits claimed to accrue from involving lay assessors into two categories: instrumental benefits and democratic benefits [6]. Instrumental benefits were those that were stated to improve the quality of inspections and minimize potential problems associated with inspection and regulation. These dominated the documents reviewed. Democratic benefits were those that promoted the empowerment of citizens within public service governance.
Under instrumental benefits, lay assessors may be seen as adding to the inspectorate workforce and compensating for the lack of local authority staff available to check standards of care [45]. “Efficiency” and cost saving are also mentioned; Adams et al. [22] state that the “mystery guest” model of assessing publicly funded care gained support because it was assumed to cost less and generate less paperwork than using professional inspectors only. Whether these cost savings occur in practice is a moot point; others were concerned about additional costs involved in training and supporting lay assessors [31, 32].
Involving lay assessors is seen as a way of adding legitimacy and credibility to inspections, at a time when public trust in the capacity of regulatory agencies to maintain standards has been undermined by scandals of poor care [21, 43, 45, 40, 31]. This justification appears to address one of the pitfalls that might befall professional inspectors [48], namely, “regulatory capture” or an over-identification of the values and practices of the organisations being inspected [24]. However, in some studies [32, 36, 45], the care home “lay” inspectors often had professional experience of working in or with care homes rather than the lived experience of receiving care themselves.
While the degree of “independence” of some of the lay assessors was questioned by some, their role was nonetheless characterized as “independent” from professional self-interest [21, 26, 31, 35, 45]. The perceived failure of inspection and regulatory bodies to identify inadequate care and hold providers to account has also been attributed to a preoccupation with procedural matters, targets and performance indicators, and paperwork [40], rather than quality of care and enhancement of quality of life of service users [43]. Hustler and Goodwin [26] suggest that coming from outside the education establishment, school lay inspectors were seen to have the capacity to introduce “some grit into the system” and constitute a constructively disruptive presence [43]. The NCSC report [31], Unwin and Leverett [36], and Dolbear [45] similarly contend that lay assessors bring a “common-sense” perspective untainted by the culture of safety and risk avoidance that predominates in many care settings and consequently have higher expectations about residents’ quality of life.
The papers reviewed suggest different mechanisms through which lay assessors are able to elicit information that inspectors might miss: they are seen as less intimidating by service users [10, 29, 32] and are perceived as more empathic [31] and capable of understanding the service users’ perspectives [29, 33, 35, 41]; they have more time, or perhaps, they make more time to speak directly to service users [31, 32, 36, 38]; they are able to pick up on “intangible” aspects of the settings, its atmosphere and “feeling” [10, 29, 32], and the body language and emotional communication of service users [40]. There are various descriptions of the communication style that lay assessors may bring to their interactions with service users, which is described as “informal” [31, 35, 49] and “direct” and not afraid to ask “naïve questions” [40].
As a result of the attributes and skills lay assessors are seen to bring to inspections, authors of the reviewed documents describe how service users were more able to “open up” to them and disclose aspects of care they were not happy about [30, 34, 35, 41, 40, 31]. It is also possible that lay assessors are able to put staff at their ease so that they are able to observe more naturalistic interactions between staff and service users [32, 38]. The mystery shopper approach [10, 20] aims to ensure that staff are observed acting in an unguarded manner. A key justification, therefore, for including lay assessors is that overall the inspection gains a fuller and more accurate picture of the setting with lay assessors being well placed to find out what is “really happening” in everyday practice [10]. On the other hand, there were challenges to the assumption that service users will necessarily prefer to speak to a lay assessor [31], with a quote from one inspector stating service users “want to talk to a proper inspector” [43]. However, having meaningful conversations with service users takes time, and some lay assessors noted that the limited time available for inspections made it difficult for them to engage effectively with service users, particularly those with communication support needs [32, 33, 31].
Democratic benefits, relating to upholding human rights through citizen involvement and transparency of regulatory systems, were less prominent, although they were evident in the stated aspirations of regulatory authorities [21, 37, 31]. The authors of research into school self-evaluation emphasized how involving students promotes democratic forms of school governance and ways of learning [27, 28] while others argue that participatory structures such as regulatory systems that include lay assessors enable more democratic decision-making [22].
3.3.2. Who Benefits?
The potential benefits of lay inspections are reported at a high level of abstraction, and one might wonder how such impacts would be evaluated. Evidence that lay assessors had indeed improved the lives of users was presented only impressionistically, despite awareness among some of the authors that evaluation of lay assessor involvement was important [21]. Weinstein’s [34] study was unusual in this respect by comparing quality assurance processes in a mental health centre that took place at different times, first professional-led and then user-led. The author found that service users were more responsive within the user-led process and felt the areas of inquiry were more relevant.
Determining the impact of involving lay assessors in improving care presents considerable challenges. The role of regulation and inspection in driving up standards in care is itself complex and somewhat contested, as evidenced in Marsden et al.’s [24] review. Moreover, regulation happens within a complex system involving multiple other stakeholders and institutional contexts that also constrain the capacity of regulation to be effective. In light of this complexity, assertions in some of the reviewed documents that involving lay assessors will lead in a linear fashion to improved quality of service provision can seem somewhat simplistic. Indeed, the lay assessors, who took part in the Learning Disability Review following the Winterbourne View scandal in the UK, themselves expressed fears that their involvement constituted “window dressing” and was more about improving the reputation of the regulator than bringing about ongoing service improvement [40]. Some of the professional inspectors surveyed by NCSC [31], and Simmil-Billing et al. [35] also cast doubt on the impact of involving lay assessors for service users with some claims that their involvement even obstructed the inspection process. Moreover, a criticism of the mystery shopping approach is that it takes staff time and resources away from “real” service users [20].
Whereas the impact of involving lay assessors on service quality may be hard to determine, participating in inspection activities was found in some studies to have a more immediate positive effect on the lay assessors themselves. The tenant inspectors described by Robertson [44] had the opportunity to gain a formal qualification as preparation for their role. Similarly, the students in O’Brien et al.’s [28] study saw opportunities to improve their academic skills and future career opportunities. Being a lay assessor can provide employment opportunities and income for individuals who often experience exclusion from the job market [33] and payment was a motivation for some [23].
As well as gaining these tangible rewards, according to the National Development Team for Inclusion (NDTi)'s evaluation of inspections of learning disability services in England, lay assessors increased their personal knowledge of services [40]. Lay assessors also developed communication skills [33] and leadership capabilities, “gaining kudos” through being part of an inspection team [40] and using this to take more effective action in their local services and were also positioned as role models for the service users they met during inspections [30, 38]. Lay assessors in several programmes mentioned the intrinsic satisfaction of feeling that they were doing something to help other service users [28, 32, 33, 35, 36, 42, 40].
There were on the other hand some suggestions that involvement in inspections can expose inspectors to negative experiences: feeling undervalued when treated with less respect or appreciation than professional inspectors [29, 30], foregoing recompense for their time and out-of-pocket expenses, and feeling emotionally overwhelmed [30].
3.4. Theme 3
Challenges
3.4.1. Unclear Roles and Responsibilities
A problem raised in the papers reviewed here and across all sectors was the lack of clarity over the roles and responsibilities of lay assessors [10, 23, 28, 32, 34–36, 38, 42]. This resulted in confusion over what lay assessors bring to the inspection process.
Simmill-Binning et al. [35] found that there were differences of opinion “between inspectors and assessors but also amongst the individuals involved” about what lay assessors should do. Notably, lay assessors felt their contribution to be more significant than the inspectors did, and lay assessors also aspired to expand their involvement further. This lack of clarity regarding roles may be a challenge for inspection teams more generally, who can also struggle with leadership issues. Boyd et al. [23] consider how role confusion may be due to the inspection team spanning organisational boundaries, so team leadership may be ambiguous. They suggest a solution might be to facilitate shared leadership in which all inspection team members can regard themselves as coleaders, yet acknowledge that the topic of leadership within inspection teams requires further research. Similarly, Wright [32] and the NCSC report [31] found that communicating their role to care home residents was also a difficulty for lay assessors, as they try to distance themselves from the role of inspector and align themselves with the citizen, thus having to navigate a liminal space, something Scourfield describes as being in “categorical limbo” (2010: 1899).
Confusion around the involvement of lay assessors in regulation and inspection was identified at a political and policy level, too. For example, Adams et al. [22] note a “blatant lack of clarity regarding the difference between unannounced inspections conducted by inspectors and use of mystery guests as a new review instrument.” This “fuzziness” over the role perhaps relates to a lack of consensus over exactly what it is that lay assessors bring to the table, despite the sorts of justifications for involvement described in Theme 2. Not only does the lack of clarity result in confusion at the policy level, create tensions within inspection teams and during the inspection process, it also creates difficulties when reviewing and researching studies of this nature.
3.4.2. Maintaining “the Naive Eye” vs Professionalization
Several studies in this review [32, 35, 38, 42, 49] highlighted the problem of the “quasiprofessionalisation” [38] of lay assessors. In other words, the more they are involved, the less “lay” they become, as they learn the ropes of inspection and regulation.
Simmill-Binning et al. [35] report that many lay assessors in social care inspection deemed it important to maintain the “naïve eye.” Lay assessors in elderly care homes similarly expressed a desire to not become over-professionalised [42]. As Wright [32] notes, some lay assessors in care homes had conducted around 60 inspections, raising “the interesting issue of what it means to be lay.” Baraitser et al. [20] also categorise this as a potential disadvantage: “the more experienced the professional patients become, the less like normal users they are.”
Simmill-Binning et al. [35] also report that although they did not regard the role of lay assessors as “assistant or quasi-inspectors,” inspectors did, however, perceive that a lack of training of lay assessors impacted the quality of the checking, as they lacked knowledge of expected standards. The difficulty evidenced in these studies is how to maintain the “naïve eye” and benefit from the unique experiences and approach that lay assessors bring, while ensuring they remain objective and act professionally/respectfully.
This calls into question the level and type of training that is provided. As Wiig et al. [49] state, “such training may lead to the professionalization of patient input, which can distance the participants from their experiences as patients.” This issue was explored by Unwin and Leverett [36] evaluating the training of older people to become “peer visitors” in care homes. The training need arose after some trainees reportedly used language that was considered offensive, thus justifying the need for professional acculturation into professional terminology and etiquette. Indeed, Wright [32] reports just this. When inspectors were asked to recall negative experiences of lay assessors, they reported lay assessors’ inappropriate behaviour, which resulted in them not being asked back for further inspections. Hustler and Goodwin [26] also reported unprofessional conduct from lay assessors, for example, when they seemed aggressive or tactless. This indicates that there are perhaps topics of lay assessor training (e.g., around language, diversity, and inclusion) which are essential for lay assessors to participate respectfully with service users but would be unlikely to affect their “naïve eye.” This insight relates to the next challenge; how to ensure the “layness” of lay assessor’s knowledge is valued, or, as Scourfield asks, “are experts by experience valued for their subjectivity or their objectivity?” (2010: 1901).
3.4.3. Including Lay Perspectives in Final Inspection Reports
Various studies in this review discuss significant difficulties around incorporating user knowledge and findings in inspection reports [10, 24, 27, 29, 33, 35, 42, 43]. Two broad areas of difficulty regarding lay information emerged under this theme: first, a perception that this knowledge lacked legitimacy, and second, that it failed to fit institutional information structures.
The debate over whose experience and knowledge define “good care” or “good quality” was raised in various studies under review (for example, [10, 24–27, 33, 34]). Although policy discourse focuses on the importance of including user’s voice [22, 27] and the “naïve eye,” professional epistemologies dominated inspection reports in these studies, with lay assessor input often being undervalued [10, 22, 24, 27, 42].
Wiig et al. [49] found that patient and family input was difficult to incorporate because regulators perceived their knowledge to lack legitimacy. In other words, their knowledge was deemed to have less “weight” than that of inspectors, which Wiig et al. refer to as “epistemic injustice” [49]. Inspectors and lay assessors may have differing understandings of what constitutes good quality services [24], and De Graaff et al. [42] posit that the lay assessors in their study had to give up their personal definition of “quality” in this setting for their views to be considered legitimate by inspection staff. In situations where inspectors and lay assessors disagree in their evaluation of services seem to adopt two strategies regarding reporting: either to give prominence to inspectors’ findings or present the lay assessors’ views separately.
The foregrounding of inspector knowledge was evident in a number of studies and individual inspectors were found to wield a significant amount of control over how individual assessor’s findings were incorporated in final reports. Adams et al. [10] and De Graaff et al. [42] conclude that lay assessor findings were largely used to illustrate issues inspectors were raising, rather than being presented as legitimate substantive findings. Lay assessors described by Boyd et al. [23] reported that their perspectives on hospital patients’ situations often did not coincide with the clinicians’ perspective, with the outcome of their inspection remarks being notably weakened in the final report. Thus, lay assessors thus may find their contribution restricted in the aspect of coassessment that McKenna [6] identifies as cojudging (deciding how well provision meets agreed benchmarks).
The alternative strategy of presenting inspectors’ and lay assessors’ views separately in final reports (e.g., [33]) creates greater epistemic parity between the professional inspector and lay assessor conclusions and provides a clear demarcation of areas of agreement and disagreement within the inspection team. However, findings presented in this way run the risk of confusing mixed messages being picked up by regulated organisations [24, 35].
There was also an issue with the reporting format used by lay assessors, which, as far as some the professional inspectors was concerned, was hard to integrate into the structure of the final report. Wright [32] describes how there was considerable variation in how lay assessors communicated their views, with only some lay assessors using a template, and others using a narrative approach which might be edited for inclusion in a final report. Adams et al. [10] also suggest that the way information was collected by mystery guests in elderly care can be a significant challenge. They note that it was difficult for regulators to incorporate lay assessors’ insights, as their reporting style and focal point varied greatly and did not necessarily align with the regulator’s standard reporting practice. This may result in lay assessors being unable to “fit” their findings into the institutional information framework, and inspectors rejecting this information as lacking relevance [27, 42, 43]. Lay assessors in some studies expressed frustration that the data collection instrument confined responses to within only a few themes defined by the regulators [42, 40]. Inspectors also reported difficulties in incorporating assessor findings with other data, categorising it as “anecdotal” [42] when it could not be included in the report.
Structural and inspection design problems were also highlighted in incorporating lay views into reports. Lay assessors are infrequently involved in aspects of coassessment that McKenna [6] labels cofocussing (identifying what needs to be assessed) and codirecting (setting quality standards). Weinstein’s [34] study was distinct in this respect, as service users set the evaluation goals in a steering group, suggesting strategies for handing authority to lay assessors in these processes.
The transitory nature of some inspection teams was also highlighted as a practical barrier to incorporating user knowledge. For example, Boyd et al. [23] describe how lay assessors had little involvement in contributing directly to hospital evaluation reports, which may be written by inspectors over several weeks, well after the team has been disbanded and the involvement of the lay assessors has ended. Consequently, limited feedback is available to lay assessors regarding how their input had shaped the final report and any changes that it had brought about [29]. Lay assessors in the NCSC report [31], Simmill-Binning et al. [35], and NDTi [40] studies felt unsure as to whether their views had been acted upon at all. This truncated experience can be unsatisfactory [43], relating once again to the need for a clearer understanding of roles and responsibilities by all involved. These findings suggest that lay assessors have limited opportunities for aspects of coassessment that McKenna [6] calls coeffecting (bringing about positive change in services).
4. Discussion
This review has demonstrated that a great variety of involvement of lay assessors is evident in coassessment across the three fields of health, social care, and education, though we found that overall lay assessors were recruited to engage in codetecting (gathering data on providers’ performance), rather than other dimensions of coassessment [6]. In this discussion, we consider how far this lay involvement has succeeded in challenging services to improve quality. We then distil three key interrelated issues that inspectorates or organisations seeking to involve users could address to ensure more meaningful coassessment.
There appear to be several benefits accrued from involving lay inspectors. It can add legitimacy and constructive disruption and embed higher expectations about users’ quality of life. Lay assessors can also elicit information that inspectors might miss and are seen as less intimidating by service users. These suggest that improvements to the overall quality of the inspection process can be achieved through the meaningful involvement of lay assessors. However, it was less clear how tangible the final impact was on service users or inspectors themselves.
There was also some evidence that suggests that lay assessors widen the resource base of inspectorates (at a time of constraints in public services). However, a note of caution should be made here. We identified that the relationship between the identities of lay assessors and that of service users was not well delineated, raising questions about the representativeness and suitability of some lay assessors, and how well they reflect the diversity and characteristics of service users. Furthermore, in several of the studies, it was revealed how coassessment requires significantly more resources and time than commonly appreciated, and in some cases, lay assessors felt shortchanged. The time commitment of lay assessors needs to be considered, to avoid the risk of increased burden [27].
To maximise the benefits, inspectorates need to be mindful of some of the challenges facing lay involvement. First, unclear roles and responsibilities are one of the primary inhibiting factors to meaningful coassessment. There are repeated accounts of a lack of clarity over terminology and roles, and how coassessors are recognised [20, 35]. Sometimes, this confusion was driven by an inability or unwillingness to clarify roles among the inspectors. For social care inspections, the confusion was sometimes felt by other service users [32]. This lack of clarity undermined efforts by lay assessors to understand their expectations of them and the scope and limits of their role [22]. At times, assessors felt valued but were frustrated by how limited their role was [33]. Coproduction in the design and the assessment process can help mediate this concern, and from here, it should be considered at the point of recruitment [35].
Second, there is often a misfiring of expectations over the degree of professionalization required. Losing the authenticity of a lay assessor’s “voice” and maintaining the “naive eye” is a justified concern. “Regulatory capture” undermines the independence of lay inspections. Protecting their authenticity can include ensuring their involvement is ringfenced in terms of duration and number of inspections, although this may mean a shrinking pool of potential lay assessors who share important characteristics with service users. While training of lay assessors is justified around language, diversity, and inclusion, most authors felt that the naïve eye was worth safeguarding.
A third, more difficult issue is the perceived divide between professional versus lay epistemologies. In some cases, this arose from a perceived lack of legitimacy among “professional” inspectors of lay knowledge; opportunities for embedding potentially valuable lay feedback are sometimes overlooked and dismissed as “anecdotal.” Failure to incorporate lay feedback suggests that institutional processes and forms are framed in a way that places lay and professional inspectors in oppositional camps. This relational framing relates to a wider challenge of ensuring the intrinsic (and assumed) power imbalance between professional and lay assessors is constructively challenged to maintain trust in the process and in each other. Inspectorates could reflect on the theme of the legitimacy of knowledge and the institutional structures that reproduce this hierarchy. Failing to involve lay assessors early in developing the inspection process and design of agreed standards, benchmarks, and inspection materials (e.g., questions to use in their interviews) further compounds this and suggests an unwillingness to adapt.
We recognize this review has its limitations. We restricted our selection of sources to those published in English meaning that some relevant articles and reports written in other languages may have been omitted, including those evident in included articles reference lists. We also encountered difficulties locating documents published online only since 2000 by now defunct organisations, such as the UK Commission for Social Care Inspection that was replaced by the CQC. The review also identified gaps in the evidence that require further research. Observational and ethnographic research would throw light on how lay assessors interact with professional inspectors and people using services and how issues of authority, expertise, and voice are negotiated. Research also needs to establish a clearer relationship between the involvement of lay assessors and improved outcomes for people using services.
5. Conclusions
We noted that over the last 20 years, including citizens in inspection teams has increasingly been seen as self-evidently beneficial by all stakeholders, following consistent pressure from service users, policy-makers, commissioners, and inspectorates for services to become more accountable to users. There are important lessons that this paper reveals that can help public bodies, commissioners, and inspectorates to avoid common pitfalls that prevent the lay assessors’ voice from being incorporated, to ensure a more accountable and empowering inspection process.
Returning to Kerrison and Pollock [1], it is heartening to see evidence that service users’ voices are becoming more present in the inspection and regulation of public services. However, issues remain around role ambiguity, levels of training, and legitimacy of knowledge. These will not be solved until there is a willingness to open a space for dialogue over what it is that lay assessors bring to the inspection report and regulation. The literature is lacking on this topic, and future research is needed as to what aspects of training for lay assessors can be given before their “lay” status becomes affected.
Additional Points
What is Known about This Topic? (i) There is consensus across public services that the views of citizens and people who rely on services should be part of the quality control of services. (ii) Regulatory bodies in health, social care, and education involve “lay assessors” in inspecting and evaluating services. What This Paper Adds? (i) Lay assessors are seen to add credibility, an independent perspective, and a layer of public accountability to inspections of public services though their specific roles, responsibilities, and the impact they bring to inspection activities are not always clear and explicit. (ii) Issues of power dynamics within inspection teams can be overlooked, and lay assessors may find their views have not been effectively integrated into inspection reports. (iii) The unique perspectives of lay assessors can add value to inspection of services, but their participation needs to be facilitated at every stage of the inspection process and their views meaningfully represented and acted on in order to avoid tokenism.
Disclosure
The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was funded by the National Institute for Health and Care Research (NIHR) and School for Social Care Research (SSCR) (Grant P160). Open access funding was enabled and organized by JISC.