|
| Author/s and year, country | Aims | Design | Participants | Findings | Limitations (author identified) |
|
| Service area: health |
|
| Baraitser et al. [20], UK | To test the feasibility of professional patients as a tool for sexual health service evaluation | Quantitative and qualitative evaluation of mystery shopping sexual health services | 40 professional patients (PPs): 16 male, 24 female, average age 26.5 (16–50+)
9 white British; 7 not stated, 24 BAME (Black & minority ethnicity) | (i) Using PPs can be a useful method for inspecting sexual health services and providing information to providers with little staff resistance | (i) Ethical issues around staff spending time with PPs rather than genuine patients |
| (ii) Interviews with PPs gave more detailed and specific data than questionnaires | (ii) Possible negative effect of poor healthcare experienced by PPs |
| (iii) PPs adopted different approaches: some tried to see services from the perspective of other patients, while others recorded their own feelings | (iii) Breakdown of trust between health professionals and those conducting the research due to deception involved in mystery- shopping |
|
| Health and Social Care Regulatory Forum [21]; Ireland | To propose a framework for involving service users and the public in the work of health and social care regulatory bodies in Ireland | Overview of current good practice in Ireland UK, Australia and New Zealand, USA, Canada, Sweden, Finland, Denmark, and The Netherlands | N/A | (i) Varying levels of public involvement and empowerment across different states | None mentioned |
| (ii) Framework for public and service user involvement in regulation proposed from level 1 (inform or educate) to level 5 (partner) |
|
| Adams et al. [10]; The Netherlands | To examine the use of “mystery guests” as a method of monitoring quality and safety in elderly care | Qualitative research and policy document analysis and analysis of 38 completed reviews; semistructured interviews | 23 participants including IGZ inspectors, inspection officials in other sectors, regulatory inspectors, experienced mystery guests | (i) Lack of clarity regarding expectations and rationale for mystery guest visits | None detailed |
| (ii) Mystery guests had the potential to get a detailed view of everyday practice |
| (iii) Mystery guests noted “softer” aspects of care, lack of access to back office features |
| (iv) Hard to integrate mystery guests’ findings into final reports |
| Adams et al. [22]; The Netherlands | To examine political expectations for increasing citizen participation in healthcare regulation and explore how these compare to regulators’ expectations and experiences | Qualitative document and web analysis, focus groups, and interviews | Employees of IGZ (Dutch health inspectorate), inspectors from the policy sectors, and organisations such as the Dutch patient and consumer federation (NPCF). 32 participants in focus groups; 19 semistructured (face-to-face) interviews | (i) Barriers to participation included: lack of understanding of different participatory strategies; surface-level involvement; individuals expecting response to personal complaints; lack of respect from care professionals; lack of understanding about role and training required | None detailed |
|
| Boyd et al. [23]; UK | To understand how inspection team members work together to conduct surveys of hospitals, the challenges teams may face, and how these challenges might be addressed | Qualitative study: interviews and survey | Interviews with 18 key national organisation stakeholders, 35 inspection team members, 25 hospital staff
Survey (with free text) of 369 team members from inspections of 19 organisation and 698 managers from inspected organisations | (i) Challenges included: the temporary nature of teams hindering activities; tensions between team members; team leader regarding experts by experience as opinionated; the perception that users are not represented in the final report
(ii) Existing processes of recruitment and selection, training, and preparation, and to some extent leadership, did not lend themselves to addressing the challenges arising from the temporary nature of the teams | Data collection is not designed for temporary teams |
| Leadership insights are drawn from observations only |
| CQC processes changed shortly after the pilot with more robust recruitment and training |
| Better evaluation of team performance advised |
|
| Marsden et al. [24]; UK | To explore what constitutes effective regulation in health and social care | Identified 42 publications via an iterative approach to finding the most appropriate publications | N/A | (i) Inspection and regulation can improve standards, but challenges include: not enough service user participation in the development and delivery of regulatory activities; user value downplayed; disagreement over definitions of quality | Language and practice of regulation varies by context, sector, and country and therefore meaning of terms may differ |
| No exclusion criteria meant papers reviewed with a range of rigour, validity, and generalisability |
| Wiig et al. [25], Norway, England, the Netherlands, and Australia | A systematic analysis of how patient and family involvement in regulation is accomplished across different health systems, mapping user involvement strategies | Multiple case studies | N/A | (i) Benefits included: Improving regulatory work; providing additional information source; fresh perspective; focus on softer aspects of car; legitimising regulation | Involvement methods can be used without being published or identified as regulatory practice |
| (ii) Barriers included: organisational procedures hinder involvement; incorporating user input into reports; user input undervalued; tokenism; lack of user interest; lack of diversity; user professionalization; costs and time; “hard to reach” groups | Hard to determine whether the diversity of methods was due to country characteristics, regulatory regimes, or other factors |
|
| Service area: education |
|
| Hustler and Goodwin [23]; UK | Examining head teachers’ views on lay inspectors within OFSTED inspections | Qualitative research | 20 secondary and 25 primary head teachers | (i) Heads questioned the need for lay assessors and did not take them seriously, concerned with “deprofessionalisation” (lack of experience with children and education) | None noted |
| (ii) Secondary heads regarded assessors as “safe,” examining marginalized areas, not “at the heart” of teaching |
|
| Brown et al. [24]; Ireland | To explore the extent to which efforts to involve parents and students in school self-evaluation (SSE) in Ireland are developing in practice | National online survey of postprimary school principals
Semistructured interviews with a range of stakeholders | 164 principals responded to survey
Interviews with 12 principals and deputy principals, 34 teachers, 20 parents and 43 students | (i) Principals mainly valued parents and teacher involvement in SSE as informants | Schools may have failed to document how policies were developed |
| (ii) Principals felt barriers to involvement included a lack of commitment, knowledge, skills, and experience among parents and students |
| (iii) Staff, parents, and students were concerned with time constraints |
| O’Brien et al. [28]; Ireland | To document an approach to student involvement in school self-evaluation (SSE) in Irish postprimary school | Qualitative case study, interviews | 5 students aged 16-17, 4 members of school staff | (i) Barriers included: Lack of clarity over the role; student concern over speaking openly in front of teachers; more resources needed to engage students as coresearchers than as data sources | None noted |
| (ii) Students felt respected/gained skills but were concerned that the impact of involvement was not clear |
| (iii) Positive impact on wider staff team: raised awareness of SSE |
|
| Service area: social care |
|
| Valios [29]; UK | To comment on the use of lay assessors in social care settings | Online opinion piece (blog): Reviews social care leaders’ opinions of the use of lay assessors in social care settings in the UK | NA | (i) Lay assessors are less intimidating than inspectors and can help gather more evidence of strengths/weaknesses | None noted |
| (ii) Service users can be scared to criticise due to repercussions |
| (iii) Lay assessors not treated as equal participants: not trained, respected, recompensed for time/expenses, tokenistic |
| (iv) Not all service users can communicate their concerns; need for other strategies, e.g., comment cards |
|
| Hibbert [30]; UK | To compile learning from the listening and responding component of social services inspectorate inspections of local authority children’s services involving care-experienced young people in inspection teams | Report from feedback workshops with inspection team members | Listening and responding teams that took part in 19 children’s services inspections in England in 2001
14 adult inspectors from associated voluntary sector organisations and 12 care-experienced young people | (i) Team members felt more training was needed | None noted |
| (ii) Young inspectors found briefing sessions hard to follow |
| (iii) Children and young people receiving services opened up more to young inspectors |
| (iv) Not enough time for discussion and reflection |
| (v) Some young inspectors found experience triggering |
| Commission (NCSC) [31], UK | To clarify characteristics, activities, contributions, and financial costs of lay assessors in social care inspections | Postal survey and qualitative interviews | Survey of 45 NCSC area managers
A national survey of 360 stakeholders—inspectors, lay assessors, and service providers
Interviews with 18 service users (older people and people with learning disabilities)
Feedback from regional workshops | (i) Use of lay assessors by NCSC not routine | Only 63% of NCSC area managers responded to the survey
Lay assessors conducted interviews with services users at the end of the inspection, danger of bias
Information on financial costs was limited |
| (ii) Lay assessors are predominantly white, female, and over 50 |
| (iii) Some mixed views, but in general contribution of lay assessors is seen as positive: they bring independence, impartiality, and informality |
| (iv) More reservations among service providers and inspectors |
| (v) Need for lay assessors to have clear roles, training, ongoing communication, and support |
|
| Wright [32]; UK | To explore the role of lay assessors in the inspection process of care homes in England and Wales | (1) Postal survey about policies/practices regarding lay assessors
(2) Semi-structured interviews with participants in 13 care home inspections (8 care homes for older people and 5 for adults with learning disabilities) | 21 survey respondents, all administrators of local registration and inspection units
73 interview participants: 14 residents, 13 lay assessors (9 women and 4 men), 13 inspectors, 13 care home managers, 20 care staff | (i) Survey showed variation in nature of lay assessor involvement, issue with representativeness | Communication barriers made interviewing people with autism and dementia challenging |
| (ii) Inspectors had little time to talk to residents/staff so valued lay assessors for this role. Lay assessors saw this as a key difference and considered themselves more approachable |
| (iii) Confusion over roles, and difficult to explain to service users |
| (iv)Some lay assessors had done 60 inspections–question over “layness” |
| (v) Variation in who contributed to reports: lay assessors felt they did not get feedback on their input |
| Smith and Hasler [33]; UK | To report on the involvement of people with learning difficulties in the inspection of registered care homes | Descriptive article | 4 people with learning difficulties supported to work with inspectors in 45 registered care homes | (i) Care home residents felt understood | None stated |
| (ii) Experts by experience valued; gained new skills and experience yet frustrated by limited role. Support needed to deal with these emotions |
| (iii) Experts by experience critical of the minimum standards; did not have enough time to get to know people who did not speak |
|
| Weinstein [34]; UK | To compare the process and outcomes of two approaches to engaging mental health service users in the quality assurance (QA) evaluation of a mental health day centre | Document analysis including service user questionnaires collected in two reviews. During the second review service users were involved in an inclusive process in the stakeholder steering group, designed a questionnaire and agreed on an action plan on receipt of the findings | The day centre is attended by 15 clients per day with severe and enduring mental health issues. Overall membership is between 70 and 90. Most members aged 40–60 | (i) Top-down QA inspection had less ownership from service users and staff. Response rate was 28%. Staff was minimally involved; users were only involved via questionnaires and 5 spoke to QA volunteers. Took 6 weeks. The outcome was to report to the board that “services met standards” | The author identified being involved with the project; she could not claim to be objective
Significant time gap between data collection and write-up stages |
| (ii) Second collaborative review: focused on different priorities and developed a new approach to getting users’ views; 73% response rate. Took 5 months |
| (iii) Joint implementation plan developed by staff and service users |
| Simmill-Binning et al. [35]; UK | To examine the reasons for the involvement of lay assessors and perceptions of the effectiveness of lay assessors in specific regulatory functions | Qualitative study: group interviews with assessors and stakeholder organisation, questionnaires for inspectors and lay assessors, follow-up telephone interviews; interviews with lay visitor providers and CSCI’s lay assessor forum members | 45 lay assessors plus prospective lay assessors. 65 questionnaires were sent out. 20 inspectors took part | (i) Lack of agreement and call for clearer guidance on roles and responsibilities | A lack of time and resources limited the authors’ strategy |
| (ii) LAs saw themselves as putting service users at ease, seeing problems otherwise missed, asking important “naïve” questions |
| (iii) Some inspectors said the lay view contradicted their findings due to lack of knowledge. LAs felt they were not heard in the inspection process/reports. Some felt it should be made clear if there were disagreements |
| (iv) Call for training but debate over level and type |
| (v) A need for diversity |
|
| Unwin and Leverett [36]; UK | To evaluate the “having your say” training course to develop peer visitors for older people’s residential care homes | Evaluation of the training programme | Participants’ ages ranged from 61 to 90.3 were residents of older people’s homes; 2 tenants in sheltered housing, remainder lived independently | (i) Some peer visitors used offensive language, raising the issue of what training should be given | None stated |
| (ii) Participants questioned role title. Alternatives were “inspector,” “lay visitor,” “volunteer visitor” |
| (iii) Participants’ mixed professional backgrounds meant some were intimidated by others |
|
| Care Quality Commission [37]; UK | Statement of commitment to involving people who use services in the work of the Care Quality Commission (CQC) | Policy document/charter | Consultation process involving local and national groups (no details available) | (i) Links between involving service users and the promotion of equality, diversity, and human rights | None stated |
| (ii) Compliance with a legal framework (Health and Social Care Act) |
| (iii) Commitment to including experts by experience in inspections as well as a wide variety of involvement activities |
| (iv) Emphasis on partnering with existing groups |
| Scourfield [38]; UK | To explore whether the choice of the word “expert” in the context of CQC lay assessors is appropriate and useful | Qualitative: critical reflection and discussion paper | NA | (i) Conceptual problems of the term “expert by experience” | None noted |
| (ii) Experts by experience felt in a vulnerable position: one to be judged/scrutinised |
| (iii) Author question’s role of experts by experience as appropriate to include service user perspectives |
|
| Care Quality Commission [39]; UK | To capture the views of experts by experience and professional advisors regarding their involvement in inspection. To determine how to improve the process and experience for future inspections | Service evaluation using interviews and a survey | 58 participants inspecting 150 services and 14 inspectors | (i) All groups including inspectors felt the experience was positive, added value to inspections, and went beyond “tokenistic” - brought additional insights to inspections | Low response rate to the survey |
| (ii) Experts by experience reported families “opened up” to them (inspectors agreed) |
| (iii) Participants felt reports accurately reflected their input |
|
| National Development Team for Inclusion (NDTi) [40], UK | To evaluate the experiences of people with learning disabilities, family carers, and professional advisors as members of CQC inspection teams within the learning disability review | Group discussions at evaluation workshops and a questionnaire | 16 self-advocates with learning disabilities
15 family carers
27 professional advisors | (i) Consensus that involvement of self-advocates and family carers added value to inspections | None mentioned |
| (ii) Enabled more in-depth discussions |
| (iii) Limited opportunities to advise services on how to improve their practices |
| (iv) Structure of the feedback form and report format felt to be limiting |
| (v) More preparation is needed, including team building and consideration of the most appropriate team to inspect each service |
| (vi) Mixed views about value of training |
| Care Quality Commission [41]; UK | Details the CQC’s testing of a range of methods for the inspection of home care services | Qualitative interviews and review of inspection types against regulations and key themes | Unclear | (i) Experts by experience felt their background helped them to know when to ask more detailed questions about particular topics | Lots of organisational change impacted on the success of the methodology and slowed down programme; the scale of study stretched resources; home visits and calls were time-consuming |
|
| De Graaff et al. [42]; Netherlands | To explore how to involve citizens to better use clients’ perspectives on quality of care to improve regulatory work in the IGJ (Dutch Health and Youth Care Inspectorate). To evaluate a pilot involving experts by experience in inspections of elderly care homes | Qualitative research: organisational ethnography; document analysis; observations; 41 interviews including experts by experience, project team, inspectors, clients, board members, care home managers, other stakeholders | 7 experts by experience (four men and three women) | (i) Professional epistemology dominated the way care information produced | Specific selection of active clients, and informal caregivers, and focus on reports and interviews rather than observations
Ethical issues (anonymity), and how legal and institutional responsibilities limit participants to explore possibilities
Lack of early involvement of clients in development of themes to use in their interviews |
| (ii) Experts by experience not involved in the design of interviews so views were structurally excluded |
| (iii) Inspectors used experts by experience input to illustrate their own views |
| (iv) Inspectors felt expert by experience information not valuable for safeguarding quality |
|
| Richardson et al. [43]; UK | Inductive analysis of the role of service users and citizens in health and social care regulation. This includes how CQC involved people as experts by experience in inspecting and rating health and social care providers | Qualitative research: 6 geographical case studies including interviews with staff, and document analysis | 61 interviews across acute care, mental health adult social care, and general practice with CQC staff and representatives of groups of service users and citizens, and voluntary sector organisations | (i) Experts by experience better able to elicit experiences from people due to greater insight | Sites were not varied; the study did not explore CQC’s national public engagement work, e.g., state of care reports; interviewing individual members of public/service users could further aid understanding of engagement practices |
| (ii) Barriers included: the current transactional nature of the relationship; professional hierarchies make using lay findings difficult; lived experiences can colour contributions; experience not always relevant to the service; service users do not always want to talk to fellow service users; lack of feedback about inspection outcome |
| Service area: housing |
|
| Robertson [44], UK | To report on new certified training for tenant inspectors | Online opinion article reporting on a group of tenants completing a 14-week tenant inspector course | 19 social housing tenants aged between 21 and 70 were trained | (i) Course trains prospective tenant inspectors: assessing void properties, refurbishments, joining scrutiny committees | None noted |
| (ii) Useful to have a tenant perspective in inspections |
| (iii) Development opportunity for tenant inspectors |
| (iv) Good value for money |
|
| Dolbear [45]; UK | To outline the learning from the project using volunteers as lay assessors to improve the quality of life of older people in care homes | Qualitative case study | Volunteers aged 50+ from the retired senior volunteer’s programme, with personal or professional experience in social care | (i) Managers overall found lay assessor role improves residents’ quality of life (when suggestions were specific and practical) but sometimes their input failed to make changes | None noted |
| (ii) Residents and families happy for independent people to hear views |
| (iii) Challenges included: high staff turnover; establishing trust and respect; communication; not falling into institutional care home culture; keeping volunteers motivated |
|
