Earlier, I posted an article introducing testing. To summarize it, testing is an essential process for ensuring quality, identifying defects, validating functionality, and improving current products/services or creating new ones. Testing also serves as a crucial tool in the development lifecycle, providing insights into the performance, reliability, and usability of software or systems. By systematically evaluating various aspects, testing helps mitigate risks, enhance user experience, and meet business objectives.

In this blog, we will discuss another commonly used testing methodology: Critical Incident Technique (CIT). We’ll explore its principles, applications, and benefits. If you prefer to read it in another language, simply click on the flag below this blog, and the text will be translated into your preferred language.

Overview of the Critical Incident Technique (CIT)

The Critical Incident Technique (CIT) is a qualitative research approach that collects direct evidence of human behavior that is crucial, influential, or decisive with the aim to evaluate, regardless of whether the result of the scenario and evaluation is positive or negative. Individuals are frequently asked to provide extensive details on important events or experiences, known as “critical incidents,” which represent situations where something significant occurs different from the norm or expected outcome. For an incident to be “critical,” it needs to happen in a situation where the observer can clearly understand the purpose or intent of the action, and the outcomes are clear enough to leave little doubt about its effects (Butterfield et al., 2005). These incidents also need to have a substantial impact on the individual or the situation being studied, often serving as turning points that trigger unanticipated feelings, ideas, or responses and offer insightful observations about human behavior and decision-making.

The goal of CIT is to develop scenarios that include both positive and negative important events, explain why each incident occurred, and provide recommendations for improving future results (Hanington & Martin, 2012). All observations are also often tracked and used to solve practical problems and develop broad psychological principles after completion (Flanagan, 1954).

So, to simplify: the essence of CIT is the exploration of incidents that are critical not because they occur frequently, but because they significantly impact outcomes.

CIT is a highly flexible and versatile approach applicable in various fields, including education, healthcare, and organizational studies, designed to help interviewees retrieve and report more comprehensive and accurate information about witnessed events or personal experiences.

Benefits of CIT

One of the main benefits of CIT is its ability to provide a detailed, comprehensive understanding of significant events and their impact from the perspective of those directly involved (Nuzzaci, 2023). The open-ended nature of CIT allows participants to share their experiences in their own words, providing rich, contextual data that can be used to identify patterns, themes, and areas for improvement (Gremler, 2004). Additionally, CIT is a flexible method that can be adapted to various research contexts and can be used to study both positive and negative incidents, making it a valuable tool for understanding complex phenomena (Butterfield et al., 2005). By focusing on specific, concrete events, CIT helps to minimize recall bias and provides a more objective, reliable account of participants’ experiences compared to other self-report methods (Flanagan, 1954).

According to Hanington and Martin (2012), CIT will also help you identify the incident cause, user actions, user sentiment, incident outcome, and ideal outcome.

The incident cause refers to the underlying factors or circumstances that led to the occurrence of the critical incident under investigation. Understanding the cause is crucial for identifying areas for improvement or change (Butterfield et al., 2005). It will also provide valuable insights into how the incident unfolded and what role the user played in the sequence of events (Landers, 2014).
User actions refer to the specific behaviors, decisions, or responses exhibited by the individuals involved in the critical incident. These actions can be positive or negative, effective or ineffective, and provide insights into the thought processes and decision-making of the individuals (Flanagan, 1954).
User sentiment encompasses the emotions, attitudes, and perceptions of the individuals involved in the critical incident. It reflects their subjective experiences, feelings, and reactions to the event and its outcomes (Butterfield et al., 2005).It can range from positive to negative sentiments and can be influenced by various factors, such as the severity of the incident, the user’s personal experiences, and the perceived level of support or resolution provided (Pang & Lee, 2008).
The incident outcome refers to the actual consequences or results that occurred as a result of the critical incident. It represents the tangible or observable effects of the incident, which can be positive or negative (Flanagan, 1954).
The ideal outcome is the desired or optimal result that should have occurred in the critical incident. It represents the goal or standard against which the actual outcome is compared, and it helps identify areas for improvement or change (Butterfield et al., 2005). It typically involves resolving the issue effectively, minimizing negative impacts, and implementing measures to prevent similar incidents from occurring in the future (Dekker, 2011).

Hanington and Martin (2012) outline also that CIT aims to create scenarios covering positive and negative critical incidents, provide explanations, and suggest improvements. Teams can prioritize recommendations and use additional research to understand the impact on user behavior.

Types of elements that can be tested using CIT

CIT can be used to test a wide range of elements related to services, products, and user experiences. Some examples include:

User interactions with software interfaces, websites, or mobile apps (Gremler, 2004)
Customer service encounters in various industries, such as healthcare, retail, or hospitality (Bitner et al., 1990)
Product usability and user experience with physical goods (Edvardsson & Roos, 2001)
Employee performance and critical job requirements in different occupational settings (Flanagan, 1954)
Critical safety incidents in high-risk industries, such as aviation or nuclear power (Butterfield et al., 2005)

Assessing feasibility, desirability, and viability using CIT

CIT can be used to assess the feasibility, desirability, and viability of requirements by collecting and analyzing user feedback on specific incidents related to these criteria. For example, by asking users to describe critical incidents related to the ease of use or usefulness of a product feature, researchers can gather insights into the feasibility and desirability of that requirement. Similarly, by examining critical incidents related to the value or benefits derived from a service, researchers can assess the viability of the service offering. The benefits of evaluating these criteria through CIT include:

Obtaining detailed, context-rich data that can inform design and development decisions
Identifying potential barriers or facilitators to adoption and use
Prioritizing requirements based on their impact on user experiences and outcomes
Validating assumptions and gathering evidence to support product or service improvements

Limitations of CIT

While CIT offers several benefits, it also has some limitations. One limitation is that CIT relies on participants’ ability to recall and articulate specific events, which may be subject to memory biases or inaccuracies (Flanagan, 1954). Additionally, CIT data can be time-consuming to collect and analyze, as it often involves conducting in-depth interviews and coding large amounts of qualitative data (Butterfield et al., 2005). CIT also does not provide a representative sample of all possible incidents, as it focuses only on those events deemed critical by participants (Gremler, 2004). Finally, CIT findings may be specific to the context in which the data was collected and may not generalize to other settings or populations (Edvardsson & Roos, 2001).

Preparing the CIT methodology

To properly prepare the CIT methodology, researchers should follow these steps:

Define the research question and objectives, specifying the type of incidents to be collected (Flanagan, 1954).
Develop clear criteria for what constitutes a “critical incident” in the context of the study (Butterfield et al., 2005).
Create an interview guide or questionnaire that elicits detailed, descriptive accounts of critical incidents (Gremler, 2004).
Pilot test the data collection instruments and refine them as needed (Edvardsson & Roos, 2001).
Train interviewers or observers to consistently apply the CIT methodology and probe for sufficient detail (Flanagan, 1954).
Determine the sample size and recruitment strategy, ensuring that participants have direct experience with the phenomenon of interest (Butterfield et al., 2005).
Summarize everything by developing a plan for collecting data, which may involve interviews, observations, or written reports.

Step-by-step walkthrough of CIT testing

Recruit participants who have direct experience with the service, product, or phenomenon being studied.
Conduct individual interviews or focus groups, asking participants to describe specific incidents that were critical to their experience.
Probe for details about the context, actions, and outcomes of each incident, ensuring that the description is complete and self-contained.
Record and transcribe the interviews or focus group discussions.
Review the transcripts and identify the critical incidents based on the predefined criteria.
Categorize the incidents into themes or patterns based on their similarities and differences.
Analyze the data by identifying and categorizing critical incidents and their underlying factors.
Analyze also the data to identify trends, insights, and areas for improvement.
Develop recommendations or action plans based on the findings.

Finalizing the CIT methodology

To properly finalize the CIT methodology, researchers should:

Ensure that the data analysis is complete and that all critical incidents have been identified and categorized (Flanagan, 1954).
Validate the findings through member checking, peer debriefing, or triangulation with other data sources (Butterfield et al., 2005).
Prepare a final report that summarizes the key findings, themes, and recommendations, using illustrative examples from the critical incidents (Gremler, 2004).
Communicate the results to relevant stakeholders and decision-makers, highlighting the implications for product, service, or process improvements (Edvardsson & Roos, 2001).
Document the CIT methodology, including the criteria, sample, data collection, and analysis procedures, to ensure transparency and replicability (Flanagan, 1954).

Tips for success and dos

To ensure the successful application of CIT, researchers should:

Clearly define the research question and scope of the study, aligning the CIT methodology with the project objectives (Flanagan, 1954).
Develop a rapport with participants and create a safe, non-judgmental environment that encourages open and honest sharing of experiences (Gremler, 2004).
Use open-ended, non-leading questions that allow participants to describe incidents in their own words and from their own perspective (Butterfield et al., 2005).
Probe for specific, concrete details about the context, actions, and outcomes of each incident, avoiding generalizations or hypothetical scenarios (Edvardsson & Roos, 2001).
Ensure that the sample size is sufficient to achieve data saturation and that the participants represent diverse perspectives and experiences (Flanagan, 1954).
- All incidents can be collected through directed storytelling, interviews, or diary studies, typically aiming for 50 to 100 incidents to achieve a workable sample size. However, depending on the complexity of the problem being studied, more incidents may be necessary (Hanington & Martin, 2012).

Things to avoid

When conducting CIT research, there are several things to avoid:

Asking leading or closed-ended questions that limit participants’ responses or impose the researcher’s preconceptions
Failing to probe for sufficient detail or accepting vague or incomplete descriptions of incidents
Overlooking the importance of context and focusing solely on the actions or outcomes of incidents
Ignoring discrepant or contradictory data that does not fit with emerging themes or patterns
Overgeneralizing findings beyond the specific sample or context of the study

Engaging stakeholders throughout the CIT process

Engaging stakeholders throughout the CIT process is essential for ensuring the relevance, credibility, and impact of the research. Researchers should:

Identify and involve key stakeholders, such as product managers, designers, or customer service representatives, early in the planning process to gather input on the research objectives and methodology (Gremler, 2004).
Communicate regularly with stakeholders throughout the data collection and analysis phases, providing updates on progress and preliminary findings (Butterfield et al., 2005).
Seek feedback from stakeholders on the interpretation and implications of the findings, incorporating their insights and perspectives into the final recommendations (Edvardsson & Roos, 2001).
Present the results to stakeholders in a clear, concise, and actionable format, highlighting the key takeaways and next steps for product, service, or process improvements (Flanagan, 1954).
Collaborate with stakeholders to develop and implement action plans based on the CIT findings, ensuring that the research translates into tangible outcomes and benefits for the organization and its users (Gremler, 2004).

Examples of techniques for CIT

The CIT involves several cognitive techniques aimed at reconstructing the context of the event and facilitating memory retrieval. One key technique is the “mental reinstatement of context,” which encourages the interviewee to mentally recreate the environmental and personal contexts surrounding the event (Fisher & Geiselman, 1992, as cited in Memon et al., 2010). This may involve visualizing the setting, recalling their thoughts and feelings at the time, and other sensory details.

Another technique is “focused retrieval,” which involves narrowing the retrieval process to specific details of the event, such as descriptions of people, objects, or actions (Memon et al., 2010). This can help the interviewee concentrate on particular aspects of the experience, potentially unlocking additional memories.

The CIT also incorporates principles of “extensive retrieval,” which involves encouraging the interviewee to report everything they can recall, even seemingly insignificant details (Memon et al., 2010). This comprehensive approach can sometimes trigger the recollection of additional information.

By employing these cognitive techniques, the CIT aims to enhance the completeness and accuracy of the information reported by the interviewee, ultimately aiding in investigative processes or research endeavors (Memon et al., 2010).

Examples of CIT in practice

Studying customer service encounters in the hospitality industry

In a study by Bitner et al. (1990), CIT was used to examine customer service encounters in the hotel and restaurant industries. Participants were asked to describe specific incidents where they experienced particularly satisfactory or unsatisfactory service. The researchers analyzed the incidents to identify key themes and factors that contributed to positive or negative service experiences, such as employee responsiveness, empathy, and reliability. The findings were used to develop training programs and service standards for the participating organizations.

Investigating user experiences with mobile banking apps

A study by Jun and Palacios (2016) applied CIT to investigate user experiences with mobile banking apps. Participants were asked to describe critical incidents where they encountered problems or had particularly positive experiences while using mobile banking services. The researchers categorized the incidents into themes related to app functionality, usability, security, and customer support. The results were used to identify areas for improvement in mobile banking app design and development.

Exploring critical safety incidents in the construction industry

Butterfield et al. (2005) used CIT to explore critical safety incidents in the construction industry. Participants, including construction workers and supervisors, were asked to describe specific incidents where safety was compromised or where effective safety practices were demonstrated. The researchers analyzed the incidents to identify common contributing factors, such as communication breakdowns, equipment failures, or inadequate training. The findings were used to develop targeted safety interventions and improve risk management practices in the construction industry.

Additional examples of usage.

The Critical Incident Technique (CIT) can be applied to create new products or services in various ways. First, CIT can be used to identify unmet customer needs or pain points that could be addressed by a new offering. By collecting and analyzing critical incidents related to customers’ frustrations or challenges, organizations can gain insights into potential market opportunities (Gremler, 2004). Second, CIT can inform the design and development of new products or services by gathering user feedback on desired features, functionalities, and experiences (Flanagan, 1954). Third, CIT can be used to evaluate the feasibility and desirability of new product or service concepts by collecting critical incidents related to prototypes or early versions (Butterfield et al., 2005).

CIT can also be applied to improve existing products or services. One way is by using CIT to identify areas for improvement based on customers’ or users’ negative experiences. By analyzing critical incidents where products or services fell short, organizations can pinpoint specific issues and develop targeted solutions (Edvardsson & Roos, 2001). Another application is using CIT to gather feedback on potential enhancements or new features, allowing organizations to prioritize improvements based on their impact on user experiences (Gremler, 2004). Finally, CIT can be used to evaluate the effectiveness of changes or updates to existing offerings by collecting critical incidents before and after the implementation of improvements (Flanagan, 1954).

Read this blog where I’ve included some examples of questions based on CIT methodology.

References:

Bitner, M. J., Booms, B. H., & Tetreault, M. S. (1990). The service encounter: Diagnosing favorable and unfavorable incidents. Journal of Marketing, 54(1), 71-84.
Butterfield, L. D., Borgen, W. A., Amundson, N. E., & Maglio, A. T. (2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative Research, 5(4), 475-497. https://doi.org/10.1177/1468794105056924
Dekker, S. (2011). Drift into failure: From hunting broken components to understanding complex systems. CRC Press.
Edvardsson, B., & Roos, I. (2001). Critical incident techniques: Towards a framework for analysing the criticality of critical incidents. International Journal of Service Industry Management, 12(3), 251-268. https://doi.org/10.1108/09564230110393197
Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. https://doi.org/10.1037/h0061470
Gremler, D. D. (2004). The critical incident technique in service research. Journal of Service Research, 7(1), 65-89. https://doi.org/10.1177/1094670504266138
Hanington, B., & Martin, B. (2012). Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions.
Jun, M., & Palacios, S. (2016). Examining the key dimensions of mobile banking service quality: An exploratory study. International Journal of Bank Marketing, 34(3), 307-326.
Landers, R. N. (2014). Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation & Gaming, 45(6), 752-768. https://doi.org/10.1177/1046878114563660
Memon, A., Meissner, C. A., & Fraser, J. (2010). The Cognitive Interview: A meta-analytic review and study space analysis of the past 25 years. Psychology, Public Policy, and Law, 16(4), 340–372. https://doi.org/10.1037/a0020518
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2), 1-135. https://doi.org/10.1561/1500000011