Lean Six Sigma Resources
Attribute Agreement Analysis
Attribute data—pass/fail, yes/no, defect/no defect—is common in many processes. While it may seem simple, attribute data carries unique challenges. Unlike continuous data, which provides detailed measurements, attribute data relies on judgment. This judgment introduces the risk of inconsistency. Attribute Agreement Analysis (AAA) evaluates whether different evaluators classify outcomes consistently and accurately.
AAA focuses on three key questions:
Do evaluators agree with themselves? (repeatability)
Do evaluators agree with each other? (reproducibility)
Do evaluators agree with the known standard? (accuracy)
The first step in conducting AAA is selecting representative samples. These samples should include a mix of clear cases and borderline cases. Borderline cases are particularly important because they reveal how evaluators handle ambiguity. If evaluators classify borderline cases inconsistently, the measurement system may not be reliable.
Next, the team selects the evaluators who will participate in the study. These evaluators should represent the people who normally classify outcomes. Including evaluators with different levels of experience can provide valuable insights into training needs.
The study typically involves each evaluator classifying each sample multiple times in a randomized order. This randomization helps prevent bias and ensures that the results reflect the true performance of the measurement system.
Once the data is collected, the team analyzes the results to determine the level of agreement. High agreement indicates that the measurement system is reliable. Low agreement indicates that the system may need improvement.
AAA often reveals issues that teams were unaware of. For example, evaluators may interpret criteria differently, leading to inconsistent classifications. The criteria themselves may be vague or subjective. The process for evaluating outcomes may lack standardization. AAA brings these issues to light and provides a path for improvement.
Improving attribute agreement may involve clarifying operational definitions, providing additional training, or redesigning the evaluation process. These improvements help ensure that attribute data is reliable and meaningful.
Attribute Agreement Analysis is essential for any project that relies on categorical data. It ensures that the data collected reflects the true behavior of the process, not the variability of human judgment. When teams invest the time to evaluate and improve attribute agreement, they strengthen the foundation for the rest of the Measure phase.
