Bias in Numbers: What Calibration Data Reveals About Rating Fairness

Now that most organizations have wrapped calibration, a pattern becomes easier to see.

Calibration season brings leaders together to review ratings across teams. They compare notes, challenge each other’s assessments, and work toward alignment, with the goal of ensuring that performance ratings are fair, consistent, and reflective of actual contribution.

So organizations invest heavily in the process.

More stakeholders in the room. More time spent debating edge cases. More effort to align on definitions and expectations.

But when you look closely at calibration data across organizations, the way we do at Klaar, a different pattern emerges.

Calibration doesn’t remove bias from performance ratings…

It reveals it.

And that distinction matters.

Because calibration isn’t just an alignment exercise. It’s one of the clearest indicators of how performance is actually being evaluated across your organization.

In this month’s blog, we’ll explore what calibration data reveals about rating fairness, from distribution patterns and manager behavior to the role of recency, visibility, and consistency in shaping outcomes. We’ll also look at what these patterns mean for leaders building performance systems that are both credible and fair.

Rating Distributions Aren’t Random

Rating distributions are often treated as outputs. But over time, they become signals.

Across organizations, teams with similar roles, tenure, and expectations often produce very different rating outcomes. In one case, three comparable teams landed at 40%, 18%, and 5% “exceeds expectations” ratings in the same cycle. Same framework. Same company. Different results.

That’s not randomness. It's a pattern.

We also see managers with zero “below expectations” ratings across multiple cycles, even when company-wide averages suggest otherwise. While possible, repeated patterns like this often point to rating inflation rather than consistently perfect performance.

Another common signal is clustering in the middle. In many organizations, 65–75% of employees are rated as “meets expectations,” with limited differentiation at the top or bottom. This often reflects risk aversion more than performance reality, defaulting to “safe” ratings to avoid difficult conversations.

If ratings were purely driven by performance, these patterns wouldn’t repeat so consistently. But they do.

Because performance isn’t the only variable shaping outcomes.

Managers Drive More Variation Than Performance Does

One of the clearest patterns in calibration data is that variation in ratings is often less about employee performance and more about how managers evaluate it.

Across organizations, managers show distinct and repeatable tendencies. Some consistently rate a high percentage of their team as top performers, while others rarely give top ratings regardless of output. Some shift significantly under calibration pressure, while others hold firm.

In one dataset, two managers overseeing similar teams produced very different distributions: one rated 35% of their team as exceeding expectations, while the other rated just 8%. After calibration, both teams were adjusted toward a more consistent distribution.

Alignment improved, but the underlying evaluation logic didn’t change.

Calibration aligned outcomes. It didn’t eliminate bias.

Because calibration doesn’t just evaluate employees. It reveals how managers interpret performance.

Recency and Visibility Influence Ratings More Than You Think

Even in structured calibration discussions, performance is often evaluated through memory.

And memory has patterns.

Late-cycle wins tend to carry disproportionate weight. In one organization, employees with a visible success in the final six weeks of the cycle were 22% more likely to receive a top rating, regardless of their performance earlier in the year.

Consistent performers without a standout moment were more likely to land in the middle.

We also see a strong relationship between feedback volume and ratings. Employees with more documented feedback entries tend to receive higher ratings…not necessarily because of stronger performance, but because more evidence is available during calibration discussions.

Visibility plays a similar role. Employees in high-exposure or cross-functional roles are more likely to be rated highly than equally impactful contributors whose work is less visible.

Not because they did more. Because more people saw what they did.

When performance is evaluated through recall and narrative, bias doesn’t need to be intentional. It simply needs to be human.

Consistency Doesn’t Mean Fairness

Most calibration processes are designed to improve consistency.

Normalize distributions. Align definitions. Reduce outliers.

And these efforts matter.

But consistency is not the same as fairness.

We’ve seen organizations enforce tight rating curves across teams, only to find that high-performing teams are compressed into the same distribution as average ones. Managers adjust ratings to fit expectations rather than reflect actual performance.

In one case, rating variance across teams dropped by 30% after calibration.

But employee perception of fairness didn’t improve. Because while outcomes became more consistent, the underlying signals still lacked clarity.

You can standardize ratings. But if the inputs are biased, the outputs will be too.

What This Means for Leaders

Calibration is not where fairness is created. It’s where it becomes visible.

For leaders, that means the focus can’t start and end with the calibration meeting itself. It has to shift earlier…into how performance is observed, tracked, and discussed throughout the cycle.

Organizations seeing the strongest outcomes are not waiting until the end of the cycle to align on performance. They are creating clearer signals along the way by:

• Making goals and progress visible in real time
• Grounding feedback in actual work, not retrospective summaries
• Identifying inconsistencies across managers before they compound
• Creating shared context so calibration is based on evidence, not interpretation

When these elements are in place, calibration becomes simpler and more effective. Discussions shift from defending perspectives to validating signals.