A shortlist is the point where quiet bias becomes an expensive decision. By the time interviews start, the narrowing has already happened. It is rarely revisited, because it no longer looks like a decision at all.

Kolstad reviewed its hiring outcomes annually, which is to say it reviewed the people it had interviewed. The much larger group filtered out before that point was never in the sample. Whatever was happening in the narrowing was invisible to the review meant to catch it.

The narrowing nobody re-reads

Every open role at the financial services firm produced a shortlist of six to eight names. Those names were drawn from applicant pools that were often ten times that size.

The shortlist itself was built by one hiring manager, working alone against a job description, usually under deadline. Kolstad’s talent team could see who got hired. It could not see who had been dropped three steps earlier, or why. That step left no record beyond the names that survived it.

Checking the list before the interviews

Kolstad now audits each shortlist against its own criteria with Yardstick Agents, before anyone is invited to interview. A skewed list is caught while it can still be corrected, rather than explained after the fact.

The audit compares the shortlist against the applicant pool it came from, and against the criteria the role declared. Where the two disagree, it names which criterion did the work: a degree, a former employer, a specific tool. It does not leave the skew unlabeled.

Roughly one in five shortlists come back flagged for a criterion that carried more weight than the role justified. The hiring manager sees that flag before the first interview is booked. A name can still be added back in at that point.

A flag is not a verdict

The audit reorders nothing, and it rejects nobody. It tells the hiring group what its own list is weighted toward. It leaves the group to decide whether that weighting is the job or an accident. No shortlist is corrected automatically. The hiring manager makes that call, and records the reason.

What the audits caught

Since the audit went live, 22% of shortlists have been flagged for a criterion that outweighed what the role actually required. Most often, that criterion was a specific former employer standing in for a skill several other candidates also had.

None of those flagged shortlists were corrected by the tool itself. 0 shortlists have been auto-rejected by the audit. Every change on record was made by the hiring manager who reviewed the flag. All shortlists at Kolstad, 100% of them, are now audited before interviews begin, while the fix is still cheap.

The audit cannot flag a criterion the role description never wrote down. An unstated preference can still slip through if nobody named it as a requirement. Kolstad is working through its job description library, role by role, to close that gap.

Extending the audit to promotions

Kolstad is extending the same audit to internal promotions. There the shortlist is often a single name, and the norm it should be checked against is written down even less often.