In RFPs, customers evaluate suppliers’ proposals then attach numerical ratings — intending to make subjective decisions look logical and rational — except of course they aren’t.
And it’s not just evaluators’ subjectivity that makes proposal ratings a guessing game. Procurement also puts its hand in to the ratings mix because after evaluators rate suppliers’ responses, procurement makes magic by:
- Averaging or totaling evaluators’ ratings into a single score per supplier
- Comparing suppliers’ total scores side-by-side
- Declaring a best-proposal, technical winner (pricing is an easier story)
What began as subjective choices by evaluators gets averaged and totaled into an objective-appearing, consensus decision.
It’s this jumbly-but-standard process that contributes to a mystery: how can customer-evaluators read the same proposals yet give them wildly different ratings?
A Subjective Example of Proposal Evaluation
Several years ago, I wrote the technical RFP questions for a corporate client and was on the proposal evaluation team.
As this was a good-sized spend, the evaluation team wanted a strong consensus on the decision. They wanted to share the risk amongst themselves of a “bad decision” blowing up in their collective face. Same career protection procurement wants.
Yet during the evaluation it was obvious how ratings varied markedly from evaluator to evaluator.
Looking at the same section, same proposal, from the same supplier one evaluator would rate it “1,” another a “5,” and another a “3.”
Only when procurement averaged and totaled ratings into one number that different opinions disappeared. The single-number replacement looked as if everyone agreed that supplier’s proposal had a 3.17 rating.
Why So Different?
Differences of opinion, of the same content, exist for many reasons. In the world of customers evaluating sales proposals these three are worth considering:
- We Are Smarter/Dumber Than We Think
- Influences on Evaluators’ Evaluations
- People are Different
1. We Are Smarter/Dumber Than We Think
When it comes to evaluating suppliers’ proposals, all evaluators are not created equal.
Research spotlights one reason individuals think differently: the Dunning–Kruger effect.
“The Dunning–Kruger effect is a cognitive bias whereby people with low ability, expertise, or experience regarding a certain type of a task or area of knowledge tend to overestimate their ability or knowledge. Some researchers also include in their definition the opposite effect for high performers: their tendency to underestimate their skills.”
This bias can help explain differing evaluator ratings. Some believe they know more about what’s offered than they actually do. Others may know lots about the offers and wrongly think that their peers know as much as they do — so no need to educate others.
2. Influences on Evaluators’ Evaluations
Other factors influence evaluators’ evaluations too. Service quality research, ServQual, shows influences that may explain differing ratings. The model identifies four influencing factors.
A) Word-of-Mouth Communications
What might evaluators have heard about suppliers from others, both personally and on crowd-sourcing social media? Even though it’s not in proposals, this information can impact evaluators’ evaluations. Specifically, if that second-hand source is trusted and respected.
B) Personal Needs
Evaluators, like everyone else, can’t isolate themselves from life. Their personal needs are, well, personal, such as work, health, family, etc.
C) Past Experiences
Evaluators’ past experiences with participating suppliers will influence evaluations, good or bad. Evaluators may want to forgive suppliers’ past indiscretions but it’s hard not to forget them.
D) External Communications to Customers
What info/media have evaluators been consuming relative to participating suppliers?
Marketing pieces, research papers, social media to mass media, and everything between can influence evaluator expectations. Even if they’re not in the proposals themselves.
3. People are Different
Evaluators’ unique personality quirks and eccentricities shape how they see the world. And this can play out in different perspectives and evaluation ratings.
Summary, Act I: Making Evaluations Better
When customers evaluate sales proposals, it’s a mystery wrapped in a riddle inside an enigma. Their evaluations can look rocky and unfathomable in the decision dust and chaos.
But what if customers/procurement used Rank Choice Voting (RCV) to select a single winner for a supplier contract (like the recent Alaska elections)?
In this process evaluators would skip numerical ratings of responses, categories, and suppliers. Instead, evaluators would rank suppliers one-time as their first or second (or third, etc.) choice. Then Procurement would tally rankings and:
- If a supplier wins an outright majority of first-preference ratings (i.e., 50 percent plus one), they will be declared the winner
- If, on the other hand, no supplier wins an outright majority of first-preference ratings, the supplier with the fewest first-preference ratings is eliminated
- All first-preference votes for the failed supplier are eliminated, lifting the second-preference choices indicated on those ratings
- Procurement does a new tally to determine whether any supplier has won an outright majority of the adjusted ratings
- The process is repeated until a supplier wins a majority of votes cast
If that’s too confusing don’t worry, evaluators would have it much easier. They’d evaluate and then rank suppliers relative to each other, and only one time. Procurement then gets to crunch numbers and tally results, like they were born to do.
Summary, Act II: Making Supplier Proposals Better
Suppliers hope customer-evaluator ratings group around a shared perception rather than wild outliers that are averaged back into submission. That’ll feel more like an honest and legitimate decision on their proposals’ merits.
However, lacking that visibility, suppliers’ best road to success is to follow proposal best practices that:
- Create customized, one-off, intellectually compelling solutions, then…
- Write unifying & emotionally engaging proposal narratives, and…
- Integrate those narratives throughout as many parts of their proposals as possible
