Ranking Alternatives Fairly: Criteria and Weighting

A list of alternatives in an order is a claim. The first entry is implied to be the best, or at least the most relevant. Readers act on that, often without thinking, because the order looks like a verdict. That gives the people who produce the ranking a responsibility: to make the order mean something, and to say what it means.

This article describes how to rank alternatives fairly: what to measure, how to weigh it, how to keep money and opinion apart from facts and how to explain the result to readers.

What a ranking should answer

A ranking is useful only if it answers a clear question. Possible questions include:

• Which alternatives do people like most?
• Which best match a particular set of needs?
• Which are cheapest for a defined use?
• Which have the most complete feature set?
• Which are most mature and best documented?

Each question has a different method. A single ranking cannot answer them all, and pretending it can produces a muddle. So begin by stating the question in one sentence, and show it above the list.

Three kinds of input

Most rankings of alternatives draw on three kinds of input. They are different in character, and mixing them without care misleads.

Community opinion

Readers vote or react. This tells you what people liked. Writing on the wisdom of crowds notes that groups tend to judge best when opinions are diverse and independent and the group is large, and that crowds do best on questions with a definite answer. Opinion rankings reflect popularity and first impressions, which is valuable, but they are not a measure of long-term quality or fit for any particular need.

Facts that can be checked

Features offered, platforms supported, licence type, date of first release, price model, certifications held. These can be verified against documentation, and a ranking based on them is auditable. They describe what a product is, not how good it is.

Money

Some sites sell placement. It is legitimate if clearly labelled, but it says nothing about quality. It must never be disguised as opinion or fact.

Keep these three separate in the interface and in the method. Present them as separate lists or columns, with labels, instead of blending them into a single score. On this site, votes cannot be bought, paid placements are labelled, and verified data is shown separately.

Choose criteria that matter to readers

If you build a ranking from criteria, choose them with readers in mind.

• Relevance. Does the criterion reflect something readers care about when choosing among these products?
• Measurability. Can it be assessed consistently?
• Independence. Does it overlap heavily with another criterion, which would double count?
• Fairness. Does it favour a type of product for reasons unrelated to the reader's need?
• Stability. Will it mean the same thing next month?

Typical criteria for software alternatives include: coverage of core features, ease of use, price model and transparency, data portability, privacy and security information, support and documentation, platform availability and project health. Wikipedia's description of a decision matrix, in which alternatives are compared against criteria weighted by their importance, offers a useful frame. List the alternatives in columns and criteria in rows, score each cell and weight the rows.

Five to eight criteria is a manageable number. More tends to hide what matters.

Define each criterion and scale

Without definitions, scores are opinions in disguise. For each criterion, write what it measures and what each score means.

Example for "Data portability":

• 1: No export, or export in a format that cannot be re-imported
• 3: Export in at least one open format, with partial coverage
• 5: Full export in open formats and a documented way to import elsewhere

Using anchored scales helps reviewers score consistently, and it lets readers understand the numbers.

Weight by what readers need

Not all criteria matter equally to all readers. Default weights reflect a typical reader, but you can offer ways to adjust.

• Publish the default weights, so that readers see what the ranking values.
• Let readers change them, with sliders or presets such as "budget", "privacy first" or "feature rich". Show the re-ordered list.
• Explain the reasoning. Why does portability weigh more than visual polish?

A ranking with adjustable weights treats readers as capable adults, and it is more honest, because it shows that the order depends on priorities.

Resist the pull of the target

Goodhart's observation is that when a measure becomes a target, it ceases to be a good measure. Alternatives rankings are a prime example. If makers know that a feature count influences rank, they may add features for the sake of the count. If votes decide, they may campaign for votes. If a criterion is easy to manipulate, it will be.

Defences include:

• Using several criteria so that gaming one has limited effect.
• Preferring verified facts over self-reported claims for key criteria.
• Protecting votes, with account checks, limits and a ban on paid or traded votes.
• Reviewing unusual changes, such as sudden jumps.
• Publishing the method so that gaming is visible.
• Revisiting criteria as behaviour shifts.

Handle ties and gaps

A fair ranking says what happens when two entries are equal or when data is missing.

• Ties. Declare a tie-break rule in advance, such as the more recent verification or alphabetical order, and apply it consistently. Alternatively, show tied entries as a group.
• Missing data. Do not treat unknown as zero. Mark it, and explain how it is handled: excluded from that criterion, scored neutrally or flagged. A product should not rank lower simply because its maker has not filled in a field, unless the ranking is explicitly about completeness.
• New entries. Explain how new products are treated until they have enough data or votes.

Show the method, briefly

Every ranking should carry a short method note, with a link to a fuller explanation.

• The question being answered
• The inputs: votes, facts, paid placement, each labelled
• The criteria and weights
• How ties and missing data are handled
• The date of the last update
• How to report an error

A reader should be able to understand in a minute what the order means.

Disclose interests

If the publisher has a relationship with any product on the list, such as owning one or receiving payment, say so on the page. A reader can judge a ranking better when they know who produced it and why. Google's guidance on helpful content emphasises clarity about who is behind a page and trustworthiness as central to quality.

Review and correct

Rankings are living things. Review them on a schedule, and when things change: a product changes its price model, a new entrant appears, an old one closes. Date the ranking. If you correct an error, say so. Keep a history so that readers can see how the order has changed, and why.

A worked example

An editor builds a ranking of five note-taking apps for "people who want to keep their notes in open formats". The question is stated at the top. She chooses six criteria: portability, privacy information, core note features, ease of use, cross-platform availability and documentation. She defines each with a five-point scale and gives default weights, with portability and privacy weighted highest because of the stated question.

Scores come from tested facts and a documented checklist, not from opinion. Separately, the page shows community reactions, labelled as "what readers liked", and a clearly marked paid placement area, labelled "sponsored". A control lets readers switch to a "most features" preset, which reorders the list.

The method note explains ties, how missing data is treated and the date of the last review. A "report an error" link sits beside it. After a month, a maker points out that her app added export in an open format, and the editor verifies it, updates the score and adds a dated note. The order changes, and the change is visible.

A checklist

• Clear question
• Three inputs kept separate
• Relevant, measurable, independent criteria
• Defined scales
• Published, adjustable weights
• Defences against gaming
• Rules for ties and gaps
• Method note and last updated date
• Interests disclosed
• Review and corrections visible

The leaderboards page here shows how ranking systems are labelled, the categories page groups tools by purpose and the search page helps readers find options for a need.

What readers can do with a ranking

Readers can treat a published method as a checklist. They can compare the criteria with their own needs, change the weights if the page allows and look at the underlying facts for the products at the top. They can open the entries ranked lower on the criteria they value most. They can notice whether paid placements are clearly marked. And they can come back after a few months to see whether the order has moved, since a ranking that never changes is probably not being maintained. A fair ranking invites this kind of use, and it is a good sign when the page itself suggests it.

Frequently asked questions

Is it fair to rank a free product against a paid one? Yes, if criteria and costs are clear and readers can adjust for what matters to them.

Should I show scores or just order? Show scores where they mean something, and always show the method.

What if makers dispute their ranking? Invite evidence, review it and correct errors openly.