Scoring methodology

How a numberearns its place.

Every LiteSurface score comes from a published procedure: a weighted scorecard, two independent evaluators, deterministic synthesis, and gates that can override the number. This page documents all of it.

9
weighted criteria
2
independent evaluators
1
median per criterion
6
gates, one of them soft

The procedure at a glanceModels judge. Code keeps the score.

Language models are good at reading evidence and forming a judgement. They are not reliable at arithmetic, consistency, or bookkeeping. So the models score, cite, and flag; everything numeric happens afterwards in deterministic code that you can rerun on the stored outputs and get the same result.

Input
Stage 1: Concept and evidence
One concept version, the scorecard, the strategy profile, and the project’s cited claims.
Model
Stage 2: Two evaluators
OpenAI and Anthropic each score every criterion independently, citing claims and flagging gates.
Code
Stage 3: Median and weights
Median per criterion, then a weighted total from 0 to 100.
Code
Stage 4: Gates
Hard gates override the number. The soft coverage gate caps it and asks for evidence.
Code
Stage 5: Recommendation
Band, recommendation, confidence, disagreement, and coverage, each reported separately.

The default scorecardFast Product Bet.

Nine criteria, each scored from 0 to 100, with weights that sum to 1.0. The weighting favors user pain, then feasibility and distribution: the things that decide whether a small team can find out quickly if an idea works. Enterprise plans can define custom scorecards on the same rules.

The nine criteria of the Fast Product Bet scorecard with weights
Criterion	Weight	What it measures	Coverage
User pain / urgency	0.16	Frequency, intensity, and willingness to change behavior.	Evidence-based
Why-now / enabling shift	0.11	A new capability, cost curve, regulation, or behavior change that makes this possible now.	Evidence-based
Differentiation	0.12	Meaningful advantage over existing alternatives.	Evidence-based
Prototype feasibility	0.13	Whether a credible validation product can be built quickly.	Evidence-based
Distribution plausibility	0.13	Reachable users through channels the team can actually use.	Evidence-based
Monetization quality	0.10	Payer clarity, pricing power, and margin profile.	Evidence-based
Defensibility accrual	0.10	Whether usage creates a durable advantage over time.	Evidence-based
Evidence strength	0.10	Quality and coverage of the supporting evidence.	Evidence-based
Strategic fit	0.05	Alignment with team strengths and project constraints.	Opinion-based, excluded
Total	1.00	A concept’s score is the sum of each criterion’s median times its weight.

Strategic fit is a judgement about your team, not a fact about the market, so it is left out of the evidence coverage calculation described below.

Dual evaluation and the medianTwo opinions, kept apart.

One evaluator from OpenAI and one from Anthropic receive the same context package: the concept version, the scorecard, the strategy profile, and the project’s cited claims. Neither sees the other’s answer. See Evaluation for how this looks in the product.

Each evaluator returns, per criterion

A score from 0 to 100 and a confidence from 0 to 1
Supporting and contradicting claim ids
The material claims its judgement depends on
Signals for each gate: pass, fail, or uncertain

Code then

Clamps every score to the 0–100 range
Takes the median across evaluators for each criterion
Applies an adjustment only where a written rule defines one; none by default
Sums median × weight into the concept score

Why the median

The median resists a single outlier and is honest about what two evaluators agree on. With two evaluators it equals their midpoint; the procedure is written for more. Run modes decide which concepts receive dual evaluation: a top subset, the shortlist, or all of them. If one provider is unavailable, evaluation continues with one evaluator and the result is marked partial.

Example

A worked synthesis

Illustrative scores for one concept. The medians, contributions, and total are calculated from them on this page with the same formula the product uses.

Example evaluator scores, medians, and weighted contributions
Criterion	Weight	Evaluator A	Evaluator B	Median	Contribution
User pain / urgency	0.16	82	76	79	12.64
Why-now / enabling shift	0.11	74	70	72	7.92
Differentiation	0.12	66	72	69	8.28
Prototype feasibility	0.13	84	80	82	10.66
Distribution plausibility	0.13	62	70	66	8.58
Monetization quality	0.10	70	64	67	6.70
Defensibility accrual	0.10	58	66	62	6.20
Evidence strength	0.10	72	68	70	7.00
Strategic fit	0.05	80	84	82	4.10
Weighted score					72.1

A score of 72.1 falls in the Strong band, so the numeric recommendation is Validate, provided no hard gate fails. The evaluators differ by 5.6 points on average, a disagreement index of 0.06.

Hard gates and the soft gateSome problems outrank any score.

Gates catch problems that a weighted average would dilute. A failed hard gate sets the recommendation to Gate failed whatever the score. The soft gate never blocks; it caps the recommendation below Build now, lowers confidence, and marks the concept as needing evidence.

Gates, when they fail, and who decides
Gate	Fails when	Decided by	Type
Prototype exceeds build horizon	The estimated prototype time is longer than the project’s build horizon. Overruns of up to 20% are marked uncertain: a scoping note, not a failure.	Rule	Hard
Requires unavailable capability	A capability the concept depends on is restricted, deprecated, or missing from the atlas. Unknown availability is marked uncertain.	Rule	Hard
Conflicts with project exclusions	The concept matches a category excluded in the strategy profile, or the evaluators agree that it conflicts with one.	Rule, then evaluators	Hard
Legal or safety critical risk	The evaluators agree there is a critical risk with no plausible mitigation, or risks are declared without mitigation while the project avoids regulatory exposure.	Evaluators, then rule	Hard
No plausible acquisition path	The evaluators agree there is no credible way to reach the target users, or the concept states no primary channel.	Evaluators, then rule	Hard
Evidence coverage below minimum	Weighted evidence coverage is under the scorecard minimum, 35% by default.	Rule	Soft

Rules first

Where a gate can be checked from structured data, such as prototype days against the horizon or capability availability in the atlas, code checks it directly and the models are not consulted.

Agreement for model signals

A gate fails on model signals only when every evaluator that reported on it says fail. A split verdict is recorded as uncertain and shown on the card.

Evidence coverage and the gap loopThin evidence is a task, not a verdict.

Coverage measures how much of an evaluation rests on evidence in your project rather than on the model’s general knowledge. It is reported beside the score, and when it is low the system goes looking for what is missing. The evidence library holds every source and claim it can cite.

Coverage for one criterion

coverage = supported material claims ÷ total material claims

A material claim is one the evaluator says its score depends on. It counts as supported only when the evaluator cites a claim that exists in the project.
Claims resting only on a model’s prior knowledge count as unsupported.
Opinion-based criteria (strategic fit) are excluded. The concept’s coverage is the weighted average across the remaining criteria.

When a concept needs evidence

A concept is flagged as needing evidence when either signal fires. The second exists because coverage can be satisfied by citing generic claims; the evaluators’ own judgement of evidence strength is the more honest check.

Coverage below minimum
Weighted coverage under the scorecard minimum, 35% by default.
Weak evidence strength
The median evidence-strength score is below 58, the top of the weak band.

The flag caps the recommendation at Validate, multiplies confidence by 0.8, and turns the result into an action: gather evidence and re-evaluate. Balanced and deep run modes do this automatically for the strongest concepts; economy mode skips it, and you can start a pass from any concept.

Step 1
Collect the gaps
The evaluators’ stated evidence gaps and the concept’s riskiest assumptions.
Step 2
Plan targeted queries
Each gap becomes a focused research question rather than a broad topic search.
Step 3
Triage for relevance
A compact judge scores every search result against the focus. Results below 0.35 are never fetched.
Step 4
Extract claims
New sources become atomic claims with locators, credibility, and freshness.
Step 5
Re-evaluate
A new evaluation is appended. The earlier one stays in history, unchanged.

If evidence is still thin after the new evaluation, the concept keeps its needs-evidence flag and the loop can run again.

Confidence and disagreementReported beside the score, never blended in.

A 74 that both evaluators agree on is different from a 74 they reached from 60 and 88. LiteSurface keeps these signals separate so a high score can never hide an argument, and low confidence can never quietly drag a score down.

Disagreement index

mean over criteria of |A − B| ÷ 100

Zero means the evaluators gave identical scores. An index of 0.20 means they were 20 points apart on average. Disagreement is also stored per criterion, so the widest gaps make good agenda items for a review.

Confidence

mean evaluator confidence × (1 − disagreement)

Then × 0.7 when only one evaluator responded, and × 0.8 when the concept needs evidence. In the worked example: 0.72 × (1 − 0.056) = 0.68.

Bands and recommendationsA clear order of precedence.

The score maps to a band and a numeric recommendation. Gates and the evidence flag can then change that recommendation, always in the same order.

Score ranges, bands, and recommendations
Score	Band	Recommendation	What it means
85–100	Exceptional	Build now	Validate or build, depending on risk and the strength of the evidence.
72–84	Strong	Validate	Shortlist and test the riskiest assumptions before committing.
58–71	Hold	Hold	Park it, or run targeted research on the weakest criteria.
0–57	Weak	Reject	Reject unless there is a stated strategic reason to override.

Precedence 1
Hard gate fails
The recommendation is Gate failed, whatever the score.
Precedence 2
Concept needs evidence
Build now is capped at Validate. Lower bands are unchanged.
Precedence 3
Otherwise
The band decides the recommendation.

People make the decision

The recommendation is advice. Portfolio status (shortlist, validate, build, hold, or reject) is set by a person, and every decision stores the prior recommendation, the rationale, who made it, and when. Validation outcomes lead to a new evaluation that records that it included them; earlier evaluations are never edited. See Portfolio and Validation.

What models do, what code doesA deliberate division of labor.

Models propose

Plans research queries and judges whether search results are relevant
Extracts atomic claims from sources, with locators
Drafts concept genomes and flags near-duplicates
Scores each criterion with a rationale and cited claim ids
Signals gate concerns and states its evidence gaps
Writes validation plans and handoff sections

Code decides the numbers

Validates every model output against a schema, repairing or rejecting it
Drops citations to claim ids that do not exist in the project
Takes the median per criterion and computes the weighted total
Evaluates rule-based gates and applies recommendation precedence
Computes coverage, confidence, and disagreement
Keeps evaluations immutable and versioned, and enforces budgets and provider policy

What a score is not

A score is a structured judgement about plausibility given the evidence in your project. It is not a forecast of success, and it is not legal, safety, or financial advice. Read the AI disclosure for the limits of model output, and the glossary for the terms used here.

See it appliedto one concept, end to end.

The sample evaluation shows both evaluators’ scores, the medians, all six gates, coverage, and the claims behind them, exactly as the pipeline computed them.

Open the sample evaluation

SharePost on X Share on LinkedIn

Hear when the method changes

Methodology changes are noted in the changelog and in product updates.

See the procedurerun on your own ideas.

Explore a sample project before you run your own, and inspect every median, gate, and coverage figure behind each recommendation.

Start free Read the FAQ

Solo and Team are free during early access
Explore a sample project first
Bring your own OpenAI and Anthropic keys