How to Separate Recruitment Quotas From Evidence Weight

A recruitment quota tells you how many people a study planned to recruit and, in some designs, how it intended to distribute them. It does not tell you how much support the resulting finding provides for a particular claim. That is the central rule for how to separate recruitment quotas from evidence weight in research synthesis.

A larger target may provide broader coverage or more precise estimates when the question, population, design, and measure are appropriate. It cannot repair a mismatched research question, an unsuitable population, a weak measure, or a limitation that changes the interpretation. A smaller study may be directly useful for a product decision while remaining inadequate for a prevalence claim.

The practical response is to keep two records. One describes recruitment and coverage. The other evaluates what the study can support. Reconcile them only after both records are complete.

What the recruitment record tells you

Record recruitment facts without turning them into an evidence ranking. At minimum, capture:

• Recruitment target: How many people the study intended to recruit.
• Quota structure: Any planned distribution across relevant groups, locations, behaviors, or other eligibility characteristics.
• Achieved sample: How many eligible participants or units were actually included.
• Recruitment route: The panel, community, customer list, intercept, referral path, or other source used to find participants.
• Eligibility and exclusions: Who could enter the study and who was removed from analysis.
• Coverage questions: Which parts of the intended population may be included, underrepresented, or absent.
• Overlap: Whether participants, households, organizations, sites, panels, or observations also appear in another study.

These fields answer questions about sampling and coverage. They do not, by themselves, answer whether a finding is directly relevant, well measured, or suitable for a decision.

Do not collapse the recruitment target and achieved sample into one number. A target is a plan. The achieved sample is what the analysis actually used. The difference may need explanation, but its meaning depends on the study design and the claim under review.

Evaluate the claim, not the study in the abstract

Evidence weight is claim-specific. Start by writing the claim the synthesis may make, then assess each study against that claim. A useful evaluation record includes the following fields:

• Question fit: Does the study address the same problem, behavior, mechanism, or outcome?
• Population fit: Do the participants resemble the people to whom the claim will apply?
• Context fit: Were the task, product, channel, setting, timing, and constraints comparable?
• Design fit: Is the method suited to the kind of support required? Interviews, usability sessions, surveys, experiments, and behavioral records contribute different kinds of evidence.
• Measurement fit: Does the measure represent the construct or outcome in the claim?
• Directness: How many interpretive steps separate the study result from the claim?
• Limitations: What could weaken, qualify, or narrow the interpretation?
• Consistency: Do other studies point in a similar direction under comparable conditions?
• Applicability: Can the result inform the decision at hand, even if it cannot support a broader generalization?
• Synthesis role: Is the study being used as direct support, contextual evidence, a counterexample, a mechanism clue, or a reason for follow-up?

This approach follows the basic logic of evidence synthesis. The Cochrane Handbook describes defining eligibility criteria and planning the structure of a synthesis. A Cornell guide describes synthesis as identifying, selecting, and combining results from multiple evidence sources. Neither principle requires sample size to become a universal ranking rule.

Build a two-part evidence ledger

A study-level ledger keeps the categories visible without pretending that every study can be reduced to one score. Use one record with two clearly separated parts.

Recruitment and coverage

Study: Identify the study, date, source, and unit being recruited.

Target: Record the planned recruitment quota and any subgroup targets.

Achieved: Record the eligible sample analyzed, along with exclusions or missing cases that matter.

Population coverage: Describe the intended population and the parts the recruitment route may or may not reach.

Independence: Note known or possible overlap with other studies and repeated observations.

Claim evaluation

Synthesis claim: State the exact claim the study may support.

Question and population fit: Explain what matches and what does not.

Method and measure: Record how the evidence was produced and what it actually measures.

Directness and limitations: Identify the inferential distance and the conditions that limit interpretation.

Consistency and applicability: Compare the result with relevant studies and the decision context, not merely with the direction of the majority.

Synthesis role: State whether the study supports, qualifies, contrasts with, or cannot resolve the claim.

A structured record also protects provenance. Documentation guidance from the University of Minnesota emphasizes recording information sources, methods, search strategies, and records-management decisions. For a product research team, that principle translates into preserving how a sample was recruited and how the final interpretation was reached. Teams should also preserve participant-level traceability when excerpts or observations contribute to a synthesis.

Why the largest quota can mislead

Imagine a hypothetical team studying why users abandon account setup. Study A has a larger recruitment target through a broad panel. Study B recruits fewer people but closely matches the product’s intended audience and examines the same account-recovery context.

The larger quota does not settle the comparison. Study A may offer wider coverage or a useful estimate for a defined population. Study B may be more direct for the product question. Both can still have limitations. The ledger should show those differences instead of awarding an automatic advantage to the larger study.

The conclusion also depends on the claim. For a prevalence claim, achieved sample, coverage, nonresponse, and measurement conditions may be especially consequential. For a mechanism claim, a smaller study with close contextual fit may help explain how a problem occurs without establishing how common it is. For a product decision, either study may be useful if the decision is narrowly bounded and the relevant uncertainty is explicit.

A synthesis should therefore compare studies by the question they answer, the population and context they represent, the method and measure they use, and the limitations they carry. A specialist guide to explaining research evidence recommends separating consistency of direction from consistency of evidence quality. That distinction is useful here: several studies may point in the same direction without providing equally direct or reliable support. When findings appear to disagree, teams can reconcile conflicting research findings by checking whether the studies actually address comparable questions and contexts.

Reconcile the records without creating a false score

Once the two parts are complete, use a decision sequence rather than a universal numerical weight:

1. Define the claim. Specify whether the synthesis concerns prevalence, a mechanism, an experience, a segment, or a product action.
2. Check eligibility and fit. Exclude or separate studies that address a materially different question, population, context, or outcome.
3. Inspect recruitment and achieved sample. Use these details to understand coverage, precision, missingness, and possible selection concerns.
4. Assess design and measurement. Ask whether the method can support the type of inference being considered.
5. Check independence and consistency. Do not count overlapping samples or repeated observations as entirely new evidence. The overlapping-samples comparison framework provides a separate check for this problem.
6. Set the claim boundary. State what the combined evidence supports, what it only suggests, and what remains unresolved.
7. Route the decision. Retain contextual evidence, narrow the claim, separate populations, seek focused follow-up, or defer the decision.

The result may be a bounded statement such as: “These studies suggest a problem in the account-recovery context for this audience, but they do not establish its prevalence across all users.” That is more useful than declaring one study stronger because it recruited more people.

Recruitment information still matters. It can reveal missing coverage, differences in population, possible overlap, and limits on how far a result can travel. Its role is to inform interpretation, not replace it. Keep the quota, achieved sample, and coverage record intact, then evaluate evidence weight against the claim the team actually needs to make.