How to Compare Research Findings When Samples Overlap

Different studies do not automatically provide independent evidence. Two studies may use different tasks, questions, or measures while drawing from some of the same people, households, organizations, sites, or panels. If a team counts each study as a fresh sample, its synthesis can sound broader or more strongly supported than the evidence allows.

A defensible comparison starts with the relationship between the studies. Define the relevant sampling unit, document known and uncertain overlap, compare what each study contributes, and route the result to a bounded claim, a separate analysis, or targeted follow-up. This preserves useful longitudinal and contextual information without treating repeated observations as new participants.

Start with the sampling unit, not the study title

The first question is what was sampled and what the synthesis is trying to count.

A usability study may sample individual participants. A field study may sample households or organizations. A survey panel may contain the same people across waves. An education assessment may use schools as sampled units even when students provide the responses. The relevant unit depends on the claim.

A claim about individual task difficulty requires attention to repeated participants. A claim about organizational adoption requires attention to repeated organizations or sites, even if the individual respondents changed. The National Center for Education Statistics describes overlap among sampled schools, illustrating why an audit should look beyond individual identities.

Write the sampling unit into each study record before comparing results. Use a specific description such as “individual account holders,” “households using the service,” or “organizations operating the workflow.” If the unit is unclear, record that uncertainty instead of assuming that participant count is the correct denominator.

Build an overlap-aware study ledger

Create one record for each study before writing the synthesis. The ledger should make differences and relationships visible without retaining more personal information than the research decision requires.

Useful fields include:

• Study identifier and date
• Research question and intended decision
• Sampling unit
• Target population and recruitment source
• Setting, task, product version, or field context
• Method and measures
• Number of sampled units, when relevant
• Known, suspected, and unknown overlap with other studies
• Evidence contributed by the study
• Limitations that affect comparison
• Permitted claim scope

The overlap field should not force a false yes-or-no answer. Use categories such as known overlap, probable overlap, no overlap established, and unknown. Add the basis for the classification, such as a shared recruitment panel, matching study codes, a longitudinal design, a common organization, or a privacy restriction that prevents linkage.

Keep the linkage record separate from the synthesis record when possible. A synthesis may need to say that two studies share some participants without storing names, email addresses, or full identifiers in the document used by a product team. This supports traceability while limiting unnecessary exposure of participant information.

Compare study boundaries before comparing findings

Overlap affects interpretation, but it does not make two studies duplicates automatically. A returning participant can contribute a new observation when the question, context, task, measure, or time point changes. That contribution may still depend on the earlier observation, so the synthesis should distinguish what is repeated from what is new.

Compare the studies across these boundaries:

Question and construct

Do the studies examine the same phenomenon? A later interview about trust and an earlier usability test about task completion may involve the same participants but answer different questions. Keep their findings as distinct strands rather than merging them into one count.

Population and setting

Are participants drawn from the same population, organization, site, panel, or recruitment source? New individuals from the same organization may add contextual coverage without providing a fully independent organizational sample.

Task and exposure

Did participants encounter the same product version, workflow, intervention, or stimulus? A later study may reflect learning, changed expectations, or prior exposure. Those conditions can be useful evidence, but they change what the comparison means.

Method and measure

Interviews, diary studies, surveys, usability sessions, and behavioral logs produce different kinds of observations. Mixed-methods work can compare evidence across methods or levels rather than treating every observation as interchangeable. A mixed-methods research overview describes mixed-methods research as a way to consider activity across different levels of a system. That supports comparing study boundaries, not automatically combining their participants.

Time and sequence

Record which study came first and whether participation in one could affect the other. Repeated measures may reveal change over time, but they should not be presented as independent replications merely because they appear in separate reports.

Separate repeated evidence from additive evidence

After comparing the studies, classify how their findings relate. This is a reasoning step, not a universal statistical threshold.

Repeated evidence comes from the same sampled unit or a linked unit and addresses substantially the same claim. It may show that an observation recurred for that unit, but it does not create a new independent participant or organization.

Contextual evidence comes from an overlapping unit under a meaningfully different task, setting, time, or measure. It can show how a finding changes or persists across conditions. Its value is comparative, not simply additive.

Additive evidence comes from genuinely new sampled units addressing a sufficiently comparable question. Even then, shared sites, panels, recruitment channels, or procedures can create dependence at a higher level.

Unresolved evidence is evidence whose relationship cannot be determined because identifiers, recruitment records, or sampling details are missing. Unknown overlap is not proof of overlap, but it prevents a confident claim that the samples are independent.

Consider this hypothetical example. A design team compares an interview study with a later usability study. Several people may have participated in both, while the studies use different tasks and measures. The team records known and suspected overlap, marks returning participants’ observations as repeated or contextual, and treats evidence from new participants as additive only where the study boundaries support that interpretation.

Choose the synthesis claim after the audit

Let the overlap assessment determine the claim’s scope.

If the samples are distinct and the questions are comparable, the team may describe convergence across studies while naming differences in method and population. If some units overlap, describe the findings as converging observations across studies with shared or partially shared samples. Avoid implying that the full participant counts are independent.

If the samples cannot be linked, use narrower language. For example, say that the findings are consistent across available studies while noting that participant independence is unknown. This is more useful than discarding the studies and more accurate than presenting them as one larger combined sample.

If the studies use materially different questions or measures, keep them as separate strands of evidence. They may support one product decision together, but they do so by addressing complementary risks or contexts rather than by replicating the same result.

Know when not to pool findings

A qualitative synthesis can compare themes, contexts, exceptions, and claim boundaries without producing one combined estimate. That may be appropriate when studies use different measures, overlap is only partly known, or the decision concerns a design risk rather than a population parameter.

Formal pooling requires more than a list of participant counts. Depending on the design, an analyst may need to account for repeated participants, clustered units, longitudinal dependence, shared controls, or correlated estimates. No universal overlap percentage determines when a synthesis becomes invalid, and a wording change cannot substitute for an appropriate statistical method.

Seek methodological help when the decision depends on a combined effect estimate, prevalence statement, confidence interval, or formal meta-analysis. The ledger remains useful because it gives the analyst the study boundaries and linkage information needed to determine whether formal adjustment is possible.

Route unresolved cases deliberately

An overlap audit should end with a decision, not just a note that samples might intersect.

• Narrow the claim when the findings remain useful but independence is uncertain. State the shared or unknown sampling relationship and avoid a combined-sample interpretation.
• Separate the evidence strands when the studies answer different questions or use incompatible measures. Use them together only to explain complementary aspects of the problem.
• Reanalyse the data when participant or unit linkage is available and the decision requires a formal estimate that accounts for dependence.
• Run targeted follow-up research when unresolved overlap prevents a decision and a new sample, site, or recruitment source could answer the narrow question.
• Preserve the uncertainty when privacy, consent, or governance rules prevent linkage. Document what is known, what cannot be checked, and how that limitation constrains the claim.

The purpose of comparing overlapping samples is not to reduce every study to a single score. It is to keep the relationship between observations and claims visible. A repeated participant can provide useful evidence about change, context, or experience over time. A shared organization can explain why findings travel together. Neither should be silently converted into independent support for a broader conclusion.

Preserve participant-level traceability when recording those relationships. Then compare research themes with rival explanations after the evidence relationship is clear.