Search Check began with a practical problem in our own research: a literature search can change many times before it becomes the search that appears in a paper, but most of the reasoning that produced the final version is easy to lose. A researcher may begin with a reasonable set of terms, discover that one of them retrieves an unrelated literature, notice that a paper they expected to find is missing, translate the search for another database, revise it again, and eventually arrive at a strategy that behaves well enough to use. By publication, however, the history of those decisions may have been reduced to the final query and a short description of where it was run.
We wanted a way to preserve more of that process without asking the researcher to hold it all in memory. Search Check grew out of that need. It is an experimental open research tool for developing, testing and documenting literature searches as they change: what the researcher intended to find, what a candidate query actually retrieved, what expected material it recovered or missed, what the researcher learned from those results, and why the next version of the search was different.
The methodological problem is discussed in more depth in How Do You Know If a Literature Search Is Finding the Right Research?. Search Check is our attempt to turn that problem into a usable research workflow.
The final search is only part of the method
Search reporting standards are important because another researcher needs to know which databases were searched, when they were searched, which limits were used and what search strategies were executed. But there is another layer of information that matters during search development. Why were these concepts chosen? Which terms produced unexpected results? Was a query revised because it retrieved too much noise, because it failed to recover known relevant work, or because its translation behaved differently in another database? When several candidate searches were tried, why was one retained and another abandoned?
Those questions are often answerable while the work is happening because the researcher remembers the sequence. They become much harder to answer later. Search Check therefore treats search development itself as something that can be recorded. The aim is not to preserve every keystroke. It is to preserve the consequential parts of the reasoning that produced the search eventually used for research.
What happens while a search is being developed
Consider a researcher looking for work on how education develops a particular human capacity. The first query may combine a term for education with a term such as “capacity,” only to retrieve papers in which capacity means institutional capacity, hospital capacity or electrical capacity. The researcher narrows the terminology and gets a more coherent set of results, but a paper already known to be central to the topic is absent. A second database retrieves it; the first does not. Another revision recovers the paper but introduces a new cluster of irrelevant results. None of these outcomes, by itself, tells the researcher what the final query should be. Together, however, they provide evidence about how the search is behaving.
Ordinarily, much of that evidence remains informal. The researcher remembers why a term was removed, why a concept was split, which database behaved unexpectedly and which known paper exposed a weakness in the query. Search Check externalizes those parts of search development. It separates the intended search object from the query used to represent it, and the conceptual query from its database-specific implementations. It can then connect those objects to retrieval results, relevance judgments, benchmark publications, diagnoses and revisions.
This matters because a search failure can have several causes that look similar from the outside. A known relevant paper might be missing because the query does not represent the concept well enough, because the database-specific translation changed its meaning, because a filter excluded the record, or because the database does not index it. Treating all of those as “the search missed a paper” hides the methodological distinction. The point is not to eliminate uncertainty, but to preserve enough information for the researcher to investigate what kind of problem they are actually facing.
A structured process without an automated verdict
We do not want the tool to decide that a search is “good,” that a paper is relevant, or that a benchmark set represents the complete literature. Those are substantive research judgments. The role of the software is narrower: make the objects, tests and decisions easier to record consistently so that the researcher can inspect them and another person can later understand what happened.
That distinction shapes the workflow. A researcher can state what a search is intended to retrieve, preserve candidate query versions, run or document database-specific translations, inspect retrieved records, record relevance judgments, identify publications they have independent reason to expect the search to recover, investigate misses, and record whether the evidence suggests keeping, revising, splitting or merging a query. The software structures the process; the researcher remains responsible for the interpretation.
Preserving the search as something that can be revisited
Reproducibility is sometimes discussed as if repeating the final query were enough. Repetition is useful, but it does not necessarily explain how the search was constructed or whether its known weaknesses were understood. A versioned search-development record allows a different kind of inspection. A collaborator can see why a term was removed. A reviewer can understand whether benchmark recovery was considered. The original researcher can return months later and recover the logic of an earlier decision instead of rediscovering it from scratch.
This is also why Search Check preserves predecessor and successor queries rather than simply overwriting an old query with a new one. Revision is normal. What matters methodologically is being able to see that a revision occurred, what prompted it, and what changed as a result. The history becomes part of the research record rather than disposable scaffolding around the final method.
What Search Check currently does
The current public version is an alpha, not a claim that search validation has been solved. It is designed around several recurring tasks: developing and comparing candidate queries; inspecting samples of retrieved records; recording why results are relevant, irrelevant or uncertain; building benchmark sets from publications that should be recoverable for stated reasons; distinguishing verified misses from non-indexing or unresolved indexing status where possible; preserving database-specific translations; and recording the decisions that lead from one query version to another.
The intended output is therefore more than a search box and a set of results. The longer-term aim is a research object that can preserve the search configuration, versions, judgments, benchmark information, diagnostics and run information needed to inspect how a search developed. That record can then sit alongside the review or study for which the search was created.
What Search Check does not establish
A structured record does not make a search complete, and a benchmark set does not become a gold standard because it has been entered into software. Search Check does not replace an information specialist, PRESS review, PRISMA-S reporting, title and abstract screening, critical appraisal or evidence synthesis. It cannot know the complete universe of relevant literature, and it should not make substantive relevance decisions on behalf of the researcher.
The broader proposition behind the tool is itself open to evaluation: whether making search development more explicit and inspectable improves researcher calibration, auditability, reproducibility or search quality. We think that is worth testing rather than assuming.
Built from research, released as research infrastructure
Search Check was not conceived as a general software product and then given a research use case. It emerged because our own studies required search-development infrastructure that we could inspect, reuse and improve. Once that infrastructure began to solve a recurring problem, keeping it as study-specific code made less sense than separating the general workflow from the particular research project that first required it.
This is part of a broader principle in how we want to work at How Intelligence Works. When doing research carefully produces infrastructure that may be useful beyond one study, we want to make that infrastructure open where we can. Openness, in this sense, is not only publishing source code. It is making more of the scientific process legible: what the system records, what it does not decide, how it changes, and where its limitations remain.
Methodological foundations
Search Check draws on established work rather than replacing it. PRESS provides a framework for peer reviewing electronic search strategies; PRISMA-S addresses transparent reporting of literature searches; and work on relative recall and benchmark/reference sets provides ways to examine retrieval performance when the complete universe of relevant literature is unknown. Search Check brings parts of these concerns into a prospective development workflow while keeping the limits of each approach visible.
Use Search Check
If you want to understand the methodological problem before using the software, read How Do You Know If a Literature Search Is Finding the Right Research?. If you want to work with the current alpha, you can open Search Check on Hugging Face. The source and development record are available in the Search Check repository on GitHub.
The tool will continue to change as we use it in real research and discover where the workflow is useful, where it is cumbersome, and where the underlying assumptions need to be challenged. That development history is part of the point: the infrastructure should be open to inspection in the same way that we want the research process it supports to be open to inspection.
