AI Research Methods and Scientific Practice

A research hypothesis in AI is a specific, testable claim about how a system will behave — not a general hope that 'AI will get better.' One of the field's founding documents illustrates the difference. The 1956 Dartmouth workshop proposal, written by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, is remembered as the document that named the field, but it is also a template for how to write a research hypothesis: it laid out a specific, if audacious, conjecture — that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it — and proposed a concrete, time-bounded study to make progress on that conjecture. Notice what the proposal does not do: it does not simply assert that intelligent machines are coming. It commits to a claim specific enough that particular lines of work — game playing, language use, abstraction, self-improvement — could count as evidence for or against it. Modern AI research proposals are held to the same standard in miniature: a hypothesis worth testing must specify, in advance, what result would count as support and what result would count against it.

Before that hypothesis can be tested, a researcher has to establish that it is actually new — or, if not new, that it addresses a genuine gap or contradiction in what is already known. This is the job of the literature review: a systematic survey of prior work that situates a proposed study relative to existing findings, so that effort is not spent re-discovering a known result or re-running an already-settled comparison. In AI, the great majority of vetted, peer-reviewed findings appear in a small set of venues — conferences such as NeurIPS, ICML, and ICLR, and journals such as JAIR — whose review processes are the field's main quality filter before a claim is treated as established. A careful literature review does more than list prior papers; it identifies which claims were rigorously supported (large-scale experiments, released code, statistically significant results) versus which were preliminary or contested, because a new hypothesis that only conflicts with an unreplicated preliminary claim is making a much weaker contribution than one that overturns a well-established, frequently reproduced result. Skipping this step is how research groups waste months rediscovering effects that were already known, or quietly reproduce a mistake the original literature had already corrected.

Once a hypothesis is formulated and situated in the literature, it has to be tested through experiments designed so the result actually bears on the hypothesis — and reported so another team can check the work. The NeurIPS conference's paper checklist, one of the most widely used reproducibility instruments in AI research, makes the required standard explicit. It asks authors to confirm that the claims made in a paper's abstract and introduction accurately reflect what the experiments actually show, rather than overstating generality beyond what was tested. It requires statistical rigor: reporting error bars, confidence intervals, or significance tests, with a clear statement of how they were computed, rather than a single number presented as if it were noise-free. It requires full experimental detail — data splits, hyperparameters, and how those hyperparameters were selected — because a result that depended on an unreported, hand-tuned setting is not really the same result someone else can obtain. And it requires disclosure of compute resources (hardware type, memory, and runtime), since an unreported compute budget can make a comparison between two methods misleading if one was given far more tuning time than the other. Crucially, the checklist's philosophy is that transparency matters more than perfection: honestly answering 'no' with a justification is treated as acceptable, while an unsupported 'yes' is not.

Reproducibility itself is not one property but several, and the ACM's Artifact Review and Badging policy — used across many computing conferences to formally evaluate reproducibility — distinguishes them precisely, following the International Vocabulary of Metrology. Repeatability means the original team can reliably repeat their own measurement using the same procedure and equipment. Reproducibility means a different, independent team can obtain the same result using the original team's own artifacts (their exact code and data) — a weaker form of external check, since it still relies on the original implementation. Replicability means a different independent team obtains the same result using their own, independently developed artifacts, tackling the same problem from scratch — a stronger form of confirmation, because it rules out the possibility that a subtle bug in one specific codebase produced an illusory effect. ACM's badging scheme rewards each stage separately: an 'Artifacts Available' badge for sharing code and data in an archival repository, an 'Artifacts Evaluated' badge (Functional or the more thorough Reusable level) for an independent audit of that material, and a 'Results Reproduced' or 'Results Replicated' badge for an independent team actually re-obtaining the claimed result. Scientific communication in AI is judged, increasingly, by how many of these stages a claim has actually passed.