Risks and failure modes

A research protocol can improve discipline without guaranteeing good judgement.

The central danger is procedural substitution: following the protocol becomes a proxy for thinking about the research. A well-documented workflow can still converge on a weak, conventional or substantively misplaced argument.

The protocol should therefore be treated as a scaffold for judgement, not an authority over judgement.

1. Delegation creep

Agentic AI makes it increasingly easy to delegate a chain of tasks rather than a single bounded task.

A researcher may begin by delegating code drafting or literature triage and gradually allow the system to:

select the next analysis;
decide which robustness check matters;
interpret the output;
draft the claim;
respond to criticism;
revise the manuscript.

The workflow can still contain a nominal human-in-the-loop while substantive monitoring has disappeared.

Warning signs

the researcher cannot explain why the next analytical step was chosen;
outputs are accepted because they look coherent rather than because they were checked;
several AI-generated stages depend on one another before any human inspection occurs;
“the model suggested it” becomes the operational reason for a research decision.

Countermeasure

Keep consequential decision rights human-controlled.

For each consequential step, require a researcher to be able to state:

what decision is being made;
what evidence is relevant;
what alternatives were rejected;
what would make the decision change.

Human monitoring must be substantive rather than ceremonial.

2. The illusion of oversight

A completed charter, log or audit can create a false sense that the research has been governed.

Documentation is evidence about a process. It is not evidence that the process was intellectually adequate.

A researcher can complete every form in this resource and still:

misunderstand the substantive problem;
use the wrong model;
fail to inspect the data;
overlook a better interpretation;
accept a weak argument because it has been carefully documented.

Countermeasure

Every governance record should point to a real research judgement, not merely show that a box was completed.

If a record contains no decision, refusal, challenge, evidential boundary or change in interpretation, ask whether the record is doing useful work.

3. Novelty erosion through convergence pressure

Repeated AI-assisted revision can pull a paper towards familiar disciplinary language and familiar claims.

The same risk appears when adversarial review is repeated many times. Each review round can make an individual objection easier to answer while gradually removing the feature that made the work distinctive.

A project may therefore become increasingly defensible and increasingly ordinary at the same time.

A recurring warning from methodological work

In iterative work on time-dependent Ball Mapper (TDABM), an approach that compares local neighbourhood representations across time, repeated adversarial review has at times pushed the manuscript away from the novel methodological contribution. Distinctive claims can be narrowed, rewritten or removed until the manuscript approaches a disciplinary mean.

The problem is not that every narrowing is wrong. Some claims genuinely need to become smaller.

The problem is failing to distinguish:

a correction required by evidence

from

a convergence towards what is easiest for a reviewer or language model to recognise as conventional.

Countermeasure: novelty-preservation check

Before a major revision, record:

what is genuinely novel;
what evidence supports the novelty;
which aspects are non-negotiable unless the evidence changes;
which claims are provisional and may legitimately shrink.

After the revision, ask:

Did the evidence make the contribution smaller, or did the revision merely make the contribution more familiar?

Novelty should never be protected from valid criticism. It should be protected from accidental disappearance.

4. Salience substitution

Generative AI often writes most fluently about the feature that is easiest to describe.

The easiest feature to narrate is not always the most important feature of the method.

TDABM example

When writing about TDABM, AI-assisted prose can gravitate towards overlap between local groups because overlap is intuitive and rhetorically accessible.

The methodological object, however, is built around the balls (local neighbourhoods around reference observations): how local neighbourhoods are constructed, what each ball represents, how observations enter those neighbourhoods and how the resulting cover carries information.

Overlap matters, but an account dominated by overlap can displace attention from the balls themselves.

The prose can therefore remain technically related to the method while slowly changing what the method appears to be about.

Countermeasure: object-of-analysis check

At every substantial rewrite, ask:

What is the primary methodological object?
What feature is merely a consequence or property of that object?
Which concept now receives the most words?
Is that allocation substantively justified?

Writing quality should not determine conceptual priority.

5. Adversarial review becomes the optimisation target

Adversarial review is valuable when it exposes a real threat to validity, identification, interpretation or presentation.

It becomes dangerous when the research starts optimising against an imagined reviewer rather than answering the substantive question.

Repeated simulated review can produce an endless loop:

claim → objection → qualification → new objection → further qualification → weaker claim

Eventually, the paper may contain little that an adversarial reviewer can attack because it contains little that is substantively interesting.

Countermeasure

Classify every major objection before revising:

validity problem — evidence or design must change;
inferential problem — claim must change;
clarity problem — representation must change;
scope disagreement — may require explanation rather than concession;
preference for convention — does not automatically require revision.

Do not treat all reviewer discomfort as evidence of error.

6. Model consensus is mistaken for independent corroboration

Several AI agents can agree because they share similar training distributions, disciplinary conventions and rhetorical preferences.

Agreement among agents is therefore not equivalent to:

independent empirical replication;
independent theoretical derivation;
independent human expert judgement;
convergence across genuinely different evidence sources.

Countermeasure

Ask whether apparent corroboration is epistemically independent.

When independence matters, introduce a genuinely different source of challenge: another method, primary evidence, a differently situated expert, an alternative theoretical frame or a falsification test.

7. Premature claim freezing

A claim-preservation contract is designed to stop prose from silently changing an authorised argument.

Used blindly, the same mechanism can prevent a better argument from emerging.

Research remains capable of changing after a contract has been written.

Countermeasure

Treat the contract as versioned and revisable.

A claim boundary may change when:

new evidence appears;
an identification problem is discovered;
a better theoretical distinction is found;
the research question itself legitimately changes.

The requirement is not immutability. The requirement is an explicit reason for the change.

8. Provenance becomes bureaucracy

Reason provenance is selective by design.

Recording every prompt, response, edit and keystroke can create a large archive with little discriminating value. Documentation burden may then crowd out analysis.

Countermeasure

Record what is needed to support claim-relevant attribution.

Prioritise:

consequential decisions;
origin of important questions or interpretations;
explicit refusals and constraints;
moments where evidence changed the claim;
transitions between generated artefact and verified evidence.

More documentation is not automatically better provenance.

9. Writing governance homogenises disciplinary voice

A manuscript-governance system can become too aggressive.

Repeated filtering for clarity, defensibility and conventional structure may remove:

disciplinary vocabulary;
unusual but useful conceptual distinctions;
authorial voice;
productive ambiguity where the phenomenon itself is unsettled;
forms of argument that are appropriate to one discipline but unfamiliar to another.

Countermeasure

Distinguish between:

claim-preserving clarity

and

stylistic normalisation.

A revision should be rejected when it makes the prose more conventional by erasing a distinction that matters to the research.

10. The human monitor lacks the capacity to monitor

“Human-in-the-loop” is meaningful only when the human has enough expertise, time and access to evaluate the relevant output.

A researcher cannot provide effective oversight merely by clicking approval.

Countermeasure

State the human competence boundary explicitly.

If the researcher cannot independently evaluate an output, the workflow needs another verification route: collaborator expertise, primary-source checking, reproducible execution, external review or a more limited claim.

A protocol for resisting the protocol

Before following any step mechanically, ask five questions:

What substantive judgement is this step supposed to protect?
Could the same step make the work more conventional without making it more valid?
Has AI shifted attention towards what is easiest to write rather than what is most important?
Is human monitoring real, or merely present in the workflow diagram?
Would I still make this decision if the protocol, model or simulated reviewer had not suggested it?

A strong research protocol should make it easier to disagree with the AI, with the reviewer, and occasionally with the protocol itself.