Is Information Systems Research Going Through an Epistemological Crisis?
I have followed Professor Cathal Doyle’s work at Victoria University of Wellington for some time. I am also taking an epistemology course with Dr. Jay Foster and listening to Stephen Straker’s lectures. So, it was interesting to come across Dr. Doyle’s paper, What Science Is, Information Systems Research Is Not.
My doctoral research focuses on Design Science Research (DSR) and Conceptual Modeling. I am still early in that work, but Dr. Doyle’s argument gives me a useful introduction to epistemology and the question of what makes research scientific.
The central question is simple:
What makes a field scientific?
Dr. Doyle’s argument raises a related question:
Can a field produce the products of science without following the practices that make science possible?
What does science require?
Different philosophers of science give different answers. Several ideas are especially relevant here.
| Philosopher & Year | Key Concept | What it requires |
|---|---|---|
| Karl Popper (1959) | Falsifiability | Scientific claims must allow empirical tests that could show them to be wrong. |
| Thomas Kuhn (1962) | Paradigms and anomalies | Science works within paradigms. Persistent anomalies can eventually force a paradigm change. |
| Imre Lakatos (1970) | Research programmes | Progressive programmes make risky predictions. Degenerating programmes mainly accommodate known facts. |
| Paul Feyerabend (1975) | Methodological pluralism | Science has no single universal method. It must still explain something real. |
| Robert K. Merton (1973) | Communism and organised scepticism | Researchers should share knowledge openly and subject claims to systematic scrutiny. |
These philosophers disagree on many points. They share one important idea.
Science must connect claims to reality and provide ways to discover when claims fail.
- Popper emphasizes testability.
- Kuhn emphasizes anomalies.
- Lakatos emphasizes risky predictions.
- Feyerabend emphasizes real-world explanation.
- Merton emphasizes openness and scrutiny.
This raises a problem for Information Systems.
A citation base is not a knowledge base
A citation base consists of textual dependencies between papers. Researchers cite earlier work, identify gaps, and build new studies on existing claims.
A knowledge base consists of claims that have survived testing, verification, replication, or rejection.
A large citation base does not necessarily produce a reliable knowledge base.
This creates the possibility of an epistemic crisis:
A discipline faces an epistemic crisis when it has a large published literature but lacks sufficient mechanisms to determine which claims are reliable.
The problem is therefore not simply whether individual studies are good or bad. The problem is whether the discipline can systematically discover which claims deserve trust.
Gap-spotting and truth-checking
Information Systems often organizes research around gaps in existing literature.
Researchers ask:
What has not been studied?
That question can generate new research. It can also create a feedback loop.
Paper A makes a claim. Paper B identifies a gap in Paper A. Paper C extends Paper B. Paper D builds on Paper C.
Researchers can continue this process without testing Paper A’s original claim.
This creates a distinction between gap-spotting and truth-checking.
- Gap-spotting expands the literature by identifying unanswered questions.
- Truth-checking tests whether existing claims are actually correct.
Researchers may expand the literature without checking its foundations. The literature can become more sophisticated without becoming more reliable.
Five substituted products
Dr. Doyle identifies a deeper problem. Information Systems can mistake visible research products for the scientific practices they represent.
| IS product | Scientific practice | Problem |
|---|---|---|
| Publication | Knowledge | Publication becomes the endpoint instead of the starting point for verification. |
| Peer review | Verification | Review can assess coherence and plausibility without establishing whether claims are true. |
| Method section | Scrutiny | A researcher’s description can replace independent inspection of data, code, and materials. |
| Citation | Cumulativeness | Citation counts can create the appearance of cumulative knowledge without testing underlying claims. |
| Practical implications | Consequence | Claims about potential usefulness can replace evidence from actual use. |
The products themselves are not the problem. Publication, peer review, methods sections, citations, and practical implications all have legitimate purposes. The problem occurs when these products become evidence that the underlying scientific practice has occurred.
- A paper was published, so we treat its contribution as knowledge.
- A paper was peer reviewed, so we treat its findings as verified.
- A paper has many citations, so we treat its claims as established.
Those conclusions do not necessarily follow.
What does this mean for design science research?
This question matters to me because my research area is DSR.
DSR aims to connect research with practical problems. Hevner et al. (2004) emphasized the importance of creating and evaluating artifacts that solve relevant problems. That should make DSR resistant to the problem Dr. Doyle describes.
However, DSR can still operate within the same academic system that Dr. Doyle found epistemologically flawed. Nagle et al. (2022) reviewed 111 DSR papers from basket journals. They found that 96 papers derived their research problems from academic literature rather than practical operations.
That finding raises an important question. If DSR aims to address practical problems, why do many DSR problems originate in the academic literature? The artifact may target the real world. The research problem can still come from the literature.
Hypernormalisation
Dr. Doyle uses the concept of hypernormalisation to describe this condition.
Alexei Yurchak used the term to describe late-Soviet society. People continued performing institutional routines even when those routines had become disconnected from reality. Dr. Doyle applies the concept to research.
Researchers produce papers for journals. They identify gaps, propose models, describe contributions, and discuss implications. The system rewards these outputs.
The question is whether the system rewards the practices that establish reliable knowledge to the same degree.
This distinction matters:
Research products: papers, peer review, citations, impact factors, and conference proceedings.
Scientific practices: openness, scrutiny, replication, correction, empirical testing, and exposure to real-world use.
Seven practices for checkable science
Dr. Doyle argues that Information Systems needs stronger institutional practices for checking claims.
| Principle | Description |
|---|---|
| 1. Openness | Researchers should make data, code, and materials available when possible. Others can then inspect the evidence. |
| 2. Scrutiny | Researchers should inspect the evidence behind published claims. Reading a methods section is not the same as checking the underlying work. |
| 3. Correction | The field should reward researchers who identify and correct errors in existing knowledge. |
| 4. Replication | Researchers should treat replication and reanalysis as normal parts of research. |
| 5. Use | Theories, models, and artifacts should face real-world use when their claims concern practical value. |
| 6. Failure | Research claims must face conditions under which they can fail. |
| 7. Consequence | Empirical failure should affect the status of a claim and the research built upon it. Without consequence, falsification becomes largely symbolic. |
Generative AI changes the problem
Generative AI makes this issue more visible.
Academic writing requires substantial effort. Researchers spend months producing literature reviews, methodological explanations, tables, and theoretical arguments. Large language models can now produce much of this form quickly.
This creates an important epistemological question: What does academic writing actually demonstrate?
If a machine can reproduce the form of academic research, then the form itself provides weaker evidence of intellectual contribution. The important question becomes whether the research produced reliable knowledge. AI exposes a distinction that already existed.
Producing research-shaped text is not the same as producing knowledge.
Recursive knowledge production
There is another concern.
Researchers can use AI to generate papers. Reviewers can use AI to evaluate papers. Future AI systems can train on the resulting academic text.
Liang et al. (2024) examine the use of language models in academic evaluation. Shumailov et al. (2024) describe model collapse, where models trained recursively on synthetic data can lose information from the original data distribution.
The same idea raises a concern for academic knowledge. If academic text increasingly generates more academic text, the literature could become increasingly self-referential.
The field could produce more polished writing while becoming less connected to the reality it studies.
Conclusion
I am still early in my doctoral research, so I do not think I can answer whether Information Systems is experiencing an epistemological crisis.
Dr. Doyle’s argument does, however, give me a useful way to think about the problem.
A discipline can have prestigious journals, peer review, sophisticated methods, extensive citation networks, and thousands of publications.
Those features do not guarantee reliable knowledge.
The deeper question is whether the discipline has mechanisms that can find out which claims are reliable. This may be the central epistemological issue.
Information Systems may not need more research products. It may need stronger practices for checking the knowledge those products claim to produce.
For me, that makes the question especially relevant to DSR. If an artifact claims to solve a real problem, the important question is not whether the paper describing it looks rigorous.
The important question is whether the artifact works when reality gets involved.