When Observations Fail to Become Data

I recently experienced a brief, confrontational interaction with a stranger. I could describe what happened, but I could not confidently explain the person’s intentions. I also could not tell whether the incident warranted further attention. Reporting might preserve something useful. It might also start an administrative process disproportionate to the evidence.

I chose not to report formally. That left me wondering: Why must I decide what an observation means before the institution can assess it?

What does an observation establish?

Consider three statements about a hypothetical encounter:

Each statement goes further than the last. They could all be correct, but evidence for the words spoken does not establish the person’s intentions or their significance to an institution.

The boundary between observation and interpretation is not clean. Norwood Russell Hanson and Thomas Kuhn challenged the idea that observation provides a wholly neutral foundation for science. Concepts, expectations, and background assumptions influence what we notice and how we describe it.

We can still distinguish a relatively direct description from a claim that requires more inference. That distinction disciplines what we say the evidence supports. An observer might have enough evidence to describe an encounter without having enough to classify its significance.

The difficulty begins when a reporting system requires the stronger claim.

Four asymmetries in reporting

The observer, the subject, and the institution hold different information. They also face different consequences.

An observer might witness one encounter while the institution holds records of others. Separately, those observations could be inconclusive. Together, they might reveal a pattern. The institution cannot compare observations it never receives.

But aggregation can also mislead. Several misunderstood encounters do not become well-founded evidence simply because they appear together. An institution must manage both risks: losing relevant observations and assigning unwarranted significance to ambiguous ones.

Why people remain silent

In Exit, Voice, and Loyalty, Albert Hirschman examined responses to organizational decline. Speaking up offers an alternative to leaving, but it carries costs and depends on what people expect it to achieve.

Morrison and Milliken’s work on organizational silence asks how organizational conditions encourage people to withhold information about problems. Silence can emerge from structures and shared beliefs, beyond any one person’s preference.

Research on teams and employee voice shows why that environment matters:

• Amy Edmondson’s study of 51 manufacturing teams found that psychological safety was associated with learning behavior.
• Detert and Burris studied 3,149 employees and 223 managers in a restaurant chain. Managerial openness was associated with employee voice, with psychological safety helping explain the relationship.

These studies concern team learning and employee voice. They do not directly establish how people use formal incident systems, or explain my encounter with a stranger. They give us reasons to examine how expectations about an organization affect communication.

That becomes a data problem when an institution treats its records as a picture of underlying events. Few reports might indicate few incidents, little willingness to report, or both. The database alone may not distinguish them.

Game theory and anticipated consequences

Crawford and Sobel’s model of strategic information transmission studies an informed sender and a receiver whose subsequent actions affect them both. Their model shows how differences in interests can limit informative communication.

Institutional reporting has a related structure, with a third party whose interests also matter:

Suppose an observer thinks an encounter is worth recording but does not warrant an investigation. If they expect reporting to trigger an investigation automatically, they may stay silent. The institution might have preferred to receive the observation and assess it before deciding what to do. The subject also has an interest in preventing ambiguity from becoming an unsupported allegation.

Nobody needs to act irrationally for information to disappear. Each party can have reasonable concerns, yet their interaction can produce an outcome nobody wanted.

This describes a possible mechanism. Establishing a strategic equilibrium would require explicit payoffs, beliefs, and rules. We do not need to assign numerical probabilities to explain why an expected response might discourage disclosure. Whether it does so in a particular institution remains an empirical question.

The diagram shows the proposed reporting process and its feedback:

Figure 1. Solid arrows show the process and inputs. Dashed arrows show how anticipated consequences influence reporting and how decisions shape future expectations. The diagram represents a proposed mechanism, not an estimated causal model.

What missing reports do to the data

Donald Rubin’s work on missing data establishes conditions under which an analysis can ignore the process that made observations unavailable. Reporting presents a related selection problem: events enter the database through people’s decisions about disclosure.

For a simplified system, define three sets:

• E: events that occur.
• O: events observed by potential reporters.
• R: observed events that are formally reported.

This model excludes reports based solely on indirect evidence and events detected independently of these observers. It also describes which events enter reporting, rather than guaranteeing that a report accurately represents them.

The probability that an event is reported can vary across people and circumstances:

Imagine two departments with comparable rates of problematic interactions. People in one trust management and report frequently. People in the other anticipate negative consequences and remain silent. Comparing their incident counts could make the second department look healthier.

That comparison risks selection bias. The institution sees events filtered through a reporting process. Estimating event frequency requires assumptions or independent evidence about that process. Selective reporting does not automatically establish a particular missing-data mechanism; that depends on what the analysis observes and what disclosure depends on.

When a schema demands certainty

Once someone submits an observation, the system must represent it. Bowker and Star’s Sorting Things Out examines how classifications shape social and organizational life. Categories make some distinctions visible while obscuring others.

An incident form might require the reporter to choose harassment, misconduct, safety concern, or other. The person may know what happened without knowing which category applies. The form still demands an answer.

I would call this premature epistemic commitment: asking someone to claim more about an event’s meaning than their evidence supports. They might choose a stronger category to secure attention, choose an uninformative residual category, or abandon the form. Each response changes what the institution receives.

The issue goes beyond interface wording. The conceptual model determines whether the system can represent an observation whose significance remains uncertain.

Separate observations, interpretations, and decisions

One design response is to represent three entities:

One observation can support several interpretations. New evidence can change an interpretation. A decision can link to the evidence and reasoning available when it was made.

The model should also allow accounts to be corrected and disputed without losing their history. Storing a statement as an observation records what someone reported; it does not certify the statement as true.

This separation need not delay an urgent response or justify unlimited retention. Privacy, access controls, retention periods, and reporting obligations still constrain the system. Its purpose is to make the steps from evidence to action explicit.

Tracing how evidence becomes a decision

The W3C PROV framework describes entities, activities, and agents involved in producing information. Provenance helps answer where a record came from, what transformed it, and who or what was responsible.

In a reporting system, that could mean distinguishing:

• The observation supplied by a reporter.
• An administrator’s interpretation.
• Additional evidence received later.
• The decision based on the evidence available at that time.

Think of this as keeping the working behind a calculation. A final answer alone cannot show which inputs changed or where a mistaken assumption entered.

If the latest classification overwrites the original description, an institution may struggle to reconstruct its reasoning. A later conclusion can begin to look as though the first observation established it. Preserving the links supports traceability of inference: the ability to follow how evidence became a conclusion.

Different thresholds for different decisions

My encounter left me with a description and unresolved questions. I could report what I experienced without knowing the stranger’s intentions. Whether that account warranted an institutional response was a further judgment.

Recording, concluding, and acting serve different purposes. Institutions need appropriate thresholds for each, alongside limits on what they collect and retain.

The evidence needed to record an observation need not be the evidence needed to act on it. A reporting system should leave room for that difference.