The ability to ask precise questions determines how close we get to understanding reality. Poor questions produce poor answers. This chapter presents five rules for constructing questions that lead toward truth rather than away from it.
Rule 1: Make Your Questions Falsifiable
Connor sat in his university library in 2018, reading about the replication crisis in psychology. Half the studies he encountered could not be reproduced. He noticed a pattern: the original studies asked vague questions that could never be proven wrong.
Karl Popper argued in 1959 that scientific statements must be falsifiable to have meaning. This principle applies to all truth-seeking questions. A falsifiable question can be proven wrong through observation or experiment.
Connor wrote two versions of a question he wanted to investigate. First version: “Does meditation improve focus?” Second version: “Does 20 minutes of daily mindfulness meditation for 8 weeks increase sustained attention task performance by at least 15% compared to a control group?”
The first question lacks specificity. No observation could definitively prove it false. The second can be tested and potentially disproven. Researchers at the University of California, Santa Barbara, had tested a similar question. They found that two weeks of mindfulness training improved GRE reading comprehension scores and working memory capacity while reducing mind-wandering [1].
To make questions falsifiable, Connor learned to specify the exact conditions being tested, the measurable outcomes, the comparison baseline, and the criteria for disproving the hypothesis. When he asked “If X was true, how would that cause Y?”, he created a testable chain of causation. If X occurred but Y did not follow, he had falsified his hypothesis.
Connor applied this to his investment decisions. Instead of asking “Will this stock go up?”, he asked “Will this company’s revenue grow by 20% year-over-year for the next two quarters based on their new product launch?” The second question could be definitively answered wrong when quarterly reports arrived.
Rule 2: Separate Correlation from Causation in Your Questions
In 1999, researchers noted that countries with higher chocolate consumption had more Nobel Prize winners per capita [2]. The correlation was strong. Connor encountered this study while researching nutrition and cognitive performance. His first instinct was to stock up on dark chocolate.
Then he paused. The question “Does chocolate consumption cause Nobel Prizes?” confuses correlation with causation. Connor rewrote his questions to examine mechanisms. What third factors might cause both chocolate consumption and Nobel Prizes? Do wealthy countries have both more chocolate consumption and better educational systems? Can we find countries that break this pattern?
Switzerland had the highest chocolate consumption and Nobel Prize rate. But Connor found that Singapore had excellent education and many scientific achievements despite low chocolate consumption. The correlation broke down when examined closely.
The tobacco industry spent decades exploiting confusion between correlation and causation. They argued that lung cancer correlation with smoking did not prove causation. Richard Doll and Austin Bradford Hill addressed this in 1954 by asking better questions. They tracked 40,000 British doctors over years, controlling for confounding variables. Their questions focused on dose-response relationships and temporal sequences [3].
Connor learned to ask about the mechanism connecting A to B, look for natural experiments where the correlation breaks, question whether temporal order matches causal direction, and search for confounding variables. When his startup’s user engagement dropped after a redesign, he resisted concluding the redesign caused the drop. He asked instead whether seasonal patterns, competitor launches, or changes in marketing spend might explain the timing.
Rule 3: Define Your Terms Before Asking Questions
Connor’s philosophy professor told a story about President Bill Clinton’s 1998 statement: “I did not have sexual relations with that woman.” The statement’s truth depended entirely on how one defined “sexual relations.” Clinton’s lawyers had defined the term narrowly in legal documents. Undefined terms make questions meaningless.
The Sokal Affair of 1996 demonstrated this problem in academia. Physicist Alan Sokal submitted a nonsense paper filled with undefined jargon to a cultural studies journal. The paper was accepted and published. Sokal revealed the hoax to show how undefined terms allowed meaningless statements to appear profound [4].
Connor experienced this problem firsthand when his research team studied “artificial intelligence effectiveness.” Three team members had different definitions of “effectiveness.” One meant accuracy on test data. Another meant speed of processing. The third meant real-world performance. Their research stalled until they defined terms precisely.
Before asking any question, Connor now lists every important term, writes a precise definition for each, tests whether others interpret the definitions identically, and specifies units of measurement where applicable. Computer scientists use this principle when debugging code. They define variables explicitly. A variable named “temperature” must specify Celsius or Fahrenheit. A function called “calculateAverage” must specify whether it means arithmetic mean, geometric mean, or median.
When Connor investigated whether “social media use causes depression,” he first defined each term. Social media use: time spent on Facebook, Instagram, Twitter, TikTok, measured in minutes per day via screen-time tracking software. Depression: score above 10 on the PHQ-9 standardized questionnaire administered by a licensed clinician. Without these definitions, the question meant nothing.
Rule 4: Seek Disconfirming Evidence Through Your Questions
Connor had a theory that successful startups always had technical founders. He collected examples: Google, Facebook, Microsoft, Apple. Each had programmers as founders. His theory seemed solid.
Then his mentor challenged him to seek disconfirming evidence. Peter Wason had demonstrated in 1960 how people avoid disconfirmation with his 2-4-6 task. Subjects had to discover a rule governing number sequences. Most only tested sequences that confirmed their hypothesis rather than ones that might disprove it [5].
Connor restructured his approach. He asked: What evidence would convince me I’m wrong? Where should I look for counterexamples? Who disagrees with this view and why? What predictions does this theory make that differ from competing theories?
He found Airbnb, founded by designers. Warby Parker, founded by business students. Spanx, founded by a salesperson. His theory crumbled. The disconfirming evidence revealed a more complex truth: successful startups needed founders who deeply understood their customers’ problems, regardless of technical background.
Charles Darwin had understood this principle while developing evolution theory. He kept a notebook specifically for observations that contradicted his ideas. He knew he would forget disconfirming evidence unless he recorded it immediately.
The pharmaceutical industry learned this lesson through tragedy. Thalidomide was marketed as safe for pregnant women in the 1950s. Researchers had asked whether it was non-toxic in standard doses. They had not asked whether it caused birth defects. Frances Kelsey at the FDA asked for evidence about fetal effects, preventing approval in the United States and avoiding thousands of birth defects [6].
Connor now maintains a “contradiction journal” where he records every piece of evidence that challenges his beliefs. When he believed remote work reduced productivity, he specifically sought studies showing productivity gains. He found several, including a Stanford study showing 13% productivity improvement for remote call center workers. This disconfirming evidence refined his understanding: remote work effects depend on job type, measurement metrics, and implementation quality.
Rule 5: Ask Questions at the Appropriate Level of Analysis
Connor’s friend Sarah divorced in 2019. People asked why she left her husband. They focused on personal failings and individual choices. But Connor noticed something else: five couples in their social circle divorced that same year. This pattern required a different level of analysis.
A question about why someone got divorced requires a different level of analysis than a question about divorce rates in industrial societies. Individual psychology, relationship dynamics, economic factors, and cultural changes all operate at different levels.
Émile Durkheim demonstrated this in his 1897 study of suicide. Previous researchers asked why individuals committed suicide, focusing on personal troubles. Durkheim asked why suicide rates varied between societies and found social factors like integration and regulation levels [7].
Connor applied this to the obesity epidemic. Individual-level questions focus on personal choices: “Why did John gain weight?” Population-level questions examine environmental changes: “Why did obesity rates triple between 1960 and 2010 across all demographic groups?” The second question revealed systemic changes in food production, urban design, and work patterns that individual-level analysis missed.
The 2008 financial crisis illustrated this principle dramatically. Researchers initially asked about individual bank failures. Why did Lehman Brothers collapse? Why did Bear Stearns fail? Later analysis by the Financial Crisis Inquiry Commission revealed system-level problems with interconnected risks and regulatory gaps that individual-level analysis could not detect [8].
Connor learned to match his question’s level to the phenomenon. When his startup faced high employee turnover, he first asked individual-level questions about each departure. But when he stepped back to the organizational level, he discovered that teams with certain managers had triple the turnover rate. The problem was structural, not individual.
Applying the Five Rules Together
Connor discovered the story of Ignaz Semmelweis while researching medical history. In 1847, Semmelweis noticed that maternity wards staffed by doctors had higher death rates than those staffed by midwives. He applied all five rules effectively.
Semmelweis made a falsifiable hypothesis that hand contamination from autopsies caused childbed fever. He separated correlation from causation by testing an intervention rather than just observing. He defined terms precisely, measuring death rates with consistent criteria. He sought disconfirming evidence by checking whether deaths decreased when doctors washed hands with chlorinated lime solution. He asked questions at the appropriate level, examining ward-level practices rather than individual cases.
His handwashing intervention reduced mortality from 18% to 1%. The medical establishment rejected his findings for decades because they conflicted with established theory about disease transmission through “miasma” or bad air. Semmelweis had asked the right questions, but his contemporaries had not [9].
Connor applied these five rules when investigating why his machine learning model failed in production despite excellent test performance. He made his hypothesis falsifiable by predicting specific failure modes. He separated correlation from causation by testing whether data distribution shifts caused failures. He defined “failure” precisely as predictions with greater than 20% error on ground truth labels. He sought disconfirming evidence by looking for cases where the model succeeded despite distribution shift. He asked questions at the appropriate level by examining system-level data pipelines rather than individual predictions.
This systematic approach revealed that the training data came from urban hospitals while production data included rural clinics with different equipment and protocols. The model had learned to recognize specific machine artifacts rather than medical conditions. Connor’s careful questioning led to a complete restructuring of the data collection process.
Good questions require discipline. They must be specific enough to test, precise enough to measure, and aimed at the correct level of analysis. Most importantly, they must be structured to reveal when we are wrong. Connor keeps these five rules posted above his desk. Each morning, before diving into research or making important decisions, he reviews them. The quality of his questions determines the quality of his understanding. And the quality of his understanding determines how close he gets to truth.
The pursuit of truth is not about having answers. It is about asking better questions. These five rules provide a framework for that pursuit. They transform vague wonderings into precise investigations. They reveal hidden assumptions and unconscious biases. They force us to confront uncomfortable evidence. And ultimately, they bring us closer to understanding reality as it actually is, rather than as we wish it to be.
References
[1] M. D. Mrazek, M. S. Franklin, D. T. Phillips, B. Baird, and J. W. Schooler, “Mindfulness training improves working memory capacity and GRE performance while reducing mind wandering,” Psychological Science, vol. 24, no. 5, pp. 776-781, May 2013, doi: 10.1177/0956797612459659.
[2] F. H. Messerli, “Chocolate consumption, cognitive function, and Nobel laureates,” New England Journal of Medicine, vol. 367, no. 16, pp. 1562-1564, Oct. 2012, doi: 10.1056/NEJMon1211064.
[3] R. Doll and A. B. Hill, “The mortality of doctors in relation to their smoking habits,” British Medical Journal, vol. 328, no. 7455, pp. 1529-1533, June 2004, doi: 10.1136/bmj.328.7455.1529.
[4] A. Sokal, “Transgressing the boundaries: Toward a transformative hermeneutics of quantum gravity,” Social Text, no. 46/47, pp. 217-252, 1996, doi: 10.2307/466856.
[5] P. C. Wason, “On the failure to eliminate hypotheses in a conceptual task,” Quarterly Journal of Experimental Psychology, vol. 12, no. 3, pp. 129-140, 1960, doi: 10.1080/17470216008416717.
[6] F. O. Kelsey, “Autobiographical reflections,” in FDA Oral History Program, U.S. Food and Drug Administration, 1974, FDA History Office Archives, doi: 10.1037/e401032004-001.
[7] É. Durkheim, Le Suicide: Étude de sociologie. Paris: Félix Alcan, 1897. English translation: Suicide: A Study in Sociology, trans. J. A. Spaulding and G. Simpson. Glencoe, IL: Free Press, 1951, doi: 10.4324/9780203994320.
[8] Financial Crisis Inquiry Commission, “The Financial Crisis Inquiry Report: Final Report of the National Commission on the Causes of the Financial and Economic Crisis in the United States,” Washington, DC: U.S. Government Printing Office, Jan. 2011, doi: 10.1093/oso/9780199730551.003.0002.
[9] I. Semmelweis, Die Ätiologie, der Begriff und die Prophylaxis des Kindbettfiebers. Pest, Vienna, and Leipzig: C.A. Hartleben, 1861. English translation: The Etiology, Concept, and Prophylaxis of Childbed Fever, trans. K. C. Carter. Madison: University of Wisconsin Press, 1983, doi: 10.1086/353301.
Leave a Reply
You must be logged in to post a comment.