In data analysis, we often rely on the conclusions drawn from research studies to guide our decisions. However, not all studies are as objective as they may seem. One subtle yet dangerous error frequently distorts the outcomes of studies is the intention-to-treat error. At first glance, this error may appear as a minor oversight, but it can lead to significant misinterpretations and flawed conclusions. Whether in medicine, business, or other fields, the intention-to-treat error can drastically alter our understanding of effectiveness and success. In this article, we’ll delve into what this error is, how it manifests in real-world scenarios, and why it’s crucial to recognize it before making decisions based on skewed data.

The Speeding Drivers Fallacy

The speeding driver example might seem like a paradox at first. On one hand, we’re told that speeding is dangerous and increases the likelihood of accidents. On the other hand, data from this example reveals that those driving faster experience fewer accidents than their slower counterparts. This seemingly contradictory finding results from a flawed data categorization, leading to a misleading conclusion. It exemplifies the intention-to-treat error by misplacing the results in the wrong group.

Let’s explore the concept in greater detail to understand why this happens. We categorize drivers into two groups: those who complete the 75-mile journey in under an hour (speeding drivers) and those who take longer to reach their destination (careful drivers). Now, if we look at the data, we can see that all drivers in the “reckless” category, who completed the journey in under an hour, could not have been involved in accidents. Why? Because they were simply driving too fast to encounter any accidents. The time spent on the road was so short that any risk of an accident was minimized.

In contrast, the slower drivers, categorized as “careful,” were exposed to potential hazards for longer because they spent more time on the road. Due to the extended duration of their travel, these drivers are more likely to be involved in accidents, increasing their chances of encountering traffic, weather-related incidents, or other potential dangers.

This illustrates how the intention-to-treat error works: by focusing only on the speed factor and ignoring the time spent on the road, the data gets misinterpreted. The slower drivers are classified as more accident-prone, even though the actual cause of their accidents is the time they spend on the road. This results in an inaccurate view of the correlation between speed and safety. The data, in essence, mislabels the “reckless” drivers as safe simply because they complete their trip faster, ignoring the larger context of how time spent traveling influences the likelihood of accidents. This fallacy shows the importance of considering all variables before concluding any dataset.

The Banker’s Debt Fallacy

The banker’s study on companies with debt versus those without illustrates how misleading data can be when improperly analyzed. The study found that companies with debt on their balance sheets had higher returns on equity and capital than firms funded only through equity. This seems like strong evidence that debt benefits business profitability at face value. However, a deeper look into the methodology and selection of companies involved in the study reveals a serious flaw: the intention-to-treat error.

To fully understand the error, we need to recognize the critical omission in the study: unprofitable companies generally do not qualify for loans. This is the hidden bias. The study did not consider that unprofitable firms that couldn’t secure loans were included in the “equity-only” category. These equity-funded companies may not have had enough capital to expand or leverage their operations, but they stayed afloat longer despite their unprofitability. However, these companies didn’t represent thriving businesses—they were simply in survival mode, with limited growth potential.

On the other hand, the firms that could secure loans, which were classified in the “debt” group, were generally healthier companies. However, the firms that eventually went bankrupt due to unsustainable debt levels were removed from the study entirely. This means that only the more successful, debt-laden companies remained in the “debt” category, distorting the comparison between the two groups.

This results in a skewed interpretation of the role of debt in business success. The study fails to recognize that those companies that went bankrupt—due to high levels of debt—were excluded from the results. The healthier companies that survived, despite the debt, are the ones that remain in the study, giving a false impression that debt inherently leads to higher profitability. The companies that failed due to debt would have created a very different picture if included. Thus, the intention-to-treat error leads to an oversimplified conclusion that borrowing is a straightforward route to success, neglecting the risks involved in over-leveraging a business.

The Pharmaceutical Study Error

The pharmaceutical study is an excellent example of how the intention-to-treat error can artificially inflate the effectiveness of a treatment. In this case, a new drug designed to reduce mortality from heart disease shows impressive results. Patients who took the drug regularly had a 15% mortality rate, while those on the placebo had a similar mortality rate. At first glance, this suggests the drug is no more effective than the placebo. However, a closer examination of the data reveals that the irregular intake group, who also received the drug, had a mortality rate of 30%. So, what’s going on here?

The crux of the issue lies in the categorization of patients. The “regular intake” group, which shows better outcomes, is likely made up of healthier patients who could adhere to the treatment regimen without significant side effects. These individuals were more likely to have better health outcomes regardless of the drug. On the other hand, the patients who could not keep up with the regular regimen likely faced severe side effects or their conditions worsened to the point where they couldn’t continue the treatment. These individuals were placed in the “irregular intake” category.

In this case, the patients who couldn’t take the drug as prescribed were more likely to have severe health issues. Their inability to adhere to the regimen was a consequence of their deteriorating health, not necessarily an issue with the drug itself. By categorizing them in the “irregular intake” group, the study falsely suggests that the drug’s effectiveness is the cause of the differences in mortality rates. What the data fails to capture is that the irregular intake group might have been disproportionately filled with sicker patients who simply couldn’t continue with the treatment. As such, the study’s conclusion—that the drug is effective—is based on the misleading assumption that the drug was the main factor in improving health outcomes rather than the patient’s preexisting health status.

This misinterpretation highlights the importance of considering patients’ behavior and health status in any medical study. When participants are categorized based on their ability to adhere to a treatment, it’s crucial to account why some patients may drop out or switch to irregular intake. The intention-to-treat error here stems from grouping those most likely to fail due to their health status with those who can continue the treatment, distorting the findings and leading to a falsely optimistic conclusion.

The Importance of Correct Data Analysis

Correct data analysis is foundational to obtaining accurate and meaningful results, especially in complex studies that involve human behavior or variable outcomes. The intention-to-treat principle is designed to prevent bias in study results by ensuring that all participants are accounted for, whether they complete the study or not. This is crucial because excluding individuals who drop out or fail to meet certain criteria can lead to a misleading representation of how effective or successful a treatment, intervention, or business strategy truly is.

In clinical trials, for example, the intention-to-treat principle ensures that all patients, regardless of whether they completed the treatment, are included in the final analysis. This approach is vital for understanding the real-world effectiveness of a drug or therapy. If a study excludes participants who dropped out due to side effects or other reasons, it risks overestimating the treatment’s benefits. The people who drop out might have had the worst experiences with the drug, and excluding them could make the treatment appear more tolerable and effective than it is.

Similarly, excluding companies that have failed or gone bankrupt in business studies or financial analyses can distort the findings. By only focusing on surviving businesses, studies may give a false sense of security to investors or entrepreneurs who think that adopting a particular strategy or practice will lead to success. The reality is that failed businesses are just as important to understand, as their failure often provides valuable lessons about the risks involved in certain business decisions or strategies.

Proper data analysis requires considering all participants, successes, and failures to ensure that conclusions are based on an accurate and comprehensive understanding of the situation. This is why the intention-to-treat principle is so crucial in research and analysis. It avoids skewed conclusions by ensuring that all data points—whether positive or negative—are included in the analysis.

Recognizing the Intention-To-Treat Error

Recognizing the intention-to-treat error is essential for evaluating the validity of any study or data analysis. This error can often be subtle, as it involves miscategorizing or excluding key data points that don’t align with the expected outcomes. To spot the intention-to-treat error, it’s important to examine the methodology of any study closely and ensure that all participants are accounted for, even if they drop out, fail to adhere to the treatment protocol or encounter failure.

When reviewing a study in medicine, business, or social sciences, look for signs that participants who didn’t meet the success criteria have been excluded. Are there reasons provided for their exclusion, or are they simply removed from the final results? If a study relies on data from only those who succeeded or adhered to a certain protocol, it’s likely that the results are skewed and may not reflect real-world outcomes accurately.

Additionally, pay attention to how failure is treated within the study. Failure should not be ignored or discarded—it is often the most valuable data point. Whether it’s a failed business, a patient who didn’t respond to treatment, or a project that fell short of its goals, understanding why these failures occurred is just as important as understanding the successes. The intention-to-treat error obscures this by omitting failures or misclassifying them, leading to an incomplete understanding of the overall situation.

By vigilantly and critically examining the methodology behind studies, we can better avoid the intention-to-treat error and ensure that the conclusions we draw from data are grounded in reality. This will lead to more informed decisions in medicine, business, or any other area that relies on data analysis.

Conclusion

The intention-to-treat error highlights the importance of thorough data analysis and careful interpretation. By miscategorizing or excluding data points based on success or failure, this error distorts the reality of a study, leading to conclusions that may not be applicable in real-world situations. Whether in business decisions, medical treatments, or financial strategies, it’s essential to consider the full picture and account for all participants, regardless of their outcomes. Recognizing and understanding the intention-to-treat error allows us to make better, more informed decisions that reflect the true complexities of the situations at hand. Always approach data critically and ensure that studies reflect the full spectrum of experience, both successful and failed.

This article is part of The Art of Thinking Clearly Series based on Rolf Dobelli’s book.