This situation illustrates what is wrong with a large swath of online reporting.
In their “me-too” eagerness to promulgate any scrap of ersatz news, none of these tech websites pointed out that this SquareTrade study, in its current form, is completely worthless for drawing conclusions about laptop reliability, scientific or otherwise. The problem lies in the unscientific nature of SquareTrade’s research.
The SquareTrade study utterly fails to provide any meaningful statistical analysis of its numbers. It finds “the average total failure rate of laptops to be 31% over 3 years”. An average can not stand on its own: to be of statistical use, it must be accompanied by at least a standard deviation of the data from which the average was taken. (A range would be helpful, too). SquareTrade’s sloppiness becomes more evident with a closer reading.
“Apple is in fourth place in laptop reliability” was the headline used by many tech websites to push this SquareTrade study. But the graph upon which the headline ranking is based (on page 6 of the report) was created using a linear projection from TWODATAPOINTS. Here’s why this is a problem: fitting a curve that will have any prognostic capabilities requires more than two data points. More data points mean more complications. So, to handle the extra data, the researcher/underpaid intern at SquareTrade must understand curve fitting, specifically the coefficient of determination (calculated to make sure the curve fits well). When you only have two data points to model, however, two things happen: (1) you can easily model the two points with a linear curve that perfectly fits the data (R2=1). (2) Your model is capable of predicting nothing. So, the SquareTrade authors have formed an inappropriate model based on sloppy data to make fallacious projections.
Ostensibly, SquareTrade put this 3 year projection in their paper to make the reliability difference between the laptop companies they analyzed seem greater. Despite the statistically ignorant data massaging, the difference between the most reliable laptop makers and fifth-most reliable is 2.7%. There is no information to tell us if this difference is statistically significant. If SquareTrade had an interest in scientifically rigorous data, they would report the standard deviation (commonly called error bars when used on graphs) and the range of their data. Instead, they provide colorful arrangements of numbers that illustrate little, prove nothing, and distract from sloppy, unscientific methodology.
What’s more, SquareTrade is a warranty company looking for customers to purchase insurance against exactly the topic of this study: laptops/portables breaking. This obnoxiously blatant conflict of interest is only mentioned as a tiny caveat in the Ars Technica piece. But, the damage has already been done: the study has been broadcast as fact across a multitude of tech websites.
Dissemination of this type of information (too low in quality to be called factual) is a symptom of a larger problem on the internet: the quantity of information available continues to rise while the quantity of factual data grounded in scientific methods fails to keep pace.
As a result, the ratio of factual, useful research to that of generally worthless information continues to dwindle.
The “professional” blogs exacerbate the overload with their echo chamber effect.
My wife Mykala is a woman who knows how to wear some sweatpants. As the owner of a veritable library of them (nearly the antithesis to my own wardrobe), Mykala reintroduced me to sweatpants through persuasion and sheer exposure.
In elementary school, I had these red sweatpants which I wore to school once. I’m sure no one said anything about my pants, but possessing the misplaced assumption that everyone was looking at me, ready to laugh, I was certain derisive snickers and outright insults were bound to rain down upon me. Mykala and I have a short-hand for this type of self-consciousnessness: I imagine a world where everyone has giant eyes — mercilessly following my every move. This (of course) is patently false now, as it was then. Even at my young age, I was unusually intent on details, and I think there was a small darn on one of the legs… as far as I can tell, all this freaked me out sufficiently to keep me out of sweatpants for about a decade afterwards. What a shame.
Now, returning home from school, I regularly put on my pair of sweatpants from my sister Katy — they are everything scrubs are not: warm, soft, stretchy, fuzzy, cozy.
When I first started designing as a hobby, I hated
everything I made. I knew it was terrible, and no matter
how hard I tried, I could never make it good enough for
myself. But I didn’t give up, and after a while something
clicked. I started to sort of like my work. But I am
still not satisfied; every day I reach higher, trying to
grasp the level of awesomeness that I can feel but can’t
recreate.
This is so unbelievably true. And the Ira Glass video about storytelling that Curtis links really really crystallized the concept for me. Of course its about satisfying your inner critic! My inner critic continues to drive me.
As Toyota built better cars than its competition for less
money, it won new customers. Some rivals matched its
successes (as Honda did); some lost market share (as
Detroit did). No such dynamic exists in health care.
William Lewis, a former director of the McKinsey Global
Institute who studies productivity, says that the
economic benefits from the various quality movements have
been quite large but that they are also largely in the
past. Most industries have incorporated Deming’s big
ideas and are now making only incremental progress.
“However, there is one big exception,” Lewis adds. “You
guessed it: health care.”
Why? In part, it is the faith
that patients have in their doctors. When people are
buying a car, they often consult Consumer Reports or Road
& Track. When they are choosing a place to have surgery,
they ask their doctor to recommend a surgeon and go to
the hospital where that surgeon works. Hospitals that
provide less than top-quality care are rarely punished in
the way that General Motors and Ford have been.
So, there’s no impetus for quality improvement. On top of that, when doctors must deviate from routine, they end up administering a battery of tests, which can increase costs dramatically. What’s the solution?
Reform theory
The basic idea of Dr. Brent Jame’s reform is most effective in these situations where doctors would normally run a bunch of tests — by establishing an exhaustive catalog of baselines for treatment (doses, protocols, equipment settings, et al.), the costs of treating patients becomes simultaneously lower and more effective. For example: inductions before 39 weeks of pregnancy were reduced from 30% to 2% of deliveries — evidence went against conventional practice, changed the flawed practice, and resulted in far fewer babies in intensive care.
Fee for health system
Wouldn’t such an obvious beneficial change be quickly adopted around the country? Not if it doesn’t make financial sense:
As long as doctors and hospitals are paid for each extra
test and treatment, they will err on the side of more care
and not always better care. No doctor or no single
hospital can change that. It requires action by the
government.
As articulated by this article, the solution becomes tricky in practice: let’s say you try to control costs by bundling — a set fee for different diseases/illnesses. BUT, this gives the hospital an incentive to turn away the sickest patients. The article’s response to this? “Medicare or private hospital groups would most likely monitor outcomes to make sure the incentives didn’t lead hospitals to skimp on care or turn away the sickest patients.” I don’t think the solution lies in more regulation, but instead in building a system with some inherent checks and balances. I do not know how that could be done.
As the article states, “…the only sensible strategy is to try anything that seems promising.”
Aggression scores increased in the rejected groups. But
the IQ scores also immediately dropped by about 25 per
cent, and their analytical reasoning scores dropped by 30
per cent.
“These are very big effects - the biggest I’ve
got in 25 years of research,” says Baumeister. “This
tells us a lot about human nature. People really seem
designed to get along with others, and when you’re
excluded, this has significant effects.”
The patent for Abacavir (ABC) (trade name Ziagen), a reverse transcriptase inhibitor (RTI) effective against some resistant strains of HIV, expires this December. Incidentally, a guest lecturer in microbiology came to us to speak about HIV. Here’s where it gets interesting:
Using public funds, researchers at the University of Minnesota analyzed primary literature and hypothesized a novel RTI. After testing revealed its efficacy, Abacavir was patented. GlaxoSmithKline then sold the drug as Ziagen… without proper rights to do so. Mark Yudof, president of the University in the late 90s (and a lawyer), decided to sue for royalties. Settlement: 400 million dollars. This is the largest intellectual property case in the U of M’s history.
The other prominent case centers around the Honeycrisp apple. After having one, many people call the Honeycrisp the best apple they’ve ever had. As noted by a grower: “It has a first-of-its-kind crispness that no previous apple has ever possessed, its juiciness is unsurpassed, and it can be held for an unusually long period of time, compared to other apple varieties, in prime condition.”
Now, I dont’ know the details of the suit over the apple — and I’m not that interested. The story of the apple, however, is rather remarkable. The University of Minnesota has been breeding apples for over 150 years and they “release” new breeds to apple growers fairly regularly: e.g. Haralson in 1922, Beacon in 1936. In 1960, crosses were done with existing apple breeds (one known: Keepsake, the other unknown) to produce, ostensibly, the first Honeycrisp seedling in 1961. Over the next decade plus, absolutely nothing remarkable was noted about the variety.
At the U of M orchard, in the late 1970s, The remaining Honeycrisp trees were marked for destruction. The entire Honeycrisp variety owes its existence to the fruit breeder David Bedford:
Apparently no one on the face of the earth tasted a
Honeycrisp apple from the time it was numbered in 1974
until Bedford picked some in 1983. Nine years! Bedford
himself never tasted fruit from the original tree. It was
out of existence for a number of years before the four
first-generation trees came into bearing, and they existed
based only on a very weak, non-descript 1974 report, the
last report ever made on the original tree’s fruit. When
Bedford marked the four trees for removal and then, upon
reconsideration, removed the discard tags that would have
forever obliterated Honeycrisp, he had never tasted the
apple.
And here we are: an incredible apple with an amazing taste — which is now the Minnesota state fruit, was very nearly tossed out as a part of regular orchard breeding program maintenance.
It seems to me that science is all about fortune favoring the prepared: nobody explicitly mapped out all the desired characteristics of an apple or an RTI drug — they make some educated guesses, and then watch for results.
