This one’s important.

The other day we had a post on when it’s ok to judge people by their worst.

Dmitri wrote in:

You know who I judge by their worst belief? Healers: doctors and such. If a doctor has one crazy health belief, I am out of there.

My mother-in-law has an acupuncturist who is into all sorts of weird Chinese traditional medicine. I could use an acupuncturist for some tendinitis but I have ruled her out because I know that some of her beliefs are totally nuts. Maybe she knows where to stick for the tendinitis but I am not taking any chances.

I told him I disagreed regarding the nutty acupuncture beliefs (more on this below), and Dmitri elaborated on his reasoning:

When I choose a health-care practitioner I am choosing on the basis of the quality of their health-care reasoning. I want them to figure out what’s wrong with me and find a way to fix it. If I have any evidence that their health-care reasoning is faulty, I want to stay away.

My comment about acupuncture was written from the standpoint of the belief that acupuncture might really work for certain kinds of ailments. I had a friend who got good results with tendinitis. But even if it’s all placebo I suspect I’ll get the best placebo effect if I have faith in the person administering the treatment.

This all makes sense, also it pleases my sense of nostalgia to see a mother-in-law joke. OK, it wasn’t really a joke on Dmitri’s part; still, it brought back memories of wacky mothers-in-law in old sitcoms.

The acupuncture paradox

But back to the acupuncture. I’ve been thinking about this for a long time. Here’s a post from 2011:

The scientific consensus appears to be that, to the extent that acupuncture makes people feel better, it is through relaxing the patient, also the acupuncturist might help in other ways, encouraging the patient to focus on his or her lifestyle.

A friend recommended an acupuncturist to me awhile ago and I told her the above line, to which she replied: No, I don’t feel at all relaxed when I go to the acupuncturist. Those needles really hurt!

I don’t know anything about this, but one thing I do know is that when I discuss the topic with any of my Chinese friends, they assure me that acupuncture is real. Real real. Not “yeah, it works by calming people” real or “patients respond to a doctor who actually cares about them” real. Real real. The needles, the special places to put the needles, the whole thing. I haven’t had a long discussion on this, but my impression is that Chinese people think of acupuncture as working in the same way that we think of TV’s or cars or refrigerators: even if we don’t know the details, we trust the basic idea.

Anyway, I don’t know what to make of this. The scientific studies finding no effect of acupuncture needles are plausible to me—but if they’re so plausible, how come none of my Chinese friends seem to be convinced?

My question here is not whether acupuncture could work (possibly through some backdoor mechanism like the needles stimulating your body in some useful way, or whatever) but on the evidence of how much it does work. As noted, I think the overwhelming impression among my Chinese friends–statisticians included–is that it does work, and not merely through some vague calming effect. But this would seem to contradict the research, so I don’t know what to think.

This does seem to be a paradox, as evidenced by some of the discussion in the 56 comments on the above-linked post.

We had another good comment thread when I brought up the topic again in 2016.

What are acupuncturists doing?

So, yeah, this paradox was bugging me for years, and then at some point I came up with a satisfying resolution.

My resolution of the acupuncture paradox might not be scientifically correct–indeed, it would be wonderful to design some experiments to study the topic and see what, if any, of my ideas in this domain hold up–but it has the virtue of being a possible solution to the problem. Which is more than I had before.

I’ve talked with a bunch of people about this idea, and I’ve mentioned it in some public lectures, but this might be the first time I’ve written it up.

My resolution of the paradox starts with the idea that the success of acupuncture, as with physical therapy, coaching, teaching, and many other things, comes from a fruitful interaction between the patient and the therapist. Good acupuncturists, like good physical therapists, coaches, and teaching do not just push buttons and follow a template; they work closely with each patient and figure out what is needed. In addition, I assume that acupuncture is like these other endeavors in that an key function of the therapist is to motivate patients to keep up with the work.

Thinking of a treatment effect as a vector with direction and magnitude

I’ve written that you can conceptualize an education intervention as a vector, where the direction of the vector is the material being learned and the length of the vector is the amount that students are motivated to work. You want the material learned to be useful–you’d like the vector to have a positive “dot product” with the vector of skills, knowledge, and understanding that will be useful going forward–but, conditional on those two vectors being roughly aligned, the real gain is in the magnitude. And this magnitude will be an interaction between the teacher and student: there’s no button to push to create motivation, and if there were a button it would already have been pushed.

What I’m saying is that, when thinking about acupuncture, or physical therapy, or coaching, or teaching, we have to go beyond what I’ve called the penicillin model of science, the idea that innovations come from nowhere and that the job of statistics is to design and analyze experiments to reject the null hypothesis of no effect, and in which the treatment in such experiments is considered as a black box, with the goal being to estimate an average treatment effect.

I don’t think the penicillin model usually applies. Most of the time in health, education, and just about any field, improvements are incremental, and the goal is to improve the process while gaining understanding. Clinical trials and offline experiment both play a role, and you’re not going to learn much by studying a treatment as if it’s a black box.

This is not to say that there cannot be new developments in any of these fields, nor is it to deny that such developments can sometimes arise serendipitously. I just think that, in any case, you have to go beyond the average treatment effect and think about the mechanism of action.

Resolution of the acupuncture paradox

OK, now on to my answer.

Let’s suppose that acupuncture really works, not just as a placebo or as relaxation or whatever, but as a set of physical manipulations that help you heal better. Let’s also suppose that the mechanism of acupuncture is not the position of the needles or qi or whatever, but the result of the acupuncturist observing you, listening to you, watching and feeling you as you move, then getting a sense of where your problems are and doing movements and giving you advice that will improve your healing. There’s no need for either or both of these statements to be true, but they could be, and suppose they are.

In that case, we should see two things:

1. The usual controlled studies of acupuncture should show no effect. If you do an experiment where the treatment group gets acupuncture with the needles in the “correct” places and the control group gets acupuncture with needles in the “wrong” places, there will be no different. If useful acupuncture is being done, with good interaction between the therapist and patient and the therapist giving informed, patient-specific therapy, then it will work in both treatment and control groups. If push-button acupuncture is being done, without that focus on the patient, it won’t work in either group. In either of these scenarios, the treatment effect in the experiment will be zero, or nearly zo.

2. Real-world acupuncture would work, and not just because of relaxation/placebo/etc.

In this setting, the usual experimental research method won’t work, because the experiment with the random needle placements is removing the very mechanism by which acupuncture works (or is assumed to work, in my scenario).

That’s a paradox for you: We can have an effect that is real but which will not show up under the usual controlled-trial design.

This is not to say that the effect could not be detected. You’d just need a different design, for example acupuncture (done however the therapist wants to do it) versus nothing, or versus some default therapy. In such a setting you’d want to gather lots of intermediate data to find out what the acupuncturists (and also the control-group therapists) and the patients are actually doing, how their bodies are moving as the weeks go on. Get rich data, thick description that can then be analyzed using multilevel models as necessary.

What about the theory?

To return to Dmitri’s original statement: What about the theory of acupuncture, the placement of the needles, the lines on the body, the qi, etc.? I don’t know. I admit I’m skeptical, and I think that acupuncture could work (and not just as relaxation etc.) even with all these theories being bogus. I’d think of the theories as a sort of checklist, a framework that gives acupuncturists some focus and gives them the space to observe the patient and figure out what to do and what to recommend.

Just as in chess it is said that planning is important, even your plan does not work out, it could also be true in acupuncture (and also in physical therapy, coaching, teaching, etc.) that even a misguided or empty theory can provide useful structure.

But we don’t always have a good language for talking about this when we talk about science. We can talk about specific mechanisms (this gene codes for this protein which unlocks this other protein which catalyzes that reaction, or whatever) or we can talk completely abstractly (this treatment works, as has been demonstrated in a bulletproof clinical trial), but we’re not so good about the steps of trying to work through a mechanism by gathering intermediate measurements and modeling them.

So, give your mother-in-law’s acupuncturist a break. Her beliefs and medical theories may be “totally nuts,” but they may be no more than a framework that she can use to do the useful things that she does.