The Childbirth Truth Squad

Can’t See the Forest Through the Trees?

Posted on: July 20, 2010

Any study that shows midwifery is more deadly, is met with a flurry of criticisms by midwives and company, supposedly based in science and statistics.

The recent meta-analysis by Wax et al in AJOG stated that homebirths are three times as deadly.   As anyone who follows the debate knows, there are no randomized controls comparing homebirth to hospitals.   Yet, one of the biggest complaints by the homebirth crowd was that the study failed to include a forest plot.    Forest plots are for meta-analyses of randomized controls.  They are merely graphics of data and therefore cannot possibly be a flaw. 

The earliest accusation that this was some kind of fatal flaw that invalidated the conclusions seems to be from Gil Gyte, a childbirth educator in London.    http://    She feels she is qualified to comment by virtue of her association with The Cochran Reviews.   The Cochran Reviews is an organization devoted to poor quality meta-analysis via its promotion of a very simplistic, cookbook method that is made worse by interpretations by its mostly amateurish researchers.    Her criticisms are little more than noting or merely claiming that the reviewers did not do it according to the Cochran algorhithm.   Among them, she claims that a forest plot should have been included.  

Amy Romano of LaMaze’s Sense and Sensibility quickly plagiarized, er, eh , I mean, chimed in with same.

“I also take issue with the fact that the researchers did not display the standard “forest plot” that customarily accompanies a meta-analysis to illustrate how the relative magnitude of observed differences in the individual studies and the pooled analysis.”

Then Jennifer Block of  excitedly passed on the misinformation as if it were a big Gotcha moment.   

“For a great break down of the science, look to Amy Romano’s Science & Sensibility. She asks why the authors did not graphically display their results using the customary “forest plot,” which usually accompanies meta-analyses (perhaps because it would have shown “a confidence interval you could drive a truck through”?).”

Gold Medal for this Birth Junkies Behaving Badly Trifecta goes to Romano.    It is not for the mistake itself, started by Gyte.   But, rather her pervasive pattern of using statistical terminology she really doesn’t understand to dress up her beliefs,  professional promotion, and self-interests  in objectivity and her behavior towards her critics and her evasiveness when found out.

Amy had the clever idea of actually linking to a Wikipedia posting on forest plots from her post  condemning its absence.  Wiki clearly tells you in the first sentence or so it is for random studies, which homebirth studies are not.    

“It was developed for use in medical research as a means of graphically representing a meta-analysis of the results of randomized controlled trials.”

So essentially, right under her nose was the information that, yes, many people borrow the technique for non-random analysis, but it is technically incorrect.    Furthermore, it is merely a graphic representation of the data.   In other words, it can’t be a methodological flaw to omit something that technically, one should not use in these circumstances.   Refraining indicates better depth of understanding of complex statistics.   And,  it is certainly not a methodological flaw to not have drawn a picture for those unable to understand the narrative!   

Ms. Romano has gotten more than one comment on her blog pointing out this and many, many other of her blatant errors in her statistical and methodological criticisms.   However, none make it through moderation.     None of the attempted comments were copied be re-posted here.  After all, no one ever imagined pointing out a blatant misuse of statistical terms would be banned as a TOS violation, profane, spam, or unladylike.

She and Block further go on to speculate that this was not done to hide “a confidence interval you could drive a truck through”.   She implies that this too is another methodological flaw.    The confidence interval indicated that homebirth was anywhere from a 1/3 more deadly to 5 times more deadly.    Earth to Amy.  Earth to Amy.  Come in Amy.    Both ends of the interval are very damning to homebirth and midwifery by proxy.    Homebirth is worse no matter what.  No mom wants to get in this truck no matter what.    

Clearly she doesn’t understand what confidence intervals mean.  Their understanding of research is much like their understanding  of birth – they watch a few and think they understand it enough to use it.

It is obvious to anyone who has studies advanced statistics or research methods, that these ladies decide a study is fatally flawed statistically and methodologically simply because the results don’t flatter  them.   Then, they proceed to pepper their condemnations with statistical terms lifted from books that have little or no relationship to the study at hand and they don’t understand.  They are not trying to educate women, they are trying to deceive us.

She wrote in response to being called on the carpet about the refusal to post:

“Fwd: [Science & Sensibility] Comment: “Read this book: How to Read a Paper” Friday, July 16, 2010 12:53 PM   This sender is DomainKeys verified

“Amy Romano” <>
Because it’s my blog, and you’re rude. 
Sent from my iPhone
Begin forwarded message:

Subject: [Science & Sensibility] Comment: “Read this book: How to Read a Paper”

Why are you erasing comments that point that you are not truly raising evidence-based statistical or methodological issues, but seem to be lifting and throwing out terminology from this book?

If you really know what these terms you are using mean, and they really apply to a situation, then you should be able to explain yourself when they are used in a completely unstandard context.

You erased one of mine and I have evidence you did same with another’s.  

All you will accomplish is the genesis of the Stuff Science and Sensibility/Amy Romano Wants to Hide blog.   It will probably come up higher in google than this.”

She reportedly  Twittered she wasn’t going to argue with someone who insults her.  So why does she insult our intelligence?

About these ads

5 Responses to "Can’t See the Forest Through the Trees?"

I would recommend, for example, seeing the PRISMA checklist on transparent reporting of systematic reviews and meta-analyses, which recommends:

“Results of individual studies: For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.”

Or, for example, Trisha Greenhalgh’s “How to Read a Paper,” in which the section on meta-analyses talks about forest plots as a standard data representation in these types of papers.

Thanks. But, research professionals don’t use Paint-By-Numbers methods.

And, I think I will skip the book because I used to teach statistics and I have tons of those.

The only purpose of these seems to be to make amateurs feel they know much more than they do. That, in turns seems to make them feel that what little they do know is etched in stone.

Really, all they are doing is reciting by rote, simplistic Playskool cheatsheets that were devised when we had to dumb things for the Statistics for Non-Majors class.

Note: it’s not a *methodological* flaw to omit a forest plot, but it makes it a heck of a lot easier to readily note whether the findings from the included papers were statistically or clinically significant.

It was designed for metas of RCTs. In fact all of meta-analysis was originally designed for RCTs.

These guys are one of the few who actually recongize this when borrowing the theories and techniques for non-randoms.

Yet, comically, two things they did and should have done to adapt the concept to non-randoms, the Birth Junkies raise as some kind of abnormality or mistake or flaw because it is way beyond their understanding.

Unflattering conclusion + Unfound in Watered Down Statistics for Non-Majors Manual = Flaw. You flunk!

Here’s objective evidence that the cookbook (Cochran et al) methods don’t do anything much to promote understanding and certainly aren’t some monolithic standard.

So, not following them might mean you are someone who is an expert and way past the cookbook level. Commenting on their absence marks you as an amateur.

All the Cochrane reviews had forest plots (2197 in total), and a random sample of 500 of these plots were included. In total, 28 of the non-Cochrane reviews had forest plots (139 in total), all of which were included.

In other words, all the Cochrane amateurs used them (it’s required!) and only about 1/5 of the professional researchers did.

“Conclusions Forest plots in Cochrane reviews were highly standardized but some of the standards do not optimize information exchange, and many of the plots had too little data to be useful. Forest plots in non-Cochrane reviews often omitted key elements but had more data and were often more thoughtfully constructed.”

Guess who won?

Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice
David L Schriger1,*, Douglas G Altman2, Julia A Vetter3, Thomas Heafner4 and David Moher5
1Department of Emergency Medicine, University of California, Los Angeles, School of Medicine, Los Angeles, CA, USA, 2Centre for Statistics in Medicine, University of Oxford, Oxford, UK, 3Stritch School of Medicine, Chicago, IL, USA, 4Saint Louis University School of Medicine, St Louis, MO, USA and 5Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada

*Corresponding author. 924 Westwood Boulevard, #300, Los Angeles, CA 90024-2924, USA. E-mail:


Background Forest plots are graphical displays of findings of systematic reviews and meta-analyses. Little is known about the style and content of these plots and whether published plots maximize the graphic’s potential for information exchange.

Methods We examine the number, style and content of forest plots presented in a previously studied cross-sectional sample of 300 systematic reviews. We studied all forest plots in non-Cochrane reviews and a sample of forest plots in Cochrane reviews.

Results The database contained 129 Cochrane reviews and 171 non-Cochrane reviews. All the Cochrane reviews had forest plots (2197 in total), and a random sample of 500 of these plots were included. In total, 28 of the non-Cochrane reviews had forest plots (139 in total), all of which were included. Plots in Cochrane reviews were standardized but often contained little data (80% had three or fewer studies; 10% had no studies) and always presented studies in alphabetical order. Non-Cochrane plots depicted a larger number of studies (60% had four or more studies) and 59% ordered studies by a potentially meaningful characteristic, but important information was often missing. Of the 28 reviews that had a forest plots with at least 10 studies, 3 (11%) had funnel plots.

Conclusions Forest plots in Cochrane reviews were highly standardized but some of the standards do not optimize information exchange, and many of the plots had too little data to be useful. Forest plots in non-Cochrane reviews often omitted key elements but had more data and were often more thoughtfully constructed.

Keywords Systematic review, forest plot, meta-analysis, graphical data representation, funnel plot

Accepted 12 November 2009

Comments are closed.


    Get every new post delivered to your Inbox.

    %d bloggers like this: