We Berate, You Deride
–
DemandDebate.comÕs survey on the scientific
consensus
surrounding global heating
Reality is a hard
nut to crack - it doesnÕt give up its secrets easily. When scientists researching
the EarthÕs climate realized that humanity was the major driving force behind
observed global heating, many people didnÕt believe it. Some couldnÕt accept
that our collective activity was overwhelming natural forces like changes in
solar irradiance and Milankovich cycles. Some wanted to wait until more and
better data was available and a scientific consensus formed.
Others simply
didnÕt care, either rejecting the science in an anti-intellectual paroxysm or
concluding that global heating would never affect them or their families. But
there were more than a few people who saw the science, understood it, and then
set about deliberately undermining the science for political, religious, or
financial reasons.
On November 8,
2007, the anti-global heating tactic of questioning the scientific consensus
behind global heating got a fresh piece of ÒevidenceÓ: DemandDebate.com issued
a press release about a survey that purports to undermine the scientific
consensus on global heating.
DemandDebate.com
claims to be Òmore worried about the intellectual climateÓ and has started up
an educational video series called ÒWe Debate, You DecideÓ that is supposed to
provide actual debate on global heating. DemandDebate.comÕs survey, however, is
pure propaganda.
According to the
professional organization American Association for Public Opinion Research
(AAPOR), there are a few basic guidelines to ensure the accuracy of any public
opinion poll: having specific goals, maximizing the response rate to the
survey, taking great care in wording questions correctly for the surveyÕs
population, and holding respondentÕs identities as privileged, among others. In
all of these areas, DemandDebate.comÕs survey methodology fails.
When designing a
survey, the designer is supposed to choose objectives that are Òspecific,
clear-cut and unambiguous.Ó The goal of such surveys is to acquire
statistically valid information about the population being surveyed, not to
produce predetermined results. The stated goal of DemandDebate.com survey was
to determine the level of consensus among the scientists involved in the
drafting of the United Nations Intergovernmental Panel on Climate Change (IPCC)
ÒClimate Change 2007: The Physical BasisÓ document. However, the discussion of
the results do not present an unbaised appraisal of the survey data. Instead,
the discussion is entirely biased against global heating:
But if thereÕs no agreement
on what the target climate should be, what precisely is the point of taking
action on global warming? What is the climatic goal at which we are aiming?
ÉSo was there no climate
change before mankind? And if there was natural climate change before man, why
not now also?
ÉBut when you ask questions
that are off the usual script, the supposed consensus seems to readily fall
apart.
And letÕs not forget that
many climate experts no longer participate in the IPCC process because they
perceive it to be biasedÉ.
These statements
do not just present the survey data, they provide readers the ÒcorrectÓ interpretation
of that data, propose alternatives to the actual questions, imply nefarious
motives to the scientists who answered the survey questions, and add
information external to the survey sample itself in order to support the
surveyÕs supposed conclusions. These actions introduce an obvious bias to both
the results and the questions themselves, suggesting that this survey was not
intended to be a valid statistical survey as claimed, but rather the equivalent
of a political Òpush pollÓ for global heating.
Quality surveys
attempt to maximize the rate of response to the survey, because more responses
mean greater accuracy. However, in this case, DemandDebate.com did nothing to
maximize the number of scientists who responded - in fact, DemandDebate.com
intentionally ignored about 45% of the scientists involved in the IPCC and
focused on the U.S.-based scientists only. If the intent had truly been to test
the level of consensus among global heating scientists, the survey would have
been sent to all 620 or so IPCC scientists, not just the 345 scientists who are
based in the United States. This introduces a selection bias into the answers
to the survey.
Furthermore, the
fact that only 54 of the U.S. scientists responded suggests that the surveyed
population detected bias in the survey and simply refused to answer the
questions posed. In fact, several of the scientists who refused to answer this
survey blogged about it, and their refusal to participate, at RealClimate.org:
The ÒHave you stopped beating your wife yet (yes/no)Ó questionnaire. The end
result is that less than 10% of the possible survey respondents actually
responded. As such, the results of the poll are strongly suspect due to
self-selection bias (ie a non-representative sample) and qualify as unscientific
and thus essentially meaningless SLOP (Self-selection Opinion Poll).
Which brings me
to the questions themselves. According to S&RÕs resident marketing expert,
Dr. Slammy, there are at least two questions (of the six total) that are biased
and specifically designed to produce results that can be spun by
DemandDebate.com and Mr. Milloy for their own political ends. Lets look at
questions 5 and 6 (results removed from questions):
Question #5. The climatic
impacts of a mean global temperature that is 1-degree Celsius warmer than today
are:
[ ] Undesirable.
[ ] Desirable.
[ ] Desirable for some and
undesirable for others.
[ ] Too difficult to
assess.
[ ] No opinion.
Question #6. The ideal
global climate isÉ
[ ] Warmer than the
present.
[ ] Cooler than the
present.
[ ] Occurring today.
[ ] There is no such thing
as an ÒidealÓ global climate.
[ ] No opinion.
The selections
for question #5 are specifically designed to use scientific nuance and
vagueness to get an answer that can be spun the way the survey designer wants.
ÒDesirable for some and undesirable for othersÓ is undoubtedly the ÒcorrectÓ
answer, but itÕs the correct answer for nearly every temperature increase or
decrease of any magnitude - some areas will always get ÒbetterÓ and some will
get Òworse.Ó
But better or worse, desirable or
undesirable for whom, or for what? People? Other species? Coastline erosion?
Sea level rise? And the temperature rise, 1 degree C, was carefully chosen to
produce the most easily manipulated results. After all, The Pew Center on
Global Climate Change said that low amounts of temperature rise would probably
benefit the United States, but that higher temperatures (like those in the 3-5
degree C range that the IPCC actually predicts by 2100) will be bad for the
U.S. These levels of nuance are get lost because the question is so poorly
designed. At least, poorly designed for producing valid statistical data.
Similarly,
question #6 is intended to get a predetermined answer, namely Òthere is no such
thing as an ideal climate.Ó Not only is this the only scientifically viable
answer - and thus the answer that most good scientists will give - but the
answer is designed to prejudice survey readers against scientists for whom the word
ÒidealÓ is nearly meaningless. For example, ÒidealÓ for people (the likely
interpretation for most survey readers) may be different from ideal for lions,
tigers, and bears. In addition, there may not actually be an ÒidealÓ climate
for human beings - something that is unknown and may be fundamentally
unknowable.
However, because
the scientifically valid answer is the one that looks the worst for the
scientific consensus, this question is both great propaganda and a no-win for
any scientist who responded to this survey. And in fact, both the press release
and the actual survey used this question as a jumping-off point for a biased,
explicitly political message, namely ÒBut if thereÕs no agreement on what the
target climate should be, what precisely is the point of taking action on
global warming? What is the climatic goal at which we are aiming?Ó So, have you
stopped beating your wife yet?
For a detailed
look at how good survey questions should look, I recommend The AAPORÕs question
wording and SurveyProÕs survey design tips.
However, perhaps
the greatest methodological error, if we can call attempted character
assassination an Òerror,Ó is the fact that Mr. Milloy failed to hold the
identity of the respondents private. In fact, in his Fox News commentary on the
surveyÕs results, Mr. Milloy threw out these gems:
One National Oceanic and
Atmospheric Administration scientist responded simply by dropping an
f-bomb-laced insult into an e-mail.
This particular response
and any institutional intolerance for climate skepticism, so I am informed, is
being investigated by NOAA chief Vice Admiral (Ret.) Conrad LautenbacherÉ.
É[It] just so happens that
[IPCC scientist John] ChristyÕs survey responses were within the 50 percent who
didnÕt think that a 1-degree Celsius rise in global temperature was uniformly
undesirable and the 86 percent who didnÕt think there was any such thing as an
ideal climate.
IÕm not going to
support expletive-laden insults from NOAA scientists, but the fact that Mr.
Milloy reported the incident for an official investigation instead of holding
the information confidential (as expected of professional public opinion
researchers) illustrates that his goal throughout the entire survey process was
to manipulate the U.S.-based IPCC scientists into appearing to be less unified
on global heating than they actually are. And his quoting the exact answer of a
scientist is also professionally unethical for a public opinion researcher
(again according to AAPORÕs best practices regarding confidentiality).
Mr. Milloy was
contacted for this article, but did not respond to a request for an interview.
Even without the
severe problems in the methodology, there are significant issues with the data
itself. As I pointed out above, there are approximately 620 scientists who
worked on the IPCC Physical Basis for Climate Change document, either as
principal writers or reviewers. However, only the 345 scientists in the United
States were surveyed, and of them only 54 actually took the survey and returned
it. And yet, given that this survey purports to be a serious,
statistically-valid survey, there is no mention of the confidence level, nor is
there any information about the margin of error for this survey.
Given the strong
selection biases in the sample, we actually cannot draw any statistically valid
conclusions from this survey. But if we could, and we applied the industry
standard 95% confidence level to this survey, we find that 54 respondents
produces a margin of error of +/- 12.75% (found using this online calculator).
Given that the target margin of error for meaningful political polls is less
than 3%, this is a huge margin of error, and it effectively erases the
differences between the questions when the differences are less than
approximately 13%.
LetÕs look at all
the questions again to determine what the statistically meaningful (separated
from other answers by greater than 13%) answers are.
For question #1
(What best describes the reason(s) for climate change?), the only statistically
meaningful answer becomes ÒHuman activity drives climate change, but natural
variability is also importantÓ, at 63% of respondents. All the other answers
are within 13% of each other and so are statistically meaningless when applied
to the larger, 620 member population of scientists.
The only answer
to question #2 (Which best describes the role of manmade CO2 emissions in
climate change?) that is statistically meaningful is ÒManmade CO2 emissions
drive climate change, but other natural and human-related factors are also
important,Ó at 70% of respondents.
The only
statistically valid answer for the full scientific population for question #3
(Which best describes the impact on global climate of controlling manmade CO2
emissions?) is ÒLimiting manmade CO2 emissions would have a strong impact,Ó at
72%.
Question #4
(Current mean global temperature is:) is the only question that has two
statistically valid answers, namely ÒUnprecedentedly warm and getting warmerÓ
and ÒWithin natural variability but moving to unprecedentedly warmer levels,Ó
at 56% and 31% respectively.
Question #5 has
two answers that are statistically differentiated from the others but not from
each other. In answer to The climatic impacts of a mean global temperature that
is 1-degree Celsius warmer than today are, only ÒUndesirableÓ and ÒDesirable
for some and undesirable for othersÓ, at 48% and 39% respectively, are
differentiated from the other answers, although they are within the margin of
error of each other and so cannot be directly applied to all the IPCC
scientists.
Question #6 (The
ideal global climate isÉ) has only a single statistically valid answer: ÒThere
is no such thing as an ÒidealÓ global climate,Ó at 61%.
In terms of the
entire population, this means we can make only the broadest of statistical
claims. And yet Mr. Milloy reports that his survey data illustrates a complete
lack of consensus among scientists, when in fact it does nothing of the sort.
In fact, were we actually make this survey statistically valid, the only
conclusions we could draw for the the entire population (all 620 IPCC
scientists) is that between 50 and 75% of scientists believe that human
activity is driving global heating and that natural forces also matter, that
between 57 and 83% feel that anthropogenic CO2 is the driving force behind
global heating, that limiting anthropogenic CO2 emissions would strongly impact
global heating, that between 74 and 100% believe that global mean temperatures
are or will be soon at unprecedented levels, and the same number (74-100%) believe
that the impacts will be somewhat or wholly bad. And again, this is applying
scientifically valid conclusions to a biased, thoroughly unscientific survey.
Even if you
accept Mr. MilloyÕs data and assume the results are perfectly accurate (which,
as IÕve illustrated above, they arenÕt), he still intentionally misrepresents
them in order to make his point. We can use his very own data to completely
reverse his conclusions.
1. Only 4% of respondents deny the role of human
activity in global warming.
2. 0% of their respondents denied the role of manmade
CO2 or human-related factors in global warming.
3. 0% of the respondents believed that limiting
manmade CO2 would have no impact on global warming.
4. Only 4% of respondents believed that mean global
temperature is within natural variability and stable.
5. Only 4% of respondents believe that a 1 degree rise
in mean global temperature would be desirable.
6. Only 2% of respondents believe that a warmer mean
global temperature would be desirable.
ThereÕs a reason
that thereÕs jokes about lies, damn lies, and then statistics, and that joke
applies to real statistics, never mind fake statistics like these.
Ultimately, this
so-called survey is nothing more than a vehicle for Mr. Milloy and DemandDebate.com
to try and discredit the very real (and statistically valid) consensus that
does actually exist among climate scientists. It is nothing more than an
attempt to use pseudo-science to create propaganda that can be used by Mr.
Milloy and his conservative allies to imply significant disagreement where
there is none.
Tomorrow: Part 2
- A closer look at the background of Steven J. Milloy, executive director of
DemandDebate.com