This Guest Blog is rather different. It’s about statistical analysis and written by a respected scientist, Professor Jeremy Greenwood. In his Guest Blog Professor Greenwood criticises the statistical basis of the UK government’s opposition to a ban on neonicotinoid pesticides during the tenure of Owen Paterson as Secretary of State for Environment, Food and Rural Affairs.
This paper is considerably more technically demanding than my usual scribblings about Hen Harriers or farmland birds and the reason it is being published here is that it touches on a subject of importance and of current debate. It’s the type of thing that it is difficult to publish in the scientific or statistical literature and even more difficult to publish quickly so that it can have an impact on the policy debate. This blog’s readership will be interested in the punchline of the paper, which is, that one of the main published scientific papers on which the UK government’s objection to a ban on neonicotinoid pesticides was based is deeply flawed and essentially worthless as a basis for policy decisions. Note that Professor Greenwood asked to see the original dataset to perform further statistical analysis but was refused access – this is contrary to scientific etiquette and procedure in my experience.
Professor Greenwood is quite blunt in his assessment of the paper by Pilling and others. He states that the analysis is ‘absolutely unnacceptable’, that ‘the experiment has told us nothing’ and that ‘It is difficult to understand how the work came to be published in a refereed journal.’. This is blunter than Godfray et al. in a very recent paper which reviews that same paper (and others) and blunter than the Parliamentary Office of Science and Technology briefing on the subject.
I’m grateful to Professor Greenwood for this paper. I am happy to consider responses to it from Syngenta or from Defra (or actually from anyone else), perhaps from the Defra Chief Scientist Ian Boyd, if they would like to comment or correct anything that they see as an error.
Professor Jeremy Greenwood is a former Director of the British Trust for Ornithology and is an Honorary Professor attached to the Centre of Ecological and Environmental Monitoring at St Andrews.
—————————————————————————————————————————————————-
1. Use and environmental distribution
Neonicotinoids are the most widely used insecticides in the world. They are toxic to most arthropods. They are widely applied as seed dressings because they act systemically, protecting all parts of the crop.
Neonicotinoids can persist and accumulate in soils. They leach into waterways. They are found in nectar and pollen of treated crops. Concentrations in soils, waterways, field margin plants, nectar and pollen have commonly been found to be levels sufficient to control crop pests; they commonly exceed the LC50 (the concentration which kills 50% of individuals) for beneficial organisms.
2. Apparent effects on bees
There have been many studies in the lab and, increasingly, in combinations of lab and field conditions on the possible impacts of neonicotinoids on both honey bees and bumble bees. They indicate that concentrations in nectar and pollen in crops are not lethal to bees but are sometimes sufficient to reduce their ability to learn, to forage and to find their way back to the hive. There is evidence that they can reduce the survival of colonies. The general conclusion of these studies is that, together with such things as loss of flower-rich habitat and Varroa mite, neonicotinoid use may have caused losses of bee populations.
The manufacturers say that only studies carried out entirely in the field are valid. This is a curious reversal of arguments in respect of the impact of organochlorine pesticides on birds in the 1960s, when manufacturers said field studies were of no value and that all effects had to be demonstrated in the lab.
3. Restrictions
Following a scientific report by the European Food Safety Authority, the EC has imposed a
2-year moratorium on use of 3 neonics from 1 December last year. The UK government opposed this, on the grounds that there was no evidence of deleterious effects from purely field trials.
4. One study under field conditions was done in Britain, by the Food & Environment Research Agency (Fera), an Executive Agency of Defra.
Thompson H, Harrington P, Wilkins W, Pietravalle S, Sweet D, Jones A. 2013 Effects of neonicotinoid seed treatments on bumble bee colonies under field conditions. See http://www.fera.co.uk/ccss/documents/defraBumbleBeeReportPS2371V4a.pdf. (Note that this work has not appeared in the peer-reviewed literature).
To determine the effects of neonics on them, bumblebee colonies were placed adjacent to oilseed rape fields in which the seed had been treated with either of two neonics or with none. Unfortunately there was no difference in the amount of neonics picked up by bees adjacent to treated and untreated fields, probably because the bees foraged beyond the immediately adjacent fields, so those next to untreated fields picked up neonics from treated fields. Thus the experiment was incapable of showing anything useful about the effects of neonic exposure on the colonies. The work was not published in a peer-reviewed journal.
The lead scientist involved in the Fera (research (Dr Helen Thompson) moved to Syngenta (a major manufacturer of neonics) shortly after the report on it was made public. (Damian Carrington. The Guardian, Friday 26 July 2013).
4. Another study under field conditions was done in France, by Syngenta.
Pilling E, Campbell P, Coulson M, Ruddle N, Tornier I. 2013 A four-year field program investigating long-term effects of repeated exposure of honey bee colonies to flowering crops treated with thiamethoxam. PLoS ONE 8, e77193. doi:10.1371/journal.pone.0077193.
Honeybee colonies were placed beside thiamethoxam-treated or control fields of maize (three replicates) or oilseed rape (two replicates). Bees from treatment hives had higher concentrations of insecticide residues. But the authors stated that the results “show no evidence of detrimental effects on colonies that were repeatedly exposed over a four-year period to thiamethoxam residues in pollen and nectar, following seed treatment of oilseed rape and maize.” This is true but their conclusion that the measures of colony performance “were similar between treatment and control colonies” does not follow. They conducted no statistical analyses of their data nor did they present their results in a way that allowed others to make a proper assessment of their claim.
5. Hand-waving is no substitute for formal statistical analysis.
The reason that formal statistical analysis was not undertaken was said to be that, because the sample sizes were so small, “such an analysis would lack the power to detect anything other than very large treatment effects, and it is clear from a simple inspection of the results that no large treatment effects were present. Therefore a formal statistical analysis … would be potentially misleading”. This is absolutely unacceptable.
The usual way to analyse experiments such as that reported by Pilling et al. is to conduct a significance test of the means of the treated and untreated subjects. In such a test, one typically sets up a null hypothesis that there is no underlying difference between treated and untreated subjects (colonies in this case), any difference apparent in the experiment being just a result of chance. Should the analysis indicate that the difference found in the experiment is too great to be reasonably attributed to chance, the result is considered significant and one rejects the null hypothesis. If, in contrast, the difference can be attributed to chance, the result is labelled as non-significant and in practice many people then accept the null hypothesis. However, in addition to the null hypothesis being true, there are two other reasons why a test result may not be significant: that there is great individual variation between colonies subject to the same treatment or that there were too few replicate colonies. Forgetting these other two possibilities, people commonly interpret a non-significant result to mean that the null hypothesis is true. When sample sizes are as small as those used by Pilling et al. such an interpretation would certainly be misleading – but it is not the analysis but the interpretation of the result of the analysis that would be misleading. In fact, accepting the null hypothesis is an error. The correct,interpretation is that the experiment has told us nothing. This might be embarrassing to those who designed the experiment but the risk of embarrassment is no reason for not doing the right thing.
To avoid being misled by wrongly-interpreted significance tests, Pilling et al. used “expert interpretation of the data by scientists with experience in undertaking such trials”. But, however experienced are the experts, they have no more data before them than would be used in a statistical analysis. Expert interpretation of the results of a statistical analysis is important but without the formal analysis it becomes an exercise in mere hand-waving and obfuscation.
Professional statisticians nowadays tend not to carry out significance tests but prefer to estimate the magnitude of the difference between treated and untreated subjects and to place confidence limits on the estimate. The limits provide an idea of the range of values in which the underlying true difference between treated and untreated is likely to lie, so the approach is more informative than the simple significant/non-significant dichotomy of an hypothesis test. Pilling et al. should have adopted this approach. Alternatively, if they wished to use significance tests they should have actually done what the inventors of significance test, Jerzy Neyman and Egon Pearson, advised. That is to set up both a null hypothesis and an alternative hypothesis. For the bee experiment, the latter might have taken the form “There is a difference in colony performance of X% or more.” There are then four possible outcomes, three of which tell us something useful:
a. Both tests significant: “There is a difference but it is less than X%”.
b. Null significant, alternate not: “There is a difference and it could be X% or more”.
c. Alternate significant, null not: “Any difference is less than X%; it may be zero”.
d. Both non-significant: “There may be no difference or it may be X% or more”.
7. Inadequate reporting of the data.
Pilling et al. present graphs that allow one to get an idea of the magnitude of the differences between treated and control colonies and they tell us how many replicates they used. All we need to get a feel as to whether their conclusion that their results demonstrate that the measures of colony performance “were similar between treatment and control colonies” is an idea of the extent of differences between the two or three replicates. Unfortunately, their graphs do not reveal this. I have asked for a data set on which to test formal analyses but have been refused access.
8. Conclusions
With replications of only two or three, no formal statistical analysis and the results being published in a form so aggregated that it is impossible to assess the variation between replicates, the conclusion that “mortality, foraging behavior, colony strength, colony weight, brood development and food storage levels were similar between treatment and control colonies” is a clear overstatement unless one defines similar so loosely as to be scientifically meaningless. It is difficult to understand how the work came to be published in a refereed journal.
[registration_form]
It is a shame that Professor Greenwood is not able to access the original data from the Pilling article as it was submitted on the understanding (according to PLoS One’s Editorial Policies) that:
2. Sharing Materials and Data
Publication is conditional upon the agreement of the authors to make freely available any materials and information described in their publication that may be reasonably requested by others.
When submitting a manuscript online, authors must provide a Data Availability Statement describing compliance with PLOS’s policy. If the article is accepted for publication, the data availability statement will be published as part of the final article.
Furthermore the authors do state in their Competing Interests Statement (incidentally all the authors are either employed by Syngenta or by an associated company and this is clearly stated) that: “This does not alter the authors’ adherence to all the PLOS ONE policies on sharing data and materials.”
So why are the authors not following the policies of the journal that agreed to publish their article?
Incidentally a potential flaw of Open Access publishing whereby the costs of publication are paid by the authors and then the article is made freely accessible to all users. Because their sole income stream is from these author fees there is the temptation, on the part of the journal, to accept articles on the basis of ability to pay the fee, rather than on scientific accuracy.
Allen – thank you. Interesting and well-informed points.
While we have not yet seen colony collapse disorder as in the US, what is clear is that colony losses are higher in areas of the UK where neonicotinoid use is more prevalent. I realise that because of other variables such as weather this proves nothing on it’s own.
My own unscientifically formed opinion however is that this is significant.
I’m afraid you are misinformed as to bee deaths here in the UK and in Europe. France lost 1,000,000 colonies in a single year in 1994 when neonics were introduced by Bayer on sunflowers (imidacloprid). We do not have a ‘honey industry’ akin to France, we merely have 55,000 hobbyist beekeepers and a few bee farmers, but losses have run at 30-50% in the English and Scottish arable crop areas where neonicotinoid use us most intense (oilseed rape) since at least 2003. Tragically the British bee establishment (BBKA) was already in the pocket of Bayer and Syngenta as early as 2000 – they accepted £17,000 a year (without the knowledge of the Membership) for endorsing Bayer and Syngenta’s pesticides as ‘bee friendly’. the secret contract continued unabated, despite violent objections from BBKA members, for TEN years an d BBKA received over £170,000 from pesticide manufacturers. The BBKA Executive has been actively ‘pro-pesticides’ since the late 1990s and has never wavered in its loyalty to Bayer, Syngenta, DEFRA and the British government and the NFU in all that time. they have consistently followed the ‘Party Line’ that neonics are good for bees and good for farming and good for birds, despite over 800 peer reviewed papers which confirm that neonics are directly responsible for the global crash of bees, pollinators, soil invertebrates and insectivorous birds. The french knew this as long ago as 2000, when they banned neonics; Europe banned them in 2013, but the BBKA, the NFU, Defra and the British Government are still firmly ‘on message’, defending Bayer’s profits to the last ditch. Tragically, the RSPB has been little better – using neonics on its own farm until very recently and to date the RSPB has been UTTERLY SILENT in public on the causal relationship between massive bird population declines and wall to wall use of neonics in UK and Europe. Now, they are slowly, ever so slowly waking up – but it is 18 years too late, the damage has been done, and one suspects there will now be a ‘revisionist’ PR policy under which RSPB will start to make anodyne statements about neonics and bird declines. Too little, too late. The British science establishment has failed us utterly. Derek Ratcliffe took the pesticide corporations on over Peregrines and DDT in the late 1950s and fought to the death to get the stuff banned; his scientific heirs have been utterly useless. RSPB, SNH and English Nature have probably got 10,000 staff between them, and the best part of £1 billion in resources, but they do not appear to have a single CONSERVATIONIST of the calibre of Ratcliffe; not one.
Neonicotinoid seed treatment use isn’t quite ‘wall to wall’ in the UK. Most of osr was treated before the ban, and probably all sugar beet still is, but for wheat it is about 30%’ winter barley and oats 10%, and many crops such as spring barley, peas and beans none at all. So it does depend on each individual’s rotation, some will be using them most years, some not at all. Maybe the frequency of use could be restricted, but this would probably be difficult to regulate.
refusing access to data on which public policy decisions are made is appalling
the very failure to allow access to data suggests dishonesty on the part of the scientists or their employers. The whole basis of science is open sharing of results so others can validate them. What don’t they want others to know?
m – it may suggest dishonesty to you, but it suggests that they would be embarrassed by it to me. Not quite the same.
Surely the publication of a “scientific” paper which makes important claims which have no basis in the “scientific evidence” the authors are putting forward to support their claims; compounded by a failure by the authors to allow independent verification of these claims is potentially a fraud on 2 levels the original deceit compounded by a cover up. One wonders if a dog ate the original data or it was lost in a flood. I’m with M Parry, this reeks of dishonesty.
In the UK the approved use of neonicotinoids on a high proportion of the major arable crops in typical rotations – winter wheat, winter barley and winter oilseed rape – means that much arable land receives continuous doses from neonic seed treatments. As crops take up only a small proportion of the active ingredient most of the neonic dose is effectively applied to soil. The persistence of neonics in soil having been understated it should be no surprise that residual amounts of neonics in arable soils will make it very likely that “untreated” control areas are compromised.
There are many smoking guns concerning neonics in the environment and what we need is for Defra to stop saying “Smoke? What smoke?”, conduct an investigation independent of manufacturer personnel and activists, and to stop hiding behind the regulatory process as if it is adequate.
As for refusal of access to original data and scientific etiquette and procedure – this is now almost standard practice in post-normal, policy-led science.
Excellent blog piece Mark. It makes me wonder if the assessment of possible non-invertebrate and human health impacts were done with the same degree of rigour. Does anyone know?
On the same topic, I hesitate to say that this (http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/environmental-audit-committee/national-pollinator-strategy/oral/10709.html) makes an interesting read; revealing might be a better description. It appears the industry is unable to provide any peer reviewed evidence in the public domain to confirm that neonics actually work in the first place…
Do neonics work? – Yes. They kill insects.
It’s the wrong question. The question is not whether neonics work, but “Does prophylactic use of neonics against insect pests on arable crops effectively prevent measurable yield loss”.
The answer so far has been “Errr …”
Or, does it do what it says on the tin?
This report is worth reading
The Impact of the Nation’s Most Widely Used
Insecticides on Birds
American Bird Conservancy, March 2013
http://www.abcbirds.org/abcprograms/policy/toxins/Neonic_FINAL.pdf
The Syngenta study (Pilling et al, 2013) has been quoted as evidence that Honeybees are not affected by thiamethoxam in the field. However the paper has profound weaknesses. As Prof greenwood highlights, of over-riding significance there was insufficient data to undertake statistical analysis – this is not merely of academic interest as one of the fields did show a consistent effect (see graph S9) and more data may have (despite the methodological failings) confirmed that there was a significant pattern.
It is additionally very worrying that Prof Greenwood has been unable to get hold of the raw data.
Other very important concerns centre on the lower than field realistic dosing rates, small size of the treatment plots and the movement of the hives to distant areas uncontaminated by pesticides. While we welcome the publication of this paper as a significant step towards transparency by pesticide companies, the paper chiefly exemplifies the depth of the weaknesses in the science that is currently hidden at the heart of the pesticide approvals process. Pilling problems:-
• There is insufficient replication of treatments and therefore the key comparisons are not statistically significant so no conclusions can be drawn. There is limited statistical analysis to the study which even the authors admit to (page 10, 12-13). This only serves to highlight the issues of laboratory/semi-field studies versus field studies and what can be determined from the findings of this field study.
• It doesn’t address thiamethoxam dust, water contamination issues, synergistic impacts or other risks.
• Bees in the “multi-exposure” (field) study were seemingly only put into the maize fields during the flowering period for a few days (typically 5 or 6 days, up to 23 days in one field) (page 9) each year, with rest of time apparently off in a “woodland site” that was uncontaminated. Such a limited exposure time is not realistic. In the real world it seems the hives would be exposed to more contaminated dust, water, soil and weedy plants.
• Field sizes were only 2ha which is a significantly small field than is normal agricultural practice and there was only 2km distance between the control and treated sites. Honeybees can forage up to 5km distance in search of food. Data on pollen foraging from target crops is presented as a range of percentages so it is impossible to tell what proportion of foraging was done on the target crop in each case.
• Despite claiming that the study uses the maximum seed label recommendations. This is not the case according to the figures presented in the EFSA report Jan 2013. They use an OSR seed application rate 37.2% of the maximum used. Neonic content of OSR plant, pollen and nectar then recorded in the experiment was negligible. They use an application rate for maize that is 84% of the maximum.
• The discussion is based very considerably on the discredited Cutler Dupree paper of 2007.
• Consideration must be made on the strength of Honeybee colonies exposed to treated (dashed line) and control (solid line) maize fields in the Alsace region of France during the four years of observations including the last overwintering. See figure S9 in the supporting information for an example i.e. there appears to be consistently lower colony strengths on one of the treated fields.
Research is rarely ‘definitive’. The Plant protection Products legislation is carefully worded to take account of the fact that there will be situations where finding proof and reaching complete consensus is not possible. In the opinion of EFSA the risk to honeybees has been established, therefore action is required.
Not everyone believes that more research is either necessary or feasible, but this approach will continue to be promoted by the pesticides companies who will continue to profit from the pesticides, long after they should have been banned, as happened with DDT in the UK.
Poor Ian Boyd, out if his depth again. He has now become a figure of ridicule amongst most of the scientific community, because of statements like this, in response to the recent Hallmann et al paper on bird populations and neonics: “I am not sure how much systematic bias is present within the Hallmann et al study but I suspect quite a lot’. Devastating scientific rigour, taken straight from one of Ernie Wise’s Plays Wot He Wrote. He also has the temerity to say: “More underpowered studies of the effects of neonicotinoids is not what we need” – when he’s relying on one of them to advise government policy.
https://ianlboyd.wordpress.com/2014/07/10/more-is-sometimes-less-a-response-to-the-hollman-et-al-paper/
But these studies, and the ones used for the EU EFSA report which sparked the 2-year ban, are just the tip of the iceberg. As Godfray et al state in their recent survey paper:”For this topic, the published literature is a small fraction of the evidence that has been collected.” Most of the data is in the hands of the agrochemical companies, and they are keeping it hidden for ‘commercial confidentiality’. Until the rules are changed on this (very similar to the rule that allows pharma companies to hide the results of drug trials), we won’t ever know which pesticides are going to be environmentally harmful until it’s too late.
Proc. R. Soc. B 7 July 2014 vol. 281 no. 1786 20140558
http://rspb.royalsocietypublishing.org/content/281/1786/20140558.full
Ah commercial confidentiality, sounds so much nicer than a convenient cloak for deception and cover up, which it so often turns out to be.
Excellent blog Mark with some useful comment contributions too. The thing that probably amazes me most about this is that every single contributor to the discredited paper is in the pay of Syngenta. How can the UK government possibly consider this research to be unbiased, regardless of its academic integrity?
Please see this short video, which points out that many bird populations are declining in areas where neonicotinoids are used.
https://www.youtube.com/watch?v=4MIUydhOgsA
Published on Jul 9, 2014
Populations of common insectivorous birds are declining in farmland areas with high levels of the neonicotinoid insecticide imidacloprid. This is shown by an analysis of detailed data on local bird population trends and environmental factors, which include imidacloprid concentrations in surface water. The scientific journal Nature published the study, written by biologists at Radboud University Nijmegen and the Sovon Centre for Field Ornithology, on July 9, 2014.