A whimsical Manitoba study shows how research can be manipulated to show agronomic benefits where none exist. Can you spot the fallacy?
Obviously, it’s silly. Or is it? John Heard, for one, thinks all of us can be lured into believing the research that we want to believe, whether it’s true or not.
To put that idea to the test, Heard sprayed diluted maple syrup on a block of canola at the University of Manitoba’s Carman research station. In fact, he went even further, designing a classical research experiment where the maple syrup concoction was replicated 20 times within the same block to simulate a multi-site study.
Read Also
How to go from managing to mentoring
https://www.youtube.com/shorts/L7Q7OZ_d3cI There’s a moment every leader remembers: that moment when someone looks at them differently. Not because of what that…
Then Heard, who by day is soil fertility specialist with the Manitoba Ag Ministry, applied the maple syrup to a spring wheat crop to measure the product’s effectiveness as a nitrogen replacement on that crop.
Within the spring wheat field, Heard experimented with an unfertilized check, 60 pounds per acre of nitrogen, 60 pounds per acre of nitrogen plus the maple syrup dilution, and 120 pounds per acre of nitrogen.
Heard then measured leaf chlorophyll levels as well as total biomass yield, and analyzed the results for both crops.
So, what do you predict he found?
Much of Heard’s project focused on statistics, which he admits might seem a dry topic. But those boring numbers can take on a life of their own “if someone chooses to be kind of unscrupulous,” Heard says. “Even if it does nothing, if the statistical probability you’re using is 90 per cent probability, two times out of 20 it will seem to be doing something.”
In this case, statistical analysis showed the maple syrup gave an average 13 per cent biomass yield bump at 11 of the canola sites… except, Heard had cherry-picked the results, grouping the numbers only from the plots where the maple syrup seemed to be creating a yield increase. It’s why you have to look at such results so carefully. Really, what it’s saying is that where there was an apparent yield increase, the increase was 11 per cent.
When the data from all 20 plots was examined, however, the results weren’t so impressive. In fact, canola yields on eight of the plots were actually lower where the placebo was applied.
It’s also misleading that the study only measured canola biomass yield, not grain yield. Though a boost in biomass can seem like a good thing, grain yield is what the farmer actually gets paid for.
Results from the wheat crop could sound promising too. There was no significant difference between the plots that received 120 pounds per acre of nitrogen and those that received the 60 pounds per acre of nitrogen plus maple syrup.
In other words, it seems like the maple syrup replaced 60 pounds of N.
However, a closer look at the results reveals no difference between the check and the 120 pounds per acre nitrogen application.
What had actually happened was that the previous year’s soybean crop had left plenty of residual nitrogen in the soil. So, in effect, the crop didn’t require any nitrogen, which explains why the placebo seemed to have the same effect as the applied nitrogen.
There isn’t a lot of unbiased, third-party testing of new nutrient products. The Canadian Food Inspection Agency has required research on the safety and effectiveness of many fertilizer products in the past. The agency doesn’t require efficacy data on all fertilizer products and is now looking at provisional registrations for new products that would allow foreign or limited domestic efficacy data.
Though there usually isn’t a lot of harm in potentially ineffective products, Heard is concerned when they replace proven products or technology.
“So if money is tight and financial resources are scarce, it would be harmful for producers to be misappropriating dollars they need for nitrogen or phosphorus into lesser needed products,” says Heard.
With technology like yield monitors available, many producers have the tools needed for on-farm testing. If time and statistical knowledge are constraints, Heard says producers may want to team up with a crop adviser. Producers doing their own testing should replicate tests for new products two or three times, rather than leaving only one check strip. Producers may also be able to pool their data with farm testing groups to get more reliable results. Such groups have been successful in Ontario and Iowa.
Heard tells producers to look beyond the scientific language. Insist on seeing the results from all test sites, along with information on soil test results and cropping history. Also look at the yields of the untreated check and standard treatments.
Says Heard, “That’s just part of good stewardship of your own crop input dollars.” CG
