By Dr. Richard Gawel
A couple of weeks ago I conducted a workshop for the judges from the Royal Perth Extra Virgin Olive Oil Show. It was the 4th such workshop conducted annually at the request of the Western Australian Olive Council. The judging workshop program evolved out of the belief that whilst show judges are experienced EVOO tasters, the act of show judging requires a variety of skills to bring it all together.
Every year I concentrate on building a particular skill required by show judges. Themes have included – measuring judge consistency, writing effective comments for the benefit of exhibitors, and understanding and dealing with diversity of EVOO styles and characters.
One of the most important requirements of a good show judge is the ability to score consistently. Agreeing with yourself is a pretty good quality to have. One would hope that a judge that gives an oil a gold medal score when tasted on one occasion would give it another gold medal score if they were asked to score it on another occasion. While it would be nice to think that this could be done by all show judges all the time, I have to say that believing so is a little naïve. A couple of years ago I measured the consistency of over 500 wine show judges (and potential wine show judges) and found that a surprising number found scoring consistently was a tricky business (Gawel and Godden, 2008). In my opinion, EVOO judging is more difficult than wine judging (I’ve done quite a bit of both) as the difference between an average quality and a high quality EVOO is much smaller than between the average and high quality wine. So it’s no surprise that consistency scoring EVOO’s for quality isn’t as easy as many people might think.
So with the aim of measuring the consistency of the olive oil judges I presented them with 12 EVOO’s which they scored for quality. Later on that day, they were presented with 15 oils, 12 of which were the same as those scored in the morning – but in a different order of course. The judges were unaware that oils were being presented for a second time as I led them to believe that we were conducting a different exercise (I can be a complete bastard!). I then correlated the scores given on the first occasion with those given on the second to the same oils. The results of the exercise allowed the judges to benchmark themselves against their peers.
Appreciating Style Differences
Every year I spend a lot of time at these workshops presenting diverse sets of EVOO’s – from different varieties and blends – early and later harvest – from different regions – mild through to very robust . All this is done to impress upon judges that it is bad form to penalize an oil because it either has or doesn’t have, a particularly flavour characteristic. The world of EVOO is wonderfully diverse – different flavor profiles and different levels of bitterness and pungency are necessary. Why? – because despite what some producers might tell you, no single style of EVOO can hope to be perfectly suited for use in every single culinary dish. While some styles might have a limited application compared with other styles, they all have their place ‘out there’ in one way or other. But there are high and low quality examples of each style, and recognizing this is the point of the judging exercise.
So here were the results of the 2009 Perth Show which took place two weeks after the workshop. I’ve given the average score for oils within polyphenols ranges. The higher the polyphenols level, the more bitter and/or peppery the oil (generally speaking), so the polyphenol level is a pretty good indicator of style. I’ve also turned it around and given the average polyphenols level for each medal ranges, best of class and best of show.
|Polyphenol Range||Average Score|
|200 – 299||13.8 Bronze|
|300 – 399||13.4 Bronze|
|Best of Class*||410|
|Best of Show||310|
*average of the highest gold medal scoring oil in each class
Lastly, if you’re like me and like seeing things in pictures, here is the plot of awarded score against polyphenol content.
The correlation between polyphenols level and score was a meager 0.005 which was as close to the magical random, ‘no relationship whatsoever’ value of 0 as you would hope to get. In short on all the measures, the amount of bitterness/pepperyness had no bearing whatsoever on the scores awarded to the oils. This is the way it should be. Good oils should be awarded based on quality not style. Fluke perhaps? – well I just spent some time on the flight back from the 2010 Royal Perth Show and calculated that the correlation between score and polyphenols level again showed that there was no relationship between score and polyphenols level (aka style) – the correlation was 0.001
Unlike EVOO shows elsewhere, the majority of Australian shows provide commentary on each submitted oil via the published show results booklet. The comments are written by the head judges of each panel, and are based on their opinion and those of the other two judges. However, the comments can get a bit jargonny at times, and while they clearly mean something to the judges who wrote them, they may not mean much at all to the exhibitor. On the other side of the coin, the comments can be so generic that they mean little at all.
So in one workshop I presented a series of comments written by judges in another show (The Australian National), and simultaneously presented the oils to which they referred. The judges attending the workshop were asked to match the comments to the oils after tasting them. We then discussed at length exactly what aspects of the comment made it easy to identify the oil, and what aspects of the comment was confusing or resulted in an incorrect match of description to oil. The logic here was, well if you can identify the oil from its description then the description must be informative. After picking apart the value of the provided descriptions, the judges rewrote the comments in a way which they felt made for the briefest yet most informative comment. While I’d admit that the art of writing brief but useful commentary on oils is a work in progress, it was a good start.
Gawel, R. and Godden, P.W. (2008) Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research, 14, 1 – 9.
Richard Gawel’s blog is Slick Extra Virgin. Reproduced with permission.