It’s hard to know where to begin. From the concept, to the outreach, to the staging and execution of the event, Ryan Neil did a great job with the Artisans Cup. For all of your effort, thank you very much! Thanks too to the exhibitors, vendors, volunteers and visitors without whom the event could not have met with such success.
Vine maple by Michael Hagedorn
In the past few weeks, much has been said about the exhibit hall and the beautiful trees. I’ve seen less attention go to an important artifact from the event – the Judging Rubric.
If you have yet to read it, I highly recommend it. It’s one of the best distillations of how to evaluate a bonsai exhibit available to the public. It’s also a good checklist of items to keep in mind when displaying bonsai at any event.
Top dressing by Greg Brenden
Above all, I’m impressed by the effort Ryan made to foster transparent and objective judging. There were five judges – enough to provide balance among different tastes. Each judge had the exhibit hall to himself during judging – a luxury that is not always possible or practical. The judges also evaluated every tree in the exhibit – a time-consuming task that provides great insight into the judging and scoring processes. The rubric also called for “honorable and consistent” assessment – the most we could ask from any judge.
As for what to judge, the Rubric calls out both technical and artistic considerations. Among the technical, “Compositions should display a high level of technical proficiency in all its forms.” Trees must be healthy and appropriately dense for their variety. My favorite point pertains to craftsmanship in cleanliness, development, pruning and wiring, plus preparation of containers, stands and accent pieces.
Accent
As for the artistic considerations, “Compositions should display a high level of artistry and proficiency in design.” This applies to the tree, container, stand and accent.
Ron Lang pot displayed by Mike Pollock
Rather than prescribing how to dole out points – trees were evaluated on a 0-60 point scale – the Rubric left all interpretation to the judges whose scores would be made public.
Have any points in the rubric brought up good questions about how to score displays? Definitely. One example is the scoring of three-point and shohin displays. Does it make more sense to tell judges how point breakdowns should work in these cases? Probably not, however, some additional guidance might help.
Stand displayed by Stephen Liesen
Likewise the breakdown between tree, container, stand and accent. I spent a good amount of time walking through the exhibit thinking about how I would score trees and found that it was hard to get started until I came up with a rough framework for how many points could go to each element of the display. The main concept I wanted to protect was stated in the rubric: “a good tree with a crappy display should not win this show.” Rather, “Winning trees should exhibit quality from head to toe in every aspect of the display as it’s presented in the exhibition.” Well said.
I found that the rubric was well suited to the Artisans Cup, and with some modification, could be suited to a variety of other events. The number of judges, for example, and the time allotted for judging could entail significant changes to the framework. That said, the points relating to the evaluation of individual displays are a great starting point for any organization looking to incorporate high quality assessment into their event.
View a copy of the Judging Rubric at the Artisans Cup website.
Subscribe to Bonsai Tonight
New Posts Delivered Every Tuesday and Friday
Ray Norris says
Well said Jonas, I think a new standard in judging. Wonderful show.
ceolaf says
Here we go again…
1) The most important point in the rubric is that last bullet in the “Expectations” section. That is the call for consistency. And yet, multiple judges made clear during the “Ask the Judges” panel that they ignored that call, that consistency is NOT to be desired. They explained that different trees SHOULD be judged by different criteria.
2) At least one judge said that he include a degree of difficulty component to his judging — a criterion that is entirely absent from the rubric.
3) Statistical analysis has show that there are further criteria that might have influenced the judges scores. While far from definitive, this suggests that the issue should be accounted for in the future.
4) The judges made clear that they each judged the tree exactly as each of them has judged trees in the past. They acknowledged not a single way in which they deviated from their (recent?) past practice. A rubric so ignored might not be worthy of such plaudits.
5) It would not be necessary to “prescribing how to dole out points” to make the judging far more transparent. Each judge would merely need to explain how s/he doles out points. Following the strictures of the rubric, s/he would just need to be consistent. Boon actually does this, I believe. The others do what experts call “holistic” judging.
6) Unfortunately, the “rubric” is really no rubric at all. It is a list of criteria — and a pretty expansive one at that. But it does not in any way either guide judges in how they scored the trees (see #4 above), nor does it clarify for owners, artists or others why any tree received the scores it did. A rubric must be more than a list of criteria, and by being more than a list of criteria, it can achieve more than this “rubric” did.
********************************************
Clearly, a points system is preferable to some partial list of ordinal rankings. But a points system so opaque as this one is only a small step better than a partial ordinal system.
I agree with the decisions to have each judges score the trees independently. I think it is difficult to argue that collectively scored (or ranked) trees is a better option — particularly with such experienced judges.
I cannot figure out the logic of dropping the high and low judges scores. I cannot understand why judges were not asked to abstain from scoring trees they have worked on or have been worked on under their oversight (perhaps in some defined amount of time).
Any scoring guide — not even speaking of rubrics — should explicitly inform the judges whether especially strong traits or aspects can compensate for weak ones in scoring, or if each trait or component of a score has a firm ceiling. Once that is decided, you can supply or ask each judge to come up with their own “rough framework for how many points could go to each element of the display.”
etc. etc..
********************************************
Obviously, traditional judging of bonsai exhibitions is a mess — has been a mess for a very long time. These problem are far from unique to the world of bonsai. These issues are so common that they need answers in every classroom of every school in the world.
My big point is that many fields actually HAVE made progress on this issues, already. Some organizations have their answers, and are able to use scoring as a better vehicle to advance their real goals. American bonsai could learn from them.
In the meanwhile…if anyone is THAT impression with TAC’s “judging rubric,” it’s just because the bar has been so low for so long.
James F. says
Terrific post, Jonas. I think ceolaf makes a good point. Although the goal of outlining a new scoring system is admirable, it is clear that the judges in this show either rated the trees by different criteria or otherwise have very different takes on what the rubric/criteria means. Kudos to the event organizers for being so transparent with the process, it is really informative and provides others with ideas on how different scoring systems might work. Reviewing the posted scores, one can see that the way in which different judges ranked trees within their own personal conception of the rubric are wholly inconsistent among the group of five judges. In some cases the highest scored tree by one judge is the lowest scored tree by another. There is no statistical correlation between scores of the different judges when evaluated as a rank order for each judge. I would go a step further than ceolaf and say that this essentially proves that the judges did not share a common standard for the rated attributes. I also agree that dropping high and low scores makes no sense and actually censors some judges from fully contributing to the process. That is, all scores from one judge were lower than most of the scores from all of the others, resulting in a statistically significant, larger fraction of that judge’s scores being dropped. The highest scores from that judge fell below 75% of the scores from all the other judges. This results in a decreased impact of that judge’s ranking on the outcome (I would say that it is the internal ranking of each judge that matters more than the numerical score assigned because the latter assumes that everyone knows what “60” means in terms of quality). There is a similar, but less dramatic effect for another judge who gave generally higher scores.
For me, the larger question is how does one judge art as good or bad. I think bonsai is more like visual arts than say a dog show in which individuals are compared to an accepted standard. The posted scores from the artisans cup show that there is tremendous variability between experts on what constitutes greatness. Because of this variability, having a relatively small number of judges (5) results in confusion and possibly less than ideal results – by that I mean there was no agreement among the judges about which trees were best, and a mediocre score from one judge potentially results in a better final ranking than the highest score from a different judge. The trees that won the show are amazing, but so are almost all of the ones that didn’t win.
I think it is impossible to quantify one’s liking of art, but it is possible to know which pieces one likes best. I would like to see what happens with a system in which a larger number of attendees (like >100) select their 2 or 3 favorite trees. The tree with the most “votes” would win by virtue of being the favorite of the largest number of people who appreciate the art. That’s kind of an American way of voting, no?
Bill Costa says
Here we go again…..Read what I have to say because I can do it better, Whatever!!!!
bonsai Mohegan says
Hi Jonas. How’s it? Thank you for posting Ceolaf’s comment and not “moderating” it to the trash. Too many people do that and we only hear dull, lame…I will digress. Keep up the great blogging.
ceolaf says
Let me be clear: I loved The Artisans Cup.
* I loved the ambition of it
* I loved the trees, the pots, the displays.
* I loved the lighting.
* I loved the ambiance.
* I loved Jim Gremel’s very traditional tree and I loved David Crust’s untraditional display. (Well, actually, I did not love it. But I was glad that it was there, and my biggest disappointment was that there was not MORE stuff that really challenged the audience, as Crust’s container did.)
It was a great experience, and I feel bad for anyone who missed it. I think that we all have a lot to learn from the things that The Artisans Cup did well — which was quite a lot!
I take issue with the scoring and the compilation of the scoring. I think those procedures could have been handled better. But the scoring and compilation of the scoring is WAY down on the list of things that matter about The Artisans Cup.
I am glad that Ryan and Chelsea want to improve bonsai judging. I am with them in that. I hope that my comments on scoring and compilation of scoring support their efforts in this area.
ceolaf says
James,
Let’s name names.
1) Boon used a narrower range of scores than anyone else, and had a lower average than anyone else. As a result, his score was the lowest (or tied for the lowest) 80% of the time. Thus, his score was tossed 80% of the time. (Boon’s score was never the high score).
2) DeGroot’s score was the high score 68% more often than Walter Pall’s, and more than twice as often as anyone else.
This is NOT their fault. I do not think that they did anything wrong. This is the natural outcome of a lack of training, guidance and calibration for the judges.
People may differ about values and the goals for judging, but there are basic techniques for dealing with situations like this. As you mention, the scores could be translated into imputed rankings and then averaged. That would have slightly altered the top trees. Or, each judge’s scored could have been standardized (i.e., subtract them mean and then divide by the standard deviation). That would, in my view, have been the most respectful of each judge’s efforts, and would have produced even more different results that the method used.
Regardless of the method, Randy Knight’s tree wold have won. This is about the rest of the rankings.
ceolaf says
James,
I want to give your point about the inconsistency across the judges its due attention.
None of the correlations between the judges were even medium strength, by the standards of social science research. Particularly notable is Colin’s near-zero correlation with Boon and negative correlation with Dave Degroot.
Again, this is not to say that anyone did anything wrong. It might be that bonsai simply cannot be judged in any way consistently, and standings are merely a result of who was selected to judge.
Obviously, we hope that any give judge would score the same on a different day, but we know from the field of assessment that this is NOT a given, even among the best trained scorers.
I am particularly troubled by the lack of correlations between these judges. They each possess impeccable credentials in the field. If The Artisans Cup’s judging was this inconsistent, I have to ask why we bother.
As for your suggestion of opening up the voting, a lot of clubs do that with their own shows. This would be a different kind of judging, and would be even more subject to predictable biases (e.g., size) that expert judging. I certainly do NOT think that that it would have been appropriate for The Artisans Cup, whose mission includes pushing American Bonsai forward. The popular vote is always going to be somewhat conservative in nature.
I have no doubt that a strong rubric would have lessened some of these issues. But they really are something that we should address as the great American Bonsai community. Why do we want scoring and judging? What do we hope to achieve by it? Are our current approaches to scoring actually likely to help us with those goals?
James F. says
I really loved this show – it was an amazing experience and I will definitely attend the next as well. So many of the trees were spectacular and the range of completely different styles, approaches and species added to the excitment. I guess I really don’t care that much about judging and scoring, but I got interested in the opportunity to analyze the numerical data from this very subjective experience to see what I could learn about this aspect of bonsai. It is probably fair to say that without judging, the element of competition would be lost and potentially some of the quality along with it – maybe not. I also wonder if clearer guidelines for judging or a select set of judges would result in participants “targeting” their submissions to meet the specific expectations. For example, large format juniper or whatever. That could change the experience of the show. I guess there are a hundred different ways one could judge a show and none is perfect. I am pretty sure that numerical scoring as done in this show actually results in a kind of random selection, which is fine when most of the trees could easily hold up as winners. I wonder about a non-numerical system. Something like : Each judge picks their 3 personal favorites and then all of the judges deliberate together to establish a final ranking of these trees. Instead of scores there would only be critiques and the final agreed upon ranking. I appreciate that the organizers were so transparent with their process for this show because I had never really thought much about the role of judging or how to do it before.
leebates says
Personally, I dislike the dark lighting. Added drama, perhaps. Certainly reduced the amount of information converyed in the photo. Were the trees judged in the dark also?
ceolaf says
1) The lighting was NOT good for photos, but worked VERY well for viewing an art exhibit. The human eye simply works differently than a CCD sensor. These pics above are not the best photos taken in that room. I know that I worked hard to figure out how to take good photos of the trees there, and eventually I figured it out.
2) As for judging, they turned on the lights. However, they did not explain their thinking on that.
It could be that the judges are most accustomed to viewing/judging trees in simpler light, and they wanted to accommodate that. It could be that they think that more would be visible in simpler light.
I think it was a mistake to present the trees to attendees differently than they were presented to the judges.
3) I believe that Ryan and Chelsea’s plan is to offer a better set of photos online, for a fee. This would be in place of the traditional commemorative book.
leebates says
When judging, every detail is important. Nothing but “simpler” lighting would do. As a bonsai enthusiast, I would prefer seeing what the Judges saw. The show lighting, while dramatic and powerful for some visitors would, for me, (without being there) like lipstick on a pig. Perhaps if I saw it, I would think differently.
bonsai Mohegan says
I have issues with controversial judging that can be called transparent when it may not be. When the winning tree (bush?) ends up being a tree belonging to the supplier of so many stolen trees (yamadori), even supplying some of the other participants trees, most if not all, stolen. It sets a very negative precedent. And now “the thief of the mountain” isn’t even keeping these stolen treasures in the states. He’s teamed up with others (W.P.,etc.)to ship the best overseas. Europe has no import quarantine, just visual inspection. So all the sickly weak trees stay and the healthy goes (T.T., etc.).
ceolaf says
bonsai Mohegan,
1) Are you suggesting that Randy Knight should have been ineligible because he is a vendor to other participants?
2) What is your definition of “stolen”? If he had relevant government permits, would that mean the yamadori are still stolen, or that they are not stolen?
3) I don’t understand your last point. If yamadori is immoral theft on its face, then why does it matter whether they end up 2,000 miles away or 5,000 miles away? Why do continental or national lines matter so much?
bonsai Mohegan says
Ceolaf, it’s not that vendors should be ineligible. Not at all. It’s the question of bias. Whether intentional or subconsciously. Stolen, because if he had a permit for the tree someone bought off of him,(claim does), did they get a copy? If they purchased this in any manner (directly, indirectly, trade, etc.), then the permit is void and the tree was collected illegally (stolen). The permits we are speaking of clearly state “for personal use only”. I’m sure those who are doing this will change from “I had a permit”, to “I collected it on privet property”. This should not be how bonsai is promoted. It is unsustainable. It encourages others to attempt collection and results in too many dead trees. We should be promoting it by growing and MINIATURIZING trees. Promote bonsai and not oneself.
ceolaf says
I have long been concerned about collection, both legally and morally. It’s an important discussion to have, and to revisit again and again over time.
Can you point me to where I might be able to see for myself the language in the permits? Are they listed on state or federal websites?
bonsai Mohegan says
Celiac, are you from the states?
bonsai Mohegan says
Sorry, my spell check changes word, Ceolaf.
bonsai Mohegan says
Hi Ceolaf. How’s it? I would first like to thank Jonas for facilitating such discussions. I consider his bonsai blog to be the very best. It’s the only one I read when time is limited. Between his misho, experiments/comparisons, allowing discussions of topics others might fear,etc.. Top notch! I hope everyone who reads his blog signs up to follow. I know me and others do not sign up to most because we do not like the whole blog post in our email, but we are going to so our support will be shone. As far as the permits, I will not divulge any information on them so as not to encourage there use. I will tell this…an elder went up the mountain, to hear tales of great hardship and survival. To be encouraged to persist. To be enlightened and inspired. What he found on that spring morning was loss and sorrow. What had lifted him for years, decades, was now gone. If anyone has a reason why rk should have the right to take what is not his. To take such power, inspiration,encouragement, beauty, then please inform me. That elder is now gone. May his spirit find again a place of such.