Dear FPA Members (Jammers),
After numerous email threads and telephone conferences, the FPA Judging committee (consisting of Claudio Cigna, Lori Daniels, Rodney Sanchez, Jan ‘Mahony’ Zahradniczek and myself) has agreed on a proposal for changes to the current FPA judging system. Based on FPA members’ input through various communication channels we tried to modify the judging system in order to:
– Encourage diverse, risky and innovative styles of play;
– Make judging more fair;
– Make judging less demanding.
The most common answer to each of these challenges is judging education: The current system is quite well-designed but often judges lack experience to apply it correctly. Because judging education is a time-consuming and ongoing task that goes beyond the scope of this committee, we focused on formal measures that are easy to implement to make things a bit better (hopefully). However, it is the Judging committee’s desire that judging education should not be considered secondary or insignificant as a follow-up to the proposed judging changes. In fact, the hope of this committee is that all FPA members who wish a viable, reasonable and fair-as-possible system will commit to learning more about the nuances (fine-points) of judging efficiently, objectively and accurately.
The pages below outline the key topics and agreement points of our discussions. We also included notes on some of the things we considered but eventually dismissed:
1. Reassembling Artistic Impression (AI) categories
2. Difficulty – Blocks or Phrases
3. Difficulty Multiplier
4. Execution and Breaks in Flow
5. Bonuses
6. Discussed but not Implemented
You, as FPA members, now have 45 days to comment on the proposed changes, so the deadline for your input is the 7th of June, 2012. At this point in the process, the idea is to fine-tune things or possibly dismiss them. New ideas will not be considered in this part of the process – sorry!
No changes will be implemented before September 2012, so you are safe to build your routines for the big tournaments this year based on the current judging system.
As you can imagine, the more one thinks about judging Freestyle disc, the more one realizes that it’s a very difficult thing. Part of the challenge is due to the lack of boundaries of a sport such as Freestyle disc (e.g. everything is allowed in Freestyle and, hence, everything has to be covered by the judging system) as well as the subjectivity of judging, especially in the categories of Difficulty and Artistic Impression. Our job as the FPA Judging Committee was largely to synthesize and operationalize what many players supported as modifications to our current judging system, and suggest implementation methods (what is doable) in a thoughtful and pragmatic way. Please give the proposed judging system ideas fair consideration, given this context.
After the fine-tuning input has been received, the proposal will go to the FPA Board of Directors for final approval and an official new judging manual will be created. After, the judging system changes will be implemented tentatively for 15 months, after which time, there will be another vote of the changes for final approval.
Thanks for your attention,
Philipp Lenarz (FPA Judging Director)
Given the hard data, I agree with the diff multiplier USING THE CURRENT DIFF AND AI FORMATS ONLY – changing those formats will itself change the balance, or why propose the change. The discussion on ex makes sense (secret – I pretty much judge ex that way already – although I rarely judge it). I lean a little on the diff blocks already as well (another little dirty secret) but only if a catch is imminent. Everything else will be extremely hard to educate properly, especially given that education and proper application is seen as our biggest problem with a system that has been around years. I feel the current system for the most part is delivering the right decisions.
Great Job Philipp! Very professional job…..here’s some feedback:
• I don’t like de-emphasizing flow as I see it – it is not an obvious skill for newer players who just wants to hit the move, and hiding it in Show will make it less rewarded…watch corey-evan rose bowl 78 Individual flow seem more aligned with Form than Show to me also.
• Online judging certification tests should be given – not practical maybe but a dream…the tests are practical, but getting everyone to take them will be the challenge
• The diff approach sounds reasonable….glad you avoided complexity of phrases which is too complex I believe
• We did the multiplier in worlds previously – glad it’s coming back
• For the crowd, show scores after each team!
• Good idea on ex –
o Need some reasonable approach on the 0.1 for brush-ups to save a co-op – this was abused previously where every brush became a 0.1 so avoid that – 0.6 deduction on a single move.
o A move where the disc goes out in front of the player in the brushing run, has to reach out to brush it or kick it, to keep the run or move going…is that a 0.1?
o I assume a clean swoop by a partner in a co-op to a planned bash to a co-op continuation would not be a 0.1 – this is where it becomes a slippery slope with maybe too much room for interpretation going against the spirit of the rule
• Love the bonus…
Really good job Philipp and committee!
By deciding that certain judging techniques are too complex or demanding on judges and using that as an excuse to dumb down the system, you’ve given up. Only a strategy that promotes educated judges, non-playing or playing, will ultimately stand the test of time. Only categories that demand as much from the judge as the player are worthy of changing the current system–which was already dumbed down from the previous system.
As long as the current system is structured as it is, with execution soooo heavily weighted, I believe the first stated objective of encouraging riskier play will be ever elusive. And although I’m all for making judging more ergonomic, if want to avoid mental demand in judging a very complex kinetic art form like freestyle, then I have to respectfully disagree. We should pride ourselves with the multiplicity of styles, controlling continuation skills, and mental approaches that our international community has developed and risen to heights we see today. It would be a step backwards to further dilute the discretionary factors that judges have as tools to be as fair as they can. If we take away categories, we take away data.
We could focus on ergonomics, posting instant results, and raising the esteem of being a competent judge to make judging more attractive to players.
In the past we tried out so many rule changes at smaller Freestyle Tournaments which gives a good feedback about the effects. Couldn’t we test the changes at a tournament before the judging system gets finally decided?
In my humble opinion one general,very important thing when judging Freestyle is:
Judge WHAT you see, not WHO you see.
There is a tendency that, especially inexperienced, judges are biased by the names of persons who are playing (motto: great player will perform great). Also masters can make mistakes, wobblers, etc.. and not perform well… and unknown players can maybe seedbust them on a good day.
Judges should focus on what is happening with the disc(s) and not on who is playing with them.
Another little step towards a more balanced judging system…
Hi
1
This is my first post to the judging discussion and I hope to have enough time to make all the posts I wish to, in the remaining short time. There will be a logical order in my posts, that’s why I number them.
First of all I would like to thank Philipp, Claudio Lori, Rodney and Jan for there big work for us players and for our sport – well done!
I would like to use my first post for some comments about the things written on the “start-page” and add some additional thoughts:
I understand we are not following any other general goals than trying to make the judging results better – demanding enough!
Goal 1: Encourage diverse, risky and innovative styles of play
I support this – this is going to make our sport even more attractive for us and for spectators
Goal 2: more fair
Of course, any system should be as fair as possible. But if judges are following the given rules of judging, they judge fair. So this is not a question of the system itself but it’s a question of how judges are handling it. To prevent unfair judging effectively we need to be able to have instrument to avoid this: for example scratch scores which are off (scratch highest and lowest scores in each category), or (less effective) having immediate and public numbers from each judge; this puts some pressure on the judge to be fair). Unfair judging can allways happen, because a judge wants someone to win or because missing judging-skills and we need to have a instrument to prevent it – on any given system.
Goal 3: less demanding
Yes, a good goal, but with limits. It’s a complex sport, and to much simplifications will not do good. The key will be education!
Education:
We need to do a much better job on that – on any given system!!
And education will start with a good manual and proper (professional) translation. I’m sure we do have a language problem in our sport. To know the exact meaning of words is not simple and it can make big differences! For example: In German we name our 4 or 5 minutes routines (I’m not sure if I write “routine” correctly?):
“Kür”, or “Vorführung”, or “Aufführung”: If I let this translate to English I get “Free program”, “Demonstration”, “Performance” if I let this translate back into English I get “Kür”, “Vorführung”, “Leistung” – something is wrong with the word “performance” – things like this can cause big missunderstanding.
Immediate scoring
On any given system we need immediate scores! Once more this was obvious at Paganello. Hundreds of spectators were enjoing the competition and were waiting and asking for results – but they didn’t wait for one hour, of course not. In any other judged Sport there are results within minutes ….. Soectators want to see the results, because they made there own ranking and they like to see the winners celebrate and they will feel with the loosers – there are so much emotions in our sport – let’s use them better for our advantage.
Where are the retired competitors? (one last general idea / maybe at the moment just a dream)
In any given judging system we could use the experiance of retired competitors.
In any sport many retired competitors start there carrere as coach or/and judge. I’m sure this would also be possible in our sport – but for this we need some money to cover their expenses and for this we also need to think if our sport is not worth to put a little more monney into it from our side – we need to start this, not big sponsors (they might come later).
More to come in my second post about judging- hopefully today
Reto
Hi all
Before I make my first comment on the judging discussion, I would like to thank Philipp, Claudio, Lori, Rodney and Jan for their huge amount of work, they have done so far for our sport and for us players – THANK YOU!!!
I understand that you wish to get comments just on the given subjects to make this process managable. Nevertheless I allow myself to give some general comments and proposals of different solutions as well, before I’m going to try to give comments on every subject in the remaining short time. I hope you (committee) don’t feel disrespected. I just have so many thoughts about many different aspects of judging which I would realy like to share with you. It would also be very difficult for me to make comments on just the proposed subjects, because I see a couple things little different.
Comments about the goals:
Encourage diverse, risky and innovative styles of play: This goal is ok for me although I think that this is not a big problem in todays system.
Make judging more fair: The judging system is fair, if it measures what we want to measure; that is an aspect of the judging system. But a judge can act unfair and this can’t be prevented by the judging system. A biased judge can judge unfair in any system. To avoid this, we need to be able to scratch scores from biased (or unable) judges. That means that we need more than 3 judges for each category. I don’t see any other solution against unfair judging (maybe immediate and public numbers from each judge would put some pressure on judges to be fair; but less effective than scratches).
Make judging less demanding: This is a good goal, but it has it’s limits. Freestyle is a complex sport and to much simplification will not do good. But I agree with the committee, that we can (must) make things easier, specialy in AI.
Main problems in todays judging (how I see it):
1. Redundant (overlaping) categories
2. quality of judging
1)
The current system is redundant in many ways: Example 1: Drop: minus 0.3 in EX, flow-deduction in AI, less DIFF in the timeblock. Example 2: player with lots of flowbreaks: less flow in AI and lover Diff. This need to be avoided (see my judging system probosal; in my next comment)
2)
The quality of judging need to be better. I absolutely agree with the committee that education is essential. And education will start with proper translation of all the manuals. I’m sure, that we have a language problem. It’s very demanding to know the exact meaning of all the words in a foreign language. Translation and training of judges is necessary in any given judging system. I think we should think about tests for judges, to have a better education. Additionaly it would be a big help, if we could keep retired competitors in our sport. In any other judged sport, many retired competitors start to judge. Where are our retired? It would be great to meet them every year as well educated judges – all it takes is some money to cover their expenses. That would be a very good investment in our sport!
More to come soon.
Reto
First, let me applaud the judging committee for taking on a task like this. Considering changes to the judging system is a thankless job. Everyone seemingly has a change to the judging system he is passionate about, yet few actually step up to wade through the competing sentiments.
Having done this a few times before, I’m going to come at this from the perspective not only of an FPA member and player but also someone who has gone deep inside our various judging systems, done statistical analysis, and considered how even small changes can throw a system far out of balance.
It’s essential to start with the goals or there is no benchmark for knowing whther we are making changes for the sake of making changes or whether our changes are getting us closer to an ideal.
Within the discussion of goals, there is a beast lurking. Some call it vision. Others call it bias. Most of the time, we don’t like to talk about it. It’s our personal idea of the kind of freestyle we want to reward in competition.
As mentioned in Philipp’s introduction, our sport is complex and so is its heritage of being freestyle. There has never been a strong movement toward narrowing the sport to a small subset of approved moves like figure skating does. Our challenge is different: how do we both give players freedom of expression and allow ourselves to measure their excellence.
As I share my thoughts in the various comment threads of this discussion, I’m going to circle back to goals – what they are, what they might be – and how the proposed changes impact them.
As others have shared, the greater need for our sport is judging education. Judging education has far more potential to change the quality of competition. Improving judging performance is just one part of the story – the more energy players invest in learning about their sport, the greater their potential as players and teachers of freestyle.
We’ve tried judging education occasionally but never made a commitment to it, and we’ve never expected our fellow players to fulfill anything but minimal commitments at the judging table. It could be that the biggest learning from this discussion is that revising the judging system isn’t needed. Instead, redefining our expectations of each other and committing to judging training is.
The three goals the committee shared were:
1. Encourage diverse, risky and innovative styles of play;
2. Make judging more fair;
3. Make judging less demanding.
Without getting bogged down in wordsmithing, I would suggest that these goals are near misses. Some of the proposed changes encourage specific elements of play rather than the general attributes mentioned in goal 1. If the sport wants it, it’s fine to decide to promote very specific things, but in that case it’s essential to be open and honest about it. I’m not hearing “encourage speed flow” as a mandate from players, but it seems like a very specific initiative stemming from Goal 1.
I would suggest restating Goal 1 as two goals:
– Welcome diversity and innovation
– Reward risk
Goal 2 is puzzling because I don’t often hear that our system is unfair. Disputes over final placements are much more rare than when I started this chapter of my playing career in 1992. I hear complaints that the judging system is imbalanced and does not always encourage technical excellence. That’s different from being fair. I would suggest restating Goal 2 as:
– Measure excellence and ensure a balanced judging system
As some have mentioned, there may have been some requests for Goal 3, but it runs at odds with quality judging. The current system has room for improvement, but with practice and education it’s easy to navigate.
This is the perspective from which I’ll share my thoughts about the specific suggestions in their threads.
2
Sorry for my double-comment yesterday. I thouht my first comment got lost and so I wrote and posted it again. Some other words, but the same comment after all.
Now I would like to share my ideas to change the judging system. Im presenting a different system than the committee, but if you read the details you will realise, that much of what I’m proposing has also been proposed by the committee. I think I’m going in the same direction as the committee, but in some parts I’m going further. It will be interesting for me to hear, if I’m going too far or if the committee could go further, since I realised, that in some parts the committee were thinking further but holded back to make some proposals, because they thought it will be too much of a change to be accepted.
I started thinking about changing the system a couple month before the committee publised their solution and I created a, what I think, possible solution. I hope the committee will still be opened enough to think about it, even it’s different to their solution. If not, it was still worth for me to write everything down.
My thoughts started with questions about the goals of judging. For me, the only goal of the judging system is that it makes the best team winning.
The best team ist the team that shows the best performance in the competition.
The best performance ist the one with the highest total of the skills technic and artistic impression.
Technic:
The technical skills are diffculty, execution and variety:
Difficulty is very important if we want to answer the question «who is the best player?». The better player is able to play higher diff. Higher diff means more restrictions and also more flow. Three moves played with a controll-delay in beween each, are less difficult than the same three moves played fluently. Good judged see the flow easily if they judge diff. So it should be part of difficulty, without beeing a redundant own subcategory.
But diff is not the only criteria to determine the best player. Better players also have bigger variety, than weaker players. And they execute their moves properly. This is the biggest difference between my solution of a new system and the system discussed so far. Difficulty alone is not enough to determine the best players, or the best technic of players. We also need variation (more different moves) and execution. We do have both you might say. Yes, I answer, but in the wrong place. Variety is within AI, although it has nothing to do with AI. Execution is a categoriy itselve, judged by someone else, although it belongs to the technic. Problems we are facing with that are well known. The diff judges and the execution judges are taking care of the same, without knowing how the other ones judge the situation. This must be done by one judge (category) to avoid redundance. This means the technical judges would give difficulty points (I leave it open if on timeblocks or phrases, not that important I think) and make marks about the execution during the routine. At the end of the routine the diff-points will be added and the total of the execution points will be substracted from the total of the diff-points, giving a first total about the technic.
So I wouldn’t drop execution, but I wouldn’t have execution as a own category anylonger, because it has all to do with the technic. Like this the same judge can decide if he wants to give the diff for a catch, (because the disc drop out of the hand) or not (because the disc was way off and uncatchable). With every diff mark the judge also put down a execution mark. For perfect playing a zero (meaning no deduction), for little uncleannesses a 0.1, for middle problems a 0.2, and for big problems like drops but also for rescues with (Ultimate)dives (since this looks really ugly in a performing sport) a 0.3.
Example1: Disc skips out of hand at the catch after a solo, player is able to pick up the disc very quickly and continue playing without any delay: I would give all the diffpoints because everything was (almost) completed and would make a 0.1 deduction because execution was not perfect at the end. Example 2: big set for a tripple spinning gitis, disc way of, no chance to make the catch and player is loosing his balance and he is falling on his bud: I would give no diff-points for the move and I would make a 0.3 deduction because this was very poorly executed.
About this I’m very convinced. I’m also convinced in having variation within technic. But I don’t have a good solution yet to make it handable for the judge to do this additionaly to diff and ex. A lot could be done also with the design of the judging sheet. Today we get no help from the sheet-designs at all, there are just empty boxes. But design of judging sheet could help a lot.
So, my thoughts about the details of that system are not finished, but I think the system would be good and we could find ways to fix the details.
Artistic impression in my next comment
Reto
A good discussion at Jammers….why not focus on education first and foremost. In regards to changing the system, why not try a simple change or 2 max in a given year and see if it works…change it back if it does not. Then try some different changes the next year….change it back if it does not. Evolutionary rather than revolutionary. I am fearful of a complete overhaul because it will take years to get back to the uneducated state we are in today…
3
Artistic impression
Freestyle is very technical, but it’s also (almost) an artform. Specially if we compete in teams. If we were just interessted in the technical skills we could have solo-competition (like super pro at Paganello)).
So we need a judging for artistic impression, that only taking care about the routine.
For me the categories for this should be:
Teamwork, Music choreography, „Flow/execution of the routine“ and maybe form
Variety and „individual flow“ are within difficulty (see my comment 2). With individual flow I mean if a player is able to connect elements in his solos or in a coop without flow breaks, see my comment 2).
General impression should not be used anylonger. If I judge AI, my likes and dislikes are allready influencing my judging of all the other subcategories of AI. With the additional subcategory general impression we give the most subjective category much to much weight.
Teamwork: Nothing to change. If we have competition in teams we want the teams to do a lot together. We just need to be carefull not to judge the difficulty of the coops, this is done by technic.
Music choreography: Nothing to change.
Flow/execution of the routine: Beside individual flow (see above or my comment 2) there is the flow of the routine. A good routine has no breaks – it flows. Flow breaks in the routine happen, when a team (or a player of a team) loses track, when they have to talk together to help each other remembering the next move or if they have to correct their placement. Flowbreaks in the routine don’t look good (even spectators can see, that something is not running how it should). It’s the synonym of execution in the technical mark (see my comment 2). The easiest way to judge this will be deductions. For smaler „problems“ minor deductions, for bigger „problems“ bigger deductions. Everytime a judge see a „problem“ (flow-break) in the routine he marks a small or big deduction. At the end he adds up and deduct from the AI interim total to get AI total.
Form
I could live without it. I never really understand the use of form. Technicly better players also tend to have a better form because they move smoother, so we have once again a redundant. And I don’t think that we want ballet like tip toe. So, it will not change a lot if we skip form.
That’s it – AI got much easier but still have the important subcategories to judge.
I think this system puts the important subcategories into the right place and this must be first step for a redesign of judging. The details within the categories and subcategories can be design as a next step.
My comments on the suggestions of the committee will be based on my thoughts in my comment 1-3
(Please excuse my writing mistakes. I’m not that good in writing in english and it’s taking a lot of time to check)
Reto
Outstanding job by the committee and great feedback from all. Good ideas in Peter Laubert’s feedback. In short the directions are good toward simplification and combining the categories and it would be feasible to do. (I also do not think that the multiplier is too burdensome as long as you have some poor soul who’s only purpose in life is to count the moves during the routines, and then feed that number to the judging panel. No way to do it while also judging.-Not really a topic here.) Also enjoyed Arthur’s fine tuning comments. Only strong feedback I have is toward the Bonus idea.
Bonus = Pandora’s Box; an inherently difficult (and herein I believe deleterious) idea though prompted by our collective joy of seeing the big move at the right place.
The inherent difficulty is that it will be next to impossible to fairly administrate a hugely subjective element such as this. Thus Bonus points will be inevitably awarded (by the proposed definitions) across the board resulting in larger point swings based upon greater subjective response than even our current system. (Or with many personalities it will be awarded based upon the similarity to our own styles thus self actualizing.) This will be dependent upon who is judging at any given time. My take is that we currently DO reward the big move and the bonus is not necessary. (I do it in difficulty and AI b/c the big move often has more risk and is efforted toward accent points in music.) – ADDITIONALLY big moves per se are not always all that difficult for the people doing them. They are ‘trademark’ in many cases and this subverts other potential rewards very much deserved by more technically demanding, often AS physically demanding moves; simply because they are more subtle, fluid, etc. and do not conclude with someone (frequently) crashing to the ground. Certainly those of us in the crash move category will be for the bonus as it’s readily interpreted as ‘big’. Yet I must harken to agree with much of what is being said in regards to expanding our scope of expectations rather than rewarding a narrowing of scope. The bonus idea, although a very, very, fun concept, seems woefully infeasible from the application standpoint and will narrow our scope. Will the bonus reward Matteo’s absolutely flawless spinning phlaud pull set to a phlaud in 2010 FPA semis? …that was BIG from a flow and continuity standpoint, yet would not be rewarded, or Pat Carrasco’s double spinning osis to end perfectly balanced and holding the pose for presentation while jamming on the sidelines (still relevant as an example though he was not competing). Those are far more compelling acts of athleticism than leaping really high for a catch. I fear that is what the bonus will reward (and I can jump really high when I want to : ) ), and that’s going backwards in my opinion because it narrows our scope of play. Simultaneously we would contend with even larger clouds of subjectivity than we do currently. — Fact of the matter is that a Bonus rule is redundant as well as overly subjective and will skew judging b/c we already reward such moves.
I’m not disinclined to have our sport be different than other sports (no other judged sport has a ‘bonus’ point system??) as that’s a great virtue of our sport and fuels our passion.) Simply don’t think that the current proposed bonus idea is necessary, or can be fairly implemented and/or workable.
Again huge thanks to the committee for their tireless efforts and for providing a forum for all to lend feedback and stimulate ideas. Fantastic to be a part of this community!!
Thanks for your hard work…