Okay. I’ve been looking at the ranking system that USAC is using and can’t really figure it out at all. I’m pretty good with numbers, but these numbers are baffling. I’ve sent a bunch of emails to USAC and only have one returned. They corrected part of the issue, but just created or maybe more accurately, exposed another. Here’s an example below. This is cut from the USAC website for the Jingle Cross UCI race last year. Here is the Elite results and ranking points awarded. Remember the lower points are better.
Now here are the results from the Master’s 35+ race.
Supposedly the higher quality the field, the lower the ranking points. Here in the Elite race, you have arguably 5 of the top 10 cross racers in the country. And in the Master’s race you have a bunch of Master’s guys. Not close to the same quality of field. But, the first two places in the Master’s race received a better ranking score than the winner of the Elite race.
I think I’ll eventually talk to someone at USAC that knows something about this ranking system, but it would be nice to be better informed before talking. So, if anyone has an interest in trying to figure out this new algorithm and could explain it to me in layman’s terms, I’d be more than happy to listen.
i attended the UCI CX Promoters Summit this past May, where Steve Johnson and Tom Mahoney from USAC gave us a presentation on how the new ranking system worked. the intent was to provide a better way to measure riders against each other than just the “points accumulation” system.
it’s based on alpine skiing, which uses current rankings between riders in a race to create the score. not being a statistician myself but being in a field that sees a lot of statistical data, it seems sound. the only issue i see with it is that there will be a bit of a “transition” period until the system accumulates enough data to make the model really accurate. it also works better when there is a more robust data set, like the UCI race with split times. soon all CX promoters are going to be asked to provide finish times for everyone. not totally sure how that will pan out or if the model compensates for an inconsistent data set.
here’s the USAC FAQ on it with the basic calculations:
http://www.usacycling.org/news/user/story.php?id=6998
hope that helps a bit!
Steve,
I make my living doing this stuff and hopefully your observations are turning into bug reports that will get fixed.
What they are trying to do is not easy.
– The chances the person writing the queries is doing their own testing are near 100%. Hint: someone like you should be doing Q/A on the data.
– I will be surprised if the ranking system ends up working at all. This is USAC we’re talking about. To be fair, there are some good people there doing right by the federation. It’s just a mess at the top.
Steve, we are facing the same thing in mountain biking.
While its not life or death for me, its weird for me to win my respective division (CAT 1 30-34) and then see the top 3 in the (CAT 1 35-39) all get a lower score than my win…… And both divisions had roughly 15 racers, and if finishing time means anything (which it probably don’t), the CAT 1 35-39’s that got lower scores all posted times slower than the 30-34 division…..can’t quite figure how that makes the CAT 1 35-39 field “Deeper”??
You have to start a system like this with some base assumptions. just from a cursory review, it looks like USAC started with the rankings going back to 2008 races, and then based on the race size, allocated base points which to build off of in the future. The oldest race I can find is here: https://www.usacycling.org/results/?permit=2008-258
At this point, non of the racers should have had any “carried” points, so they just gave the winners ~500 points, and went from there.
At some point, it does look like bigger races may have been allocated better points.
I believe the points for each race are probably based on the quality of the field to some degree, and the USAC event type…
This being said there is an inherent problem built into the system, namely that racers that do not race each other often have point systems that are independent of each other, so the rankings become arbitrary at this point. For example, masters racers in California probably rarely race masters from New England. The points may rank them well against people they race against often, but that is not necessarily true for people they don’t race against.
It also doesn’t control for course conditions, but that probably is a lesser effect. However, since it only takes your best 3 scores into account instead of some averaging system, it could also make the number unreliable.
If it would make sense, it wouldn’t be the USAC!
I forgot to mention their process is a black box from the outside.
As much as I know about how to do the job unless someone outside is willing to publish the process it and then the most important part, **listen and make changes**, it won’t do any good to try guessing out here.
Steve, Played around with this and found that the points awarded are points for the category that you raced, not total overall points. ie I raced Jingle cross as a 3 the first day, and in the open (SS) race the 2nd day. the results from my ss results would not count toward my rankings as a cat 3. I would imagine that the same would be true in a masters race. your points from a cat 1/2 would not count towards your ranking as a master. This is how I interpreted it.
Steve
Thanks for looking into the ranking system. I have no idea how they are doing it or what. In fact I have always wondered why we are ranked. Other thank individual pride is there a prize for a top ranked athlete that isn’t a pro or elite racer?
Over time, the points for elite guys should go quite low – single digit-ish. If a top five guy races the other top five (lowest possible Race Quality Value) and then wins, he will almost always lower his points, possibly by 10%. This lowers the Race Quality Value next time around, etc.
Seeing elites with rankings in the mid-200s indicates the system is too new still. (Everything gets seeded at maximum points – 540 for the winner, 590 for last place)
Alpine ski racing uses a similar system. It’s been around for decades and every race gets scored. It’s not perfect, but it works well. Six point results are the gold standard to earn World Cup starts. 200 point guys are the equivalent of cat 4s.
It’ll take time and consistency for the cyclocross rankings to prove their value, but they’ll get there.
I looked at Todd Wells’ results according to the USAC website, and he doesn’t have too many ‘cross races listed. Considering that they calculate the “Race Quality” by averaging the 5 best ranked riders that finished in the top 10, I’m not too surprised that the Elite ranking is a little lower than the masters race, where a lot more results were posted.
It looks like a good system, but unless every single result is posted it fails just as much as the previous system.
why should a seperate 35+ race get any UCI recognition? IMHO cat 1s should not be allowed to race 35+.. AT ALL! the USAC upgrade system is extremely flawed.. we need regionalized selection races as qualifiers. if a cat 1 35+ guy can’t qualify then he should have to down grade to race the 35+ and earn his way back to cat 1 status. or simply not race and wait for next year. At the local road level we now have 2 seperate races of nearly equal ability levels. it discourages fuller fields and higher quality competition and discourages lower category 35+’ers from participation.
my bad, those are usac points and not UCI points.. but a flawed system IMHO none the less..
I am two spots behind you steve. Here are some
standouts. It doesnt matter as they have uci points
but it shows bad math.
49 218.26 Jonathan Page Northfield, NH
102 239.52 Brian Matter Sheboygan, WI
122 247.50 Mitchell Kersting Louisville
134 252.70 Molly Cameron Portland, O
at the risk of putting a USAC fanboy hat on, i really want to encourage people to note two things that have been mentioned here:
1) it’s not perfect, but nobody said it was. it’s better than the old system by many bike lengths.
2) the more data that’s put into the system, the better the results. having to use old data to base the current rankings on isn’t ideal, but it’s all the data there is. it should sort itself out by the time Nationals rolls around, when it’s actually meaningful.
3) absolutely email/call USAC and give them your feedback (politely). if there’s one term that’s used more in USAC management more than anything else, it’s “customer service”. you pay them money, give them your input. they aren’t uncaring people, but they do have a nationwide membership of tens of thousands of people they have to serve. don’t expect a response right away. hell, i’m in the Local Association and i can’t even get a prompt response a lot of the time.
Thanks. I understand the system pretty well and still don’t understand how the Elite race could be ranked lower than the Master’s race. Most those guys started racing in Cross Vegas and Jingle Cross wasn’t until Thanksgiving. That is plenty of time for them to amass points.
They (USAC) really need to assign someone full time on this project if they are planning to implement it this season. Especially before Cross Nationals in Madison.
So far, just checking my results, they are completely screwed. I’ve raced 4 Cross races this season, with three being UCI races and they have no ranking points assigned for any of the races in the Elite category.
Then they go on and assign points the next week to other races with the same riders. That is just going to screw up the actually results more and more. The only way this works if they take into account the chronological order of the races and re-tabulate all the the results each time they enter a previous race.
So, my batting average is 000%. Lots of mistakes last season, wrong addition on the current ranking (that they will use if they enter the races I competed in) and no ranking points for any races I’ve done this year.
If my rankings are the norm, then they need to come up with some other system to line up cross nationals in 3 months.
Steve, I’m sure you must have found this: https://www.usacycling.org/news/user/story.php?id=6998
Pretty much describes how the system works. As Andrew noted, the problem is likely to be the seed values. It probably would have made more sense to seed the first ranked races with different max/min values to create some separation between the categories (and/or seed from Nationals results). In theory this will self-correct, but is dependent on having some racers migrate between categories and regions.
I don’t think there is anything wrong with the system (assuming its doing what it says its doing) per se, a similar system has been working decently on crossresults.com
Old Ranking System vs. New Ranking System
Without a doubt, this new ranking system is light years better than ranking systems they had in the past. Based on amateur riders that I know it has them ranked in the correct order and changing weekly based on most current results (best 3 results in 12 month period).
You and a few others have pointed out some obvious flaws in select races. I believe these flaws will work themselves out with more stats gathered. The best I can figure – this dynamic complicated model self corrects statistically over time. The jury is still out but this is the best ranking system I have seen in a long time. (Provided you race at least 3 races in a given 12 month period.) If you are a “Super Stud” and get 100 points Race 1, 100 points Race 2 and don’t have a 3rd Race you would have 266.67 score. (100 + 100 + 600)/3
Statistically, I like the concept and for those who really care about rankings it’s better than what they had before…
It’s a little confusing how they are saying there is only one ranking value no matter which class you race in. It seems like a problem if the method of calculation (copy below) can result in the above result. In fact based on the calculation method everyone in the 35+ race could have been slower than every rider in the Pro race, raced a shorter race and still get more points.
It seems that there could be a momentum problem. It appears that the 35+ riders are likely to continue to keep better and better rankings as their relative ranking increase above the pro field and there would be no incentive to race to pro event. Soon there will be better rankings available in the 35+ than the Pro event. I don’t see how this will correct itself unless the pro riders start racing the 35+ even if they can.
So my current cross ranking of 464.80 isn’t going to get me on the front row this year?
Walid is faster than you bitch.
Hey Dammit ! I am not faster than him ! Leave me out of this. I’m just an innocent.
They shoulda just paid Colin and used crossresults.com
Seems to work pretty well to me there. I bet he’s pretty steamed as it seems to follow his model (which he got from cross country skiing), my scores between the systems are very close.
In putting on my cynic hat, I see USAC to try and get more races USAC sanctioned. If none of your races from other sanctioning bodies count: then you won’t get a call up.