You really should spend your time on something else man. I guess you are just trying too much.
only problem with this is the results will get skewed more and more inaccurate as you keep basing guesses on older guesses.
Everybody reads them...even the ones who like to take time out of their day to lie and say tldr. The trouble is comprehension. But I wrote this one so it will make sense to whoever has to actually edit the programming not the general public.
The DBR counted by using your method would be very inaccurate. There is one big problem. The in-game skill brackets are huge, especially the very high bracket.
Let's say the brackets ranges are like this:
Normal: 0-2000
High: 2000-4000
Very High: 4000+
An average very high game can have an average MMR of 5500.
A low-end very high game can have an average MMR of 4200.
A page 1 game can have an average MMR of 8000.
If you compare a page 1 game with a low-end very high game, the difference in MMR is huge but both of those games are "very high".
That would mean that winning a page 1 game would give you the same increase in DBR as winning a low-end game. Eventually, players who play low end very high games and players who play page 1 games will achieve the same DBR value as long as their win rate is the same.
Same applies for normal and high games. A 'high' game can have an average of 3800 MMR but, at the same time, another 'high' game can have an average MMR of 2200.
The current skill brackets are just too huge and therefor it is impossible to estimate DBR by using your method. If there were more skill brackets (10 for example) than this would be possible.
how many players with stats on would need to be in a game for u to have a good enough idea on the average dbr of the game?
wat would u think is statistically significant and wat wouldnt be
DBR is always somewhat inaccurate as are all such TSR type models. As many of you will recognize, when you start a new account your rating changes quickly trying to converge at on a best estimate of your skill rating. For the purposes of matching (ie MMR) this is important to get fair matches. But every time you win or lose it changes. The fact that you can go from Normal to Very High in 5 to 10 games shows you a realistic idea of how these type of systems behave. That is not some kind of mistake Valve made. That's just how the math works.
The rating of every player (even if we stats on for all of them) has some uncertainty if they have just been on a winning or losing streak that uncertainty can be very large. When it is very large your rating changes quickly. When you have been winning and losing in a stable near 50/50 pattern it changes slowly.
The only difference is the standard algorithim is set up to guess at all ratings on the team in an iterative fashion based on the previous value alone. Those guesses would be better if we had data on all players, but we really don't need it. Bad guesses (ie high uncertainty) just makes convergence take longer. So your DBR will change more slowly than with an algorithim that has all the information for all players.
If you understand what I wrote you will see that this does not cause wild fluctuations in your score. It does the opposite. Games you play with few known ratings will cause a smaller change to your score. Game with many of the players stats on with have a larger impact since the data is more certain. Exactly how much can be adjustable easily. It would be a scalar factor in the equation. You could arbitrarily decide that the assumed values for DBR of unknown players have only half the impact of known or a quarter. Whatever makes you feel better about it. The smaller impact you assign to them the slower the scores will change.
If there were feedback into the actual matching from the guesses at DBR you could have a problem with the bad guesses from the large uncertainties generated by just using three numbers to describe Normal, High, and Very High. But there is no feedback. MMR controls the matches. DBR is just an attempted description of it. So there is no problem. As your MMR goes up, you will play games against players with higher DBR and games with more people with stats on too. And your score will be rechecked and adjusted every time based on who you played against.
All that is needed to estimate what the MMR system did. You don't have to get it right. You just need to know that it went up a bit for a win and down a bit for a loss. The DBR model will follow the MMR score despite the uncertainties. It will just follow it slower when MMR changes a lot.
Hi Relentless,
Thanks for making this post, it was fun to read and contemplate.
In my opinion, DBR and systems like it can only be effective when they have a factual history for each player that can be used to accurately determine the impact of the next action. If a fact isn't available, we have to calculate it. If it's not certain (it never is), we consider the uncertainty into the next calculation. If we don't know, we can't proceed as doing so is unsafe and will endanger the facts we already have.
Unfortunately, that's where it ends for me. If we don't know the skill of a single player in a match, the best we can do for calculating a rating for players in that match is a guess (and not a particularly accurate one, given the factors involved).
After you've accepted that there's a guess involved, we have two options:
- We can truthfully convey the lack of confidence, the player skill certainty value will go down and eventually we won't have a usable rating for any player at all. This doesn't work for obvious reasons.
- We can pretend that we have more confidence than we do and players will have ratings, but they'll be wrong and they'll get more wrong every match they play. I certainly couldn't stand behind this as being accurate, so we couldn't deploy it.
It's like the saying "every time you have sex with someone, you're having sex everyone they ever had sex with". Except that in this case you're having sex with 10 people every 40 minutes, only 2 of which are using protection and everyone has a different STD that you're eventually going to get and die young. Sounds bad.
Do you have the capability to simulate a smaller test set of data? I bet if you go back in your history and write up my algorithm...
Pick a time and cut out the people who did not turn stats on. Then run it forward a week and see how the DBR values compare to what they actually were. I bet you get nearly the same numbers for everyone. There would just be some lag. DBR will lag behind MMR because it will change slower with larger uncertainty.
You will end up showing the scores people had maybe 3 to 5 games back. Anyway that's how this type of mathematical system behaves with large uncertainty. It doesn't get less accurate, it just converges slower.
There is no reason to be less confident in the results after many games. I would say, yes it will be less accurate for a few games. But that doesn't matter. DBR does not impact the matches. People want to know where they are after lots of games. Not exactly how they ticked up and down to get there.
Just to be clear...it is not possible for DBR to stray off track from MMR with my suggestion. Every match will recheck and bring it back because its only a description of MMR. It does not feedback into the matchmaking. At the top ranking that people really care about the uncertainties will be less because so many more people in the matches have stats on anyway.
You've explained guidelines that would help write a system, but I can't get over the fact that we need to be truthful about certainty, and without a steady gain in certainty for all players the system doesn't work. We'd need an algorithm for accurately determining the skill of a group of transient unknown actors who are unpredictable in size and origin.
Take this match: http://dotabuff.com/matches/250295976
What is the change in rating and certainty for HOTTOWN?
I can't answer that question with a straight face. If I had every detail of how Valve MM works, I'd still probably be hesitant to even make a guess. MMR/TSR systems work by gaining knowledge over time. There's no knowledge to be gained from that match - for all we know he's platinum queueing with his bronze friend. It could be the greatest win of all time or he should lose points for even playing it. We don't know and can't know.
Its not that hard to answer the question. The simple answer is the change is small. If you want you could make the change for matches with only 1 player with stats on 10% of the full certainty change. You know he lost. So have it go down a tiny bit. It really does not matter exactly if he loses 0.25 DBR for this match or 0.1 or 0.5. His later matches will correct it.
---side point, if its possible to lose points for winning or gain for losing then you have a rounding error in the program somewhere.
Or if you really want to be on the safe side have the change be zero when you are below a certain confidence. That's what you've got right now. No change in DBR for 6 months.
But I can tell you now that nearly all of your players matches with low confidence in stats are going to cancel out and have no impact on the system. I am 100% certain of this because they will all have nearly 50% win rates when they get to a lot of games and all those games will be in normal. If someone is at+5 wins in normal, he will have higher DBR than if he is at -5 wins in normal. At that is truly as accurate as you could hope for if you had full data. Unusual circumstances where someone wins a few easy games and loses a lot of hard ones will tend to cancel out over many games. And they will be partly reflected by the scores of those people who have stats on. Major changes will change skill brackets and again reflect the difference in DBR. Games won in High will be worth more than Normal. And you always carry your DBR into the game. If you got placed into High on a stack and your DBR is only 900...well its going to move you up a lot if you win, just like MMR does to everyone who stacks. Then later if you try to play on your own with that High MMR you will lose and go back down. DBR will follow those changes.
Anyone who is up more than 5 wins when they get to 100 games will be playing in High or Very High with lots of other players that have stats on. You should give up on the idea that the system has to perfectly track MMR. MMR itself is uncertain anyway and everyone knows it. Instead make something that works.
If you want to reflect the greater uncertainty in an upfront fashion then don't have DBR update until it is 3 games old. That will give it enough time to catch up to the lag caused by the assumed values.
============================
EDIT: I forgot to answer half Jasons question!!! :P
The uncertainty value that you need to chain each game together...I recommend adding some zeros in place of the unknowns (or a tiny number if you have to be non-zero) such that this changes little and only based on the few players whose stats are on. This number is for the higher order derivatives and honestly doesn't do much. Newton's Method is really just as good and it doesn't need this number at all. All you need to do it keep re-measuring every game.
OH I Just realized something important that some people may have misunderstood. The TSR equation usually deals with an uncertainty that it uses to move the score by larger amounts when the uncertainty is larger. (ie the number in the above EDIT)
This is not the number I am talking about. I mean you add another factor. A scalar multiplier to the size of the change which is based on the uncertainty caused by assigning skill bracket based values for the players with stats off. This will cause changes to be small when fewer of the players stats are known. These small changes will eventually catch up as your DBR stabilizes around your typical level because you start to go to 50% win when MMR is stable.
When your MMR changes fast, if you had a lot of no-stats players in your games...DBR would lag behind for a while. But if you keep performing at the new level it will catch up. However if you swing back the other way, DBR will already be there waiting for you and won't have to take such a drastic turn.
==========================
If you take a look at the outlying winrates from the perspective of the model designers...they are scores with large uncertainty. So Murs account where he had like 85% win on 200+ games...that account MMR could not stabilize because it could not find a top to Murs skill level. So it has to just keep going up until someone beats him and then drop back down based on the rating of whoever finally does it. Anyway all those players who have far from 50% win. The uncertainty number in their MMR is large and it never gets small because there are not enough people who can beat them and really establish their score solidly from above and below.
If you gain/lose a small amount of skill by winning or losing and most of your opponents are unknown, and most people also hover around 50% win rate, then what you're describing is a system that will have almost nothing to do with skill. If anything, it'd be more about luck (of being paired with the right people who have DBR on and haven't improved in the last 6 months). It might "mostly work" for 5% of the player base, but will fail miserably for almost everyone else.
Sorry but I just don't see this as something we're going to devote resources to. We have a lot of promising projects in the queue.
I'm sorry you feel that way. You are incorrect that it would only track the score of a few players accurately. It would track all of them accurately...it would just track only the top end closely...the majority would lag more since most of the low MMR games have stats off for many players.
But I'm glad you have some other projects in the works. I look forward to seeing what you come up with next.
I think Relentless's idea can work quite well for solo queue games. For me a "Solo queue DBR" is good enough.
However, in the games with stacks, these calculations can be quite inaccurate. If a bronze player stacks with a diamond, then the difference between the estimated values and the real ones will be quite high if game is played on normal bracket.
Sorry man the English is too much in this.
ENGLISH SKILLZ. In other words there are a lot of words that I don't know the meaning of.
I find Relentless idea to be quite valid as some have stated.
I also believe it wouldn't take much to implement, maybe some of the community members can even help build up an example of this just for the sake of it.
I mean, as what we (don't) have now, what bad could it do to try?
Just my 2 cents
DBR in it's short lifespan was awful enough as it was without incomplete data
if DBR hadn't been removed and kept the same algorithm the top 50 would be 90% abusers of it's weaknesses
you don't see that now but if you see how much effort people put into getting on the DB winrate page, which compared to the DBR top 50 nobody actually cared about, you'd probably not want a DBR system for the sake of your matchmaking experience
@Yoshi
Look at the amount of whine/flame posts about DotA2's matchmaking in this forum, considered it's
a) actually quite decent and
b) not Valve's forum, so the critic (if you would want to call it that (please don't)) does not reach them.
Now imagine the amount of comparable posts about a new DBR integration, considered it's
a) based on a lot of presumptions, likely leading to incorrect values, and
b) this is the forum of the actual developers of the feature.
Aside from the questionable values themself, of course ;-)
@Socram
I give you right for that, can't quite say the opposite (almost! :-) ) but there's just one thing that makes me curious, if we can say that way.
So far:
1. We may know this idea might be inaccurate (not sure, since i'm not so good in calculations but i take what everyone said till now for a good start)
2. We know it will cause whine as it's causing now the MM system (that i also don't quite like to be honest)
But what we also know is that this kind of systems will always attract whiners (even if sometimes are only good criticism really :-) ) as will any other aspect of the game.
People will whine about heroes being imbalanced, map being also, etc. etc.
That being said, i was (as said above) just curious about this system to have a very generic idea of where i am placed.
Smart people will always use the tools provided properly, abusers will always abuse.
Anyway, again I perfectly understand your point of view and in part share it
![d[-_-]b ZeKon <(^^,)>](https://steamcdn-a.akamaihd.net/steamcommunity/public/images/avatars/76/76d348ec67f9b483edde80a8e9b3fe928af34be4_full.jpg "d[-_-]b ZeKon <(^^,)>")
could you somehow use the stats gatehred on http://dota-academy.com/home/ to help with the very-high tier??
Stop trying hard untill Valve changed their mind. The failed MM is feeding too much poisonous data (yes I believe it's even worse than drawing players totally random to form a match) to any legitimate model.
The dota skill of one player is an extremely sparse space. How he play 1 out of the 100 heroes against the enemy team. It is totally impossible to compare the positions of one player in such a space with huge dimension. As well impossible to form a single number to compare two players.
One approach is to use hero score for each hero to build the rating system. The MM system is totally destroying this system. As you win a game with one hero, the next game MM put a tard in your team so you lose. Now you can pick one hero too make it lose all the lost game, and other heroes got high win rate, as well as high stats.
Look at how this guy did:
http://dotabuff.com/players/125538203
And I tried a bit myself on Sunday
http://dotabuff.com/players/138775413
The only way to solve it is build an AI into the game, just like someone is watching the game as it progress, and evaluate the skill of the player based on the role they played in the team. Support, Carry, Ganker, Initialter, Pusher, and so on. Well it's a bad thing for professional obs... this take the role they played.
Look at how the old dota 1 days, lobby games are good option, build large lobby rooms like GG did before, players can join any game in a lobby they wish. Noobs themselves will stay away from the high skill lobbies . Obviously valve tried something new, but pretty much failed at this point. lol
Please sign in to post comments.
I expected that Dotabuff would be able to find a way around the issue of Valve blocking stats by default, but since they have not I decided to give an example of how to construct an algorithm to do it. I think there is a lot of interest in having DBR update again. The part of the community that is interested enough in Dotabuff to buy Plus would certainly support the return of DBR values. And if you go back to your polls on the subject (http://dotabuff.com/polls/dotabuff-rating) the rest of the community mostly likely will as well.
Without complete information it is more difficult to model a system, but it is not impossible. It is merely impossible to be exact. So you need to treat the DBR values like any system whose behavior you are modeling with a sample of measurements because measuring everything is never possible.
In this case what you need to do is have estimated values for the DBR of the anonymous players. This is could be done simply by using the Normal, High, and Very High brackets. The result is an uncertainty in the measurement. The uncertainty will be much larger for Normal then High, and larger for High than Very High...but people who really care about their DBR and buy Plus are in High or Very High anyway. And I will show that it will function well enough even for Normal Bracket players too.
For example, lets say that you have a game in Very High with 7 players with stats on. 7 of them will have an exact DBR, 3 will be assigned an estimated DBR for the purpose of calculating the updated DBR values. The estimated value can be determined by taking into account the MMR restrictions of the skill bracket...so its reasonable to assume the DBR of the players will also be close. If you have a team with 5 stats on, then you have the average DBR for that team exactly. If you have one team with 4 and one with 3 with stats on let the unknown players be given an assumed "Very High" rating of (whatever your data shows the typical Very High player to be) lets say 2,000 DBR. Then you will have ratings for both teams.
For each rating that you assumed based on skill bracket you have an uncertainty. If only 1 rating is assumed in a Very High game the uncertainty in the team average DBR will be 1/10 of an estimated range of reasonable scores. Lets say that estimated range of reasonable scores is 3 standard deviations from an average of Very High and is maybe 400. So the average for the team if the 4 known players are 1500, 1900, 2100, 2300 would be 1960 +/- 80. Make this uncertainty inversely proportional to the change in DBR for a given match. So while a fully known match would have 100% of the normal change in DBR a match with this example of 2% uncertainty would only change by 98% of the value calculated with the assumed values.
In Normal bracket matches where few players have stats on and the range of possible unknown DBRs is larger the uncertainty will be large and the overall impact on the system model will be that players who play only in Normal games will find their DBR slow to change. However it will still be possible for the model to function, remain stable, and accurately update scores...although it will take more games for them to become stable.
Now for a "worst case" example.
In a normal game perhaps the range of reasonable scores is 0-1600 with an average of 1200 maybe 3 standard deviations would be 800. If we had only 1 exact DBR on a team of 500 DBR the average would be 1060 +/- 640. On the other it would be 1200 +/-800. The total uncertainty of the game is 64% so the change in DBR awarded from this game would be only 46% of the value assigned if all scores were known exactly.
You can see from the worst case that even Normal bracket players scores would only take twice as many games as with total certainty to accurately adjust.
By using assumed DBR based on skill bracket to bring an uncertainty into the measurement you will only delay the convergence of the system to stable values. If you use this method you can re-write the algorithm to calculate DBR that will update for everyone despite not knowing the values for people who have stats off.