Ask HN: How does a game like Just Dance score movement?
Hi HN, playing Just Dance on Switch while I am stuck inside. Can anyone explain how a game like Just Dance determines the accuracy of your dancing given a single Joy Con per dancer? Of course it uses telemetry such as the accelerometer, but what algorithms are at work? How does the game process the telemetry? Sometimes the scoring seems a bit arbitrary / not sensitive enough / too sensitive. What movements actually matter? For calibrating the songs, do they capture many dancers/repetitions and average it, or do they take the data from one good performance?
1 comments
> what algorithms are at work? How does the game process the telemetry?
The term you likely want to put into Google is "gesture recognition".
Sadly it's hard to tell exactly how Ubisoft are doing it - since they're not about to put Just Dance on GitHub any time soon - but Schlömer's 2008 paper [1] about the Wiigee gesture recognition library [2] describes one possible approach using k-means clustering, hidden Markov models and Bayesian classifiers to recognize gestures based on the accelerometer in the Wiimote (similar to the one in the Joycon).
I'm guessing that the game regularly polls the Joycon's accelerometer, fires off events when a gesture (like a hand motion) is recognized, and scores you based on how close in time you were to the expected time of that gesture (with additional points for being closer to an "ideal" gesture).
> For calibrating the songs, do they capture many dancers/repetitions and average it, or do they take the data from one good performance?
The former, both to gather gesture training data and to playtest it.
Since it's just a game, they can take a lot of liberties with what constitutes "accurate" dancing - it's far more fun to play if the game occasionally classifies a misstep as a perfect dance move and gives you a freebie, than if it consistently treats correct moves as missteps.
[1] http://wiigee.org/download_files/gesture_recognition_with_a_...
[2] http://wiigee.org/
The term you likely want to put into Google is "gesture recognition".
Sadly it's hard to tell exactly how Ubisoft are doing it - since they're not about to put Just Dance on GitHub any time soon - but Schlömer's 2008 paper [1] about the Wiigee gesture recognition library [2] describes one possible approach using k-means clustering, hidden Markov models and Bayesian classifiers to recognize gestures based on the accelerometer in the Wiimote (similar to the one in the Joycon).
I'm guessing that the game regularly polls the Joycon's accelerometer, fires off events when a gesture (like a hand motion) is recognized, and scores you based on how close in time you were to the expected time of that gesture (with additional points for being closer to an "ideal" gesture).
> For calibrating the songs, do they capture many dancers/repetitions and average it, or do they take the data from one good performance?
The former, both to gather gesture training data and to playtest it.
Since it's just a game, they can take a lot of liberties with what constitutes "accurate" dancing - it's far more fun to play if the game occasionally classifies a misstep as a perfect dance move and gives you a freebie, than if it consistently treats correct moves as missteps.
[1] http://wiigee.org/download_files/gesture_recognition_with_a_...
[2] http://wiigee.org/