Well, there is a really hackish way to do this, actually. You could preload the gif and then use something like http://slbkbs.org/jsgif/ to jump around the gif frames.
It would be unfeasible to do so. Assuming the displayed image is a reasonable size, say a dimension of 320x240, you would run out of pixel space in a jpeg due to dimension limitations after about 38 minutes @ 24fps. Granted, that's not using the manifest, just straight skipping around in the sprite.
To make this anywhere near possible on a large scale would be feasible though. The easiest way would to be create the manifest while encoding the video. Take the same compression scheme but save the encoding diffs for every frame.