The elements and everything used during the scroll checks is in fact cached (which you'll be able to verify by looking at those internals you mention) and throttled per my comment below. As another commenter pointed out, the project page itself is full of drop shadows, text shadows, and css3 gradients, and opacity. Those are probably slowing you down way more than the plugin. Do you see the same performance problem from the relatively unstyled examples?
Plugin author here, I can help clarify that. The scrolling is throttled so that all the checks aren't run with every single scroll event, as that would choke everything. So the small delay you see there is some 0-100 millisecond value while it hasn't triggered the event yet.
You can change the throttle by setting $.waypoints.settings.scrollThrottle = 50, or whatever number you like, to try to reduce flickers in cases like these. Just be aware its a balance you'll be striking with executing more scroll checks. But all the selectors and offset values are cached so the performance should still hold up should you choose to lower this value.