Yeah, not rendering nodes out of view came up a little early on in the current implementation of infinite scrolling as something I was looking to do. The only way I could think of not rendering previous rows was to lock a specific row height, which was not something I was excited about (and it breaks down even further on nested tables, which have a state independent of its parent grid).
It's definitely something Ryan and I (or anyone else that might want to contribute) will be revisiting in the near future, potentially as a config option.
In our case, we needed to filter on hidden columns (as we still cared about the data behind the scenes), but I'm thinking that this is something that should be optional.