We need to worry more about LEO. Orbits decay automatically, but less and less the higher you go while still within LEO. There are many polar launches which makes debris come potentially from any direction, while in GEO all satellites go in roughly the same direction.