Do you think we still have the source code to all C64 binaries out there? We don't. And in 5 years, we won't have the source to all the stuff we are running now.
The point is that if you have only x86 and ARM binaries, you can't run them directly on a new architecture. It isn't a matter of changing an implementation.
It's like trying to run a C64 binary today. The only practical way is emulation, which is slow - but thankfully fast enough in this case. In general though, it means new architectures will run more slowly than existing ones. That's not a good thing.
GP was right, including just x86 and ARM is not good enough. Heck it even ignores x86_64 right now, and won't support new SIMD extensions that come out from time to time. Not to mention be a barrier to anyone introducing a new arch, as GP said.