I always felt that the Feynman Lectures were really, "an incredibly deep understanding of freshman level physics, targeted towards physics graduate students." (As opposed to freshman level physics targeted towards pre-med students.)
Even after I got my undergrad degree in physics I would reread the Lectures and would see things on a whole new level than I ever understood them the first time.
Edit: even "Surely you're Joking" had jokes that only someone that had graduate level general relativity would get (talking about "gee-mue-nue, gee-mue-nue")
This is the key. The person you explained it to could probably not turn around and explain it to someone else, but could walk away thinking that they understood it.
Well he made that statement in 1965, and then wrote QED, which is a layman's explanation of his Nobel Prize work, in 1985. So perhaps he set out to prove himself wrong.
QED does explain quantum electrodynamics without any calculus or other math (for example, he uses imaginary clocks as a way to explain wave interference), but I think if the average person read it they wouldn't walk away with even a partial understanding of it.
I think it's partly motivational. There is definitely a level of understanding where it feels like you understand a concept, and then when you try to teach it to someone else, you realize when they ask basic questions that you don't understand it as deeply as you thought you did.
So, if you think you understand something, it's worth trying to explain it simply, so you can push your own understanding of the subject.