Exactly. I also grew up to be bilingual from a young age, and I didn't learn my second language in a way an adult would, by connecting foreign words with the words you already know from your first language, but in a way a toddler would, just by listening to it and maybe some reading.
After I began to think in English (my second language), I found that I wasn't speaking my first language anymore, I was just translating to it from my second one. And because the two weren't connected in my brain, I started frequently forgeting words and having trouble with phrasing.
Edit: found it: https://www.jitbit.com/alexblog/249-now-thats-what-i-call-a-... . It was on Hacker News 275 days ago, but received only 4 comments. I guess I saw it on reddit.