Yep, that makes a huge difference and it'd bother me all the time.
I can see those things go missing very easily. I popped mine out when I had a battery issue and needed to shift the transmission to neutral without any power. I made sure I put it in the change pocket right away, though.