Certainly the dealers low ball the trade-in value for the above reasons. They are in the business of making money like everyone else.
It all comes down if its worth to trade-in or put up with the hassle to sell private. You need to do your due dilligence. I'm sure most people don't.
The same goes when you buy a car. I for one, I always offer $500 over the CCC wholesale price and ask every dealer around if they accept the offer until one does. I don't care where I buy the car from as long as I get the price I want for the car I want.
The dealers do the same when the they offer the trade-in price. If it works out for you, it's fine, if not, move on and sell private.