Are they really that different? To me, it seems these are the last SUVS that can actually go off-road. I'm looking towards an auto transmission.
How long have you driven each?
The manual 6 speed FJ is quite a different drive than the automatic. The "operating" difference between either of them and the Xtrail is even more of a difference.
The FJ is just simply a larger drive. It's harder to hold on the highway in strong winds. The stock tires are useless. You need to take them off, store them, and reinstall when you sell it.
The Xtrail with the off road package is more conventional obviously and more urban friendly. We had a 2008 FJ that we brutalized, floated in water propelled by the wheels. The bottom was just smashed to crap and it still ran like a champ until it caught on fire.

I would not recommend an FJ for urban use. The units with the roof racks, which is a Canadian thing primarily, won't go in many underground parking lots.

The X trail is probably more suitable for running around Ottawa, but still tackling most off road. Put a Warren winch on it.