I will give you my thoughts: We have a 2004 Sienna (FWD, not AWD), so same body as the 2006 you are looking at, along with a 2009 Subaru Forester (the new body style). So the set up is very similar to the options you are looking at.
Hands down, the family asks, when given a choice, to travel in the Sienna. More room, better ride, less noise. And with good winter tires, it has gotten everywhere it needed to go, so AWD is not really needed for us. And we have a stretch of highway that has some of the deepest snow accumulations, and wet snow at that, in BC.
I use my Forester for my forestry work (oh the irony

), and for getting to remote sites for camping/canoeing. The Sienna just can't go to those places. That is the Forester's biggest advantage.
But you can not beat a mini-van for over-all versatility.
As far as actual fuel usage goes, we have seen 34-36 mpg on the highway with the Sienna and Forester, but the Forester gets a bit better in town.
We have had no significant issues with either, both bought new. However, the AWD Sienna will have some problem areas, like the power doors, and having to use run-flat tires.
Bottom line, if I had to pick one over the other, it would be the Sienna for its superior versatility.
And as noted, check whatever you get to make sure that you can get the car seats in. And with 3 adults, and twp kids, you will need to check on seating versatility as well. You will probably find the Sienna is the winner, then decide if AWD is
really needed for you.
Also, got to keep the wife happy
