Topic: Autos.ca Comparison Tests Steering Committee

[jyarkony]

Hey folks, we wanted to create a place for us to discuss all your suggestions and respond to your query about our Comparison Tests. This can include everything from the vehicle selection process, scoring system to format of the published article.

I think you have all contributed a lot of great ideas that then get lost in the archiving of older threads, so it will be nice to have an easy way to reference your suggestions, explain our procedures, and even coordinate when Forum Members can join us to assist in the execution of these highly complex events.

We never get to see how the cars were scored or rated in each category form Autos or AJAC. Must be some propitiatory system that the "readers" would not understand 

Yup they must have a conspiracy of rating Mazda first.
First the compact CUV and now the compact cars, what's next mid-size too?

Anyways surprised at how well the Corolla did. Although I didn't think it would be last, surprised it surpassed... okay on second thought Corolla might be a good second choice for me too.

Good for Mazda but that is not the issue I have. Considering this was an in-house test the data is available to publish in a simple score sheet with several categories. I fail to understand why this can't be done. Jacob even avoided the question when asked by Boff.

We don't publish the score sheets because that is not what i want readers to focus on. We are a review site, and it is our aim to deliver great editorial content, and we want to report on the results of our comparison in context, rather than simply providing a bunch of detailed numbers that are to some degree arbitrary. When I get into work, I will create a Autos.ca Comparison Test Process thread in which we can discuss these issues.

Who wants to be on the Autos.ca Comparison Test Committee? and are you available next Thursday?

Let's keep this thread for discussion of the vehicles and this review itself.

[Snowman]

We never get to see how the cars were scored or rated in each category form Autos or AJAC. Must be some propitiatory system that the "readers" would not understand 

Yup they must have a conspiracy of rating Mazda first.
First the compact CUV and now the compact cars, what's next mid-size too?

Anyways surprised at how well the Corolla did. Although I didn't think it would be last, surprised it surpassed... okay on second thought Corolla might be a good second choice for me too.

Good for Mazda but that is not the issue I have. Considering this was an in-house test the data is available to publish in a simple score sheet with several categories. I fail to understand why this can't be done. Jacob even avoided the question when asked by Boff.

We don't publish the score sheets because that is not what i want readers to focus on. We are a review site, and it is our aim to deliver great editorial content, and we want to report on the results of our comparison in context, rather than simply providing a bunch of detailed numbers that are to some degree arbitrary. When I get into work, I will create a Autos.ca Comparison Test Process thread in which we can discuss these issues.

Who wants to be on the Autos.ca Comparison Test Committee? and are you available next Thursday?

Let's keep this thread for discussion of the vehicles and this review itself.

I agree that great editorial content is important and I enjoy reading the reviewers comments on the test cars but I want to see that data and I don't think I'm alone on this.  What time on Thursday   

[JacobBlack]

We never get to see how the cars were scored or rated in each category form Autos or AJAC. Must be some propitiatory system that the "readers" would not understand 

Yup they must have a conspiracy of rating Mazda first.
First the compact CUV and now the compact cars, what's next mid-size too?

Anyways surprised at how well the Corolla did. Although I didn't think it would be last, surprised it surpassed... okay on second thought Corolla might be a good second choice for me too.

Good for Mazda but that is not the issue I have. Considering this was an in-house test the data is available to publish in a simple score sheet with several categories. I fail to understand why this can't be done. Jacob even avoided the question when asked by Boff.

If you read the article, the scoresheets and categories are mentioned many times.
I don't expect we'll be publishing scoresheets for this one, but it is something we've discussed many times.
My personal opinion (which is absolutely not our position and is subject to change at any time anyway), is that our scoresheets represent an enormous amount of our own work and effort (intellectual property). You'd never expect anyone else to show their IP so freely and easily.
I prefer to keep those sheets as our way of accounting ourselves - if we're going to release it to the public, all that will happen is:
A: Other people might use them for their own tests - therefore removing our opportunity to be different and have different results.
B: These threads will turn in to 100 pages of, "Autos.ca scores are all wrong, why would you judge X out of 5 and Y out of 10?! Who cares about heated steering wheels?! I hope you get throat cancer and die Jacob".   

[Snowman]

We never get to see how the cars were scored or rated in each category form Autos or AJAC. Must be some propitiatory system that the "readers" would not understand 

Yup they must have a conspiracy of rating Mazda first.
First the compact CUV and now the compact cars, what's next mid-size too?

Anyways surprised at how well the Corolla did. Although I didn't think it would be last, surprised it surpassed... okay on second thought Corolla might be a good second choice for me too.

Good for Mazda but that is not the issue I have. Considering this was an in-house test the data is available to publish in a simple score sheet with several categories. I fail to understand why this can't be done. Jacob even avoided the question when asked by Boff.

If you read the article, the scoresheets and categories are mentioned many times.
I don't expect we'll be publishing scoresheets for this one, but it is something we've discussed many times.
My personal opinion (which is absolutely not our position and is subject to change at any time anyway), is that our scoresheets represent an enormous amount of our own work and effort (intellectual property). You'd never expect anyone else to show their IP so freely and easily.
I prefer to keep those sheets as our way of accounting ourselves - if we're going to release it to the public, all that will happen is:
A: Other people might use them for their own tests - therefore removing our opportunity to be different and have different results.
B: These threads will turn in to 100 pages of, "Autos.ca scores are all wrong, why would you judge X out of 5 and Y out of 10?! Who cares about heated steering wheels?! I hope you get throat cancer and die Jacob".   

No, no, no......I have suggested that a simple score card similar to what Car and Driver publishes. The data collected would populate a template with 10-12 categories which summarizes and score with specif weight into the final rating. Surely to fack that is what you do now to come up with a final score.

[MarkStevenson]

Personal opinion:

I don't think the scores from each judge should be shown for similar reasons Jacob mentioned. However, it would be very interesting to see the overall order of vehicles per judge, in addition to what Snowman is mentioning.

[sailor723]

We never get to see how the cars were scored or rated in each category form Autos or AJAC. Must be some propitiatory system that the "readers" would not understand 

Yup they must have a conspiracy of rating Mazda first.
First the compact CUV and now the compact cars, what's next mid-size too?

Anyways surprised at how well the Corolla did. Although I didn't think it would be last, surprised it surpassed... okay on second thought Corolla might be a good second choice for me too.

Good for Mazda but that is not the issue I have. Considering this was an in-house test the data is available to publish in a simple score sheet with several categories. I fail to understand why this can't be done. Jacob even avoided the question when asked by Boff.

If you read the article, the scoresheets and categories are mentioned many times.
I don't expect we'll be publishing scoresheets for this one, but it is something we've discussed many times.
My personal opinion (which is absolutely not our position and is subject to change at any time anyway), is that our scoresheets represent an enormous amount of our own work and effort (intellectual property). You'd never expect anyone else to show their IP so freely and easily.
I prefer to keep those sheets as our way of accounting ourselves - if we're going to release it to the public, all that will happen is:
A: Other people might use them for their own tests - therefore removing our opportunity to be different and have different results.
B: These threads will turn in to 100 pages of, "Autos.ca scores are all wrong, why would you judge X out of 5 and Y out of 10?! Who cares about heated steering wheels?! I hope you get throat cancer and die Jacob".   

No, no, no......I have suggested that a simple score card similar to what Car and Driver publishes. The data collected would populate a template with 10-12 categories which summarizes and score with specif weight into the final rating. Surely to fack that is what you do now to come up with a final score.

I have to say I agree with Snowy on this one. I'd be interested in a simple spreadsheet that would allow you to easily compare each cars strong and weak points

[nlm]

Hey folks, we wanted to create a place for us to discuss all your suggestions and respond to your query about our Comparison Tests. This can include everything from the vehicle selection process, scoring system to format of the published article.

I think you have all contributed a lot of great ideas that then get lost in the archiving of older threads, so it will be nice to have an easy way to reference your suggestions, explain our procedures, and even coordinate when Forum Members can join us to assist in the execution of these highly complex events.

Let's keep this thread for discussion of the vehicles and this review itself.

Would this be a "living committee" in that it is ongoing? If so, one recommendation, unless it is premature for recommendations as you might only be looking at administration for the committee at this point, is to include direct link to this thread on all application comparison articles. That would help steer readers into contributing on this thread rather than the specific comparison thread. Unless I've misunderstood the intent of this thread/committee?

[blotter]

I'm on the fence.  While the scoring data would be interesting, but I do think a well written article is far more important.  I can usually read into why someone didn't like this or that and it'll give me an idea if this lands more towards a personal choice (style points, seating) and make up my own mind if I feel that would apply in my situation. 

Some data is good to help have a simple comparison breakdown between all the vehicles.
So maybe a standard can be developed that lands somewhere in-between.

[jyarkony]

That does sound like an interesting idea.

Our current scoresheets are too large and detailed and not oriented towards producting a presentable result, especially with so many vehicles, but perhaps we can average certain larger categories in the Comparison Sheet to arrive at something publishable. However, the Scoresheets are not necessarily oriented for this purpose, so it will take some reorganizing, and that will be a goal of this upcoming mid-large SUV comparison.

[Black Hatch]

How hard can this be?
Just rate each vehicle based out of five like all the test drives you are doing right now.

Assuming that 2.5 is average or the low mark and 5 being the top mark.
Basically just so that we as the audience can separate them into tiers.

[sailor723]

That does sound like an interesting idea.

Our current scoresheets are too large and detailed and not oriented towards producting a presentable result, especially with so many vehicles, but perhaps we can average certain larger categories in the Comparison Sheet to arrive at something publishable. However, the Scoresheets are not necessarily oriented for this purpose, so it will take some reorganizing, and that will be a goal of this upcoming mid-large SUV comparison.

It doesn't have to be overly complicated. I'd just be interested in knowing things like which car was considered best value, best performance, most luxurious/best fit and finish, etc etc

[Northernridge]

I'm not interested in score sheets. Unless you guys are attaching instrumentation, everything is subjective anyhow and that's perfectly fine because the reviews are the impressions of the authors which is what I'm looking for. I'm also interested in who says what, don't obscure individual opinion, emphasize it…especially where the internal opinion differs…or state when it's unanimous for that matter.

For comparos I need to know WHY you like one car over another, not just that you think it's better. My priorities may differ from your own so I can live you drawing a different conclusion than me, but it helps to know why…or all I can do is disagree and not understand your conclusion.

The most valuable thing you guys can offer is relative comparison, you get to drive all the cars. I don't need you to tell me what I think of a car but it's very helpful (and interesting) to see what you have to say about a car relative to its competitors, previous models and the many others you have driven. I can not replicate that.

I also appreciate personal impressions much more than describing spec sheets and option lists, I can look those up. I want to read your impressions.

[jyarkony]

Hey folks, we wanted to create a place for us to discuss all your suggestions and respond to your query about our Comparison Tests. This can include everything from the vehicle selection process, scoring system to format of the published article.

I think you have all contributed a lot of great ideas that then get lost in the archiving of older threads, so it will be nice to have an easy way to reference your suggestions, explain our procedures, and even coordinate when Forum Members can join us to assist in the execution of these highly complex events.

Let's keep this thread for discussion of the vehicles and this review itself.

Would this be a "living committee" in that it is ongoing? If so, one recommendation, unless it is premature for recommendations as you might only be looking at administration for the committee at this point, is to include direct link to this thread on all application comparison articles. That would help steer readers into contributing on this thread rather than the specific comparison thread. Unless I've misunderstood the intent of this thread/committee?

good suggestion... we'll be screening other comparo threads for comments that will be more constructive here, and make this sticky whenever we have a major Comparison under discussion. can't put a link in the article itself, but I'm sure once the comments inevitably crop up we'll be able to post the thread link in the comments.

[Weels]

That does sound like an interesting idea.

Our current scoresheets are too large and detailed and not oriented towards producting a presentable result, especially with so many vehicles, but perhaps we can average certain larger categories in the Comparison Sheet to arrive at something publishable. However, the Scoresheets are not necessarily oriented for this purpose, so it will take some reorganizing, and that will be a goal of this upcoming mid-large SUV comparison.

That sounds like a good idea to me.  I also like Mark's suggestion of showing the overall ranking of each vehicles per judge - i'd have no problem addressing questions afterwards of why did xx finish last for you... etc

[normancw]

I'd like to see in-car videos.  Very little editing/production would be required, like the POV videos made by Winding Road, which have no commentary.  They would be simple to record - just strap a video cam to your head and try to hold your head steady!  They would provide a good insight into the interior dash layout and controls, outward visibility, NVH, engine sound...  If a picture is worth a thousand words, then a video is worth... ten thousand?

Check some out here.
http://www.youtube.com/user/windingroadmagazine/videos

Here's one on the Mazda 3
http://www.youtube.com/watch?v=t21D9VlBqC0

I'm not interested in score sheets. Unless you guys are attaching instrumentation, everything is subjective anyhow and that's perfectly fine because the reviews are the impressions of the authors which is what I'm looking for. I'm also interested in who says what, don't obscure individual opinion, emphasize it…especially where the internal opinion differs…or state when it's unanimous for that matter.

For comparos I need to know WHY you like one car over another, not just that you think it's better. My priorities may differ from your own so I can live you drawing a different conclusion than me, but it helps to know why…or all I can do is disagree and not understand your conclusion.

The most valuable thing you guys can offer is relative comparison, you get to drive all the cars. I don't need you to tell me what I think of a car but it's very helpful (and interesting) to see what you have to say about a car relative to its competitors, previous models and the many others you have driven. I can not replicate that.

I also appreciate personal impressions much more than describing spec sheets and option lists, I can look those up. I want to read your impressions.

^^^ I agree.

[jyarkony]

How hard can this be?
Just rate each vehicle based out of five like all the test drives you are doing right now.

Assuming that 2.5 is average or the low mark and 5 being the top mark.
Basically just so that we as the audience can separate them into tiers.

so who rates each car? the author for that write-up? then we'd have cars in 7th with higher scores than cars in 4th depending on writer's subjective opinion of assigned car... it has to be a collective score, and we're talking about adding work here, which ain't free... but yeah, that star rating system is the obvious answer - just have to figure out how to translate our score sheet averages into that system, although because if its more limited broader categories, might still have a lower-scoring car receiving higher ratings.

[jyarkony]

That does sound like an interesting idea.

Our current scoresheets are too large and detailed and not oriented towards producting a presentable result, especially with so many vehicles, but perhaps we can average certain larger categories in the Comparison Sheet to arrive at something publishable. However, the Scoresheets are not necessarily oriented for this purpose, so it will take some reorganizing, and that will be a goal of this upcoming mid-large SUV comparison.

It doesn't have to be overly complicated. I'd just be interested in knowing things like which car was considered best value, best performance, most luxurious/best fit and finish, etc etc

this is something I've tried to emphasize:  for each car, we mention which categories it won, and which it was worst in, but most importantly, why. however, it always seems like there is a little bit more that we could detail, but this is already a massive article and possibly beyond many people's attention spans. smaller groups are easier to cover in greater detail.

[Weels]

I'd like to see in-car videos. 

... I have some of exactly that (although not from this comparison...)  Could do video for the mid-size suv test coming up if there is enough interest

[Black Hatch]

How hard can this be?
Just rate each vehicle based out of five like all the test drives you are doing right now.

Assuming that 2.5 is average or the low mark and 5 being the top mark.
Basically just so that we as the audience can separate them into tiers.

so who rates each car? the author for that write-up? then we'd have cars in 7th with higher scores than cars in 4th depending on writer's subjective opinion of assigned car... it has to be a collective score, and we're talking about adding work here, which ain't free... but yeah, that star rating system is the obvious answer - just have to figure out how to translate our score sheet averages into that system, although because if its more limited broader categories, might still have a lower-scoring car receiving higher ratings.

I thought everyone would rate each car and you came out with an average star value out of five

And this way you could separate to individual results as requested above...if you wanted.

[jyarkony]

I'd like to see in-car videos. 

... I have some of exactly that (although not from this comparison...)  Could do video for the mid-size suv test coming up if there is enough interest

We should definitely at least record it. However, as simple as it would be to publish raw video, that is something we'd have to consider internally, balancing value against other factors – publishing raw amateur video might not reflect the level of professionalism we hope to convey to our readers. Now, posting some extra video in the forums, there's no rule against that for sure, but the times I've worked with video it has always seemed to me to take ten times as much time as I expect it to.