Dear metanoia,
I remember the ArchiCAD/Revit discussion here. It was very interesting and emotional. But, you are right, it is a very complex problem - to compare and rate different CAD/BIM systems. I doubt even if two very professional and impartial users can judge - the problem is so complex.
I think, that more public approach can be more effective. Project-based.
Imagine, we have a detailed description of working process under some very wide-spreaded and well-known type of architectural project. For example, like that:
1. Client come to some architectural studio. He wants an individual cottage (sorry for terms - I am very far from western architectural practice).
2. Client has some documents: the list and the documents itself
3. He describes his requirements: ...
4. He wants to see proposals: description
5. Architect generates some idea: the idea - detailed description
6. Architect order some additional information, as photos, GoogleEarth etc.: the list and the documents itself
7. Architect makes the proposals: pictures, documents.
8. Client looks at the proposals and request to change something (some kind of variations, which are very common)
9. Architect does
Etc.
It have to be a very detailed scenario from the first request to the last minute of work process with all kind of changes, variants etc.
If we have this, we can start our competition - just on the Web.
1. Select the most important BIM-related decisions for working under this type of project at particular system.
2. Discuss the best methods for each particular stage.
3. Proceed with creating BIM model, step by step with detailed description - how we do something and why we do it in this particular way.
4. We will describe all problems, which come during our work. And our "income" too.
5. If some expert has a fresh idea about the best method of doing something in a particular system, Ok we can discuss and change
I guess it could be a very useful resource - both for BIM comparison and systematic description of the best practice. But writing such kind of scenario itself - is a very complex job. It needs at least some architect who logs everything
...
Cheers,
Leo