Philip D. Plowright, Matthew L. Cole



Explicit measures of design quality are almost unknown in architectural design, particularly since judging quality in architecture, as in most design disciplines in general, resists quantification. Resistance in the judgement of design quality can be traced to requirements for evaluating complex intra- and inter-discipline factor integration, as well as cultural knowledge which is tacit in nature. This paper aims to introduce the TIOSE Qualitative Measure, an outcome evaluation tool developed for a study on team interaction, cognitive style, team processes, and quality in design. The TIOSE contains five factors for judging architectural design success: thoroughness, informativeness, organisation, synthesis, and evocativeness. The methodology for the study involved evaluating the intra- and inter-rater reliability of six architectural critics who utilized the TIOSE to judge team project quality and success among 84 graduate students working in 12 teams. Analysis found high intraclass correlations within and between critics across three rounds of ratings, suggesting the TIOSE showed excellent intra- and inter-rater reliability, respectively. Study results provide evidence that evaluating the innovation and creativity of architecture and design projects can reliably extend past ratings of efficiency (time) and productivity (volume) as objective performance measures as the testing mechanism for success. In conclusion, this study demonstrates that qualitative measures of project success can be quantified in a reliable manner and can attain internalized agreement between architectural critics. The TIOSE measure provides a rating scale with explicit definitions for the qualitative measure of architectural design quality.  


Design judgement, inter-rater reliability, qualitative factors, project success

Full Text:



Basadur, M., & Head, M. (2001). Team performance and satisfaction: A link to cognitive style within a process framework. Journal of Creative Behaviour, 35(4), 227-248.

Bland, J. M., & Altman, D. G. (1986). Statisticalmethods for assessing agreement between 2 methods of climical measurement. Lancet, 1(8476), 307-310.

Bradley, J. H., & Herbert, F. J. (1997). The effect of personality type on team performance. Journal of Management Development, 16(5), 337-353.

Cools, E., & Van Den Broeck, H. (2007). Development and validation of the cognitive style indicator. The Journal of Psychology, 141(4), 359-387.

Cummings, J. L., & Teng, B. (2003). Transferring R&D knowledge: The key factors affecting knowledge transfer success. Journal of Engineering and Technology Management, 20(1-2), 39-68.

Dorst, K. (2001). Creativity in the design process: Coevolution of problem–solution. Design Studies, 22(5), 425-437.

Ericsson, K. A. (2006). The Influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, R. R. Hoffman & P. J. Feltovich (Eds.), The

Cambridge handbook of expertise and expert performance (pp. 685-706). Cambridge: Cambridge University Press.

Flemenbaum, A., & Zimmermann, R. L. (1973). Interand intra-rater reliability of the brief psychiatric rating scale. Psychological Reports, 32(3), 783-792.

Hill, A. W., Dong., A., & Agogino, A. M. (2002). Towards computational tools for supporting the reflective team. In J.S. Gero (Ed.), Artificial intelligence in design)pp. 305-326). Dordrect, The Netherlands: Kluwer Academic Publishers.

Jackson, P.W. & Messick, S. (1964). The Person, the product and the response: Conceptual problems in the assessment of creativity. Princeton, NJ: Educational Testing Service.

Keirsey, D. (1998). Please understand me II: Temperament, character, intelligence. Del Mar, CA: Prometheus Menesis.

Kickul, J., & Neuman, G. (2000). Emergent leadership behaviours: The function of personality and cognitive ability in determining teamwork performance and KSAs. Journal of Business and Psychology, 15(1), 27-51.

Kruger, C., & Cross, N. (2006). Solution driven versus problem driven design: Strategies and outcomes. Design Studies, 27, 527-548.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychology Methods, 1, 30-46.

Priola, V., Smith, J. L., & Armstrong, S. J. (2004). Group work and cognitive style: A discursive investigation. Small Group Research, 35(5), 565-595.

Rönn, M. (2011, April). Quality in architecture - A disputed concept. Paper presented at ARCC 2011 - Considering Research: Reflecting upon Current Themes in Architectural Research, Detroit, MI, USA.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 408-420.

Svensson, C. (2010). Speaking of architecture: A study of the jury’s assessment in an invited competition. In M. Rönn, R. Kazemian & J. E. Andersson (Eds.), The architectural competition (pp. 209-232). Stockholm: Axl Books.

Vanderheyden, K., Lommelen, B., & Cools, E. (2010). Cognitive styles and teamwork: Examining the impact of team composition on team processes and outcomes. Vlerick Leuven Gent Management School Working Paper Series. Gent, Belgium: Vlerick Leuven Gent Management School.

Volkema, R. J., & Gorman, R. H. (1998). The influence of cognitive-based group composition on decision-making process and outcome. Journal of Management Studies, 35(1), 105-121.


  • There are currently no refbacks.

Copyright (c) 2015 International Journal of Architectural Research: ArchNet-IJAR


- ISSN (Online) #1938 7806 - ArchNet-IJAR is covered by ArchNet@ MIT Libraries, Avery Index to Architectural Periodicals, EBSCO, CNKI, Pro-Quest, Scopus-Elsevier, Web of Science.

- Published work in ArchNet-IJAR is licensed under Creative Commons: CC-BY--NC-ND license, see

Copyrights © Archnet-IJAR 2007-2018


Hit Counter
Visitor Hits Since 15 Jan 2014