I must confess that I found the title of Michѐle Lamont's book, How professors think: Inside the curious world of academic judgment, quite intriguing at first.  The work promised to reveal a secret landscape of reasoning and decision making activities that could not easily be uncovered by the general public.  I approached the book with the same hesitation I would devote to mystery fiction novels, a genre of narratives known to bestow upon readers the fascinating role of secret observers of others. 

In all seriousness, as I moved through the chapters of How professors think, the expectation of excitement and surprise quickly disappeared to be replaced by one fundamental question:  What can readers learn about peer reviewing from this text that they did not already know or imagine?  This reasonably well-written text with a wealth of data and interpretations to ponder has proven to be afflicted with more questions than it can phantom to answer.  The copious data, by and large, seem to offer validation to the informal wisdom that novices and connoisseurs of academia alike have held all along.  Consequently, I will dutifully recount in a succinct form my reactions to the many pages of text that promised a fairly fresh and thorough transparency to the exercise of peer reviewing in academia, but delivered a reiteration of the already known.

The text is an attempt to understand the 'black box' of peer reviewing, an essential component of academic life, mostly by asking questions to those who participate in multidisciplinary grant review panels.  Undeniably, the text attempts to address difficult and, at times, controversial issues.  Its boldness and breath of topics is remarkable, but the actual treatment of each of the selected topics reveals neither innovation in methodology nor novelty in findings.   Although the purpose of the author's research is valuable and her effort admirable, a few fallacies may be detected in her methodological approach that I believe require elucidation.

Generally, our cognitive life involves operations performed without any awareness.  Awareness of an outcome, such as a decision regarding a grant application, is not the same as awareness of the operations completed to reach that specific decision.  Difficult to imagine is the way that self-reports by individuals involved in peer reviewing could provide a complete and accurate view of online reasoning and decision making.  

The author cleverly examines different content areas and treats them as building blocks for her investigation of the workings of review panels:  schemas that panelists use to illustrate their appraisals of application materials, and the meaning and relevance attributed by panelists to the criteria and standards upon which their appraisals are conducted.  Unfortunately, the evidence is so muddled that excluding the influence of the unique characteristics of participants and settings from the data and conclusions carefully amassed by the author is difficult.  

The author seems to believe that the public expression of evaluation criteria by members of review panels can be considered an ideal setting for surveying competing points of view regarding 'excellence' across different fields of work.  Yet public expressions of evaluation criteria do not always translate into adoption and uniform application of such criteria even if the panel has agreed on their merit.  To what extent do people's intentions, public or otherwise, match or not their thoughts and actions?  Cognitive psychological evidence addressing the relationship between thought and action in reference to the issue of awareness or lack thereof in our cognitive life remains largely and disturbingly ignored.

The construct of 'fairness' as applied to the process of evaluation is central to Lamont's investigative reports and subsequent analyses. The author seems to believe that a way of conceptualizing 'fairness' in the peer review process exists, but the operational definition of this concept remains obscure from the beginning.  Even if peers can agree on a set of criteria, how will they apply them 'fairly'?  To what extent, and at what point will their status as product evaluators, their institutional and personal affiliations influence their thought processes and ultimately their decisions?  Is 'fairness' a concept that can be concretely implemented?  To provide some tentative answers to these pressing questions, the author closes her investigative argument by stating that 'it may be possible to determine the fairness of particular decisions, but it is impossible to reach a definite, evidence-based conclusion concerning the system as a whole.'  Did we know this answer all along?

  © 2009 Maura Pilotti

 Maura Pilotti, Ph.D., Department of Psychology, Hunter College, New York