How do we know when a particular university is doing well compared either with other institutions, or against external standards? When we evaluate the performance of something such as a car, a business, or an employee, we usually focus on certain features that will indicate, or lead us to conclude the degree to which its performance is satisfactory. In higher education, performance indicators have been used in the process of accreditation, evaluation, and rankings. With institutions facing increased competition for diminishing resources and greater accountability demands from stakeholders, the study and use of performance indicators has never been as crucial. One main reason for focusing on the relation between institutional characteristics and student outcomes is in response to the rapidly growing popularity of annual college and university rankings such as Maclean’s (Bruneau & Savage, 2002) and U.S. News and World Report (Hossler, 2000). Clearly, one of the main purposes implied in these rankings is to provide an objective evaluation of the educational quality of universities that is easily accessible. One of the main benefits of annual rankings is that with the help of popular media, information about lesser known schools that may have gone unchecked by students or parents are more widely accessible (Webster, 1992).

Not surprisingly, it seems that virtually every aspect of the commonly used performance indicators in education have been sharply criticized (see Bruneau & Savage, 2002). The most commonly recognized problem with most indicators is that they seem to be based largely on the assumption that they are related to student outcomes, and thus, accurately reflect the level of educational quality of an institution (Ball & Halwachi, 1987). Similarly, Nedwick and Neal (1994) point out that many of the institutional characteristics used in rankings have been selected on the basis of convenient data collection rather than a defensible theory. This reliance on measures that are quantifiable and easy to collect, leads to a “quick fix” approach that presents an impressive, but largely unsupported evaluation. While the institutional characteristics used as process indicators are indeed important (e.g., adequate libraries, qualified instructors), most of them are measured at such a general level that they obscure the fact that each institution does not pursue the same educational goals (Ball & Halwachi, 1987). As such, many of these indicators are far too removed from the students’ actual learning experiences (McGuire, 1995).  For instance, these indicators do not look at classroom processes that have a direct impact on student learning such as frequency of higher-order questioning (Renaud & Murray, 2007). Although there are several other limitations with respect to rankings (see Bruneau & Savage, 2002), these appear to be among the more substantial. 

Based on much of the published empirical research as reviewed by Pascarella and Terenzini (2005), the main conclusion is that after controlling for incoming student ability, commonly measured institutional characteristics (e.g., library size, class size, percent of faculty with PhD, operating budget) seem to have little to do with the degree of measured student learning and development (e.g., GRE/MCAT/LSAT scores, critical thinking, subject matter competence, interpersonal skills). However, a few main considerations are worth noting such that the findings of research in this area are interpreted with appropriate caution. If much of the variation due to incoming characteristics has been considered, and with such a relatively small proportion of the remaining variance attributable to process indicators as measured in these studies, this suggests that a sizable proportion of what determines student learning and development does occur during the educational experience in university. Pascarella and Terenzini (2005) offer three other relevant considerations. First, these studies were quite varied in terms of methodological rigor, which makes it difficult to draw consistent conclusions. Second, the link between institutional characteristics and student outcomes is difficult to confirm because the level and the effects of individual characteristics, such as ambitions, aspirations, and motivation, can change substantially during the university experience. Finally, of the studies that did report a significant effect, the findings usually were found only with some specific subgroups.

Given the numerous concerns surrounding the validity of the indicators in annual university rankings, the following main recommendations for future research should help clarify our understanding of how institutional characteristics contribute to student learning and development.

One general explanation for the weak relation between institutional characteristics and student learning is that the former are far too removed from both the actual practice of teaching in the classrooms and the student interactions that have a much stronger connection with learning (McGuire, 1995). In other words, more valid measures of educational quality may be found by focusing on specific teaching practices applicable across a range of disciplines. For instance, one potentially useful process indicator that has received relatively little attention is the course-level academic process described in several studies by Braxton and Nordvall (Braxton & Nordvall, 1988; Nordvall & Braxton, 1996) and by Bruneau and Savage (2002). Basically, the course-level academic process is intended to represent the cognitive demands placed on students in terms of the level of questioning (i.e., application, analysis, synthesis, and evaluation) on assignments and exams according to Bloom’s taxonomy of cognitive objectives (Bloom, 1956). Two other related course-level processes are the amount of student feedback (Cabrera, Colbeck, & Terenzini, 2001) and narrative evaluations (Astin, 1999).

There is accumulating evidence confirming the positive effect of the amount of student effort on various outcomes (see Pascarella and Terenzini, 2005) such as number of nonassigned books read with critical thinking (Terenzini, Springer, Pascarella, & Nora, 1995) and student-instructor interactions with overall competence (Martin, 2000). Moreover, it appears that student involvement explains a sizable portion of the remaining variance in outcomes once incoming ability has been controlled. The implication here is that educational quality in universities could be assessed by examining the impact of what an institution does to promote greater levels of student involvement. While most studies found traditional measures of quality and student outcomes to be weakly related, it appears that certain institutional features may actually have an indirect influence on student outcomes mediated through their involvement such as academic and social integration. For example, Ethington and Smart (1986) found that selectivity appears to show a positive, indirect link with future educational aspirations that is mediated through academic and social integration. Finally, knowing that any particular institutional characteristic (e.g., library size) is not likely to contribute, directly or indirectly, to every student’s achievement in the same manner, the effects of these processes would be more validly measured by considering the student’s level of involvement or exposure. As such, Astin (1978) suggests that exposure should be addressed in terms of both time or quantity and intensity.

If the measured characteristics of an institution (i.e., what the school does or has) have little relation with the intended outcomes (i.e., educating students), then perhaps either the wrong indicators are being used, or they are not being assessed as they should be. Because universities have varied objectives, resources, and student populations, it is important to acknowledge that the direction of this research is not necessarily intended to develop new indicators to be applied consistently across all schools. It may well be that with better methods, we could find that some of the institutional characteristics that are currently popular in annual rankings do indeed contribute toward student outcomes either indirectly through students’ experiences or for specific subgroups. 

Perhaps the most telling implication of what the findings of this body of research suggests is summed up by Pascarella and Terenzini (2005):

"...based on a flawed conception of educational quality that prompts misleading comparisons. As a result, decisions with high price tags for both individuals and public treasury are based on a demonstrably invalid set of variables for describing college’s impact on learning outcomes." (p. 642)

In other words, the information in annual rankings seem to be telling us very little in terms of knowing which universities are doing an adequate job of fostering student learning and development. If universities are truly committed to providing the best possible learning environment for their students, then the financial and personal costs of making uninformed and inferior choices based on scant evidence is one that universities simply cannot afford to continue paying.

References

Astin, A. W. (1978). Four critical years: Effects on college beliefs, attitudes, and values. San Francisco: Jossey-Bass.

Astin, A. W. (1999). How the liberal arts colleges affects students. Daedalus, 128, 77-100.

Ball, R. & Halwachi, J. (1987). Performance indicators in higher education. Higher Education, 16, 393-405.

Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: Cognitive domain. New York: McKay.

Braxton, J. M. & Nordvall, R. C. (1988). Quality of graduate department origin of faculty and its relationship to undergraduate course examination questions. Research in Higher Education, 28, 145-159.

Bruneau, W., & Savage, D. C. (2002). Counting out the scholars: How performance indicators undermine universities and colleges. Toronto: Lorimar.

Cabrera, A. F., Colbeck, C. L., & Terenzini, P. T. (2001). Developing performance indicators for assessing classroom teaching practices and student learning: The case of engineering. Research in Higher Education, 42, 327-352.

Ethington, C. A., & Smart, J. C. (1986). Persistence to graduate education. Research in Higher Education, 24, 287-303. 

Hossler, D. (2000). The problem with college rankings. About Campus, 5, 20-24.

Martin, L. M. (2000). The relationship of college experiences to psychosocial outcomes in students. Journal of College Student Development, 41, 292-301.

McGuire, M. D. (1995). Validity issues for reputational studies. In R. D. Walleri & M. K. Moss (Eds.), Evaluating and Responding to College Guidebooks and Rankings (pp. 45-59), Jossey-Bass: San Francisco.

Nedwick, B. P. & Neal, J. E. (1994). Performance indicators and rational management tools: A comparative assessment of projects in North America and Europe. Research in Higher Education, 35, 75-103.

Nordvall, R. C., & Braxton, J. M. (1996). An alternative definition of quality of undergraduate education: Toward usable knowledge for improvement. Journal of Higher Education, 67, 483-497. 

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research, (vol. 2). San Francisco: Jossey-Bass.

Renaud, R. D., & Murray, H. G. (2007). The validity of higher-order questions as a process indicator of educational quality. Research in Higher Education, 48, 319-351.

Terenzini, P. T., Springer, L., Pascarella, E. T., & Nora, A. (1995). Influences affecting the development of students’ critical thinking skills. Research in Higher Education, 36, 23-39.

Webster, D. (1992). Are they any good? Rankings of undergraduate education in U.S. News & World Report and Money. Change, 24, 19-31.

Last Modified: September 28, 2011