Good Practices in Assessment of Student Learning and Development

I.Good Practices in Outcomes-Based Assessment Program Review

This study describes "good practices" in outcomes-based assessment program review in an effort to demonstrate how improvements in student learning and development are made based on a systematic institutional evaluation process. Basic illustrative components of outcomes-based assessment program review are presented as well as criteria for identifying good practices within outcomes-based assessment program review. Institutional case studies in outcomes-based assessment program review are interwoven throughout the study to illustrate principles of good practice. Finally, suggested steps for implementing sustainable outcomes-based assessment program review are outlined.

This study is intended for faculty and administrators responsible for implementing and sustaining outcomes-based assessment program review at their respective campuses. In addition, the study can help faculty and administrators understand the "what" and "why" of outcomes-based assessment program review. Furthermore, it will explain the value of the outcomes-based assessment program review process as identified by the participating good practice institutions.

This study is not a "how-to-do-assessment" presentation. Rather, this study utilizes the results of a grounded theory study of multiple institutions to posit answers to the questions: what does good outcomes-based assessment program review practice look like from an institutional perspective? Who is doing replicable work? In short, it couples fine assessment practices with cyclical program review so that the single process of outcomes-based assessment informs many purposes: program review, strategic planning, professional accreditation, institutional accreditation, and possibly even the assessment of general education.

Assessment and Accountability
This study is a timely contribution to literature on outcomes-based assessment. Because legislatures and government agencies are increasingly observant of institutional practices and are involved in institutional decision-making, accountability has become a prevalent concept and programmatic initiative (Allen & Bresciani, 2003; Ewell, 1997a; Ewell, 1997b; Ewell, 2003). To respond to growing demands for institutional accountability with regard to student learning and related expenditures, higher education requires an articulated definition of "quality education," rather than the ill-defined conceptualizations currently in use. For example, those engaged in quality of education conversations traditionally have used persistence and graduation rates as reliable indicators of quality education (Pike, 2001; Ewell, 1997a). Such indicators are easy to define and measure, but do they identify the quality of the education experienced by students? Using these indicators, can one understand how well a student has learned the content and mechanics of her discipline or program? How well do faculty know whether a student has mastered problem solving or application of theory to practice? Indicators of persistence and graduation may indicate institutional type and mission, but they typically neglect more meaningful questions of quality learning outcomes.

Nationally standardized testing, whose purpose is ostensibly to evaluate student learning across institutional types, is another traditional medium through which to measure the quality of education. Though they offer a relatively simple method of data collection, many tests are designed without consulting the curricula; thus, the tests are not measuring the content of the course or program being delivered (Lopez, 1997; Maki, 2004). One solution is to require institutions to align their curricula with the content of the standardized test, a practice that typically reduces education to "teaching-to-the-test." Additionally, such an approach neglects the particularities of institutional culture and its unique student populations in order to satisfy a standardized approach to higher education. Another consequence may be a decrease in the specialized information required for institutional and societal improvement. This trade-off may recreate the pattern of educating elites only, which practitioners have eschewed in favor of more equitable and accessible education (Anderson et alia, 2001). Yet another consequence may be that the information contained in the standardized tests may remain behind the learning curve of the research that is taking place in higher education, therefore furthering the gap between postsecondary classroom learning and scholarly research. Additionally, standardized tests may be unable to accommodate varying learning styles (Anderson et alia, 2001; Maki 2004). Finally, the question is raised as to whether using one standardized test to evaluate all students' learning is even responsible research. What researchers do we know who engage in sound research methodology but use only one means of evaluation to formulate a theory or an assumption of quality?

Despite their claims of promoting quality and standards, legislators who call for strict quality monitoring actually appear to promote "watering down" educational principles because their proposals take simplistic approaches to evidence of learning. Moreover, the proposals often prevent the expression of faculty expertise and varying institutional values and resources (Bresciani, 2003). Some of these legislated indicators even distort conversations about quality. Institutions may feel compelled to seek traditional types of students and promote "cookie-cutter" institutional values in an effort to meet some standardized definition of a quality institution (Ewell, 2003). In reality, of course, different types of institutions are needed to serve different types of students with varying educational goals, such as an institution that educates adult single parents to become more effective managers, a research extensive university where 60 percent of the student body leaves on a two-year religious mission, or a private liberal arts college whose students do not typically work more than 20 hours a week and who are most likely to obtain a bachelor's degree in four years (Bresciani, 2003). It is not difficult to see the obstacles in using any one test or set of indicators to assess the actual quality outcomes of the curriculum.

The challenge of finding meaningful quality indicators is exacerbated by a model of higher education that dates back to the mid- to late-1800s (Goodchild & Whecshler, 1989). Patterns and habits have become entrenched. Change in the delivery of higher education comes slowly, even when clearly necessary. When government officials ask faculty and administrators to verify student outcomes, institutional responses are defensive, subversive or non-existent. Such avoidance behaviors manifest themselves because the inquiries and required indicators come from outside the academic discipline and institution (Bresciani, 2003). And, equally important, those stakeholders outside the academy-legislators, donors, parents, other educators, granting agencies, and so on-hold not only the purse strings for our endeavors, but also control the barometer of public opinion, prestige, and general well-being for the academy. In short, refusing to answer questions from these stake holders about academic quality is no longer an option.

The scrutiny of quality in higher education is not diminishing. In fact, educators face growing pressure to produce evidence of student learning, while faculty grow increasingly frustrated that their disciplinary expertise is under-appreciated or unrecognized (Bresciani, 2003). How, then, do institutions (a) report in a meaningful way data on student learning and development? (b) convey details about individual student learning so that constituents can understand the quantity and quality of learning? and, (c) inform decisions at a higher level while maintaining faculty autonomy and investment in teaching and learning?

An Overview of the Survey and Selected Participating Institutions
The criteria for this good practice study was founded on Seymour Studyt's (1991) constructionist learning philosophy. "Studyt's philosophy was used in the context that those delivering the learning are learning about how to improve student learning when they evaluate the delivery of their teaching, or those delivering the services are learning how to improve the services when they are evaluating the effectiveness of their services" (Bresciani, in press, p.2). In this context, the concept of good practices emerges as those delivering the good practice engage in constructionist learning. "It is in fact the teacher or the service provider who is constructing the best possible learning experience or service experience based on what they are learning about how well the entire package is working that constitutes opportunities for continuous improvement. This type of learning from doing and applying decisions to improve that which you are teaching in a manner that will align with the institutional culture within which you are working actually may appear to be more of a form of constructed relativism" (Bresciani, in press, p. 2). For purposes of this study however, we base the study on a foundation of constructionist learning.

Therefore, good practice criteria were constructed and refined based on a set of criteria which emerged primarily from three key resources. The first is the Nine Principles of Good Practice for Assessing Student Learning developed under the auspices of the American Association for Higher Education (AAHE) Assessment Forum, December 1992 found at http://www.buffalostate.edu/offices/assessment/aahe.htm. The second document is that of Eckel, P., Green, M., and Hill, B. (2001) publication entitled, On Change V-Riding the Waves of Change: Insights from Transforming Institutions published by the American Council of Education. And the third is criteria from Lopez, C. (2002). Levels of Implementation published by the Higher Learning Commission of the North Central found at http://www.ncahigherlearningcommission.org/resources/assessment/index.html

Making reference to these criteria, twenty-three of the most highly published assessment scholars in the United States were asked to identify institutions that they felt met these criteria in the practice of their outcomes-based assessment program review. It was made clear that the research team was looking for institutions that had effective, efficient, and enduring practices and that we were interested in seeing good practice institutions at varying stages of their implementation. A list of 43 institutions was generated and circulated to all of the assessment scholars for further comment. No institutions were removed from the list and none were added.

All forty-three institutions were invited to participate in the study which required participation in a survey of good practice, which was designed as a part of a graduate research class using the criteria in the aforementioned references. The survey was pilot tested in the class and descriptive statistics were used to analyze the survey. In addition construct validity analysis was run but the participant numbers in the survey were too low to establish any meaningful coefficients.

Participants were also asked to submit a case study illustrating their good practice. Case study submissions ranged from 7 to 84 pages in length. Using grounded theory (Strauss and Corbin, 1990), the case studies were reviewed by the primary author and graduate assistants to identify additional criteria not originally placed in the survey.

This study is a compilation of the summary of the good practice survey and the case study submissions. The research team truly modeled Studyt's constructionist learning philosophy when compiling this work as we allowed for each institution to refine the application of criteria to fit its needs and therefore, the criteria and example practices present in the study serve as guidelines. This is not a "one-size fits all, lock step" application of good practice. It is intended to provide the reader with ideas for consideration for adapting to the reader's own institutional culture.

Some good practice institutions elected not to participate in this study and others may have been inadvertently overlooked because they have not yet become known for their exemplary assessment practices. Texas A&M University was inserted into the study because they are following the implementation of this good practice model and thus, are expected to emerge as a good practice institution. The author is indebted to those institutions who took the time to submit their work so that all can benefit from the lessons they have learned

In addition, while this study is intended to assist institutions in establishing institution-wide outcomes-based assessment program review processes, the contents can be adapted to the implementation of outcomes-based assessment program review in colleges, divisions, departments, or programs.

It bears repeating that this study is not intended to promote outcomes-based assessment program review as a process established merely to sustain itself. Rather, the researcher intends to illustrate good practice in self-reflection that contributes to improved quality in student learning and development, teaching, research, and service. In other words, the process is not a means to its own end; rather, it is a way to systematically engage in daily critical inquiry about what works well and what needs to be improved (Maki, 2004).

The results of this study are being published by Stylus Publishing.

References

Allen, J. & Bresciani, M. J. (2003, January/February). Public institutions, public challenges: On the transparency of assessment results. Change Magazine.

American Association of Higher Education (1994). Nine Principles of Good Practice for Assessing Student Learning. Retrieved month day, year, from http://www.aahe.org/assessment/principl.html.

Anderson, J. A., Maki, P., & Bresciani, M. J., (2001, July). Expanding Faculty Involvement in Assessment-Based Undergraduate Academic Program Review: A Case Study. Presentation at the American Association of Higher Education Assessment Conference. Boston.

Bresciani, M. J. (2003, October). Expert Driven Assessment: Making It Meaningful to Decision Makers. ECAR Research Bulletin, 21. Boulder, CO: EDUCAUSE.

Bresciani, M. J., Zelna, C. L. and Anderson, J. A. (2004). Assessing Student Learning and Development. A Handstudy for Practitioners. United States: NASPA.

Eckel, P., Green, M., and Hill, B. (2001). On Change V-Riding the Waves of Change: Insights from Transforming Institutions. Washington DC: American Council of Education.

Ewell, P. T. (2003, month). Specific Roles of Assessment within this Larger Vision. Presentation given at the Assessment Institute at INDIANA UNIVERSITY-PURDUE UNIVERSITY-INDIANAPOLIS. Indianapolis, IN.

Ewell, P. T. (1997a). From the states: Putting it all on the line-South Carolina's performance funding initiative. Assessment Update, 9 (1), 9, 11.

Ewell, P. T. (1997b). Identifying indicators of curricular quality. In G. J. Gaff, L. J. Ratcliff and Associates (Eds.), Handstudy of the undergraduate curriculum: A comprehensive guide to purposes, structures, practices, and change. San Francisco: Jossey-Bass.

Ewell, P. T. and Jones, D.P. (1996). Indicators of Good Practice in Undergraduate Education. A Handstudy for Development and Implementation. Boulder, CO: NCHEMS.

Goodchild, L. F. & Wechsler, H. S. (1989). The History of Higher Education: Association of the Study of Higher Education Reader. Boston: Glenn Press.

Lopez, C. (2002). Assessment of student learning: Challenges and strategies. The Journal of Academic Librarianship 28(6), 356-367.

Maki, P. (2004). Assessing for Student Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing, LLC: Sterling, VA.

Studyt, S. (1991). Situating Contructionism. In Constructionism. Edited by Studyt and Harel. Cambrideg, MA:MIT Press.

Pike, G. R. (2001) Assessment Measures. Assessment Update 13 (1), 8.8. San Francisco: Jossey-Bass.

Strauss, A. & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage.