Office of University Evaluation

Academic Program Assessment is an ongoing process used to measure the extent to which an academic program has achieved its objectives regarding the knowledge, skills, and abilities of program graduates and to identify changes that will help the program to better achieve those objectives.

Assessment provides the answers to these questions: 

What do graduates know and what can they do? 

What programmatic changes are necessary to improve the knowledge and skills of program graduates?

Academic Program Assessment should not be thought of as a periodic activity with a finite beginning and end.  It is a continuous and ongoing process; each cycle provides information about the degree of success from the previous cycle and informs decisions and activities in the subsequent cycle.

No.  Both are important activities, necessary for continued improvement, but the purposes and timelines are different. 

Academic Program Assessment is an ongoing process that occurs over cycles of one or two years and focuses entirely on student-specific outcomes (knowledge, skills, and abilities) of a single academic major, minor, or certificate program. 

Consistent with Arizona Board of Regents policy, an Academic Program Review occurs every seven years. It is used as a comprehensive evaluation of the overall effectiveness of an entire academic unit that may administer multiple degree programs at both the undergraduate and graduate levels. An Academic Program Review self-study will include information from the assessment activities of the unit’s programs, but will also provide information about the unit’s human and non-human resources, faculty qualifications, scholarship and service activities, and more.

Academic Program Assessment is an internal activity in which the program faculty examine program effectiveness, as measured by student learning outcomes, and make decisions about program changes designed to enhance effectiveness.

Academic Program Review is an external process in which peer reviewers respond to the unit’s self-study and make recommendations for improvement.

At ASU, Academic Program Assessment is coordinated by the Office of University Evaluation, and Academic Program Review is coordinated by the Graduate College.  The two offices work cooperatively to support the assessment, program review, and accreditation activities of the University.

As educators, we want to do everything possible to prepare our graduates to perform in society, in the workplace, or in graduate school.  Although a program may not have prepared a structured list of the student learning outcomes, a curriculum map that outlines how those outcomes will be achieved, or a systematic approach to measuring student outcomes, the faculty will have at least a common understanding of what they expect of program graduates and have opportunities to observe the varying levels to which students master those outcomes.

Engaging in a structured system of assessment planning and reporting allows program faculty to report the specific learning outcomes they desire for their graduates, and to collect solid evidence of how well those outcomes have been achieved. This structured approach will provide an important tool that will help guide decisions about curricular changes and improvements.

In recent years, the increased emphasis on accountability in higher education has increased demands for evidence that universities are preparing graduates for employment or further study. The quality of an academic program – or an entire institution – cannot be measured solely by its buildings, its faculty, or its athletic teams. The key measure by which an institution and its programs are judged is the quality of its graduates. The consistent use of a well-designed system of academic program assessment will provide the kinds of evidence needed to satisfy accountability requirements.

The system of academic program assessment implemented by Arizona State University will satisfy many internal and external accountability requirements:

Institutional accreditation. ASU is accredited by the Higher Learning Commission (HLC) of the North Central Association (NCA) of Colleges and Schools.  NCA and the other regional accrediting bodies are certified by the U.S. Department of Education, which has faced increased pressure by Congress to reconsider or revoke certification for the regional accrediting bodies if they continue accreditation for institutions that fail to provide solid evidence that their graduates possess the knowledge and skills they should.  HLC evaluates institutions on five Criteria for Accreditation. One of those criteria relates specifically to assessment.  The other four criteria also reference assessment. An institution that lacks a strong system of academic program assessment is at risk of losing HLC accreditation.  Without institutional accreditation, a university can no longer award federal financial aid, can no longer have an NCAA athletic program, and may automatically lose any specialized college or program accreditations that require HLC accreditation.

Specialized accreditation.  Many of ASU’s colleges, schools, and programs are accredited by agencies such as the Commission on Collegiate Nursing Education, the American Bar Association, the National Architectural Accrediting Board, and others.  Specialized accrediting bodies such as these require evidence of ongoing assessment of student learning as part of their requirements for continued accreditation.

Arizona Board of Regents. ASU is required to provide an annual report to ABOR in which we document the number of academic programs actively engaged in academic program assessment.

All academic programs – majors, minors, and certificates, graduate and undergraduate – are expected to participate.

An assessment plan is a document in which program faculty articulate the specific knowledge and skills graduates are expected to possess, and the means by which the program faculty plan to measure student learning.

An assessment plan will contain the following elements:

Department mission statement

• flows from and supports ASU mission statement

Program mission statement

• flows from and supports department mission statement

Program objectives (learning outcomes)

• provide answers to the question, “What should program graduates know and be able to do?”
• tied to program mission
• observable and measurable
• possible to demonstrate effects based on programmatic inputs
• potentially too many objectives to adequately measure in one year, but programs have option of identifying subsets of objectives to target across multiple years.

Curriculum matrix of instruction and experiences that will support intended learning outcomes

How student performance will be measured

• emphasis on direct measures, although appropriate indirect measures are also acceptable
• two or more measures for each outcome
• emphasis on program outcomes rather than classroom performance, although classroom data can be appropriate measures for some outcomes
• how validity and reliability of measures will be determined

Sampling strategy

Level of performance on each measure required to satisfy objectives

Names, titles, and roles of persons involved in development of assessment plan

ASU has adopted a standard assessment plan template for online submission of assessment plans.  The use of a standard template by all programs will enable OUE to best support assessment activities and will facilitate the analysis of assessment data at the institutional level. A .pf version of the online form can be downloaded for review.

An assessment report is a document in which program faculty report the results of the data collection activities previously described in the assessment plan and describe any curricular decisions resulting from their findings.
An assessment report will contain the following elements:

• Summary of data collection/results
• how validity and reliability of measures were determined
• Indicator whether each program objective was satisfied
• if not, discussion of possible factors
• Curriculum or other changes planned or implemented as a result of assessment results
• Assessment plan for next round of data collection, with any necessary revisions
• If curriculum changes are made as a result of assessment results, the assessment plan may need to be modified in order to collect relevant data.
• If an objective was not met during the previous cycle, that objective should receive follow-up during the new cycle.
• If a subset of objectives was included in the initial cycle, the second subset should be reflected in the new assessment plan.
• Program faculty may elect to revise objectives or measures.
• Names, titles, and roles of persons involved in development of assessment plan

Program objectives are the intended learning outcomes of an academic program.  They are the answers to the question, “What should program graduates know and be able to do?”  Program objectives must be observable, measurable, and directly linked to instruction and experiences provided by the program.

Programs are encouraged to identify between three and seven program objectives during a single assessment cycle.  Programs with more than seven important objectives, particularly those specified by an external accrediting body, should divide those into subsets for consideration across two or more assessment cycles.

A curriculum matrix, or curriculum map, is a grid that indicates the specific instructional or experiential opportunities for students to acquire the knowledge, skill, or ability represented by each program objective.  A very basic form of a curriculum matrix is shown below.

 

 

Additional information about program objectives and curriculum mapping, including examples can be found on the next page.

Two measures – direct and indirect – should be provided for each program objective.  Examples of direct and indirect measures are shown in the table below.

When identifying measures of performance on program objectives, it is important to focus on the overall learning of groups of graduates rather than the individual classroom performance of individual students.  That does not mean that assessment data should consist entirely of data collected outside normal classroom activities; many forms of classroom data are suitable for inclusion in the assessment process.

Although classroom grades may seem a convenient means to monitor student learning on program objectives, their use should be avoided.  Dr. Gloria Rogers, a nationally-recognized expert on academic program assessment discusses the pitfalls associated with classroom grades as assessment data in Do Grades Make the Grade for Program Assessment?

Program faculty may decide to include data for all program graduates or for a sample of graduates.  The decision needs to be included in the assessment plan, and will be driven by such factors as program size, feasibility of collecting and analyzing some types of assessment data, and cost.  Regardless of the sampling strategy employed, it is essential that the resulting sample be representative of the group of program graduates about whom inferences will be drawn.

For each program objective, one or more performance criteria will be used to determine the level of performance necessary to determine that the program objective has been satisfied.  Performance criteria must be identified prior to the collection and analysis of assessment data. When setting performance criteria, it can be tempting to set unreasonably high “nothing but the best” standards or to set unreasonably low “guaranteed to show success” standards.  Both of these practices can be defeating.  Over time, it is far more beneficial to a program and its students to set reasonable expectations and work toward meeting them. Consider the use of primary and secondary performance criteria. A primary performance criterion, the kind most commonly used, will state that “at least xx% of students will perform at or above ….”  A secondary performance criterion for the same measure will state that, “no more than xx% of students will perform at or below …” The use of primary and secondary performance criteria can provide richer information about student learning that can inform decisions about needed improvements.

Assessment plans and reports are important pieces of the assessment process, but assessment is an ongoing process without starting and finishing points.

After assessment data have been collected, program faculty review the results, consider what changes in the curriculum and the assessment process might improve student learning as well as the measurement of that learning, and begin the process anew. Assessment is an iterative process of assessment planning and reporting that flows directly into a new cycle of assessment planning and reporting.

This recurring cycle, and the use of assessment information in curricular decision making is known as closing the loop.  Because this crucial step of closing the loop ties together the phases of assessment planning, data collection, analysis, and decision making, it is at the core of any sound assessment program and supports curricular decision-making that is sound, defensible, and most likely to show positive change. 

The Office of University Evaluation has a wide array of resources available to assist units with academic program assessment, academic program review, and accreditation. Our expert staff are available to consult with program faculty on such issues as:

• designing an assessment strategy appropriate for your academic discipline
• sample assessment materials from other departments and institutions
• development of program objectives
• identifying high-quality outcome measures
• sampling strategies
• setting performance criteria
• planning for data analysis
• survey research
• providing survey data for use in program assessment, program review, and accreditation
• test proctoring and other services through University Testing Services
• identifying appropriate standardized tests
• referrals to faculty peers with specific assessment expertise
• feedback and coaching on self-study assessment sections
• feedback and coaching on ASU assessment submissions
• other topics related to testing and measurement, program evaluation, and educational research

We have started to assemble a collection of useful articles and other materials about program assessment. New readings will be added as they are found. We encourage you to read Assessing Assessment. Although it was written about the assessment of mathematics education, the concepts apply to any discipline. The author provides nearly two dozen examples of how high-quality assessment data can be obtained from ordinary classroom assignments and examinations that are part of the curriculum for most disciplines.