Basic of Outcomes Assessment

Basic Concept of Outcomes Assessment

Outcomes assessment is a process which gauges a program's educational quality and provides for continual teaching and learning improvement. The focus is on providing a meaningful and relevant student learning experience. The process is generally correlated to an institutional mission.

The following figure gives a general flow of the key elements of a quality assurance outcomes assessment process.

Program Assessment Cycle

Figure A: Outcomes Assessment Cycle

Let's say your program intends to engage in a meaningful, outcomes-based educational practice. The goal is to improve the teaching and learning experience, which ensures a high-quality educational program. The best practices of assessment for your program are needed. The assessment flow cycle presented here is a good starting point regardless of the type of your program. The flow cycle above was adopted from Gloria Rogers' work and has been modified to suit our needs. For more of Gloria Rogers' excellent ideas and work in assessment, please visit www.abet.org. For consistent discussion, we adopt the same terminologies used by ABET for outcomes assessment.

As indicated in the flow cycle, program assessment begins with the alignment of program educational objectives (PEO) and student outcomes (SO) with the university's mission.

 Level 1: PEO Assessment:

Here the program educational objectives are the statements that describe the expected accomplishments of graduates during the first few years after graduation. The student outcomes are statements that describe what students are expected to know and be able to do by the time of graduation. The key distinction between the PEO and the SO is the time frame in which student performance is measured. From the program perspective, the SO is what can be measured and should be used as a guide to steer students towards the PEO after graduation. So most program assessment centers on meeting the SO, not the PEO. After the SO is properly measured and assessed, the action items generated should then be used to further assess if the PEOs are really being achieved. As indicated in the flow cycle, this level of assessment is termed Level 1: PEO assessment. Since the PEO looks at a time frame three or four years after a student graduates, the assessment cycle for Level 1 is expected to be conducted once every three to five years. This cycle is sufficient to gauge the PEO relevancy to the needs of the program's constituents. Since action items are the result of outcomes assessment, which involves a different cycle of assessment, it is clear that the outcomes assessment for addressing the PEOs is eventually met.

 

 Level 2: Student Outcomes Assessment Cycle:

Performance Criteria:
To assess student outcomes meaningfully, we must have performance criteria to guide us. Performance Criteria are specific measureable statements identifying the performances required to meet the outcome with confirmable, thorough evidence. Once performance criteria have been defined, the methodologies and direct or indirect approaches used in assessment vary from program to program and also implementation strategies. The following table gives some examples of assessment approaches:

 Table A: Examples of direct/in-direct assessments

Method  Direct  Indirect  Method  Direct Indirect
Exit Interview    check mark Locally Developed Exams  check mark  
Simulations  check mark   External Examiner  check mark  
Behavioral Observations  check mark   Surveys    check mark
Archival Data    check mark Portfolios  check mark  
Focus Groups    check mark Oral Exams  check mark  
Performance Appraisal  check mark   Standardized Exams  check mark  

 

Educational Practices:
Educational practices and strategies may mean how you would implement various combinations of assessment methods listed above, and the frequency and interval of those assessment methods. Here is where most of people differ about good practices or excessive/overly burdensome process of assessment. How often should we collect and evaluate results? Who should be involved and responsible for the process? Should each outcome be assessed every year? How much objective evidence is considered sufficient? How many courses should be used to gauge the success level in meeting a specific student outcome? Should we use embedded course assessment? Once you answer these and other questions, you will arrive at an educational practice that may work for you. The next question is "Assessment."

Assessment:
Assessment is the processes that identify, collect, analyze, and report data that can be used to evaluate achievement. In this case, achievement is the student outcomes or course outcomes. Assessment is generally a time-consuming data collection process and an analysis of evidence for meaningful evaluation, the next logical step.

Evaluation:
Evaluation is a process of reviewing the results of data collection and analysis and making a determination of the value of findings and actions to be taken. In view of this definition of evaluation, evaluation is a critical step to ensure the quality improvement process. Without proper evaluation, the assessment of data collection is rendered meaningless and eventually leads to failure and frustration for the parties involved.

Action Items: 
With evaluation comes action items. Perhaps the most meaningful step in the overall flow cycle of assessment is the action items that allow closing the loop for real quality assurance. With action items as the feedback loop to change or improve student outcomes, performance criteria, and educational practices, the cycle of outcomes assessment is repetitive or a work-in-progress. This form the Level 2: SO Assessment.

 

 Level 3: Course Outcomes (CO) Assessment:

If the educational practices in Level 2 SO Assessment use embedded course assessment, then this gives rise to Level 3: CO Assessment. That is, outcomes of core courses in the curriculum are being assessed to see if student learning meets the course outcomes. Course outcomes assessment involves the same cycle of assessment as student outcomes. That is, performance criteria for outcomes have to be defined, the assessment process of collecting and analysis of evidence is conducted, and the evaluation process of interpretation of evidence needs to be performed, which eventually leads to action items for improvement at a course level. This level of assessment is the most time consuming and rewarding.

In conclusion, the overall assessment cycle consists of three levels of interacting assessment cycles: (1) the PEO assessment (3-5 year time frame); (2) the SO Assessment (1-3 year time frame); and (3) the CO Assessment (every term). These three levels of the assessment cycle needs different toolsets since their assessment time frames are different. To find out more, please read Three Levels of Assessment.