Evaluating Course Performance
Was it worth it?
Evaluating the effectiveness of a course is an essential part of a course life cycle. There are many questions an evaluator may ask.
- How do you evaluate the effectiveness of a course?
- Does it matter how the course is received by learners?
- Does the course provide the experience desired by learners?
- Does the return provided by the course justify the cost and effort?
We'd like to discuss a model used to evaluate the effectiveness of course content so that you can use quantitative and qualitative metrics to measure the various aspects of a course. The evaluation can then be used to give feedback on the course about possible and appropriate modifications to continue to improve the course experience.
It is essential to break down a course to its core components and evaluate each component individually. While there are many evaluation models, Kirkpatrick’s Model is considered one of the best approaches to course evaluation.
Popular evaluation models include: Kirkpatrick’s Model, Kaufman’s Model, Brinkerhoff’s Model, Anderson’s Model, and Phillip’s Model. The model we will discuss is Kirkpatrick’s Model.
Kirkpatrick’s Model is built upon multiple components such as Reaction, Learning, Behaviour, Results (sometimes called “Business Impact”), and Return on Investment. Each component or level of the model evaluates the course through a different lens. First, we will start at the base of the model, Reaction.
The first component in Kirkpatrick’s Model is Reaction. Reaction is how the learners responded to the training or course. An example of this is surveying learners to discover what parts of the course went well and what parts didn't go so well.
- Are the learners satisfied?
- Is there specific feedback that learners can provide?
- Would the learners recommend this course to someone else?
This metric is often called a Net Promoter Score.
Once Reaction is evaluated we can move to our next component, Learning.
The next component of the model is Learning. The Learning component of Kirkpatrick's Model asks if the individual portions of the course are producing the expected learning outcome
- Do the assessments perform as expected?
- Do the classes perform as expected?
The outcomes of the assessments and classes are often used to measure the effectiveness of different parts of the course. This component also allows the evaluator to ensure that the training is functioning correctly and providing the information needed by the learner.
Individual aspects are often flagged during this stage so both content and assessments that learners are struggling to master can be revisited. These modifications can be evaluated over time to ensure updates to the course are having a positive effect.
This brings us to the next component of the model, which determines if the course changed the learner’s behaviour.
Behaviour is the next key component in Kirkpatrick’s Model.
- Did the learning that was demonstrated by the previous metrics result in behaviour change for the learners?
- This may be measured by a follow-up survey for the learner and then subsequently the managers of the learner.
- To what extent has this course changed the way the learner interacts with other courses or materials?
- If the behaviour has not changed as expected, what parts are failing expectations?
Once we are able to identify the changes in behaviour, we can consider the results of that behaviour.
- With the newly changed behaviours provided by the course, what are the measurable outcomes for the learning content?
- Can we establish not just correlation of the outcomes but causation?
- Has the newly learned content and changed behaviours provided the desired outcomes?
After considering the results of the course, we can transition to the final portion of the model, Return on Investment (ROI).
Return on Investment
Return on Investment (ROI) is a newly added component to Kirkpatrick’s Model. When we sum up all the expenses required to create, deliver, take, and maintain the course, is the sum more or less than the value provided by the course to the organization? In other words, this component evaluates metrics to understand if the course contributed positively or negatively to both the financial bottom line as well as goals of the organization.
Metrics influencing the model
What other metrics can go into a Kirkpatrick's Model? Often, an institution will already have many metrics available to plug into different components of the model. Some possible examples may be:
- Learner overall assessment scores.
- Learner scores on specific competency-based topics/skills.
- How did the learner perform on additional classes or activities?
- Is the number of learners attending the course growing?
- Has the learner been able to be a more productive member of the organization?
- Survey from the tutor teaching the course. What are the learner's strengths and weaknesses as perceived by the tutor?
- Survey of learners taking the course: what is the attitude of the learner towards the course? Was the learner satisfied? Was the course relevant to the learner?
- Does the learner’s manager or coach notice a change in behaviour or a marked improvement in confidence and use of the new skills learnt?
- Would learners recommend this course to others?
These metrics are by no means a definitive list one may use but are a starting point. Institutions often start with the data they currently have and then expand the metrics they are measuring over time.
Which metrics are the most important? Since each organization is different, each metric score should be considered in the context of the organization. In other words, some organizations may put more emphasis on tutor survey results while others may put more emphasis on the learner results.
The complexity arises due to the fact that most institutions and courses have a unique set of goals. These unique goals allow us to pick and choose which metrics make more sense for that course. Each application of Kirkpatrick’s Model can be slightly different depending on the course created and the delivery of that course.
Now that you have the evaluation scores what can you do with them? The prime benefit of evaluating a course is identifying where there is room for improvement. The value of evaluating the performance of learners is to provide a baseline to compare feedback and updates for that course. It may also be tempting to compare the score of one course to another and then state one course is “better” or “worse”. That usage of the evaluation should be discouraged as differences in each course may be an apple to oranges comparison unless the dynamics of the courses are very similar.
In conclusion, evaluating the effectiveness of a course is an essential process to ensure the improvement of the overall experience for learners. Using Kirkpatrick’s Model allows us to evaluate different facets of a course and provide appropriate feedback to clients and ensure that learners continue to improve year on year.
At Connect with Language we aim to work with all of our clients to provide the support and metrics required to guarantee the best levels of learner satisfaction, the highest levels of improvement and ROI delivered to our clients.
Do all your training suppliers provide you with exhaustive analysis and moreover a respected methodology to achieve this?