As mentioned in a previous article about Evaluating Teaching in Libraries: Kirkpatrick’s First Level, I’ve taught a ton of in-person classes, facilitated workshops, and created online tutorials but I didn’t really have a strategy for evaluating all that effort until I learned about Kirkpatrick’s 4 level model of evaluation. The model provides strategies for evaluating your instruction at four levels: reaction, learning, behavior, and results. I wrote about evaluating Reaction in the previous article about the first level. Read on to learn about evaluating learning with the second level.
How Do I Get Started with Level 2?
Before you can measure learning (level 2) and behavior (level 3), you have to do something that is simultaneously simple and difficult. You need to have learning objectives. Objectives describe succinctly what you hope your participants either learn or do as a result of your training.
I facilitate a multiple-week immersive program focused on career development. As an early step in creating that program, I developed the following learning objectives:
- During the program, participants will accurately identify the roles of the employee, the supervisor, and the Libraries in employee development.
- During the program, participants will identify at least two examples of employee actions that have led to others’ career growth.
- During the program, participants will articulate their career goals with clarity and confidence.
- During the program, participants will create a development plan with realistic action steps.
- Within 6 months of completing the program, participants will implement three or more actions in their career development plan.
- One year after completing the program, participants will report positive progress toward their career goals.
Notice that these objectives attempt to articulate:
- a performance (ex: identify the roles, articulate their career goals, create a development plan, implement…actions, report…progress),
- a condition (ex: during the program, within 6 months of completing the program, one year after completing the program), and
- a criteria used to evaluate the performance (ex: accurately, at least two examples, with clarity and confidence, with realistic action steps, three or more actions, positive).
For learning objectives, the person doing the action is always the learner. In my case, I described the learner as the “participant.”
This style of writing learning objectives is based on the classic work of Robert Mager described in Preparing Instructional Objectives: A Critical Tool in the Development of Effective Instruction. If you are interested in reading one book on this topic, I recommend that book. Full disclosure: if you follow that link and make a purchase, I will receive a small amount of money through the Amazon Affiliates Program to help me to support this website. This book is widely held in libraries and summarized often online, too.
OK. I’ve Got Objectives. How Do I Measure Learning?
As you might have guessed, you base your evaluation strategy on those learning objectives. And, you have lots of options for doing so.
For the career development program described above, I measure learning (level 2) in a few different ways. First, by interacting with the learners as they are working in small groups and in one-on-one conversations, I watch for signs that they are accomplishing the formal learning objectives. Are our conversations filled with accurate examples of the roles of the employee, the supervisor, and the Libraries in employee development?
Second, we engage in activities like identifying skills, interviewing experts, and filling in worksheets to create a learning plan throughout the program. Throughout those activities, I look for evidence that each person is achieving the learning objectives #1 through #4.
Kirkpatrick’s Four Levels of Training Evaluation (2016) also considers “what they think they’ll be able to do differently as a result, how confident they are that they can do it, and how motivated they are to make changes” as part of Level 2.
To get at these concepts, I ask the following questions in a survey towards the end of the program:
- Describe action steps you plan to complete in the future.
- How confident are you that you will complete each of these actions within 6 months?
- What barriers might prevent you from completing actions within the year?
Finally, in that same survey, I ask an open-ended question about what, if anything, they got out of the program. Here are a few responses from participants that demonstrated learning:
Interested in learning more about the career development program? Chandra Lane and I co-presented about it at the 2018 Annual Conference of the American Library Association. Check out this handout for lots of details.
All of this data is part of how I measure learning (and Level 2 of the Kirkpatrick model) for a career development program. Each piece of evidence corresponds to the learning objectives the program set out to support.
Once you’ve measured Reaction (level 1) and Learning (level 2), you’ve actually collected a lot of useful information about the effectiveness of your program. Why stop now, though? Behavior is the 3rd level in the Kirkpatrick model…and that is the topic for a new post.
I’ve shared ways that I evaluate Learning for a career development program. I’d love to read your examples of evaluating learning for your teaching in the libraries in the comments!