IBL Workshops Work! Validating Our Theory of Change with the Theory of Planned Behavior (Preliminary)

This blog post is co-written with Dr. Tim Archie, CU Boulder, Ethnography and Evaluation Research, Tim.Archie@Colorado.EDU, and this is work done by him and Dr. Sandra Laursen, Devan Daly, Chuck Hayward, CU Boulder. This effort is funded by collaborative NSF grants DUE-1525077, DUE 1525058.

---

“A lever is a simple machine used to move an object at one location by applying a force somewhere else. When we try something and see that it is working, we have gained leverage on the problem.” -Levers for Change, AAAS Link

In this piece, we look at one of the key levers for change, professional development workshops in higher education. We'll start with our theory of change, and then share results from our research team. Specifically the data is based on participants, who attended intensive 4-day IBL workshops. These workshops provided intensive training to more than 500 college math instructors, impacting hundreds of courses and thousands of students per year. Peer-reviewed articles will be published separately, and if you have questions about technical aspects of this work, please contact Tim. In this piece, we focus on the broad story about how professional development can change instructor behavior in the classroom.

Our Theory of Change is that instructors need professional development to make the switch from passive instructional methods to creating active, student-centered, inquiry-based learning environments.  This professional development must address instructors’ knowledge and skills to carry out IBL in their own classrooms, but also their beliefs about instruction, and must mesh with their own identities as teachers and understanding of their own students and teaching contexts.  Achieving substantive change in teaching can be addressed via intensive summer workshops that provide these resources and a strong, steady support system that follows each instructor through the evolution of their professional career.  Yet we also need to find ways to build the pipeline of instructors who are aware of and receptive to IBL methods.  To widely offer these varied forms of professional development will require a larger, more flexible group of skilled professional developers than is now available.  Building capacity to design and deliver effective professional development is necessary for increasing uptake in college mathematics.

College math instructors are not fully trained in teaching. We may receive some training as TAs, and then we are off to start our teaching careers. Active, student-centered teaching methods, such as IBL, require skills and practices for maximizing effectiveness. While some instructors are able to learn and develop these skills on their own or by attending conferences, professional development workshops can provide a big boost to help instructors new (or newish) to IBL teaching.

The IBL workshop is a 4-strand model. The fours strands are (1) analyzing video of IBL classes, (2) discussing the nuts and bolts of running an IBL class, (3) developing a target IBL course, and (4) studying articles from the mathematics education literature.  The four strands were designed to address specific obstacles instructors face in their classrooms.

How do we know a workshop works? To answer this question, we need to define two terms, IBL capacity, and IBL intensity.

IBL capacity is an aggregate of several variables. It includes a participant’s beliefs about the effectiveness of IBL, knowledge of IBL, and skill in using IBL. What the evaluation team did is ask participants to self-rate their capacity before the workshop, immediately after the workshop and one year after the workshop.

Our data show in the figure below that workshops increase IBL capacity. We see a statistically significant increase in capacity measures from pre to post workshop and that these gains are sustained through to the one year follow-up.

(The slight drop from post-workshop to follow-up is not statistically significant.)

IBL intensity is a way to measure how much or how intensely an instructor is using IBL. We know the workshops are effective because a large percentage of participants, 94%, reported using IBL methods after attending the workshop.  We also asked participants about the frequency of use of a range of teaching practices (e.g. group work, student presentations, lectures, etc.) before they attended the workshop and again one year after they attended the workshop. Higher scores indicate more intensive use of IBL and lower scores indicate lower IBL use.  Based on their responses, we created an IBL intensity scale shown on the y-axis of the chart below. We see a statistically significant increase in IBL intensity from pre-workshop to follow-up.

Crucially there is more to the story on top of the increases to IBL capacity and intensity. The evaluation team also conducted additional analysis to explain more precisely how the workshops work, using the theory of planned behavior (Ajzen 1991). The theory of planned behavior path model explains how the IBL workshops are influential in changing teaching practices and the relationships between IBL capacity, IBL intensity, and some other key factors.

Let's go down the left side of the model. Belief that IBL is effective is a factor that influences teaching. Simply put, the stronger your belief is in the effectiveness of IBL, the more likely you intend to implement IBL, which in turn affects IBL intensity. The IBL workshop has a strand that provides an opportunity to learn about the research supporting the use of IBL, see successful examples of IBL, and learn about real-world stories of student success. 

Subjective norms include factors such as department or institutional support and supportive colleagues.  For instance, the more supportive your department chair is, the higher your intent to implement is, which then affects IBL intensity. Generally, the more supportive the environment is, the more instructors use IBL. Subjective norms can also be a barrier. If you lack support, then you are less likely to implement. Norms matter, because teaching is a cultural activity, and local environments play a role.

The last factor on the left side is perceived behavior control.  This is the perception of how much you control your behavior, which is highly dependent on IBL capacity. If you feel confident in your skills and knowledge how to implement IBL, then this directly impacts both intent and IBL use. This is where IBL workshops play a direct role. Skills and practices are central topics of the workshops, and as shown we have data supporting participant gains in IBL capacity. 

There are other contextual factors that play a role. These factors are prior IBL experience, whether a course is coordinated (in a way that is supportive of IBL), and class size. All of these factors can potentially be barriers, such as coordination that steers instructors away from IBL or large class sizes.  

Factors such as gender, career stage, rank and position, and institutional type do not influence IBL capacity or IBL intensity (i.e. were controlled for).  This is a striking result, because what this means is that anyone can teach via IBL anywhere! While individuals do face different challenges in the field due to their identity, group differences are not detected in the data. Of course identity matters in teaching, but what we know from experience and from data that no matter what group you are in, it is possible to teach via IBL in whatever setting. This means that the challenges we face are surmountable, and the focus should be on finding ways to support instructors.

Technical Question: What does r = 0.18 mean in the relationship between attitudes that IBL works and intent to use IBL? These are standardized regression coefficients which describe the relative strength of association between an independent and dependent variable. Standardized coefficients have standard deviations as their units, making the coefficients comparable when variables have different levels of measurement.  Please also see “An Effect Size Primer: A Guide for Clinicians and Researchers” https://doi.org/10.1037/14805-020

Summary of key findings

1. The theory of planned behavior model explains 21% of the variability of behavior change (IBL intensity). (Note: R^2 equal to 21% is a significant result for social science. For more see this Link) 
2. Workshops increase IBL capacity. 
3. Workshops increase IBL intensity.
4. Department norms and institutional support matter for better or worse.
5. Other factors like class size, course coordination, environment, course coordination, beliefs, play a role influencing IBL intensity. 
6. Anyone can teach IBL anywhere (under the right conditions)

Consequently, we can confidently say that IBL workshops work. Instructors change practices in meaningful ways. We can measure IBL capacity and intensity, and further we can see more clearly how the pieces fit together in a coherent story. Some policy implications include increased and sustained investment in professional development workshops, providing more support for faculty to access training, conducting outreach by individuals and professional societies, targeted efforts to inform and train department chairs and course coordinators, and investing in addressing institutional barriers to adoption of IBL methods (e.g reducing class size, mitigating the effects of large class sizes).

References

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179-211.

Archie, T., Laursen, S., Hayward, C. N., Yoshinobu, S., & Daly, D. (2020, November 5-7). Findings from 10 years of math instructor teaching professional development [Link to Poster]. This Changes Everything, AAC&U Virtual Conference on Transforming STEM Higher Education.

Ferguson, C. J. (2016). An effect size primer: A guide for clinicians and researchers. In A. E. Kazdin (Ed.), Methodological issues and strategies in clinical research (p. 301–310). American Psychological Association. https://doi.org/10.1037/14805-020

Why I Use Dr. Y or Professor Y When I Teach

When I started teaching, I was given some advice to use my first name with my students. The idea was to be more friendly and make things more comfortable in class. Many of us, perhaps nearly off of us, got that message. I didn’t think about it too hard then, and I did that for many years, all the while not knowing that what I was doing was making things harder for women in academia.

Last month Dr. Jill Biden was the target of a disdainful WSJ op ed, basically saying she’s not a real doctor and full of sexism and anti-intellectual tones. It’s one example of a long history of sexist put down of successful, smart women in the academy.

Men get to be Dr. ___, even if they use Stan or Paul in class, but women get called Ms.___ or Alice, and it’s not always respected or assumed that the woman is a Dr. ___ and an expert.

This is how systemic biases can work. We do things without knowing that it undermines a group of people. A well intended suggestion I received early in my career, which sounded like good advice at the time, actually had some hidden( to me) negative aspects that oppresses women in higher education. Later I learned about the biases that make it so that women have to work harder and deal with more, and the extra challenges and burdens women have to deal with.

Consequently, I decided years ago to use Dr. Y or professor Y with my students. It’s out of solidarity with women and minoritized groups, and it’s with the long-term goal of contributing to shifting norms towards equity and social justice. If we all did it, then that’d be the standard way students would address *all* faculty.

In countries like Japan, all teachers in elementary and secondary schools, professors, medical doctors, other leaders are given the title, “sensei.” It is an honorable title for those who teach or help others in society. Japan is not a society that earns high marks for gender equity, so I’m not trying to say that the term is some magic bullet. Thinking about what sensei means in Japan, however, does provide useful insights. Doctor can mean more than one thing, and we generally lack norms that could help our education system be more inclusive and equitable for women.

The closest thing we have to sensei is professor, although professor has been earmarked as a rank. For those who are lecturers and without a Ph.D., we do not have a professional title. Given all this, I argue that it is appropriate for all college instructors to use the title “professor” in classes, just as sensei is used by K-college instructors in Japan. Words and titles can mean more than one thing, and their meaning can be easily understood in context.

Thus, I invite all my colleagues to consider using Dr. or professor with our students, if you haven’t done so already, because it contributes to shifting norms in a decisively positive direction. We could explain to our students why we are using our titles, and why it’s important to respect all educators for gender equity. Where possible, we can use our titles, identities, and positions at our institutions to level the playing field for women in higher education.

How much professional development is enough?

This post is a summary of some research findings for a series of workshops we have conducted with NSF PRODUCT. The evaluation team for the project is Dr. Sandra Laursen, Dr. Tim Archie, and Devan Daly, CU Boulder E&ER (Link to work on professional development, link to people). The talk was presented at the Joint Mathematics Meetings on January 7, 2021.

One of the many research questions they studied is, "How much is enough professional development?" 

Some key points:

• We offered two general types of workshops. 
• One week, residential or online via zoom summer intensive workshops (IWS). These were 30 hours+ of workshop time with a year of follow-up email mentoring.
• Shorter traveling workshops (TWS), where two facilitators would travel to a conference or department and offer a workshop that lasted from a few hours to a day or day and a half. 
• IBL capacity is a measure of skills and practices related to IBL teaching. It's sort of like a battery pack for teaching. More capacity means more skills and knowledge.
• One result from the analysis is that both formats had a positive and significant impact on increasing IBL capacity.
• IWS participants implement a more intensive version of IBL compared to TWS participants. 
• TWS reached a different subset of the teaching population and was effective at increasing interest in IBL. 
• TWS reaching different subset of teaching population is due in part by outreach efforts to send facilitators to groups that are not doing IBL yet. For example, sending teams to 2-year colleges. The IBL community in math roots in the Mathematical Association of American, which skews toward 4-year and advanced degree granting institutions. Travel funding can be a barrier for IWS, and TWS commitment level is lower. (It's a commitment to spend a week of summer at an IBL Workshop.) 
• Rather than the dosage analogy, a another analogy is TWS and IWS are different tools, and could be used strategically for different main purposes.  
• Takeaway: Use TWS to increase awareness, interest, reach new instructors, and get people trying IBL methods.  Use IWS to increase depth of IBL implementation.