Report on Major Project

Using Just-in-Time-Teaching Techniques in an Algebra-based Physics Class

James E. Heath, Jr.

Associate Professor of Astronomy and Physics

Austin Community College

Abstract

The technique of “Just-in-Time-Teaching” (JiTT), developed by Gregor Novak and Evelyn Patterson, has greatly improved physics teaching at institutions such as IUPUI and the Air Force Academy. I implemented this technique in my General Physics I classes at Austin Community College to assess its effectiveness. I prepared a series of Web-based “warm-ups” modeled after the ones employed by Novak and Patterson. The “warm-ups” were integrated into some of my sections of GPI, while other sections were held as “controls.” Unfortunately, the JiTT techniques appeared to have no effect on class attendance, student retention and student performance.

Report on Major Project

Using Just-in-Time-Teaching Techniques in an Algebra-based Physics Class

James E. Heath, Jr.

Associate Professor of Astronomy and Physics

Austin Community College

In one of the “Physics Workshops for the 21st Century” offered by the Two-Year College Physics Workshop Project, participants learned about the technique of “Just-in-Time Teaching” (henceforth known as JiTT) from two of the originators of the technique, Gregor Novak of IUPUI and Evelyn Patterson of the U. S. Air Force Academy. Novak and Patterson demonstrated the technique, which involves using Web-based “warm-ups” to achieve two primary goals: to motivate students to prepare for class, and to provide instructors with information about students’ pre-existing knowledge. Novak and Patterson claim that JiTT techniques improve not only student performance in Physics, but also student retention, a real concern for many teachers of “hard science.”

Many years ago, I incorporated “reading quizzes” into my astronomy classes – brief quizzes over assigned reading to motivate students to prepare for class. Time concerns caused me to discontinue these quizzes, and student performance suffered in subsequent semesters. I was also one of the first instructors at Austin Community College to incorporate the World Wide Web into my teaching, using it as an essential source for information and class assignments. Based on these experiences, I was interested in giving JiTT techniques a try in my Algebra-based General Physics I classes. At Austin Community College, as with so many institutions, we have a problem with student retention; 40% or more of the students that begin our physics classes withdraw by semester’s end. An alarming number of our students are not finding success in physics, and we obviously want to change that.

This research project stretched across three semesters, from the Spring Semester 2004 to the Spring Semester 2005. During that time I taught a total of six sections of our algebra-based introductory physics course, General Physics I (GP1). Each semester I required one class to do a series of 20 “Web Warm-ups” as 10% of the grade. The other section each semester did not use the warm-ups, and that 10% of the grade was devoted to attendance. I gathered data for all sections in a number of categories:

· Student attendance

· Percentage of students that fail to complete the course

· Final Exam grades

· Total semester grades

· Score on the Force Concept Inventory, pre- and post-tests

· Gain on the Force Concept Inventory

Creating the Web Warm-ups

The first step was obviously to create the twenty “Web Warm-ups.” The first warm-up was essentially a way to gather some information about the students, and to introduce them to the system. The rest follow my course syllabus, with a warm-up for almost every day that was not devoted to testing or review. To create these warm-ups, I followed the model provided by Novak and Patterson. Each warm-up consisted of three questions:

The first question was always a “short answer” style question. Most of these questions incorporated the various styles of “TIPERS,” such as Conflicting Contentions Tasks and Ranking Tasks. The subject matter for these questions was usually chosen to address common misconceptions that I have encountered in my years of teaching. For example, in one warm-up students are asked to choose between two people who present different views on the meaning of negative acceleration.

The second question was a multiple-choice question, usually with four options. Some of these questions were also used to address common misconceptions (such as the acceleration of an object in uniform circular motion), while others were intended to help students familiarize themselves with the meanings of certain basic equations. For example, students were asked in one question to ponder what factors determine the amount of work that a force does on an object.

The final question in each warm-up involved a basic equation. Typically they were very simple calculations, just intended to get the students acquainted with the equation. For example, in a series of warm-ups, students would calculate various quantities for a high-performance sports car: acceleration, net force, work, and power.

Students could access the warm-ups from any computer via the World Wide Web. They were asked to submit answers to the warm-up questions on or before the day that the material in the warm-up was to be discussed in class. Answers were due a few hours before class, to give me time to look over their answers and adjust my plans accordingly. Student answers were given credit for completeness only, not correctness.

In order to assist students in reading the parts of the text relevant to the warm-ups, I included page numbers with each one. Our department changed textbooks halfway through this project, from Giancoli’s algebra-based text (Prentice Hall) to Cutnell and Johnson’s (Wiley). Very little had to be changed about the warm-ups. In their current form, the warm-ups are keyed to Cutnell and Johnson. Note that order of topics diverges in places from the order in the text, as I take an non-traditional approach to the first semester curriculum.

The warm-ups are all available in HTML format, along with the “cover page,” in the “GPI” folder on the CD.

Results

Completion of Warm-ups

Unfortunately, few of the 42 students in the three sections that were required to do the warm-ups submitted answers for a large number of them. Only one student did all 20 warm-ups, and only 15 (36%) did more than 15 warm-ups. The average number of warm-ups done by a student was 13. Despite the fact that each warm-up missed deducted half a percentage point from their final grade, students still did not make completing the warm-ups a priority.

Student Attrition and Attendance

It was hoped that inclusion of the Just-in-Time teaching technique would help to relieve some of the perennial problems of attrition that we face in our physics classes. Comparions between classes with and without the warm-ups for the three semesters, as well as data from semesters before the project, are found in Table 1. A student is considered to have “started” the class if he or she took the first exam. Some students withdraw very early in the semester when the course is not what they expected; such students can hardly be considered to have given the warm-ups (and the course in general) a chance. A student “finishes” the course by receiving a final grade. Average number of absences is calculated only for students who finish the class.

Table 1: Comparison of Student Retention and Attendance

Semester		Number of students starting	Number of students finishing	Attrition	Average number of absences per student
Spring 2004	Warm-ups	20	14	30%	2
	No Warm-ups	19	13	32%	1.4
Fall 2004	Warm-ups	20	17	15%	0.60
	No Warm-ups	15	13	13%	1.1
Spring 2005	Warm-ups	17	11	35%	1
	No Warm-ups	19	15	21%	1.5
Previous Semesters 2001-2003		124	103	17%	1.3

Although there was some difference in attrition levels between semesters, there was no noticeable difference in attrition between sections in the same semester. Thus, the inclusion of warm-ups appears to have no effect on attrition. Likewise, the number of absences per student finishing the course was unaffected by the inclusion of warm-ups. The Web warm-ups appeared to have no effect, good or bad, on whether or not students came to class, and whether or not they finished the course.

Performance in the course between groups

I tried to determine the effect of the Web warm-ups on student performance by comparing the two groups (warm-ups vs. no warm-ups) in a number of categories. First, I compared scores on the Final Exam, a 200-point exam on which about a third of the points came from conceptual essay questions, and the remainder from worked problems. Second, I compared overall scores for the course. As mentioned earlier, 10% of the grade for the first group of students required completion of the Web warm-ups. For the other course, that 10% was taken up by attendance. The remaining 90% of the course grade was the same for both groups:

· Three exams – 30%

· Final exam – 20%

· Lab reports – 25%

· Homework Assignments – 10%

· Short Quizzes – 5%

The first week of each semester, I gave the Force Concept Inventory test to both groups. Then, halfway through each semester, I administered the FCI as a post-test. The pre- to post-test gain was calculated in the usual manner:

Gain = (post-test score – pre-test score) / (maximum score – pretest score)

Results for each indicator of performance are presented in Table 2 for each semester, and for all semesters combined:

Table 2: Comparison of Student Achievement

Semester		Average Final Exam Score (out of 200)	Average Final Total Class Score (out of 100)	Average FCI Post-test Score	Average FCI Gain
Spring 2004	Warm-ups	150	79	15	0.29
	No Warm-ups	145	83	16	0.37
Fall 2004	Warm-ups	141	82	18	0.42
	No Warm-ups	148	84	19	0.41
Spring 2005	Warm-ups	153	82	17	0.30
	No Warm-ups	162	82	18	0.39
Overall	Warm-ups	147	81	17	0.35
	No Warm-ups	152	83	17	0.38

From the table, it can be seen that there are no significant differences between the two groups in any category. In fact, it could be argued that the classes without warm-ups had better results that the classes with warm-ups!

In a further effort to ascertain if the Web warm-ups had any effect on student performance, effect size calculations were made using the following formula:

d = (average with warm-ups – average without warm-ups) / (total standard deviation)

Results are summarized in Table 3 below. All of the calculated effect sizes are less that 0.25, so again it can be concluded that including the warm-ups had no effect on student performance, by any measure.

Table 3: Effect sizes, warm-ups (N = 42) vs. none (N = 43)

Quantity	Final Exam	Total Grade	FCI Post	FCI Gain
d	-0.213	-0.23	-0.097	-0.15

Effect of Number of Warm-ups Completed on Student Performance

Looking only at the data for classes that had the Web warm-ups, I made graphs of the number of warm-ups completed versus the following variables: attendance, final exam grade, total class score, FCI post-test score, and FCI gain. The data and the resulting graphs are presented in the Excel file on the CD. In this case as with the between-groups data, there appears to be no significant correlation between number of warm-ups completed and student performance. Values of R² for the combined data are presented in Table 4 below.

Table 4: R² values for overall data graphs

Number of Warm-ups Completed vs.	With outlying data point (N = 42)	Without outlying data point (N = 41)
Total Score	R^2 = 0.3318	R^2 = 0.1992
Final Exam Score	R^2 = 0.0412	R^2 = 0.0031
FCI Post-test	R^2 = 0.0007	R^2 = 0.0020
FCI Gain	R^2 = 0.0048	R^2 = 0.0218

These R² values are very small; the only substantial correlation appears to be for warm-ups completed vs. total grade. This is likely due to two factors. First, there should be some correlation, since 10% of the course grade was for completing the warm-ups. Second, much of that correlation seems to be from a single data point. Removing the data from that very poorly performing student (who gave up doing warm-ups early, along with most other aspects of the class) lowers the value of R² for warm-ups completed vs. total grade.

Looking at the data, I was surprised at the lack of effect that the Web warm-ups had on my students’ performance. Looking at how few of the warm-ups many students completed, however, it is perhaps less surprising. It could be that lack of student motivation to do the warm-ups in the first place hampered their ability to be effective. Students made very few favorable comments about the warm-ups, and mostly saw them as yet another requirement of the class to “get out of the way.” It is sad but true that so much of a student’s performance depends not on what we do as teachers, but what the student does.

How did the warm-ups affect the way that I taught? One thing that they did do was provide me with a context for discussing certain “sticking points” in the curriculum. Before discussing a particular misconception or hard-to-grasp concept, I could project the warm-up onto a screen, or just remind the students of the question, and solicit their answers as a starting point for discussion. The warm-ups gave students a “sneak peek” at some of the main points of our upcoming class discussion, which I hope helped some of them in their understanding. Also, the warm-ups gave me some idea of things that the majority of students already knew (or had learned by preparing for the warm-up), so that I could spend less time on those well-known concepts. Still, I am disappointed by the lack of good results in my project, and hope that others have more success with Just-in-Time teaching.