Tracing Code: an attempt to understand and improve students' process

Lynda Thomas, University of Wales, Aberystwyth

Motivation

Most CS teachers have had the experience of a student coming to them with a program and saying 'it doesn't work'. I've found over the years that when I start drawing pictures (rough interaction diagrams) and tracing the program out, students are often impatient, then amazed that my approach works. Unfortunately many students don't seem to then carry this forward into their own practice.

First Year programming students at Aberystwyth take multiple choice tests as part of their assessment. Some of the questions on these tests ask the students to trace out what happens when a few lines of code are executed. All test takers are provided with a sheet of scratch paper and these sheets are nearly always returned blank. Yet marks on this kind of question are typically awful.

I've observed that neither I, nor any expert I know, does a tracing problem without drawing some kind of picture. I want to find out what students currently do instead and whether it is possible to change their behaviour to something more productive.

Study's Focal Questions

What do students currently do when faced with a 'tracing problem' (whether on a test or in a programming situation)?
Can we 'scaffold' students with interaction diagrams and thus encourage them to use this technique on their own? Does that improve their understanding and performance?

Confession of 'premature experimentation'

I was thinking about this problem before Bootstrapping last summer and got so thrilled with it that in the autumn term I performed an experiment that attempts to answer the second question. I now realise that what I should have done is try and figure out what is going on with the first question first (.......... sigh). So, I'll write about the second question first because I've done more thinking about it. (And no doubt be chastised in June.)

Question 2: Scaffolding with Interaction Diagrams

Background and Justification

Research has shown that methods of teaching and learning that combine significant student autonomy, conjecturing and articulation with dynamic scaffolding by the teacher are highly effective [Tanner, 1999].
Interaction diagrams model dynamic behaviour in an OO program. There are two types: sequence and collaboration [Fowler]. I use a sort of 'sloppy' collaboration diagram with blobs for objects and arrows for references.
I believe that interaction diagrams are helpful in being able to trace code. In addition to 'gut feeling' there is a fair amount of related evidence. I've been influenced here by the work of Mary Hegarty who has made a study of cognitive models that impact on the understanding of mechanical systems.
In a paper with Narayanan they outline a cognitive model of understanding dynamic systems that supposes that the viewer decomposes the system into simpler components, and then constructs a static model by making representational connections to prior knowledge and other components. This is then followed by integrating information between different representations (e.g. Text and diagrams); hypothesizing lines of action and finally constructing a dynamic mental model by mental animation. The authors have empirically validated the design guideline that people learn more from being induced to mentally animate a system before viewing an animation than by passively viewing the animation [Hegarty].
If we examine Interaction Diagrams we can see that the first two steps outlined in the Hegarty model are essentially the creation of the basic diagram and the last three steps involve the actual tracing of the program code with reference to that diagram.
Our interactive learning environment provides a way to:
- give the scaffolding to the students,
- observe whether scaffolded students perform better on tests,
- check and see if they then incorporate the technique into their future practice.

Approach

All the students in the introductory programming class were shown interaction diagrams and examples of their use in tracing programs. We divided the class approximately in half for the experiment. One section, the control group, were given pieces of OO code and asked to trace them in a formative test. The other group were provided with both the code and an incomplete interaction diagrams. We compared the results to see how correct the subsequent tracings were (expecting that the experimental group would do better).

Several weeks later we gave students in both groups further questions that involved tracing. We compared their correctness results again and we also collected their scratch sheets to see if the experimental group used the interaction diagrams technique with which they had been scaffolded.

Analysis

I haven't done this yet - I haven't even glanced at the results. But I intend to do some kind of statistcal analysis to see if scaffolded students did better in the first test and in the second test (t-Test??). In addition I will compare scratch sheets of control and scaffolded students. Not sure what test to use on that?

First Question

Which I should have done first. I think that I should basically get some students and ask them to trace some programs and see what they do. Pretty obvious really, but I'm a bit lost on how to go about this kind of observation. (I know that McCracken had a paper in CSEET 2002 that touched on this kind of observation.)

Bibliography

[Fowler] Martin Fowler with Kendall Scott, "UML DIstilled", Addison Wesley, 2000.
[Hegerty] N. Hari Narayanan and Mary Hegarty, "Communicating Dynamic Behaviors: Are Interactive Multimedia Presentations Better than Static Mixed-Mode Presentations?", in Theory and Application of Diagrams, Diagrams 2000, September 2000, Michael Anderson, Peter Cheng and Volker Haarslev editors, Springer Lecture Notes in Artificial Intelligence 1889
[Tanner] Tanner, H. and S. Jones, “Dynamic Scaffolding and Reflective Discourse: the Impact of Teaching Style on the Development of Mathematical Thinking”, Proceedings of the 23rd Conference of the International Group for the Psychology of Mathematics Education, Haifa (1999)