Emotive Motion: Analysis of Expressive Timing and Body Movement in the Performance of an Expert Violinist

Sheena Chandran, John Stoecker, Matt Wright

We have collected huge amounts of data, but only begun to analyze it. Our goal in this midterm presentation is to receive lots of feedback about our ideas for data analysis, and to receive new ideas for approaches to try. Please write on this handout! We especially solicit corrections, criticisms, questions, and suggestions; obviously, these will be much more helpful to us if you give this handout back to one of us so we can read it.

Tempo Curves

The musical “tempo” is the rate at which beats occur, generally in units of beats per minute. Modulation of tempo, i.e., speeding up and slowing down, is an essential element of Mr. Shiffman’s musical expressiveness. John painstakingly marked the time at which each of 17 measures began in each of the eight performances, and so far has hand-marked the onset of all 171 notes in three of the eight performances. From these data we can derive tempo curves:

Figure 1: Tempo curves per-bar (smoother) and per-note (more jagged), for “normal” performance

This calculation is extremely simple:

1)take each inter-onset interval (time between successive onsets)

2)take the reciprocal

3)scale by each note's duration in the score (to account for the fact that you expect eighth notes to be twice as long as sixteenth notes)

4)scale by the right constant to make the units be beats per minute.

The calculation for the per-bar curve is the same, but with a different constant.

We believe that the per-note tempo curve is too jagged, and does not match actual perception of tempo. We are interested in smoothing the tempo curve, to decompose the expressive timing into a true tempo curve (matching a model of human expectation of the next beat) and per-note deviation from the time predicted by the tempo curve.
Position Paths

Figure 2: Position paths of the center of pressure of all eight performances

Obviously, the overall amount of movement is much less in the “no expression” and “least motion” performances compared to the others. Certainly comparing the X and Y ranges and standard deviations will show this difference. Below is a close-up of the left edge of the “angry” versus “playful” performances; note the qualitative difference in curvature.

Figure 3: Detail of transition periods of two performances. Note the jagged appearance of the "angry" plot compared to the rounded curves of the "playful" plot.

What other differences do you see? How could we quantify them?
Temporal Correspondence

Figure 4: X and Y positions of center of force over time, with barlines overlaid, "normal" performance.

The X position of center of force is clearly swaying back and forth on the time frame of one or two bars of music. Below we see another parameter (moment about the Z axis) which is clearly related to hand-marked times (note onsets). We expect that elements of the joint kinematics will corresponding to time frames of musical notes, beats, bars, or phrases. How can we quantify this temporal patterning of the movement data?

1)To evaluate a hypothesis such as “X position follows a pattern every two bars,” we can segment the position curve into 2-bar chunks, normalize the time durations to be equal, and then take the point-wise mean across all chunks. If the mean has a clear shape (e.g., rising and falling) rather than being flat or indistinct then that tells us something about the pattern. The standard deviation of the difference between the individual segments and the mean shape tells us something about how consistent his motion is.

2)We could synthesize some kind of continuous curve based on the time points, e.g., a parabola, one half period of a sine wave, or a simple ramp, with each segment spanning the duration of, e.g., two bars. Then we could see how well this synthesized curve correlates to the actual movement data.

3)Get the variables of interest into the frequency domain and look for spectral peaks. (See below.)

Figure 5: Moment about Z axis over time, note onset times overlaid, "no emotion" performance

Frequency Domain

Figure 6: Magnitude spectrum of X force data, "normal" performance

We had hoped that examining the motion data (in this case, X position of center of force) in the frequency domain would reveal aspects of the periodicity structure. Unfortunately, this graph doesn’t seem to show much. What are we doing wrong?

1)Shift the data so it’s zero-mean: X_zeromean = X_orig - mean(X_orig);

2)Apply Hamming window:

  • X_win = X_zeromean .* hamming(length(X_zeromean));

3)Take one huge FFT of the whole thing, take magnitude part, put on dB scale:

  • X_fft = db(abs(fftshift(fft(X_zeromean, Nfft))));

There are many other frequency-domain techniques that might shed some light on our data: short-time Fourier transform, spectrogram, autocorrelation, etc.