Lab 7: SignStream lab

This page has three parts:

• Introductory notes.
• In-class exercises.
• Take-home exercises. Due Mon Dec 6.

Introductory notes

SignStream is a system, developed here at Boston University, for analyzing video correlated with a transcription database, primarily geared toward analyzing signed languages such as ASL. We will do a short lab using it to search for linguistically-relevant aspects of ASL.

The materials for this lab probably have not been installed on the CAS 330 computers. You'll need to download some subset of the following files. These materials are password-protected in the same way as the course readings (you use the same login name and password). Note that the movie files are large, so if you plan to download them over a dialup connection, have a relatively long book to read while you wait.

• SignStream 2.22 (the program file)
• SignStream guide (the manual)
• cbs-2.mov (news anchor talking about debates prior to Election 2000).

Other resources of interest

• abc.mov (news anchor talking about Election 2000)
• abc2.mov (excerpt from an interview with a voter asked to run a "reaction" machine during one of the debates)
• rather.mov (news anchor talking about Election 2000)
• biz.mov (excerpt from an interview on CNNfn)
• abc-demo1 (SignStream 2.04 database for abc.mov, the first sentence, with the words already transcribed)
• abc-sample (SignStream 2.04 database for abc.mov, the second sentence with the words already transcribed).

In-class exercises

The task: Transcription practice with SignStream, looking at eyebrow movement.

In class, we went through an introduction to SignStream using the abc.mov file listed above, transcribing the eyebrow movements. The steps here only gives a sort of rough overview of what we looked at; you will find the SignStream manual (see above) helpful as well.

Steps:

Start Signstream.

• This is accomplished by double-clicking on the SignStream 2.22 icon. See above for how to get the files. Note for Mac OS X users: SignStream runs in "Classic" mode on Mac OS X, but it still should work for what we're trying to do.

Extract the audio file from the abc.mov movie file that we're going to be working with.

• To do this, choose Create sound file... from the File menu. You will be asked to first pick the movie file from which you want to extract the sound. Find the cbs-2.mov file and select it. Then, you will be asked to name the sound file that will result when you extract the sound; you can call it cbs-2-sound and save it in the same folder as the movie file.

Open the SignStream database with the first half of the words transcribed.

This file is called cbs-database. To open it, Choose Open Database.. from the File menu and pick cbs-database. (If you have cbs-database.sit, rather than cbs-database, then double-click on it to "unstuff" it).

• When you first open it, you may receive some warning messages (although it might happen to be able to find the files itself). Since we are opening it on a different computer from the one on which it was created, SignStream sometimes isn't initially able to find the cbs-2.mov file for the video and/or the cbs-2-sound file for the audio.
• If it tells you that it cannot find the video and/or sound file, click Yes when asked if you want to locate it and locate the cbs-2.mov file. You will be warned that you need to pick the same sound file that we started with, click Proceed. Find the file in the list (you may want to click the Desktop button to get you to the "top level" of the computer's hard drive if you don't find the file right away). The location of the files on the machines in CAS 330 is given at the top of this page. You may also want to continue the search for files if it asks you this, in which case it will ask you to find the other file (the movie/sound file that you didn't just find).

Once you have the database open, we can start to look at what the eyebrows are doing and when the speaker blinks.

First, we need to add fields for these things in our transcription window. If you can't see this window (the one with the words transcribed in it), press the Gloss button in the upper left corner of any of the open windows.

In the gloss window there is a button called Show field which will let us add some fields to our gloss. Since eyebrow movement and blinking are already in the predefined list of fields, we can just pick them. Double click on eyebrows (or, alternatively, select it in the list and click OK), then click the Show field button again and double-click (or click & OK) on eye aperture.

Scan your way through the movie using the sliding bar at the top of the Gloss window (or fine tuning with the frame advance/back buttons at the right of the sliding bar).

• You might find it helpful to resize the Gloss window to its widest possible size for the screen, since that will give you better control over the movie position in the sliding bar. To resize the window, grab the box in the lower corner of the window by clicking down the mouse button on it and keeping the button depressed while pulling it to the right.

Look for places where the newscaster's eyebrows seem to be at the peak of their motion. For example, just at the end of time when he says the word "twenty-three".

When you've found a peak, or part of a peak, in eyebrow height, double click on the part of the green "play head" line that is in the eye brows row. In the resulting dialog box, choose raised from the list and click ok. This will create a dot on the green line on the eye brows line.

Now, we want to extend the raised indicator to cover the whole time that the newscaster's eyebrows are raised. To do this, click on the dot you just made and drag either left or right until it seems like the newscaster's eyebrows are just starting to come down.

Then, while the raised object is still selected (you can tell if it selected by looking to see whether it has little "bumps," or "handles," at each end; if it is not selected, you can click on it once to select it), click on the "go to end of object" button, which looks like a "T" lying on its side, with the flat edge on the right. Then, click the frame advance button (upper right corner, on the right edge of the sliding bar) once to move the movie ahead one frame (you may not see the picture change), and then click somewhere in the eye brows row other than on the little bar you've just created. This will de-select the eye brows object we created, you should see that the boxes above the glosses (that contain the start and end frames of the current object) will go blank.

Since we want to make a segment for "eye brows going down", click the Set button by the second (end frame) box (upper middle of the screen), which should bring up that dialog box again. This time, choose end, to signify that the eyebrows are coming down.

This will create a new little segment. You now want to drag that segment (if it a dot, drag the dot; if it is a little segment, drag the bump under the right edge of the segment) to the right until it seems from the image in the movie that the eyebrows have come all the way down.

We will not code "eyebrows lowered"; we will assume that when nothing appears in the eye brows row, the eyebrows are lowered.

If you were following these instructions exactly, we still need to adjust the start time of the raised segment to coincide with the point where the eyebrows have stopped raising and are at their full height. To do this, click on the raised segment once (to select it, which will produce little "handles" under each edge), and drag the "handle" on the left to the left until you feel that you have found the point where the eyebrows first hit their peak height.

Then, same trick as before, but on the other side; with the raised segment selected, click on the "go to beginning of object" button (the reclining "T" with the flat side on the left), then click somewhere in the eye brows row (to de-select the object we were working on), and click one of the Set buttons to make a new object. Make this one a start object to signify that the eyebrows have begun their ascent.

When I did this the first time, it seemed to assume that I wanted an object that started at the beginning of the movie, whether I hit the "start frame" Set button or the "end frame" Set button. It doesn't matter, though, just drag the left edge of the new object over to the right until you find the point where the eyebrows have just begun to raise.

Congratulations, one eyebrow raising is now coded. And you now know how to do it.

What we did in class is coded the rest of the eyebrow raises, some of the class using the same cbs-database database.

We also coded eye blinks, which you can do in the eye apert line in the Gloss window, using the same techniques.

Things to remember: Save as you go along !

Take-home exercises

Your task: Transcribe one of the two videos in terms of eyebrow movement, blinking, and other things that may be relevant, and try to determine if there is any systematic correlation between the content of the utterances and the gestures that occur.

Take the cbs-2.mov (or you could pick abc.mov, abc2.mov, or biz.mov, athough then you would need to transcribe the words as well).

On transcribing the words:

The main gloss line works similarly to the other fields in the Gloss window, except that it is a place where you can enter text, like in a word processor. What I would suggest you do first is go through the movie at regular speed (by hitting the "Play" button on the slider bar in the Gloss window) and then type the words that are being said in the main gloss field. Don't worry about where the words are relative to the film clip until you have typed in the whole phrase.

Then, once all the words are typed in, work your way from the left and set the start and end time for each word (or small phrase if it turns out that some of the words are just too short). To make this easier, open the sound waveform window by clicking the Media button in the upper left of the Gloss window, and then clicking the Open button that is next to the audio file entry on the Media window. This too will probably benefit from being made as wide a window as you can make it on the screen (drag the "resize box" in the lower right corner of the Audio window). If the wave form is too zoomed in, you can Zoom waveform out under the View menu.

You will probably also want to slow down the audio for this part. To do this, click the Media button in the upper left corner of any of the main windows to bring the media controller window to the front, and change the speed to 66% or maybe even 50%.

From there, it's a bit of trial-and-error hit-and-miss to get the words and the audio lined up. Use the same techniques of setting the start and end points with the Set buttons and positioning the green line "play head" in the waveform window, which can give you some clues (sometimes) as to when words start and end.

What to do:

• Transcribe the words and line up the main gloss objects with the video. Note that any word on the main gloss line that is linked with a start point and an end point on the video will be underlined.
• Transcribe the eyebrow raises, as we did in class.
• Transcribe the eye blinks as we did in class.
• Play the movie a couple of times, look at where the eyebrows raise, where the eye blinks happen. Does it seem like these correlate with anything else?
• For example, do eyebrow raises seem to go with stressed syllables/words? Do they seem to happen when the new information in a sentence is being uttered? Do the blinks seem to happen during words, or between words? Do they seem to have anything to do with syntactic phrases? Note that I don't know what you'll find, but these are the kinds of thing you might look at.
• Find something else to code that seems like it might be relevant, and code it.
  • To create a new line in the Gloss window (a new "tier" or "field"), open the Field Specifications editor under the View menu. You probably will be a new nonmanual field, so click the New nonmanual button.
  • This brings up a new window that lets you set up your new tier.
  • Choose a descriptive field name (e.g., "Stressed syllables") for the Field name, and then choose a short version (for display in the leftmost column of the Gloss window; e.,g, "str syll") for the Field label. Leave the Value prefix blank.
  • Click the New Value button and put the cursor in the New Value Name field at the bottom of the dialog box, and type the name of one of the values objects on your new tier can have (this is the analog to start, end, raised, etc., from the eye brows tier). As with the tier name, the Name is the longer, descriptive name, and the Label is the short version for showing in the Gloss window.
  • After having entered the values you think you will need, click Done.
  • Then, you can add another field, or you can click Done.
  • At this point, you should be able to click the List fields button in the Gloss window and add your new tier to the ones already in the gloss window (just as we added the eye brow tier initially).
• For example, if you felt that stress correlated with something, create a line in the Gloss window for "stressed". Or for "constituent". Or whatever seems relevant.
• Once you've glossed this new field, how did your conjecture fare? Does it seem like the eyebrow raises or eye blinks did correlate the way you thought they might? One (very real) possibility is that you could find that, say, eye blinks happen only under some certain condition, but do not happen every time that condition holds. This is still interesting, to be able to say that eye blinks do not happen except when that condition holds.
• Write up a paragraph or two describing your conjecture about what eyebrow raising and blinks might correlate with and describe what the data showed.
• Turn in the database files (emailing them to me is probably the easiest way to do this; the iMacs in the CAS 330 lab do not have disk drives, so saving them is difficult) to me when you're done. Note: Do not send me cbs-2.mov or abc.mov or abc2.mov or biz.mov or anything with .mov at the end of it; these are the movies, and they are huge. Just send me the database file you created at the beginning and have been saving at regular intervals.

Things to remember: Save as you go along !

You may find it helpful to consult: Notes on saving in CAS 330.