Recent NSF award

     Professors Stan Sclaroff (Chair, BU Computer Science) and Carol Neidle just received a $900,000 grant from the National Science Foundation for a collaborative project with Vassilis Athitsos (BU PhD 2006, now at the University of Texas at Arlington). The funding for "Large Lexicon Gesture Representation, Recognition, and Retrieval" will support research on computer-based recognition of ASL signs. One goal of the research is development of a "look-up" capability for use as part of an interface with a multi-media sign language dictionary. Although printed dictionaries exist for ASL, they are generally organized according to the closest English translation of the ASL sign, since there is no written form for ASL.  There are obvious problems resulting from the fact that there is no one-to-one correspondence between English words and ASL signs (imagine if you could only get information about French words--or words in any other spoken language--by looking them up under their English translations). This also poses an insurmountable difficulty for language learners, a kind of Catch-22:  you can only look up a sign you don’t know in the dictionary if you already know what it means.

     The proposed system will enable a signer either to select a video clip corresponding to an unknown sign, or to produce a sign in front of a camera, for look-up.  The computer will then find the best match(es) from its inventory of thousands of ASL signs.  The technology to make this possible will exploit computer vision algorithms combined with knowledge about linguistic constraints of sign production to improve recognition. One goal is to use this technology in combination with multi-media dictionaries, such as the one currently under development by DawnSignPress, a company co-founded by Ben Bahan (BU PhD, 1996, currently Professor of ASL and Deaf Studies at Gallaudet University), which will offer definitions, information about etymology and variants, and examples of usage, all in ASL, through video. The technology for sign identification will have many other applications as well, e.g., for computer-based automatic translation. This project includes plans to develop a “sloogle,” to do google-like searches through streams of ASL video.

     Prof. Neidle is also continuing her NSF-funded collaboration with Dimitris Metaxas (Computer Science, Rutgers University) on "Advances in recognition and interpretation of human motion: An Integrated Approach to ASL Recognition."  Her interdisciplinary collaborations with computer scientists make use of the large corpus of ASL data that has been collected from Deaf native signers in the National Center for Sign Language and Gesture Resources at Boston University, which has also been funded by NSF.  Linguistic annotations, carried out using SignStream™ (a computer program developed by Prof. Neidle and her colleagues specifically for this purpose), provide “ground truth” for developing and testing computer vision algorithms.  These data and software tools are made publicly available for use by the linguistic and computer science communities.  A new Web interface for access to this data is nearing completion, and a large set of newly annotated data has just been released.  These annotated data have also been of great utility for Professor Neidle’s linguistic research on the syntax of ASL.  Further information about the American Sign Language Linguistic Research Project is available from http://www.bu.edu/asllrp/.