Dr. Shengzhi Zhang
Fooling the Machines: Dr. Shengzhi Zhang Tackles Security Issues in Deep Neural Networks
Assistant Professor, Computer Science
PhD, Penn State University; BS, Tongji University (Shanghai, China)
What is your area of expertise?
My research focuses on cyber security, especially security issues that could impact people’s daily life, such as AI security, IoT security, automobile security, and smartphone security.
Please tell us about your work. Can you share any current research or recent publications?
Currently, I have three ongoing projects. The first is to study the security issues in machine learning, especially deep neural networks (DNN). We craft hard-to-notice “perturbations” into audio/video/image, which can deceive the DNN-based recognition systems and cause a misprediction. For instance, a voice recognition system, e.g., Google Assistant, may decode “call 911” from a song integrated with our perturbation, but people listening to the song would not be able to interpret that. Such “adversarial attacks” in machine learning not only exist in voice/image recognition, but also in object detection (widely used in autonomous driving) based on our research.
This research was covered by The Register (https://www.theregister.co.uk/2018/01/30/boffins_songs_ai_assistants) and was included in the following publications: “CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition” (Proceedings of the 27th Usenix Security Symposium, Baltimore, Maryland, 2018) and “Practical Adversarial Attack Against Object Detector” (arXiv, preprint arXiv:1812.10217).
The second project is to comprehensively protect the execution environment for unmodified applications running on ARM-based IoT devices. By taking advantage of ARM TrustZone technology, we construct a trusted execution environment for security-critical applications, which is isolated from the untrusted operating systems. Publications in this area include “TrustShadow: Secure execution of unmodified applications with ARM trustzone” (Proceedings of the 15thACM International Conference on Mobile Systems, Applications, and Services, Niagara Falls, New York, 2017) and “Building a Trustworthy Execution Environment to Defeat Exploits from both Cyber Space and Physical Space for ARM” (IEEE Transactions on Dependable and Secure Computing, 2018).
Finally, there is a joint project with Ford Motor Company designing a security model for the headunit systems on future cars. Car manufacturers, such as Ford, are developing the next generation headunit systems with software modules and connectivity, giving the passenger/driver a seamless experience and increasing safety while driving the vehicle. We are collaborating to propose a novel security model that integrates cryptography, network security, and system security approaches to eliminate the threats against car security. You can read more in “A security model for dependable vehicle middleware and mobile applications connection” (VEHITS 2018: Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems).
How does the subject you work in apply in practice? What is its application?
I will take the first aforementioned project as an example to explain the practical impact of our research.
Recently, Deep Neural Networks have advanced artificial intelligence in many areas, such as speech recognition, face recognition, strategic games, and, especially, in some safety critical tasks, such as autonomous driving and medical diagnostics. However, deep neural networks are known to be vulnerable to adversarial examples, which leverage a few perturbations on original inputs to fool neural networks into misclassification. Most of the recent research is limited to image classifiers, rather than speech recognition or object detectors. Our research reveals that the adversarial attacks can also be crafted against speech recognition and object detection systems. Such findings demonstrate that using the deep learning techniques without security in mind will significantly impact the safety of everyone’s daily life.
For instance, our work “CommanderSong” crafts small perturbations into a song, thus enabling the “revised” song to be decoded by speech recognition system as a valid command to operate—even while a human would not be able to interpret the command. Consider a modern home with Amazon Echo connected to smart locks, lights, switches, and more. The resident gets home, tired, and starts music streaming from YouTube. If the song from YouTube happens to be the “CommanderSong” uploaded by us (supposing we are hackers), Amazon Echo will be triggered to decode valid commands from the song—for instance, “Echo, open the door” (the exact command to be decoded can be controlled by the perturbations we added into the original song). Then Echo will operate the command and unlock the door. Due to the small perturbations, the command in the revised song can’t be interpreted by human ears, and it would even be hard for a person to notice any anomaly, based on our survey. Considering the popularity of speech recognition in smart homes, smartphones, and even cars, security countermeasures are highly demanded when using deep neural networks in speech recognition.
Another example of an adversarial attack is to deceive the modern DNN-based object detectors, widely used in many fields such as autonomous driving. For instance, most of the existing autonomous driving cars rely on cameras to capture the surrounding environment, from which the object detectors recognize stop signs, traffic lights, pedestrians, and so forth. Our research demonstrates two kinds of adversarial attacks against object detectors: a “hiding” attack that fools the detector, rendering it unable to recognize the object; and an “appearing” attack that fools the detector into falsely recognizing a non-existent object. For the hiding attack, we attached our carefully crafted perturbations onto a stop sign, which the object detector was then unable to recognize from different angles and distances. For the appearing attack, the object detector marks our crafted adversarial image as a traffic light, but a driver would not interpret it the same way. Since object detectors play a significant role in autonomous driving, such adversarial attacks require immediate attention to assure the safety of passengers.
What course(s) do you teach at MET?
I teach Foundation of Analytics with R (MET CS 544).
Please highlight a particular project within this course(s) that most interests your students. If you previously worked in industry, what “real-life” exercises do you bring to class?
My students are encouraged to choose datasets based on their own interest, and apply the analytics techniques learned in class to analyze the data. Since students taking CS 544 are with diverse background—from departments of biology, finance, mechanical engineering, psychology, etc.—they typically have their own specific dataset that they want to analyze. The project allows students to target their datasets, as well as the ones that are closely related to their own research, which makes the project highly “practical” and “appealing” to students.