CISE Fall Workshop Asks How We Know When AI Can Be Trusted
When an AI system helps decide who gets treated, who gets taught, or what gets flagged, saying “it seems to work” is not good enough. How do we test whether these systems can be trusted, keep them pursuing the right goals, and make them work well with the people who rely on them?
On November 6, the Center for Information & Systems Engineering (CISE) took on these questions at its Fall Workshop on AI Evaluation and Alignment for High-Stakes Systems. The event brought together researchers from machine learning, operations research, systems engineering, and the social sciences, and was organized by Vasilis Digalakis, Questrom (OTM) and ENG (SE); Brian Kulis, ENG (ECE, CS, SE); and Jinglong Zhao, Questrom (OTM) and ENG (SE).
Why “Good Enough” Is Not Enough

Elise Morgan, Dean of the College of Engineering, in her opening remarks, pointed out that in high-stakes settings, “good enough” has real consequences. “It’s about whether that system can really be relied upon when safety, when privacy, when opportunity, when health are on the line,” she said. Linking the topic to the College’s mission “to expand the power of engineering to transform lives at scale,” she reminded attendees that an AI model never operates in isolation. It works within a system of people and institutions, the perspective CISE has championed for 25 years.
When Help Gets in the Way

That systems-level perspective carried into the first technical talk, where Harvard University’s Finale Doshi-Velez explained that making AI interpretable requires more than simply revealing selected parts of a model. Her ICU work produced compact policies doctors could inspect. But in a study with psychiatrists, the explanation they liked best, because it was quick to glance at, also caused the biggest drop in accuracy when the AI was wrong. “Stopping the bad stuff isn’t going to be enough to get us to the good stuff that we really need,” she said, arguing that tools must invite engagement.
Hamsa Bastani of the University of Pennsylvania carried this theme into the classroom, finding that AI tools that make tasks too easy can undermine learning. Students who practiced with an unrestricted ChatGPT tutor scored 17% worse on a later exam without AI, and chess players given a “just tell me the next move” button improved less than those who received system-regulated advice. “You have to struggle through it, but you can figure it out,” Bastani said. “That’s where learning really happens.” Good design, she argues, can support that struggle rather than remove it.
Testing the Tests

If explanations can mislead people, the benchmarks used to judge models mislead too. Boston University’s Venkatesh Saligrama, ENG (ECE, SE, CS), opened with a provocative result: on AI-written visual reasoning questions, eight of 15 models scored 86–95%, beating human participants. Game over? Not quite.
On human-written questions, model accuracy collapsed (one model fell from 86.5% to 56%) while human accuracy slipped only five or six points. The models’ edge, it turned out, came from the test rather than from stronger visual reasoning: AI-written questions often probe what a caption says rather than what is in the image. “Human level is a claim about a model and the set of questions,” Saligrama said. That matters well beyond one benchmark. When an AI system is trusted with decisions about who gets treated or what gets flagged, a headline claiming that it matches or beats humans means little until someone checks what it was tested on.
“Human level is a claim about a model and the set of questions.”

Dylan Hadfield-Menell of MIT found similar cracks in safety evaluations. Across fine-tuned models, different medical safety benchmarks correlated at a median of only 0.23, and one judge even marked an empty response as unsafe. If the tests disagree this much, he argued, it is hard to know whether a model is safe or just looks safe.
That is the problem with post-training alignment, which he described as an attempt to “put lipstick on a pig”: it polishes a model’s behavior after the fact without changing what the model learned. His proposed fix is to build values in from the start. “Alignment pretraining is the future of the field,” he said.

Once a system is deployed, the question becomes how to keep watching it. Vasilis Syrgkanis of Stanford argued that safeguards need both randomized auditing and smarter monitoring. His team found that agents can carry out a planted malicious plan without ever stating it, and that even strong monitors missed roughly a third of such attacks. Monitors, he said, should “read for function and not just for phrasing.”

Tijana Zrnic of Stanford then addressed a related practical question: when an AI system makes far more decisions than humans can review, which ones should be audited? Her team’s selective auditing method uses a confidence score to decide which decisions go to a human, with a small random spot-check rate even for high-confidence ones so that hidden failure patterns aren’t missed.
How much auditing is saved, she noted, depends on how reliable the AI is to begin with and on how informative its confidence score is. Getting that score is the catch: with a black-box model, researchers must infer confidence from whatever they can observe, and knowing what to observe, she said, remains the biggest open question.
The Next Generation

The workshop also put students’ work in the spotlight. During the Poster Showcase Pitch, graduate students from several universities presented research that brought the day’s questions into real-world contexts. One project explored how small language models could help public health teams report outbreaks, where an AI mistake could delay a critical response. Another examined how often chatbots direct people experiencing mental health distress to professional support, testing whether AI responds safely when it matters most. Together, the presentations underscored that the future of trustworthy AI will be shaped not only by established researchers, but also by the students entering the field.
Is the Tool Really Helping?

The afternoon panel, moderated by Jinglong Zhao, turned to a question every organization adopting AI eventually faces: how can you tell whether an AI tool is actually helping? MIT’s Dean Eckles warned that interference between coworkers and shared data can distort experiments. Ron Berman of the University of Pennsylvania urged teams to decide, before running a trial, how big a result would have to be to change their decision.
Harvard University’s Michael Lingzhi Li described an experiment that injected errors into AI output, which helped reviewers catch real mistakes 40 to 50% more often. He also cautioned that speeding up a step that isn’t the bottleneck won’t improve organizational outcomes. In other words, time saved on one task is not proof that an AI tool is helping. What counts is whether the organization’s overall results improve.
The panelists agreed on the need to watch for overreliance and deskilling, and to talk directly to employees rather than trust dashboards alone. Asked where students should focus, Berman’s advice was simple: “differentiate, don’t compete.”
Looking Ahead

In the day’s closing discussion, BU Vision Talks underscored that working effectively with AI requires clear principles and meaningful collaboration. Yannis Paschalidis, Distinguished Professor, ENG (ECE, SE, BME) and Director, Hariri Institute, described how domain knowledge and human oversight shaped three projects: the Beacon outbreak-monitoring pipelines, protein language models for antibody design, and the NSF-funded programmable cloud labs. The takeaway is that bigger is not always better. Small, carefully trained models, he noted, can match or beat much larger general-purpose ones, and the true impact comes not from the model alone but from “AI agents and humans working in concert.”
Chris Dellarocas, Richard C. Shipley Professor, Questrom (IS) and Former Associate Provost of Digital Learning & Innovation, argued that domain expertise is necessary but not sufficient for working well with AI. Curricula should therefore also teach students to frame ill-defined problems, scrutinize AI output, and recognize the limits of their own knowledge, since agentic systems can erode the very expertise they depend on. That makes education part of the safety system: if the people overseeing AI lose the judgment to catch its mistakes, no benchmark or audit can make up for it. Keeping people sharp is as much a part of trustworthy AI as testing the models themselves.
“It seems to work” is not enough.

From the opening remarks to the final discussion, one message resonated throughout the day: trustworthy AI demands more than a system that appears to work. It must be rigorously tested through meaningful benchmarks and audits, protected from explanations and shortcuts that can erode human judgment, and designed with the people who rely on it in mind. Meeting these challenges takes more than one discipline. It reflects the very premise behind CISE’s founding: the most complex problems cross disciplinary boundaries, and evaluating AI is no exception.