"Understanding Language Models through Discovery and by Design"

Abstract

Whereas we understand technologies like airplanes or microprocessors well enough to fix them when they break, our tools for fixing modern language models are coarse. This is because, despite language models' increasing ubiquity and utility, we understand little about how they work. In this talk, I will present two lines of research for developing a deep, actionable understanding of language models that allows us to discover how they work, and fix them when they fail. In the first line, I will present structural probing methods for discovering the learned structure of language models, finding evidence that models learn structure like linguistic syntax. In the second line, I will show how we can understand complex models by design: through the new Backpack neural architecture, which gives us precise tools for fixing models.

 
 Biography

John is a PhD student in Computer Science at Stanford University, working with Percy Liang and Christopher Manning on discovering the learned structure of neural language models, and designing them to be more understandable, diagnosable, and fixable. He was an NSF Graduate Fellow, and received a B.S.E in Computer and Information Science from the University of Pennsylvania. John has received an Outstanding Paper Award at ACL 2023, a Best Paper Runner Up at EMNLP 2019, an Honorable Mention for Best Paper at the Robustness of Few-Shot Learning in Foundation Models Workshop (R0-FoMo)@NeurIPS 2023, and an Outstanding Paper Award at the Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackBoxNLP)@EMNLP 2020.

Research Interests/Area

Natural Language Processing

" “Parallelism First”: New Foundations for Provably Efficient and Safe Parallel Programming "

Abstract

In recent decades, architectural advances have brought parallelism to the mainstream. However, due to a variety of performance and correctness issues in practice, developing parallel software remains difficult—even for experts. This difficulty is exacerbated by the fact that mainstream languages are designed for sequential execution by default, and do not provide strong guarantees on safety and performance for parallel programs.

To address the difficulty of parallel programming, my research puts parallelism first: we assume parallel execution by default, and rethink fundamental abstractions from the ground up to provide guarantees on both safety and performance. In this talk, I highlight two contributions in particular: (1) disentanglement, which enables provably efficient parallel garbage collection, and (2) automatic parallelism management, which provides a solution to the long-standing granularity control problem. All of this work is implemented in MaPLe: an open-source compiler and run-time system that we built from the ground up for provably efficient and safe parallel programming. MaPLe is currently being used at Carnegie Mellon University to help teach parallel programming to over 500 students every year, and our empirical results show that MaPLe can compete with the performance of hand-optimized code written in languages such as C/C++. To conclude, I discuss my future research plans, working towards making it simpler and safer to develop high-performance parallel software.

 
Biography

Sam Westrick is a post-doc at Carnegie Mellon University, working with Umut Acar on parallel programming languages, compilers and run-time systems, and parallel algorithms. He received his PhD from Carnegie Mellon in 2022, and he is the lead developer of MaPLe, a high-level language for efficient and safe parallel programming. His work has been recognized with multiple distinguished paper awards, and in 2023 he received the ACM SIGPLAN Dissertation Award for his work on Efficient and Scalable Parallel Functional Programming Through Disentanglement.

Research Interests/Area

parallel programming, programming languages, compilers and run-time systems, parallel algorithms

CS Department Events

CSPAC Workshop Series

Other Events & Opportunities

Northwestern Medicine Healthcare AI Forum

3rd Annual Traditional Spring Pow Wow- Hosted by NAISA

CS News

NU News