"Making Machine Learning Predictably Reliable"

Abstract

Despite ML models' impressive performance, training and deploying them is currently a somewhat messy endeavor. But does it have to be? In this talk, I overview my work on making ML “predictably reliable”---enabling developers to know when their models will work, when they will fail, and why. 

To begin, we use a case study of adversarial inputs to show that human intuition can be a poor predictor of how ML models operate. Motivated by this, we present a line of work that aims to develop a precise understanding of the ML pipeline, combining statistical tools with large-scale experiments to characterize the role of each individual design choice: from how to collect data, to what dataset to train on, to what learning algorithm to use.

Biography

Andrew Ilyas is a PhD student at MIT, advised by Constantinos Daskalakis and Aleksander Madry. His main interest is in reliable machine learning, where he seeks to understand the effects of the individual design choices involved in building ML models. He was previously supported by an Open Philanthropy AI Fellowship.

Research Interests/Area

Machine Learning

"Rethinking Data Use in Large Language Models"

Abstract

Large language models (LMs) such as ChatGPT have revolutionized natural language processing and artificial intelligence more broadly. In this talk, I will discuss my research on understanding and advancing these models, centered around how they use the very large text corpora they are trained on. First, I will describe our efforts to understand how these models learn to perform new tasks after training, demonstrating that their so-called in context learning capabilities are almost entirely determined by what they learn from the training data, challenging a widely held belief. Next, I will introduce a new class of LMs that fundamentally rethink how models use their training data. These new models—nonparametric LMs—include not only learned parameters but also massive text corpora, from which they retrieve information for improved accuracy and flexibility. I will describe my work establishing the foundations for such models, showing they are more performant with fewer parameters and can easily stay up-to-date. I will also discuss how they open up new avenues for responsible data use, e.g., by segregating permissive and copyrighted text and using them differently. Finally, I will envision the next generation of LMs we should build, focusing on efficient scaling, better information-seeking, and responsible data use.

Biography

Sewon Min is a Ph.D. candidate in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Her research focuses on language models (LMs): studying the science of LMs, and designing new model classes and learning methods that make LMs more performant and flexible. She also studies LMs in information-seeking, legal, and privacy contexts. She is a co-organizer of multiple tutorials and workshops, including most recently at ACL 2023 on Retrieval-based Language Models and Applications and upcoming at ICLR 2024 on Mathematical and Empirical Understanding of Foundation Models. She won a paper award at ACL 2023 and a J.P. Morgan Fellowship.

Research Interests/Area

Natural language processing; Machine learning

"Integrating Theory and Practice in Complex Privacy-Preserving Data Analysis"

Abstract

With growing concerns about large-scale data collection and surveillance, the development of privacy-preserving tools can help alleviate public fears about the misuse of personal data. The field of differential privacy (DP) offers powerful data analysis tools that provide worst-case privacy guarantees. However, the transition of differential privacy from academic research to practice introduces many new technical challenges, which range from fundamental theory to large-scale deployment and broader privacy concerns. 

In this talk, I will discuss my research efforts in tackling two major challenges: 1 How to reason about more complex privacy accounting? 2. How to identify and protect privacy beyond DP’s individual-oriented focus? 

To answer the first question, I will show the optimal composition theorems for concurrently composing multiple interactive mechanisms, even in scenarios where we adaptively select privacy-loss parameters and create new mechanisms. These results offer strong theoretical foundations for enabling full adaptivity and interactivity in DP systems. Towards the second challenge, I will talk about a broader non-individual privacy for protecting sensitive global properties of a dataset, e.g., proprietary information or IP contained in the data. I will introduce an attack to identify the dataset-level privacy vulnerabilities; and a solution, consisting of a theoretical framework to capture this global property privacy and mechanisms to achieve that. I will conclude the talk with my future directions.

Biography

Wanrong Zhang is an NSF Computing Innovation Fellow in the Theory of Computing group at Harvard John A. Paulson School of Engineering and Applied Sciences. She is also a member of the Harvard Privacy Tools/OpenDP project. Her primary focus is to address new challenges introduced by real-world deployments of differential privacy. Before joining Harvard, she received her Ph.D. from Georgia Institute of Technology. She is a recipient of a Best Paper Award at CCS (2023), a Computing Innovation Fellowship from CCC/CRA/NSF. She was selected as a rising star in EECS in 2022 and a rising star in Data Science in 2021. 

Research Interests/Area

Data privacy, responsible computing

CS Department Events

CSPAC Workshop Series

Other Events & Opportunities

AI@NU Graduate Student Group Speaker Aimee Rinehart: How AI Is Transforming Journalism

The Balance Between Model Performance and Interpretability in Demand Forecasting

Abstract: In the rapidly evolving landscape of demand forecasting, businesses increasingly push the forecast accuracy limits by looking to new “State of the Art” models to predict market trends and consumer behavior. However, the complexity of these models often comes at the cost of interpretability, posing significant challenges for data scientists and decision-makers. This talk aims to address the critical balance between leveraging the predictive power of complex, 'black box' models and maintaining the interpretability necessary for strategic business applications.  Our discussion will begin by exploring the latest advancements in time series forecasting and discussing their strengths and limitations. We will delve into the intricacies of models that, while powerful, often function as black boxes, making it difficult to understand the 'why' behind their predictions. This opacity can hinder trust and adoption in business environments where understanding the reasoning behind forecasts is as crucial as the forecasts themselves.

To bridge this gap, we introduce SHAP (SHapley Additive exPlanations) values, a cutting-edge approach in model interpretability. SHAP values, grounded in cooperative game theory, provide a robust framework to decipher the contribution of each feature to the prediction of a complex model.

The talk will include real-world examples of how many businesses today decide on how to deploy machine learning into their business practices.

Bio: Jeff Tackes is Sr. Manager of Data Science for Kraft Heinz, headquartered in Chicago, IL. With over 10 years of industry experience, Jeff has developed a deep understanding of the intricacies of demand forecasting and has successfully built best-in-class demand forecasting systems for leading fortune 500 companies.

Jeff is known for his data-driven approach and innovative strategies that optimize forecasting models and improve business outcomes. He has led cross-functional teams in designing and implementing demand forecasting systems that have resulted in significant improvements in forecast accuracy, inventory optimization, and customer satisfaction. Jeff's expertise in statistical modeling, machine learning, and advanced analytics has enabled him to develop cutting-edge forecasting methodologies that have consistently outperformed industry standards. Jeff's strategic vision and ability to align demand forecasting with business goals have made him a trusted advisor to senior executives and a sought-after expert in the field.

Towards Human-centered AI: How to Generate Useful Explanations for Human-AI Decision Making

CS News

NU News