Seminars:

Monday 6th March

Guidance Helps Where Scale Doesn't In Language Modeling (Ofir Press) | 12PM

Monday 13th March

Data Structures for Fast Systems (Alex Conway)

Events:

Stanford Postdoc Lab Opening

CIERA Conference | March 13 -17th

CEH Seminar: Dr. Phebe Vayanos | March 13th 11AM
Dow - 2023 BEST Symposium | April 30th

News

CS Seminars

Monday / CS Seminar
March 6th / 12:00 PM
Mudd 3514

Title: Guidance Helps Where Scale Doesn't In Language Modeling
Speaker: Ofir Press

Abstract:
Language models (LMs) are at the core of almost all state of the art natural language processing systems on almost every benchmark. Recent papers, such as Brown et al. 2020 and Hoffmann et al. 2022 have shown that scaling up the size of these models leads to better results. But is scaling all we need in order to improve language models?

In this talk I argue that the answer is no, by presenting three studies that show properties of LMs that are not improved with scale. In addition, I will show how to tackle these issues without actually increasing the size on disk, memory usage, or runtime of the LM. In each case, I accomplish it by adding a new kind of guidance to the model.

In Press & Wolf 2017 we showed that the decoding mechanism in LMs contains word representations, and that in models of different sizes, the decoder word representations are of lower quality than the ones in the encoder. We then show that by using the same representations twice (in both the encoder and the decoder) we improve LM performance while decreasing its size.

Memory constraints imply that LMs have to be trained on limited segments of text. For example, GPT-3 (Brown et al. 2020) was trained on text segments that are 4,096 tokens long. Can these models summarize text sequences that are longer than the ones they observed at training? Can they make code predictions for code files that are longer than the ones they were shown during training? In Press et al. 2021 we show that existing LMs cannot process text segments that are longer than the ones they were trained on. We present a new method (ALiBi) that allows LMs to efficiently consume sequences that are longer than the ones they observed at training. ALiBi achieves this by guiding the LM to pay less attention to words that are further away.

Finally, in Press et al. 2022 we show that LMs are able to reason over facts observed during training to answer novel questions that they have never previously seen. But in about 40% of cases, they are not able to accomplish basic reasoning over facts that they are able to recall, and this does not improve with scale. We show that by adding guidance to the way we prompt LMs, by having them ask and answer sub-questions before answering the main complex question, we are able to substantially improve their reasoning capabilities.

These methods have been integrated in many state-of-the-art language and translation models, including OpenAI's GPT, Google's BERT, BigScience's BLOOM and Microsoft's, Meta's and Amazon's translation models.

Biography:
Ofir Press is a PhD candidate (ABD) at the Paul G. Allen School for Computer Science & Engineering at the University of Washington, where he is advised by Noah Smith. During his PhD he spent two years as a visiting researcher at Facebook AI Research Labs on Luke Zettlemoyer’s team where he mainly worked with Mike Lewis. Prior to that, in the summer of 2019 he interned at Facebook AI Research with Omer Levy. Towards the end of my PhD he spent half a year as a visiting researcher at MosaicML on Jonathan Frankle’s team. Before starting his PhD he completed his Bachelor’s and Master’s degrees in Computer Science at Tel Aviv University (where he was advised by Lior Wolf and also worked with Jonathan Berant). Between his Bachelor’s and Master’s degrees he was a software developer for a year.

//

Monday / CS Seminar

March 13th / 12:00 PM

Mudd 3514

Title: Data Structures for Fast Systems

Speaker: Alex Conway

Abstract:

In this talk, I'll show how algorithms can be used to solve decades-old problems in systems design. I'll present an algorithmic approach to co-designing TLB hardware and the paging mechanism to increase TLB reach without the fragmentation issues incurred by huge pages. Along the way, I'll introduce a new hash-table design that overcomes existing tradeoffs, and achieves better performance than state-of-the-art hash tables both in theory and in practice. Key to these results are "tiny pointers," an algorithmic technique for compressing pointers.

Biography:
Alex Conway is a senior researcher at VMware. He received his PhD from Rutgers, where he was advised by Martín Farach-Colton. His work has primarily focused on randomized data structures and their use in storage systems, and covers the full research stack, from theory to systems to product. He is the co-creator and research lead of SplinterDB, an enterprise-grade key-value store deployed in VMware products.

Upcoming CS Seminars:

Wednesday 8th March: Yujia Jin

Monday 13th March: Gokul Subramanian Ravi

Monday 27th March: Milijana Surbatovich

Monday 3rd April: Jennifer Sun

Other Events

The pay rate is $16/hour and meals will be provided.  Even if you’re only available for a half day, your help will be greatly appreciated!  You must currently be in the Northwestern Payroll system (note: all graduate students already are; undergraduates who have student temp or work-study positions also are).   Responsibilities may include: moderating Zoom rooms for remote attendees, signing folks in at tables in the front, general crowd control during meal times, etc.

If you have availability during this week to help out on site, please contact Madeline Wilson (madeline.wilson@northwestern.edu) with your availability each day from 8am – 5:30pm by Tuesday, March 7.

If you are a grad student, please fill out the Permission to Work form and notify your advisor. https://www.northwestern.edu/tgs/about/for-staff/funding/graduate-student-permission-to-work-request.html

Once you express interest we will follow up with required paperwork. Please reach out if you have any questions.

This year’s BEST symposium will be held in person in Midland, MI on July 24th – 27th , 2023. The symposium is primarily intended to introduce Black, Latinx, and Native American U.S. doctoral and postdoctoral scientists to the wide range of rewarding careers in industrial research, and in particular, the many opportunities with one of the world’s largest and leading materials science companies, Dow.  This conference, developed jointly by our minority scientists and Ph.D. recruiting team, demonstrates our commitment to a diverse work force.

This opportunity is for graduate students and post-doctoral scientists. Applicants must be pursuing degrees in: Chemistry, Chemical Engineering, Mechanical Engineering, Materials Science, Physics, or other closely related fields and should have a doctorate degree or expect to receive one by December 2024.

Additional information may be found at their website. All applications are due by April 30th, 2023.

Participants in the conference may be considered for future employment at Dow. However, participation neither obligates the student to apply for employment, nor guarantees future consideration for employment by Dow. For those wishing to learn more about opportunities, please visit our careers page.

For further information, please contact the symposium chair Karena Lekich.

News

Northwestern Department of Computer Science

Mudd Hall, 2233 Tech Drive, Third Floor, Evanston, Illinois, 60208

Unsubscribe