About Me

Greetings! I am a 4th year PhD student at the Language Technology Institute of Carnegie Mellon University. I am very fortunate to be advised by Prof. Yonatan Bisk. I received my Bachelor's degree in Computer Science & Mathematics with first class honors from Hong Kong University of Science and Technology.

Research Keywords: Algorithmic Reasoning, Mechanistic Interpretability, Generalization

  1. Mathematically understand machine reasoning
    • Role-filler binding with learned roles in neural networks.
    • Algorithm induction from input-output pairs.
  2. Mechanistically understand machine reasoning
    • How Transformers form circuits that accomplish tasks sequentially or in parallel
    • How Transformers (approximately) implement and execute memory.
    • How Transformers treat functions differently from handling primitive concepts.
✨ I'm actively looking for collaborators on mechanistically understanding Transformers' expressivity and their limits. ✨


Publications

Yingshan Chang and Yonatan Bisk. "Language Models Need Inductive Biases to Count Inductively" arXiv:2405.20131 (under review)
Jimin Sun, So Yeon Min, Yingshan Chang, Yonatan Bisk "Tools Fail: Detecting Silent Errors in Faulty Tools" EMNLP 2024
Shaurya Dewan, Rushikesh Zawar, Prakanshul Saxena, Yingshan Chang, Andrew Luo, Yonatan Bisk. "DiffusionPID: Interpreting Diffusion via Partial Information Decomposition" Neurips 2024
Yingshan Chang, Yasi Zhang, Zhiyuan Fang, Yingnian Wu, Yonatan Bisk, Feng Gao. "Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation" European Conference on Computer Vision (ECCV) 2024.
Akter, Syeda Nahida, Sangwu Lee, Yingshan Chang, Yonatan Bisk and Eric Nyberg. “VISREAS: Complex Visual Reasoning with Unanswerable Questions” In Findings of the Association for Computational Linguistics: ACL 2024.
Liangke Gui, Yingshan Chang, Qiuyuan Huang, Subhojit Som, Alexander G Hauptmann, Jianfeng Gao and Yonatan Bisk. “Training Vision-Language Transformers from Captions” In Transactions on Machine Learning Research, pp. 2835-8856. 2023.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. “Webqa: Multihop and Multimodal QA” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16495-16504. 2022. Oral.
Yingshan Chang, and Yonatan Bisk. “WebQA: A Multimodal Multihop NeurIPS Challenge” In NeurIPS 2021 Competitions and Demonstrations Track, pp. 232-245. PMLR, 2022.

Current Research

1. What architectural innovations address the dispersion effect of softmax in self-attention, making it friendlier to length extrapolation?
2. Can we enable extrapolation by letting the model "adjust its reference frame" dynamically? Taking reference from adaptive contrast in the retina.

Previous Research

Language Models Need Inductive Biases to Count Inductively

  • Read more

2024

Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation

  • Read more

2023-2024

Efficient Visual Grounding via Patch Affinities

  • Read more

2022

WebQA: Multihop and Multimodal QA

  • Read more

2020-2022

Low-Light Video Enhancement Using Deep Learning

  • Read more

2019-2020

Event-to-Sentence Using BERT in Automated Story Generation

  • Read more

Summer 2019

A Blockchain and Smart Contract Application

  • Read more

Summer 2019

Selected Course Projects

Neuro-Concepts

CMU 85707  Spring 2022

  • Read more

Conlanging

CMU 11823  Spring 2022

  • Read more

Sociolinguistics

CMU 11724  Fall 2021

  • Read more

Internet Computing

HKUST COMP4021  Fall 2019

  • Read more

Machine Learning

Georgia Tech CS4641  Spring 2019

  • Read more

Information Visualization

Georgia Tech CS4460  Spring 2019

  • Read more

Language Modelling

HKUST COMP4901K  Fall 2018

  • Read more

Customer Revenue Prediction with Spark

HKUST COMP4651  Fall 2018

  • Read more

Education

Carnegie Mellon University

PhD in Language Technologies  2022 -

Carnegie Mellon University

MS in Language Technologies  2020 - 2022

Hong Kong University of Science and Technology

BS in Computer Science & Mathematics  2016 - 2020

Georgia Institute of Technology

Exchange  Spring 2019

Peking University

AEARU Summer Camp  Summer 2018

Honors

Carnegie Mellon University Research Fellowship   2020 - 2022
Academic Achievement Medal   Hong Kong University of Science and Technology 2020
Bachelor Degree First Class Honor   Hong Kong University of Science and Technology 2020
Dean’s List   Hong Kong University of Science and Technology2016 - 2020
University’s Scholarship Scheme for Continuing Undergraduate Students   2016-2019
The Cheng Foundation Scholarships for Chinese Mainland Undergraduate Students  2018 - 2019
The Hong Kong Electric Co.Ltd. Scholarship  2017 - 2018
Mingxi Youth Award Scheme  2017 - 2018

Hobbies

Community Services

Date Activity & Organizer Duration
Apr 2018  Volunteering @Fung Yuen Butterfly Reserve - Tai Po Environment Association 5 hrs
Jan 2018  Teaching & Community Project, Galle - Sri Lanka Diriya Sahana Foundation 30 hrs
Nov 2017  Volunteering @Peak to Fong - Hong Kong Dog Rescue 4.5 hrs
Nov 2017  HKUST Bread Run - Feeding Hong Kong 3 hrs
Sep 2017  Blood Donation Promotion Campaign - Hong Kong Red Cross Blood Transfusion Service 3 hrs
Sep 2017  Playright Game Day - Playright Children’s Play Association 7 hrs
Apr 2017 The Salvation Army Flag Day - The Salvation Army 3 hrs