Andy's CV
The PDF format is available here.
Education
- B.Eng. in Data Science, Lanzhou University(LZU), 09/2018 - 07/2022
- GPA: 92.8/100, Ranking: 1/192
- Honors:
- China National Scholarship (Top 0.1% Across Nation)
- Merit Student in Gansu Province (Top 1% Across Province)
- M.Comp. in Artificial Intelligence, National University of Singapore(NUS), 08/2022 - Now
- GPA: 4.42/5,
- Advisor: Prof. Yang You (Director of HPC-AI Lab)
Research Experience
Master Dissertation in HPC-AI Lab, National University of Singapore, 05/2023 – Now
Keywords: Data-Centric methods, Mixture-of-Experts Model
- Working on OpenMoE project (second author) with Fuzhao Xue, which is the first open-source, decoder-only MoE language model. We released the code and checkpoint and got 1k+ stars on GitHub.
- Investigated publicly available pre-training corpus (English, Chinese, multilingual, code, etc), preprocessing methods and tokenization techniques. Do experiments comparing tokenizers. Prepare the pre-training, SFT and evaluation datasets in TFDS format.
- Worked on the Pytorch implementation of OpenMoE with the ColossalAI team. Performing model evaluations and contributed to the paper writing.
Work Experience
Artificial Intelligence Engineer Intern, Beijing, HPC-AI Tech, 07/2023 – 11/2023
Keywords: Data-Centric methods, Long Context LLM, Retrieval Augmented Generation
- Extended the LLaMA’s vocabulary for Chinese text and participated in the data cleaning and preparation process in the Colossal-LLaMA-2 project (achieved 18k+ downloads on Huggingface so far).
- Context length extrapolation: Investigated common context extrapolation techniques (e.g. PI, NTK, LongLLaMA , LongLoRA, etc.), training corpus with long data and long-text evaluation methods. Working on constructing Chinese long text training data and doing multi-GPU training to extrapolate Colossal-LLaMA-2.
- Participated in the ColossalQA project, a retrieval-based QA framework based on Langchain.
- Involved in the writing of the book ‘Practical Large AI Models’, edited by Professor Yang You.