Fudan University MOSS 16B

  • Announced Feb/2023.
  • Open-sourced Apr/2023.
  • MOSS is English-first, limited Chinese. Fudan said it: ‘trained on 300 billion English words and only 30 billion Chinese words.’
  • Less params than ChatGPT (Fudan: ‘tens of billions of parameters’ and ‘parameters of MOSS are an order of magnitude smaller than that of ChatGPT,’ MOSS=16B vs ChatGPT=175B).
  • Chinchilla-aligned. 1 token≈0.7 words. 1 word≈1.3 tokens. 330B words * 1.3 = 430B tokens trained to 16B parameters would be around 27:1 (compared to GPT-3’s 1.7:1 and Chinchilla’s 20:1).
  • Dataset may be unlike Chinese models such as Wudao and PanGu Alpha, more like Tsinghua’s GLM-130B which prioritised English data from The Pile.
  • Aligned with Anthropic’s HHH values: helpful, harmless, and honest.
  • Named after the character of MOSS from The Wandering Earth series, 流浪地球 liúlàng dìqiú (wiki, IMDB). The character of MOSS is similar to AUTO from WALL-E or HAL 9000 from 2001: A Space Odyssey.
  • Major bandwidth issues at launch ‘MOSS instantly went viral on Chinese social media, generating tens of millions of hits on China’s Twitter-like Weibo’ – via Reuters (21/Feb/2023).
  • Public interface will be: https://moss.fastnlp.top/
  • More info: https://txsun1997.github.io/blogs/moss.html

Compare the MOSS chatbot with:

  1. OpenAI ChatGPT (Nov/2022).
  2. DeepMind Sparrow (Sep/2022).
  3. Anthropic Claude (Dec/2022).
  4. Bing Chat (Sydney) (Feb/2023).
  5. Google Bard (Feb/2023).

Download

MOSS examples

 

Get The Memo

by Dr Alan D. Thompson · Be inside the lightning-fast AI revolution.
Informs research at Apple, Google, Microsoft · Bestseller in 142 countries.
Artificial intelligence that matters, as it happens, in plain English.
Get The Memo.

Alan D. Thompson is a world expert in artificial intelligence, advising everyone from Apple to the US Government on integrated AI. Throughout Mensa International’s history, both Isaac Asimov and Alan held leadership roles, each exploring the frontier between human and artificial minds. His landmark analysis of post-2020 AI—from his widely-cited Models Table to his regular intelligence briefing The Memo—has shaped how governments and Fortune 500s approach artificial intelligence. With popular tools like the Declaration on AI Consciousness, and the ASI checklist, Alan continues to illuminate humanity’s AI evolution. Technical highlights.

This page last updated: 9/Feb/2024. https://lifearchitect.ai/moss/↑