China is Overtaking the US in AI Research with its Faster and Stronger Wu Dao 2.0

Know more about the latest intelligent model system

The work behind Wu Dao 2.0, which is dubbed as China's first homegrown super-scale intelligent model system, was led by BAAI Research Academic Vice President and Tsinghua University Professor Tang Jie. He was supported by a team of over 100 AI scientists from Peking University, Tsinghua University, Renmin University of China, the Chinese Academy of Sciences, and other institutions. Wu Dao 2.0 is actually the successor to Wu Dao 1.0, which was unveiled by the BAAI earlier this year. Wu Dao 2.0 truly is China's bigger and better answer to GPT-3. Firstly, unlike GPT-3, Wu Dao 2.0 develops both in Chinese and English with skills acquired by analyzing 4.9 terabytes of images and texts. Wu Dao 2.0 also has partnership agreements with 22 brands including smartphone maker Xiaomi and video app Kuaishou. The Chinese model has been trained on 1.75 trillion parameters, which is nearly 10 times greater than the 175 billion parameters GPT-3 was trained on. Wu Dao 2.0 can also write poems in traditional Chinese styles, answer questions, write essays, and write text for images.

Wu Dao 2.0 has also unveiled Hua Zhibing, the world's first Chinese virtual student. Hua can learn, draw pictures and compose poetry. In the future, she will be able to learn to code. This learning ability of Wu Dao 2.0 is in stark contrast to GPT-3. Other details of how and exactly Wu Dao 2.0 was trained are not available yet, making it difficult to compare it with GPT-3 directly. However, the new language model is a testament to China's AI ambitions and its superb research programs. There is no doubt that AI innovation will increase in the coming years, and many of these innovative developments will help advance many other industries.

One of the AI luminaries and investors, who helped build at least 7 AI-powered unicorns driven by AI, Dr. Kai-Fu Lee, recently gave a talk at the Hong Kong Science and Technology Park where he explained the power of transformers and fine-tuning the massive pre-trained models such as Wu Dao 2.0. These models can be fine-tuned for multiple industries and a large number of applications such as education, finance, law, entertainment, and, most importantly, healthcare and biomedical research. The applications of transformers in biomedical research are likely to yield new discoveries that will benefit humans regardless of where they live. And we sincerely hope that despite the trade wars, the governments will consider collaborating on biomedical research.

Wu Dao 2.0 was trained with FastMoE, a system similar to Google's Mixture of Experts (MoE). The idea is to train different models within a larger model for each modality. A gating system permits the larger model to select which models to consult for each type of task. FastMoE, in contrast with Google's MoE, is open-source and doesn't require specific hardware, which makes it more democratic. It allowed BAAI researchers to solve training bottlenecks preventing models such as GPT-3 from reaching the 1-trillion-parameter milestone. They wrote in BAAI's official WeChat blog that "[FastMoE] is simple to use, flexible, high-performance, and supports large-scale parallel training." The future of large AI systems will certainly pass through these training frameworks.