AI Weekly: China’s massive multimodal model highlights AI research gap

This week, researchers at the Beijing Academy of Artificial Intelligence (BAAI) announced the release of Wu Dao 2.0, a multimodal AI model capable of generating text indiscernible from human-crafted prose — and more. Containing 1.75 trillion parameters, the parts of the machine learning model learned from historical training data, Wu Dao 2.0 is 10 times larger than OpenAI’s 175-billion-parameter GPT- 3.

Wu Dao 2.0 is the latest example of what OpenAI policy director Jack Clark calls model diffusion, or multiple state and private actors developing GPT-3-style AI models. For example, Russia and France are training smaller-scale systems via Sberbank and LightOn’s PAGnol, while Korea’s Naver Labs is investing in the recently created HyperCLOVA. Clark notes that because these models reflect and magnify the data they’re trained on, different countries care about how their own cultures are represented in the models. The Wu Dao 2.0 announcement, then, is part of a general trend of nations asserting their own AI capabilities via training frontier models like GPT-3.

Wu Dao 2.0, which arrived three months after version 1.0’s March debut, is built on an open source system akin to Google’s Mixture of Experts, dubbed FastMoE. Mixture of Experts, a paradigm first proposed in the ’90s, keeps models specialized in different tasks within a larger model using a “gating network.” BAAI says Wu Dao 2.0 was trained with 4.9 terabytes of Chinese and English images and text both on clusters of supercomputers and conventional GPUs, giving it more flexibility than Google’s system because FastMoE doesn’t require proprietary hardware.

Wu Dao 2.0’s mulitmodal design affords it a range of skills, including the ability to perform natural language processing, text generation, image recognition, and image generation tasks. It can write essays, poems, and couplets in traditional Chinese, as well as captioning images and creating nearly photorealistic artwork, given natural language descriptions. According to Engadget, Wu Dao 2.0 can also power “virtual idols” and predict the 3D structures of proteins, like DeepMind’s AlphaFold.

“The way to artificial general intelligence is big models and big computer,” BAAI chair Dr. Zhang Hongjiang said in a statement. “What we are building is a power plant for the future of AI. With mega data, mega computing power, and mega models, we can transform data to fuel the AI applications of the future.”

AI nationalism

Wu Dao 2.0’s release comes during a surge in tech nationalism globally, particularly in China and parts of the Eurozone. Last November, China imposed new rules around tech exports, with the country’s Ministry of Commerce adding 23 items to its restricted list. Following Nvidia’s announcement that it intends to acquire U.K.-based chipmaker Arm, the majority of U.K.-area IT experts said the government should intervene to protect the country’s tech sector, according to a survey from the industry’s professional body (The Chartered Institute for IT).

Former U.S. chief technology officer Michael Kratsios, among others, has suggested state adversaries are pursuing uses of AI technologies that “aren’t in alignment with American values.” In February, the White House said it would bump non-defense-related AI investment to $2 billion annually by 2022, while U.S. President Joe Biden has proposed an increase in the amount of federal R&D spending to $300 billion over four years. And a U.S. Senate panel last month approved the Endless Frontier Act, pending legislation that would authorize more than $110 billion for basic and advanced technological research over five years.

But U.S. superiority in AI is an increasingly dim prospect. France recently took the wraps off a $1.69 billion (€1.5 billion) initiative aimed at transforming the country into a “global leader” in AI research and training. In 2018, South Korea unveiled a multiyear, $1.95 billion (KRW 2.2 trillion) effort to strengthen its R&D in AI, with the goal of establishing six AI-focused graduate schools by 2022 and training 5,000 AI specialists. And China, whose AI Innovation Action Plan for Colleges and Universities called for the establishment of 50 new AI institutions in 2020, is expected to leapfrog past the European Union within the next several years if current trends continue.

BAAI is funded by the Beijing government, which put 340 million yuan ($53.3 million) into the academy in 2018 and 2019 alone. A Beijing official pledged to continue support in a 2019 speech.

In March, former Google CEO Eric Schmidt urged lawmakers to ramp up funding in the AI space to prevent China from becoming the biggest player in the global AI market. Schmidt suggested doubling the nation’s budget for R&D in AI each year until it hits $32 billion in 2026. Citing the U.S. National Security Commission on Artificial Intelligence, Schmidt also said lawmakers need to incentivize public-private partnerships to develop AI applications across government agencies.

“The government is not today prepared for this new technology,” Schmidt told CNN’s Fareed Zakaria, noting that the use of AI to produce and spread harmful information poses a “threat to democracy” and could ultimately be used as a weapon of war. “We believe this is a national emergency and a threat to our nation unless we get our act together with respect to focusing on AI in the federal government and international security.”

So how might the U.S. make up for lost ground, despite the many challenges ahead? Last July, the President’s Council of Advisors on Science and Technology (PCAST) released a report outlining what it believes must happen for the U.S. to advance “industries of the future,” including AI. PCAST recommended driving opportunities for AI education and training, in part by securing pledges to scale investments for training and education of the U.S. workforce in AI; developing AI curricula and performance metrics at K-12 through postgraduate levels and for certificate and professional programs; creating incentives, recruitment, and retention programs for AI faculty at universities; and increasing National Science Foundation and Department of Education investments in AI educators, scientists, and technologists at all levels.

Last August, in a step toward these goals, the White House established 12 new research institutes focused on AI and quantum information science. But the release of Wu Dao 2.0 highlights the work that must be done before the U.S. can close the AI gap with other world superpowers.

For AI coverage, send news tips to Kyle Wiggers — and be sure to subscribe to the AI Weekly newsletter and bookmark our AI channel, The Machine.

Thanks for reading,

Kyle Wiggers

AI Staff Writer

Source Link