Nvidia AI performs Minecraft, wins AI convention award

A paper describing MineDojo, Nvidia’s generalist AI agent that may carry out actions from written prompts in Minecraft, gained an Excellent Datasets and Benchmarks Paper Award on the 2022 NeurIPS (Neural Info Processing Methods) convention, Nvidia revealed on Monday.

To coach the MineDojo framework to play Minecraft, researchers fed it 730,000 Minecraft YouTube movies (with greater than 2.2 billion phrases transcribed), 7,000 scraped webpages from the Minecraft wiki, and 340,000 Reddit posts and 6.6 million Reddit feedback describing Minecraft gameplay.

From this information, the researchers created a customized transformer mannequin known as MineCLIP that associates video clips with particular in-game Minecraft actions. Because of this, somebody can inform a MineDojo agent what to do within the sport utilizing high-level pure language, equivalent to “discover a desert pyramid” or “construct a nether portal and enter it,” and MineDojo will execute the collection of steps essential to make it occur within the sport.

MineDojo goals to create a versatile agent that may generalize discovered actions and apply them to completely different behaviors within the sport. As Nvidia writes, “Whereas researchers have lengthy skilled autonomous AI brokers in video-game environments equivalent to StarCraft, Dota, and Go, these brokers are often specialists in just a few duties. So Nvidia researchers turned to Minecraft, the world’s hottest sport, to develop a scalable coaching framework for a generalist agent—one that may efficiently execute all kinds of open-ended duties.”

The award-winning paper, “MINEDOJO: Constructing Open-Ended Embodied Brokers with Web-Scale Data,” debuted in June. Its authors embody Linxi Fan of Nvidia and Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar of varied educational establishments.

You possibly can see examples of MineDojo in motion on its official web site, and the code for MineDojo and MineCLIP is obtainable on GitHub.