Deep reinforcement learning with MarsExplorer

By Bryan L. Lawrence On May 17, 2023

[ad_1]

Exploration of the solar system is accelerating, with unmanned expeditions to Mars and the Moon seeking to uncover details about our cosmic neighbors that will inform scientific advancements and future crewed expeditions. To help missions navigate new worlds, an article published in the journal Electronic introduced a deep reinforcement learning suite, MarsExplorer.

To study: MarsExplorer: Exploration of unknown terrains through deep reinforcement learning and procedurally generated environments. Image Credit: HelenField / Shutterstock.com

Explore new worlds

Scientists have been sending unmanned probes and rovers into the solar system for decades. With a large number of objectives and targets, these unmanned vehicles returned an impressive array of scientific data.

Currently, different spacecraft in orbit around Mars include Perseverance, Tianwen-1, and Hope. Other missions are planned for Mars as well as several that will land on the lunar surface in the coming years. NASA also plans to return astronauts to the Moon over the next decade.

Navigate in uncharted territory

However, there is a critical issue for semi-autonomous unmanned spacecraft: navigating unfamiliar and potentially dangerous terrain. While a human operator can react to unexpected terrain with their senses, an unmanned robotic vehicle cannot rely on human operators, especially at such great distances. They have to use their programming and having the software to adapt to unfamiliar terrain is essential.

The ability to explore and map previously unknown areas is essential for missions using autonomous vehicles. This can impact critical mission parameters (such as fuel depletion) and, ultimately, can be the difference between success and failure of the mission itself. Deep Reinforcement Learning is one area of â€‹â€‹IT that can provide software solutions to this critical challenge.

What is deep reinforcement learning?

Deep reinforcement learning is an area of â€‹â€‹machine learning that combines the principles of deep learning and reinforcement learning. Reinforcement learning helps computers learn by trial and error, and deep learning helps them make decisions using unstructured input data without the need for manual entry and engineering.

Recent advances in hardware and algorithms have accelerated the capabilities of reinforcement learning software, giving it performance levels equal to or better than human performance in tasks such as gaming. Go, robotic manipulation and multi-agent collaboration.

An important step in reinforcement learning has been taken with the development of a framework that standardizes common problems, namely open-AI-gym. This has led to new generations of reinforcement learning frameworks that can be tuned to tackle open AI gym setups.

State coding convention. (a) Graphic environment; (b) State s

MarsExplorer: Using Deep Reinforcement Learning to Explore Unknown Alien Lands

Breakthroughs in deep reinforcement learning have led to their application to trajectory planning and exploration tasks for robots. Since rovers and other unmanned spacecraft must be able to operate autonomously and navigate previously unmapped extraterrestrial areas, deep reinforcement learning frameworks are being developed and explored for this purpose. end.

MarsExplorer is a deep reinforcement learning suite that was developed by a team of Greek scientists. The software trains robots with procedurally generated environments that have various terrain diversities. These allow robots to grasp and understand the underlying structure of terrains and apply it to areas they have never explored. Robots are trained several times in the MarsExplorer environment and have strong generalization capabilities.

In the MarsExplorer environment, real-world robotic platforms can use learned policies. The team evaluated four state-of-the-art reinforcement learning algorithms – Ranbow, PPO, SAC, and A3C – with the system. Human performance in the environment was used to further assess the performance of the algorithms.

The best performing algorithm was found to be PPO, and a follow-up evaluation of this algorithm was conducted with varying degrees of difficulty. In addition, a scalability study was conducted alongside a comparison with non-learning methodologies.

The aim of the research was not to create another realistic simulator but to provide a framework for reinforcement learning methods as well as non-learning methods. This will be used to provide a baseline for exploration and coverage tasks. In reality, the execution time of the different algorithms impacts their suitability for the tasks.

Presentation of the experimental architecture. Image credit: Koutras, D et al., Electronics

The future

Robotic explorers face multiple challenges on other planetary bodies. According to the study, MarsExplorer is the first reinforcement learning framework compatible with open AI and optimized for the exploration of unknown terrains. MarsExplorer is a deep learning system that helps solve this problem.

Commonly used out-of-the-box algorithms can be used in the system, providing a software package for the exploration of extraterrestrial environments without the need to adapt the system to the individual dynamics of the robot in which it is installed. This will help future unmanned missions. explore the solar system by giving robotic missions levels of autonomy that are difficult to achieve with conventional software and hardware.