Researchers have developed a new artificial intelligence (AI) approach aimed at helping machines learn continuously without forgetting previously acquired skills—a challenge known as “catastrophic forgetting.” This is especially important for AI systems that must adapt to changing environments over time, such as robots operating in dynamic settings or software agents facing evolving tasks. The study introduces the Evolutionary Neuroplastic Agent (ENA), which combines evolutionary search techniques with a specialized memory system to better retain and reuse learned behaviors as conditions shift.
Key Takeaways
- ENA evolves small neural networks without relying on traditional gradient-based training, using evolutionary search methods instead.
- It stores (“archives”) effective policies—sets of actions or strategies—tagged by the environmental conditions in which they performed well.
- ENA uses a Bayesian retrieval method to detect the current environment and select the most suitable previously learned policy to apply.
- In tests on simulated control tasks with changing dynamics, ENA demonstrated strong retention of skills across different scenarios, though it adapted more slowly than some standard reinforcement learning algorithms.
Continual reinforcement learning involves an AI agent learning to make decisions while dealing with environments that change over time. A common problem is that when the agent learns new tasks, it often “forgets” how to perform earlier tasks—a phenomenon called catastrophic forgetting. The new research explores whether creating specialized “experts” for different environmental conditions, discovered through an evolutionary search process, can serve as a reusable memory system. This approach contrasts with the more common gradient-based training methods used in deep learning.
The Evolutionary Neuroplastic Agent (ENA) works by evolving a population of small neural networks—models inspired by the brain’s structure—that represent different policies for acting in an environment. Instead of adjusting these networks through gradient descent (a standard training technique), ENA uses evolutionary search, which mimics natural selection by iteratively selecting and refining better-performing networks. When the environment changes, ENA detects these changes and retrieves the policy best suited to the current conditions from its archive, using Bayesian policy reuse. Bayesian methods provide a statistical way to estimate which policy is likely to perform best based on prior experience.
The researchers tested ENA on two classic reinforcement learning benchmarks with non-stationary dynamics: CartPole and HalfCheetah. CartPole involves balancing a pole on a moving cart, while HalfCheetah is a simulated robot that must learn to run. In the CartPole tests, ENA achieved an average success rate of over 83% across different environment variations and showed more consistent retention of skills compared to some simplified versions of the system. However, ENA adapted more slowly than Proximal Policy Optimization (PPO) on CartPole and achieved lower performance than Soft Actor-Critic (SAC) on HalfCheetah, indicating room for improvement in learning speed and efficiency.
While these results do not prove that ENA outperforms existing continual learning methods, they highlight the potential benefits of combining evolutionary search with environment-aware policy selection to mitigate forgetting. The approach’s reliance on gradient-free optimization distinguishes it from many current methods, although the study did not isolate this factor as the sole reason for improved retention.
Looking ahead, this research opens avenues for building AI systems that better maintain past knowledge when facing changing tasks—a crucial step toward more robust and adaptable autonomous agents. Future work may focus on enhancing ENA’s learning speed and scaling the approach to more complex real-world problems, potentially benefiting fields like robotics, adaptive control systems, and lifelong AI learning.
Based on research published on arXiv by Blas Eugenio Vicco Zarate, Miguel Hernán Vicco.
