ERDM: When «incorrect» behavior is a logical response to the wrong environment

ERDM demonstrates why an AI agent can learn correctly but act incorrectly when its rewards, choices, and environment conflict. A behavior may seem irrational if we examine it in isolation from the environment that produced it. But if we consider the pressures, the available options, and the system’s reward expectations, the same pattern may prove consistent with what it has «learned» to protect itself. This is the central idea behind the Evolutionarily Recurrent Decision Model (ERDM), a new computational framework for reinforcement learning proposed by Andrew Hu.

This paper examines how evolutionary mismatch, limited rationality, and the satisficing can give rise to adaptive and maladaptive strategies. Complex patterns, such as avoidance, aggression, healthy friendliness, or learned helplessness, are not introduced as predefined scenarios. They emerge from repeated simple decisions within a stochastic environment.

For professionals who design AI agents, automations, or decision-making systems, the practical takeaway is clear: when a system exhibits undesirable behavior, the problem may not lie solely in the model. It may lie in the reward landscape, the information it perceives, the options it is allowed to choose, or the sequence of events. It is the same type of problem we encounter when the AI finds unexpected paths to a goal.

What Does ERDM Actually Model?

ERDM is a single-agent reinforcement learning system operating in a stochastic environment. It was not designed to find the optimal policy for a well-defined task. It was designed to study what happens when an agent must operate in situations where its goals conflict and no single action is always correct.

The environment includes three categories of recurring problems: threat, pursuit of prey or a target, and opportunity for alliance. These serve as abstractions of evolutionarily recurring challenges. The agent observes its internal and external state, selects an action, accepts the consequences, and updates its value estimates.

The study links this framework to the theory of evolutionary mismatch: mechanisms that evolved in one environment may produce adverse effects in another. The ERDM does not prove that this theory explains human mental disorders. It provides a simulation environment where hypotheses about environment, motivation, and behavior can be made specific and testable without conducting actual experiments on humans.

This distinction is also important for practical applications. The value of the study does not lie in the fact that it provides a ready-made algorithm for customer support or e-commerce. Rather, it demonstrates in a controlled manner why the agent’s environment should be considered part of the system and not merely a backdrop. The same principle is also analyzed in world models in which the agent and the environment must be evaluated as a single system.

How Rewards Become Conflicting Motivators

The agent learns through three competing reward dimensions. The first concerns the subsistence or physical state, which the paper denotes as L. The second is the defensive social value, DB, which is linked to strategies of dominance and threat neutralization. The third is the social value of care, NB, which is linked to prestige and to providing benefits to others.

An action can strengthen one dimension and weaken the others. Fighting off a threat, for example, may improve the agent’s defensive position but may also strain its physical condition and its ability to support others. Because the consequences depend on the state and environmental parameters change stochastically, there is no single policy that maximizes everything.

This forces the system to do satisficing: to choose solutions that are adequate given current constraints, not theoretically perfect. For a business AI workflow, the analogy is direct but must remain just that—an analogy. If an agent simultaneously optimizes speed, cost, quality, and compliance, a choice that appears poor on one metric may actually be the best available balance based on the signals we’ve provided.

This distinction has practical value. A business agent may «get stuck» because every available option violates a guardrail or because no option meets the minimum utility threshold. In the first case, a safe escalation or the ability to reverse the decision may be needed. In the second, a better-defined objective or more tools may be needed. The same apparent failure may require a different fix.

Learning, Inertia, and Risk Threshold

Each action has its own independent value network. The network takes the state as input and produces the expected reward for that specific action. The architecture described in this paper has two hidden layers, with 128 and 64 units, respectively, and one output layer. Selection is performed using an epsilon-greedy policy, so that the agent sometimes exploits what it already considers the best option and sometimes explores.

The episode ends when the physical condition or the primary dimension of social survival reaches zero. Rewards are scaled according to available resources: the same loss carries greater weight when the agent is already in a vulnerable state. The state is divided into distinct zones, ranging from “dire” to “stable,” with greater sensitivity to low resource levels.

This explains why an action does not have a fixed meaning. An action that is acceptable when resources are abundant can become catastrophic when the agent is close to its sustainability threshold. For production AI, accordingly, thresholds and escalation rules must take context into account: confidence, financial risk, data sensitivity, impact on the customer, and the possibility of human intervention.

State-aware guidance is not simply a more detailed prompt. It needs to adjust the allowed actions and thresholds when the state changes. That is why it is valuable to link reward design with state-dependent AI agent control and not just based on a general policy.

How the adversity test was set up

For validation, the study modeled adverse childhood experiences as three levels of threat intensity. Low adversity used threat intensities of 20–40 on a scale of 0–100, moderate adversity ranged from 40 to 60, and high adversity from 80 to 100. The frequency of the three types of environments was kept constant so that the primary difference would be the intensity of the adverse events.

The main experiments consisted of 200 independent simulations with 1,500 training steps for DB-dominant agents, NB-dominant agents, and random agents across the three conditions. The random agents did not exhibit learned helplessness or apathy in the action space, because these behaviors require value estimates.

The study does not represent childhood experience in all its human complexity. It uses controlled abstractions to examine whether known patterns can emerge from the combination of rewards, risk, and learning. This is computational validation, not clinical validation, and does not serve as a basis for diagnosis or treatment recommendations.

Constant loss, different policies

One of the most useful results is the distance between the convergence of the learning loss and the convergence of the behavior. For DB agents, the loss decreased from 92% to 97% depending on the condition, while for NB agents, the reduction exceeded 99% under all conditions. The late-window slopes of the loss and reward did not differ from zero, indicating that training had stabilized.

Nevertheless, the variety of actions did not disappear. The DB agents retained 34%–44% of their initial entropy, and the NB agents retained 34%–49%. The system was therefore able to maintain a stable learning process without all agents converging on the same strategy. Under high adversity, the correlation between late-window loss and behavioral entropy became stronger, indicating that learning stabilization and behavioral consistency were more closely linked in that environment.

This is crucial for evaluating AI agents. A smooth loss curve or a good aggregate score does not prove uniform or desirable behavior. We need to examine the dispersion of trajectories, minority failures, and the stability of policies across different sequences of events. The validation of reinforcement learning policies It should focus on how the system performs in critical situations, not just on whether an average metric has improved.

What Changed in the Face of Great Adversity

Among DB-dominant agents, positive expectancy remained at approximately 85% under conditions without ACE and with moderate adversity, but decreased to 67% under high adversity. Learned helplessness, which was absent in the two milder conditions, emerged with a mean value of 15.2%. Negative expectancy increased from approximately 15% to 33%.

The result was not simply a general increase in aggression. Misdirected aggression decreased from 39% to 29.4%, and relational aggression decreased from 30.8% to 24.7%. The study’s interpretation is that, when a threat crosses the risk threshold, inaction resulting from learned helplessness can supplant previous dominance strategies.

The NB-dominant agents exhibited a different profile. Healthy friendliness remained the dominant behavior, reaching approximately 47% under high adversity, while learned helplessness was absent. This finding demonstrates how much a difference in reward priority can alter the emergent repertoire, without complex, integrated scripts having been encoded in advance.

Why Did Resistant and Vulnerable Subgroups Emerge?

Additional analysis of 1,000 DB-dominant runs under high adversity revealed two latent subgroups. The Gaussian mixture model strongly favored a two-component solution and used a threshold of 2.04% learned univariability. The 296 runs were classified as resilient, with a mean value of 0.43%, and the 704 as vulnerable, with a mean value of 20%.

The environmental parameters were the same. What varied was the stochastic sequence of early events, which gave rise to different reward histories. The resilient agents exhibited higher protective behavior—31.64% versus 7.92%—and a positive expected value of 82.65% versus 62.35%.

Healthy friendliness was strongly negatively correlated with learned helplessness in the simulation: Pearson r = -0.91 and R² = 0.83. In contrast, help-seeking vulnerability did not show a significant relationship. This does not mean that seeking help does not protect people. In this specific model, DB agents undervalue it because it is perceived as a cost to their defensive social position.

The operational analogy is that the same configuration does not guarantee the same trajectories. The sequence of tools, retries, exceptions, and human interventions can alter an agent’s subsequent state. This is one reason why the Training AI agents for changing environments It requires multiple sequences, not just a single "happy path.".

From the paper to corporate reward design

ERDM is not a marketing, customer support, or e-commerce model. However, its architecture sheds light on a general problem in agentic design. Teams often set a goal—more conversions, faster service, lower costs—and treat side effects as unforeseen bugs. An agent, however, may simply be consistently following the incentives it was given.

A customer support agent who is primarily rewarded for short handling times may end conversations before the problem is resolved. A recommendation system that focuses solely on immediate clicks may sacrifice trust or variety. A marketing agent who measures only leads may generate low-quality volume. These are illustrative examples of the principle and not cases tested in the study.

Best practice is to identify conflicting goals before deployment. Which metrics compete with each other? What behavior would maximize each metric in an undesirable way? What costs are hidden in the model? What safe action is missing? The discussion about the sparse reward and guidance through hints, critics, and teachers It also shows that the quality of the feedback determines what the system can learn.

The NIST AI RMF treats AI risk as a function of probability and impact within a specific context and warns that metrics can be oversimplified or subject to gaming. This aligns with the practical lesson of ERDM: an organization’s incentives and boundaries are part of security. It is not enough to demand «good behavior» from the model if the operational environment rewards something else.

Why Averages Hide Failure Modes

The distinction between resilient and vulnerable agents shows that an aggregate result can mask two entirely different categories of behavior. If the evaluation had tracked only the average learned helplessness score, it would not have clearly revealed the bimodal structure of the 1,000 runs.

For production AI, this translates to a need for cohort analysis, tail-risk tests, and replaying different event sequences. An overall success rate is not enough. We need to know which users, which intents, which resource levels, and which tool sequences lead to failure modes. The time series can be just as important as the individual inputs.

It is equally important to distinguish between technical stability and desired behavior. A policy can be stable and problematic at the same time. Dashboards must include behavioral diversity, refusal patterns, escalation quality, rollback rate, and undesirable strategies—not just latency and task completion. This is the core of behavioral testing for AI agents.

The analysis must also link each failure to a cause. One failure may be because the agent did not have the right tool, another because it received an incorrect state signal, and yet another because the reward reinforced a shortcut. This logic is related to systems where the AI learns from the cause of failure and not just based on a binary measure of success.

Pre-deployment testing framework

A team can apply the logic of the study without copying the same model. The goal is to transform the reward landscape, state transitions, and exceptions into verifiable artifacts. Each step must be linked to a specific use case and to an observable decision.

The rollout requires clear checkpoints. An offline benchmark is no substitute for shadow mode, adversarial scenarios, rollback, and human supervision. The AI agents with constraints They are reliable when their autonomy is based on the quality of their judgment and the magnitude of the potential impact.

Supervision must have the actual ability to intervene. If an agent views every option as excessively negative, it needs a safe escalation path and a better action space—not pressure to continue. If no action meets the reward threshold, the objective may need to be redesigned. Trust is built when context, data, and human oversight are associated with executable rules.

The scope of the study and the main conclusion

The paper states clear limitations. First, the agent is assumed to perceive the situation correctly and without bias, whereas human cognition and production AI systems may be subject to noise, distortions, or incorrect calibration. Second, the environment is reduced to three states and a few conserved actions. It does not fully account for empathy, failed attempts, or specific community values.

Third, this is a single-agent framework; therefore, it does not represent group dynamics, cultural transmission, or social evolution among multiple active agents. Fourth, there is no accompanying experimental validation study in humans. The results remain computational, and the recommendations for interventions are hypotheses to be tested in the future.

These limitations do not invalidate the model; they define its proper scope of application. We must not confuse a simulation with a diagnosis, nor a correlation within the ERDM with a treatment recommendation. The value lies in the fact that mechanistic assumptions about the environment, rewards, and behavior become visible and testable.

For businesses, the key takeaway is simple: don’t evaluate an AI agent solely based on whether it «learned» or achieved an average target. Consider what it has learned to value, what risks it perceives, what options it has, what it fails to see, and how a sequence of events can turn a locally rational decision into an undesirable business outcome.