How to represent a modular FSM for AI using ECS?

I recommend you the chapter on State-Driven Agent Design in Programming AI by Example, By Matt Buckland, since a general approach of Finite State Machines is shown there. It proposes a singleton generic FiniteStateMachine System, which handles the state updating and transitions of all the agents. The states, by the way they are designed (generally and in the book) are also of the kind of singleton classes, but you could make it Component-wise, so they have their reference to their own agents, so you can handle each agent separately (not applied on my example, but it would be just a matter to add a reference to their Agents, and delete the agent parameter on their methods).

Here is a very generic example, the ECS would be separated by Agents (Entitites), States (Components) and a Finite State Machine per Agent Type (System):

/// Entity
public abstract class FSMAgent<T> : MonoBehaviour where T : MonoBehaviour
{
    public StateComponent<T> currentState, previousState;

    private void Update()
    {
        FiniteStateMachine<T>.UpdateState(this);
    }
}

/// Component
public abstract class StateComponent<T> where T : FSMAgent<T>
{
    /// I put it virtual, since maybe there'd be transition actions that you'd like to avoid in certain states.
    public virtual void Exit(T agent) { /*...*/ }
    public virtual void Enter(T agent) { /*...*/ }
    public virtual void Execute(T agent) { /*...*/ }
}

/// System
public class FiniteStateMachine<T> where T : FSMAgent<T>
{
    public static void ChangeState(T agent, StateComponent<T> state)
    {
        /// Store and execute exit of actual state. Execute new state's entrance.
        if(agent.currentState != null)
        {
            agent.previousState = agent.currentState;
            agent.previousState.Exit(agent);
        }
        agent.currentState = state;
        agent.currentState.Enter(agent);
    }

    public static void UpdateState(T agent)
    {
        /// Execute state and check for conditions and transitions...
        if(agent.currentState != null) agent.currentState.Execute(agent);
    }
}

So you just have to define classes that inherit from the base, I'll take the example with Halo's Agents, since I read "Brutes" on another answer:

public class Grunt : FSMAgent<Grunt> { /*...*/ }
public class Elite: FSMAgent<Elite> { /*...*/ }
public class Jackal : FSMAgent<Jackal> { /*...*/ }

public class GruntIdle : StateComponent<Grunt> { /*...*/ }
public class EliteIdle : StateComponent<Elite> { /*...*/ }
public class JackalIdle : StateComponent<Jackal> { /*...*/ }

/// And so on...

And they just would call the static reference of the Finite State Machine of their own kind.

Hope it helps.

I dont really understand, why you would want several components with their own system. You basically just need one BehaviourComponent, whice either has only data (System works on Components) or also has functions (System calls Components functions), depending on your liking.

Just to make it clear, i got no idea of unity. And in my following explanation i assume you only use components with data.

Lets assume you have 3 types of enemies with equivalent equipment:

• Brutes that charge and attack your Character in Close Combat. They wont block or retreat

• Balanced enemies that approach you carefully, block and attack you mostly the same time. if Hurt, they may retreat

• anxious enemies that wont approach you, mostly block and may flee if a friend gets hurt or even dies

  There three enemy types are basicly the same except for their behavior.

Now, for the sake of it, lets assume they each have three states in their fsm: idle, aware and hurt. On creating an enemy, you set the BehaviourComponentwith behavior pattern. In Idle, the brute may actively search for an enemy, called Search, when he found an enemy, he is Attackingand when hurt he is still Attacking. Each of these Actions you implement as Enums.

enum Action {Search, Attack}

Now every state needs to know, when to transition to another state. You could do something like simple events.

enum Event {FoundCharacter, GotHurt}

So. how to implement this? For every state you have you could implement a struct (if you are using something like c#).

public struct State{
    public Action 
    public Event
}

This would be the most simple case. On initiating your BehaviourComponent you create to states Idle and Aware. For Idle set Action to Search and Event to FoundEnemy, for Aware set Action to Attack and leave Event just empty, as he fights to the death. Your BehaviourComponent would then have a starting state, e.g. Idle and the information to go from one state to another.

Your BehaviourSystem then iterates over the BehaviourComponent, checks if an event occures and in case switches the state, then if follows the set Action.

To make it a bit more difficult, our balanced enemy tries to attack and block when in State of Aware. For that you could set several actions, like in an array or another struct to give them probabilities.

Your anxious guard may also have several states. When switching from Idle he might go to Aware or directly to Run or something else. This would be the same like several different actions.

The best with this approach is, your BehaviourSystem only needs to know what to do on different Actions, as the switch of states is 'sort of' implemented in the BehaviourComponent.

If you want it even more complex, you would need something like a script language, to load the 'BehaviourComponents' for each enemy type and maybe even for every Action, so you dont even have to implement them in the system. This could be loaded at runtime every time you would need it.