# `Imperial`
[🔗](https://pushin.eu/zteln/imperial)

A replicated state machine.

You implement the state machine as callbacks. Imperial replicates commands
to the servers of a cluster, one per node, and applies them in the same
order on each.

## Defining a state machine

    defmodule KV do
      use Imperial

      def start_link(opts), do: Imperial.start_link(__MODULE__, opts)

      @impl Imperial
      def handle_init(_opts), do: {:ok, %{}}

      @impl Imperial
      def handle_command({:put, key, value}, _meta, state),
        do: {:reply, :ok, Map.put(state, key, value)}

      def handle_command({:get, key}, _meta, state),
        do: {:reply, Map.get(state, key), state}
    end

`use Imperial` defines a `child_spec/1` and default implementations of
`c:handle_snapshot/1` and `c:handle_restore/2`. Override the last two to let
the log be shortened. See `c:handle_init/1` and `c:handle_command/3` for a
state machine that stores its state durably.

## Starting a server

    children = [{KV, data_dir: "/var/lib/kv", nodes: [:"kv@host2", :"kv@host3"]}]

Options:

  * `:data_dir` - required, the directory the log is stored in
  * `:nodes` - required, the other nodes of the cluster, `[]` for a cluster
    of one. Only read the first time the server starts: after that the
    members are taken from the log
  * `:name` - the name the server is registered under, the module by
    default. It must be the same on every node
  * `:snapshot_threshold` - the number of applied commands after which a
    snapshot is taken, `1_000_000` by default

All other options are passed to `c:handle_init/1`.

## Sending commands

    {:ok, :ok, leader} = Imperial.command(KV, {:put, :a, 1})
    {:ok, 1, _leader} = Imperial.command(leader, {:get, :a})
    {:ok, 1, _server} = Imperial.command(KV, {:get, :a}, local?: true)

A command may be sent to any server, which redirects it to the leader. The
third element of the result is the server that replied: send the next command
there to avoid the redirect.

With `local?: true` the command is not replicated. The server that receives
it runs it on its own state and discards the new state, so use it for reads
that may be out of date.

Options are `:local?`, `:timeout` (30 seconds by default) and `:redirects`
(the number of servers to try, 5 by default). The errors are
`{:error, reason}`, where `reason` is `:no_leader`, `:not_leader`,
`:too_many_redirects`, or the reason the call exited, such as `:timeout`. A
command that returned an error may still have been applied.

## Changing the members

    {:ok, :ok, _leader} = Imperial.add_member(KV, :"kv@host4")
    {:ok, :ok, _leader} = Imperial.remove_member(KV, :"kv@host2")

Members are node names. Start the server on a new node, with the current
members as `:nodes`, before it is added. One member changes at a time: the
additional errors are `:not_ready` (the leader is newly elected, try again),
`:change_in_progress` and `:last_member`.

# `meta`

```elixir
@type meta() :: %{index: integer(), term: integer()}
```

Describes the log entry a command comes from: its `index` in the log and the
`term` it was appended in.

# `handle_command`

```elixir
@callback handle_command(command :: any(), meta :: meta(), state :: any()) ::
  {:reply, reply :: any(), state :: any()}
```

Applies a committed command, in log order, exactly once per log index.

`meta` holds the log index and term of the command. A durable state machine
must store the index atomically with the effect of the command and report it
from `c:handle_init/1`. The effect must be durable when this callback returns.

A local query is not a log entry. Its `meta` holds the index of the last
applied entry and the current term of the node.

## Failures

A failure the caller should know about belongs in `reply`, for example
`{:reply, {:error, :not_found}, state}`. The command then counts as applied.

Raising, or returning anything other than `{:reply, reply, state}`, means the
command could not be applied. The server stops, and applies the command
again when it is restarted. No server skips a command, since the others may
have applied it. A command that fails on every server therefore keeps the
cluster stopped until the cause is removed.

A local query that fails does not stop the server: the caller gets
`{:error, reason}`.

# `handle_init`

```elixir
@callback handle_init(opts :: keyword()) ::
  {:ok, state :: any()} | {:ok, state :: any(), last_applied :: integer()}
```

Returns the initial state machine state.

A durable state machine also returns the index of the last log entry it has
applied, so that entries up to that index are not applied again after a
restart. `{:ok, state}` means nothing has been applied durably.

# `handle_restore`

```elixir
@callback handle_restore(snapshot :: any(), state :: any()) :: state :: any()
```

Replaces the contents of the state machine with those of `snapshot`.

Called on startup when the state machine is behind the stored snapshot, and
when the leader sends a snapshot because the log entries needed to catch up
no longer exist.

The server cannot continue without its state. If this callback fails, the
server stops.

# `handle_snapshot`

```elixir
@callback handle_snapshot(state :: any()) :: {:snapshot, any()} | :no_snapshot
```

Returns a snapshot holding everything the state machine has applied, or
`:no_snapshot` to keep the log as it is.

A snapshot only shortens the log. If this callback fails, no snapshot is
taken and the server continues.

# `add_member`

Adds the node `member` to the cluster.

The server on `member` must already be started, with the current members as
its `:nodes`. Returns `{:ok, :ok, leader}` once a majority has stored the
change, or straight away when `member` is already a member.

Takes the `:timeout` and `:redirects` options of `command/3` and returns the
same errors, as well as:

  * `{:error, :not_ready}` - the leader is newly elected and has not
    committed a command yet. Try again
  * `{:error, :change_in_progress}` - an earlier change is not committed yet

# `command`

Sends `cmd` to the state machine and returns `{:ok, reply, server}`.

`server` is a name, a `{name, node}` tuple or a pid, and may be any server
of the cluster. `reply` is the reply of `c:handle_command/3` on the leader,
given once a majority has stored the command. The returned `server` is the
one that replied.

## Options

  * `:local?` - when `true`, `server` runs the command on its own state
    without replicating it and discards the new state. The reply may be out
    of date. `false` by default
  * `:timeout` - milliseconds to wait for each server, `30_000` by default
  * `:redirects` - the number of servers to try, `5` by default

## Errors

  * `{:error, :no_leader}` - `server` does not know of a leader
  * `{:error, :not_leader}` - the leader lost its role before the command
    was committed
  * `{:error, :too_many_redirects}` - no leader was reached within
    `:redirects` servers
  * `{:error, reason}` - the call exited with `reason`, such as `:timeout`
    or `:noproc`

After `:not_leader` or `:timeout` the command may still be applied.

# `remove_member`

Removes the node `member` from the cluster.

Returns `{:ok, :ok, leader}` once a majority has stored the change, or
straight away when `member` is not a member. The leader can be removed: it
steps down when the change is committed. The server on `member` is not
stopped.

Takes the same options and returns the same errors as `add_member/3`, as
well as `{:error, :last_member}` when `member` is the only member.

# `start_link`

Starts a server for the state machine `mod`, registered locally under `:name`.

The options are listed in the module documentation. `:timeout`, `:spawn_opt`,
`:hibernate_after` and `:debug` are passed to `:gen_statem.start_link/4`.

Returns `{:error, :invalid_opts}` when `:data_dir` or `:nodes` is missing.

# `stop`

Stops the server and closes its log.

The server stays a member of the cluster. Use `remove_member/3` first to
take it out for good.

---

*Consult [api-reference.md](api-reference.md) for complete listing*
