A replicated state machine.
You implement the state machine as callbacks. Imperial replicates commands to the servers of a cluster, one per node, and applies them in the same order on each.
Defining a state machine
defmodule KV do
use Imperial
def start_link(opts), do: Imperial.start_link(__MODULE__, opts)
@impl Imperial
def handle_init(_opts), do: {:ok, %{}}
@impl Imperial
def handle_command({:put, key, value}, _meta, state),
do: {:reply, :ok, Map.put(state, key, value)}
def handle_command({:get, key}, _meta, state),
do: {:reply, Map.get(state, key), state}
enduse Imperial defines a child_spec/1 and default implementations of
handle_snapshot/1 and handle_restore/2. Override the last two to let
the log be shortened. See handle_init/1 and handle_command/3 for a
state machine that stores its state durably.
Starting a server
children = [{KV, data_dir: "/var/lib/kv", nodes: [:"kv@host2", :"kv@host3"]}]Options:
:data_dir- required, the directory the log is stored in:nodes- required, the other nodes of the cluster,[]for a cluster of one. Only read the first time the server starts: after that the members are taken from the log:name- the name the server is registered under, the module by default. It must be the same on every node:snapshot_threshold- the number of applied commands after which a snapshot is taken,1_000_000by default
All other options are passed to handle_init/1.
Sending commands
{:ok, :ok, leader} = Imperial.command(KV, {:put, :a, 1})
{:ok, 1, _leader} = Imperial.command(leader, {:get, :a})
{:ok, 1, _server} = Imperial.command(KV, {:get, :a}, local?: true)A command may be sent to any server, which redirects it to the leader. The third element of the result is the server that replied: send the next command there to avoid the redirect.
With local?: true the command is not replicated. The server that receives
it runs it on its own state and discards the new state, so use it for reads
that may be out of date.
Options are :local?, :timeout (30 seconds by default) and :redirects
(the number of servers to try, 5 by default). The errors are
{:error, reason}, where reason is :no_leader, :not_leader,
:too_many_redirects, or the reason the call exited, such as :timeout. A
command that returned an error may still have been applied.
Changing the members
{:ok, :ok, _leader} = Imperial.add_member(KV, :"kv@host4")
{:ok, :ok, _leader} = Imperial.remove_member(KV, :"kv@host2")Members are node names. Start the server on a new node, with the current
members as :nodes, before it is added. One member changes at a time: the
additional errors are :not_ready (the leader is newly elected, try again),
:change_in_progress and :last_member.
Summary
Types
Describes the log entry a command comes from: its index in the log and the
term it was appended in.
Callbacks
Applies a committed command, in log order, exactly once per log index.
Returns the initial state machine state.
Replaces the contents of the state machine with those of snapshot.
Returns a snapshot holding everything the state machine has applied, or
:no_snapshot to keep the log as it is.
Functions
Adds the node member to the cluster.
Sends cmd to the state machine and returns {:ok, reply, server}.
Removes the node member from the cluster.
Starts a server for the state machine mod, registered locally under :name.
Stops the server and closes its log.
Types
Callbacks
@callback handle_command(command :: any(), meta :: meta(), state :: any()) :: {:reply, reply :: any(), state :: any()}
Applies a committed command, in log order, exactly once per log index.
meta holds the log index and term of the command. A durable state machine
must store the index atomically with the effect of the command and report it
from handle_init/1. The effect must be durable when this callback returns.
A local query is not a log entry. Its meta holds the index of the last
applied entry and the current term of the node.
Failures
A failure the caller should know about belongs in reply, for example
{:reply, {:error, :not_found}, state}. The command then counts as applied.
Raising, or returning anything other than {:reply, reply, state}, means the
command could not be applied. The server stops, and applies the command
again when it is restarted. No server skips a command, since the others may
have applied it. A command that fails on every server therefore keeps the
cluster stopped until the cause is removed.
A local query that fails does not stop the server: the caller gets
{:error, reason}.
@callback handle_init(opts :: keyword()) :: {:ok, state :: any()} | {:ok, state :: any(), last_applied :: integer()}
Returns the initial state machine state.
A durable state machine also returns the index of the last log entry it has
applied, so that entries up to that index are not applied again after a
restart. {:ok, state} means nothing has been applied durably.
Replaces the contents of the state machine with those of snapshot.
Called on startup when the state machine is behind the stored snapshot, and when the leader sends a snapshot because the log entries needed to catch up no longer exist.
The server cannot continue without its state. If this callback fails, the server stops.
Returns a snapshot holding everything the state machine has applied, or
:no_snapshot to keep the log as it is.
A snapshot only shortens the log. If this callback fails, no snapshot is taken and the server continues.
Functions
Adds the node member to the cluster.
The server on member must already be started, with the current members as
its :nodes. Returns {:ok, :ok, leader} once a majority has stored the
change, or straight away when member is already a member.
Takes the :timeout and :redirects options of command/3 and returns the
same errors, as well as:
{:error, :not_ready}- the leader is newly elected and has not committed a command yet. Try again{:error, :change_in_progress}- an earlier change is not committed yet
Sends cmd to the state machine and returns {:ok, reply, server}.
server is a name, a {name, node} tuple or a pid, and may be any server
of the cluster. reply is the reply of handle_command/3 on the leader,
given once a majority has stored the command. The returned server is the
one that replied.
Options
:local?- whentrue,serverruns the command on its own state without replicating it and discards the new state. The reply may be out of date.falseby default:timeout- milliseconds to wait for each server,30_000by default:redirects- the number of servers to try,5by default
Errors
{:error, :no_leader}-serverdoes not know of a leader{:error, :not_leader}- the leader lost its role before the command was committed{:error, :too_many_redirects}- no leader was reached within:redirectsservers{:error, reason}- the call exited withreason, such as:timeoutor:noproc
After :not_leader or :timeout the command may still be applied.
Removes the node member from the cluster.
Returns {:ok, :ok, leader} once a majority has stored the change, or
straight away when member is not a member. The leader can be removed: it
steps down when the change is committed. The server on member is not
stopped.
Takes the same options and returns the same errors as add_member/3, as
well as {:error, :last_member} when member is the only member.
Starts a server for the state machine mod, registered locally under :name.
The options are listed in the module documentation. :timeout, :spawn_opt,
:hibernate_after and :debug are passed to :gen_statem.start_link/4.
Returns {:error, :invalid_opts} when :data_dir or :nodes is missing.
Stops the server and closes its log.
The server stays a member of the cluster. Use remove_member/3 first to
take it out for good.