Imperial behaviour (Imperial v0.1.0)

Copy Markdown View Source

A replicated state machine.

You implement the state machine as callbacks. Imperial replicates commands to the servers of a cluster, one per node, and applies them in the same order on each.

Defining a state machine

defmodule KV do
  use Imperial

  def start_link(opts), do: Imperial.start_link(__MODULE__, opts)

  @impl Imperial
  def handle_init(_opts), do: {:ok, %{}}

  @impl Imperial
  def handle_command({:put, key, value}, _meta, state),
    do: {:reply, :ok, Map.put(state, key, value)}

  def handle_command({:get, key}, _meta, state),
    do: {:reply, Map.get(state, key), state}
end

use Imperial defines a child_spec/1 and default implementations of handle_snapshot/1 and handle_restore/2. Override the last two to let the log be shortened. See handle_init/1 and handle_command/3 for a state machine that stores its state durably.

Starting a server

children = [{KV, data_dir: "/var/lib/kv", nodes: [:"kv@host2", :"kv@host3"]}]

Options:

  • :data_dir - required, the directory the log is stored in
  • :nodes - required, the other nodes of the cluster, [] for a cluster of one. Only read the first time the server starts: after that the members are taken from the log
  • :name - the name the server is registered under, the module by default. It must be the same on every node
  • :snapshot_threshold - the number of applied commands after which a snapshot is taken, 1_000_000 by default

All other options are passed to handle_init/1.

Sending commands

{:ok, :ok, leader} = Imperial.command(KV, {:put, :a, 1})
{:ok, 1, _leader} = Imperial.command(leader, {:get, :a})
{:ok, 1, _server} = Imperial.command(KV, {:get, :a}, local?: true)

A command may be sent to any server, which redirects it to the leader. The third element of the result is the server that replied: send the next command there to avoid the redirect.

With local?: true the command is not replicated. The server that receives it runs it on its own state and discards the new state, so use it for reads that may be out of date.

Options are :local?, :timeout (30 seconds by default) and :redirects (the number of servers to try, 5 by default). The errors are {:error, reason}, where reason is :no_leader, :not_leader, :too_many_redirects, or the reason the call exited, such as :timeout. A command that returned an error may still have been applied.

Changing the members

{:ok, :ok, _leader} = Imperial.add_member(KV, :"kv@host4")
{:ok, :ok, _leader} = Imperial.remove_member(KV, :"kv@host2")

Members are node names. Start the server on a new node, with the current members as :nodes, before it is added. One member changes at a time: the additional errors are :not_ready (the leader is newly elected, try again), :change_in_progress and :last_member.

Summary

Types

Describes the log entry a command comes from: its index in the log and the term it was appended in.

Callbacks

Applies a committed command, in log order, exactly once per log index.

Returns the initial state machine state.

Replaces the contents of the state machine with those of snapshot.

Returns a snapshot holding everything the state machine has applied, or :no_snapshot to keep the log as it is.

Functions

Adds the node member to the cluster.

Sends cmd to the state machine and returns {:ok, reply, server}.

Removes the node member from the cluster.

Starts a server for the state machine mod, registered locally under :name.

Stops the server and closes its log.

Types

meta()

@type meta() :: %{index: integer(), term: integer()}

Describes the log entry a command comes from: its index in the log and the term it was appended in.

Callbacks

handle_command(command, meta, state)

@callback handle_command(command :: any(), meta :: meta(), state :: any()) ::
  {:reply, reply :: any(), state :: any()}

Applies a committed command, in log order, exactly once per log index.

meta holds the log index and term of the command. A durable state machine must store the index atomically with the effect of the command and report it from handle_init/1. The effect must be durable when this callback returns.

A local query is not a log entry. Its meta holds the index of the last applied entry and the current term of the node.

Failures

A failure the caller should know about belongs in reply, for example {:reply, {:error, :not_found}, state}. The command then counts as applied.

Raising, or returning anything other than {:reply, reply, state}, means the command could not be applied. The server stops, and applies the command again when it is restarted. No server skips a command, since the others may have applied it. A command that fails on every server therefore keeps the cluster stopped until the cause is removed.

A local query that fails does not stop the server: the caller gets {:error, reason}.

handle_init(opts)

@callback handle_init(opts :: keyword()) ::
  {:ok, state :: any()} | {:ok, state :: any(), last_applied :: integer()}

Returns the initial state machine state.

A durable state machine also returns the index of the last log entry it has applied, so that entries up to that index are not applied again after a restart. {:ok, state} means nothing has been applied durably.

handle_restore(snapshot, state)

@callback handle_restore(snapshot :: any(), state :: any()) :: state :: any()

Replaces the contents of the state machine with those of snapshot.

Called on startup when the state machine is behind the stored snapshot, and when the leader sends a snapshot because the log entries needed to catch up no longer exist.

The server cannot continue without its state. If this callback fails, the server stops.

handle_snapshot(state)

@callback handle_snapshot(state :: any()) :: {:snapshot, any()} | :no_snapshot

Returns a snapshot holding everything the state machine has applied, or :no_snapshot to keep the log as it is.

A snapshot only shortens the log. If this callback fails, no snapshot is taken and the server continues.

Functions

add_member(server, member, opts \\ [])

Adds the node member to the cluster.

The server on member must already be started, with the current members as its :nodes. Returns {:ok, :ok, leader} once a majority has stored the change, or straight away when member is already a member.

Takes the :timeout and :redirects options of command/3 and returns the same errors, as well as:

  • {:error, :not_ready} - the leader is newly elected and has not committed a command yet. Try again
  • {:error, :change_in_progress} - an earlier change is not committed yet

command(server, cmd, opts \\ [])

Sends cmd to the state machine and returns {:ok, reply, server}.

server is a name, a {name, node} tuple or a pid, and may be any server of the cluster. reply is the reply of handle_command/3 on the leader, given once a majority has stored the command. The returned server is the one that replied.

Options

  • :local? - when true, server runs the command on its own state without replicating it and discards the new state. The reply may be out of date. false by default
  • :timeout - milliseconds to wait for each server, 30_000 by default
  • :redirects - the number of servers to try, 5 by default

Errors

  • {:error, :no_leader} - server does not know of a leader
  • {:error, :not_leader} - the leader lost its role before the command was committed
  • {:error, :too_many_redirects} - no leader was reached within :redirects servers
  • {:error, reason} - the call exited with reason, such as :timeout or :noproc

After :not_leader or :timeout the command may still be applied.

remove_member(server, member, opts \\ [])

Removes the node member from the cluster.

Returns {:ok, :ok, leader} once a majority has stored the change, or straight away when member is not a member. The leader can be removed: it steps down when the change is committed. The server on member is not stopped.

Takes the same options and returns the same errors as add_member/3, as well as {:error, :last_member} when member is the only member.

start_link(mod, opts)

Starts a server for the state machine mod, registered locally under :name.

The options are listed in the module documentation. :timeout, :spawn_opt, :hibernate_after and :debug are passed to :gen_statem.start_link/4.

Returns {:error, :invalid_opts} when :data_dir or :nodes is missing.

stop(server)

Stops the server and closes its log.

The server stays a member of the cluster. Use remove_member/3 first to take it out for good.