Attachment #117767 for bug #159897

View | Details | Raw Unified | Return to bug 159897 | Differences between
Collapse All | Expand All




    <sect2>
      <title>Synopsis</title>

      <para>High availability is one of the main requirements in serious
	business applications and highly-available storage is a key
	component in such environments.  Highly Available STorage, or
	<acronym>HAST<remark role="acronym">Highly Available

	  drives.</para>
	</listitem>
	<listitem>
	  <para>File system agnostic; works with any file
	    system supported by &os;.</para>
	</listitem>
	<listitem>

	total.</para>
      </note>

      <para>Since <acronym>HAST</acronym> works in a
	primary-secondary configuration, it allows only one of the
	cluster nodes to be active at any given time.  The
	<literal>primary</literal> node, also called

      </itemizedlist>

      <para><acronym>HAST</acronym> operates synchronously on a block
	level, making it transparent to file systems and
	applications.  <acronym>HAST</acronym> provides regular GEOM
	providers in <filename class="directory">/dev/hast/</filename>
	directory for use by other tools or applications, thus there is

	For stripped-down systems, make sure this module is available.
	Alternatively, it is possible to build
	<literal>GEOM_GATE</literal> support into the kernel
	statically, by adding this line to the custom kernel
	configuration file:</para>

      <programlisting>options	GEOM_GATE</programlisting>

	  class="directory">/dev/hast/</filename>) will be called
	<filename><replaceable>test</replaceable></filename>.</para>

      <para>Configuration of <acronym>HAST</acronym> is done
	in the <filename>/etc/hast.conf</filename> file.  This file
	should be the same on both nodes.  The simplest configuration
	possible is:</para>

      <programlisting>resource test {
	on hasta {

	  alternatively in the local <acronym>DNS</acronym>.</para>
      </tip>

      <para>Now that the configuration exists on both nodes,
	the <acronym>HAST</acronym> pool can be created.  Run these
	commands on both nodes to place the initial metadata
	onto the local disk, and start the &man.hastd.8; daemon:</para>

      <screen>&prompt.root; <userinput>hastctl create test</userinput>

	  available.</para>
      </note>

      <para>A HAST node's role (<literal>primary</literal> or
        <literal>secondary</literal>) is selected by an administrator
        or other
        software like <application>Heartbeat</application> using the
	&man.hastctl.8; utility.  Move to the primary node
	(<literal><replaceable>hasta</replaceable></literal>) and
	issue this command:</para>

      <screen>&prompt.root; <userinput>hastctl role primary test</userinput></screen>

      <para>Similarly, run this command on the secondary node
	(<literal><replaceable>hastb</replaceable></literal>):</para>

      <screen>&prompt.root; <userinput>hastctl role secondary test</userinput></screen>

      <caution>
	<para>When the nodes are unable to
	  communicate with each other, and both are configured as
	  primary nodes, the condition is called
	  <literal>split-brain</literal>.  To troubleshoot
	  this situation, follow the steps described in <xref
	  linkend="disks-hast-sb">.</para>
      </caution>

      <para>Verify the result with the
	&man.hastctl.8; utility on each node:</para>

      <screen>&prompt.root; <userinput>hastctl status test</userinput></screen>

      <para>The important text is the <literal>status</literal> line,
	which should say <literal>complete</literal>
	on each of the nodes.  If it says <literal>degraded</literal>,
	something went wrong.  At this point, the synchronization
	between the nodes has already started.  The synchronization
	completes when <command>hastctl status</command>
	reports 0 bytes of <literal>dirty</literal> extents.</para>


      <para>The next step is to create a filesystem on the
	<devicename>/dev/hast/<replaceable>test</replaceable></devicename>
	GEOM provider and mount it.  This must be done on the
	<literal>primary</literal> node, as
	<filename>/dev/hast/<replaceable>test</replaceable></filename>
	appears only on the <literal>primary</literal> node.
	Creating the filesystem can take a few minutes, depending on the
	size of the hard drive:</para>

      <screen>&prompt.root; <userinput>newfs -U /dev/hast/test</userinput>
&prompt.root; <userinput>mkdir /hast/test</userinput>


      <para>Once the <acronym>HAST</acronym> framework is configured
	properly, the final step is to make sure that
	<acronym>HAST</acronym> is started automatically during the system
	boot.  Add this line to
	<filename>/etc/rc.conf</filename>:</para>

      <programlisting>hastd_enable="YES"</programlisting>


	<title>Failover Configuration</title>

	<para>The goal of this example is to build a robust storage
	  system which is resistant to the failure of any given node.
	  The scenario is that a
	  <literal>primary</literal> node of the cluster fails.  If
	  this happens, the <literal>secondary</literal> node is there to
	  take over seamlessly, check and mount the file system, and
	  continue to work without missing a single bit of data.</para>

	<para>To accomplish this task, another &os; feature provides
	  utilize another feature available under &os; which provides
	  for automatic failover on the IP layer &mdash;
	  <acronym>CARP</acronym>.  <acronym>CARP</acronym> (Common Address
	  Redundancy Protocol) allows multiple hosts
	  on the same network segment to share an IP address.  Set up
 	  <acronym>CARP</acronym> on both nodes of the cluster according
	  to the documentation available in <xref linkend="carp">.
	  After setup, each node will have its own
	  <devicename>carp0</devicename> interface with a shared IP
	  address <replaceable>172.16.0.254</replaceable>.
	  The primary <acronym>HAST</acronym> node of the
	  cluster must be the master <acronym>CARP</acronym>
	  node.</para>

	<para>The <acronym>HAST</acronym> pool created in the previous


	<para>In the event of <acronym>CARP</acronym> interfaces going
	  up or down, the &os; operating system generates a &man.devd.8;
	  event, making it possible to watch for the state changes
	  on the <acronym>CARP</acronym> interfaces.  A state change on
	  the <acronym>CARP</acronym> interface is an indication that
	  one of the nodes failed or came back online.  These state change
	  events make it possible to run a script which will
	  automatically handle the HAST failover.</para>

	<para>To be able to catch state changes on the
	  <acronym>CARP</acronym> interfaces, add this
	  configuration to
	  <filename>/etc/devd.conf</filename> on each node:</para>

	<programlisting>notify 30 {
	match "system" "IFNET";

	action "/usr/local/sbin/carp-hast-switch slave";
};</programlisting>

	<para>Restart &man.devd.8; on both nodes to put the new configuration
	  into effect:</para>

	<screen>&prompt.root; <userinput>/etc/rc.d/devd restart</userinput></screen>

	<para>When the <devicename>carp0</devicename>
	  interface goes up or down (i.e. the interface state changes),
	  the system generates a notification, allowing the &man.devd.8;
	  subsystem to run an arbitrary script, in this case

	  &man.devd.8; configuration, please consult the
	  &man.devd.conf.5; manual page.</para>

	<para>An example of such a script could be:</para>

<programlisting>#!/bin/sh


	;;
esac</programlisting>

	<para>In a nutshell, the script takes these actions when a node
	  becomes <literal>master</literal> /
	  <literal>primary</literal>:</para>

	<itemizedlist>
	  <listitem>
	    <para>Promotes the <acronym>HAST</acronym> pools to
	      primary on a given node.</para>
	  </listitem>
	  <listitem>

	      <acronym>HAST</acronym> pool.</para>
	  </listitem>
	  <listitem>
	    <para>Mounts the pools at an appropriate place.</para>
	  </listitem>
	</itemizedlist>



	<caution>
	  <para>Keep in mind that this is just an example script which
	    should serve as a proof of concept.  It does not
	    handle all the possible scenarios and can be extended or
	    altered in any way, for example it can start/stop required
	    services, etc.</para>
	</caution>

	<tip>
	  <para>For this example, we used a standard UFS
	    file system.  To reduce the time needed for
	    recovery, a journal-enabled UFS or ZFS file system can
	    be used.</para>
	</tip>

      <sect3>
	<title>General Troubleshooting Tips</title>

	<para><acronym>HAST</acronym> should generally work
	  without issues.  However, as with any other software
	  product, there may be times when it does not work as
	  supposed.  The sources of the problems may be different, but
	  the rule of thumb is to ensure that the time is synchronized
	  between all nodes of the cluster.</para>

	<para>When troubleshooting <acronym>HAST</acronym> problems,
	  the debugging level of &man.hastd.8; should be increased
	  by starting the
	  &man.hastd.8; daemon with the <literal>-d</literal>
	  argument.  Note that this argument may be specified
	  multiple times to further increase the debugging level.  A
	  lot of useful information may be obtained this way.  Consider
	  also using the <literal>-F</literal>
	  argument, which starts the &man.hastd.8; daemon in the
	  foreground.</para>
     </sect3>

      <sect3 id="disks-hast-sb">
	<title>Recovering from the Split-brain Condition</title>

	<para><literal>Split-brain</literal> is when the nodes of the
	  cluster are unable to communicate with each other, and both
	  are configured as primary.  This is a dangerous
	  <literal>split-brain</literal>.  This is a dangerous
	  condition because it allows both nodes to make incompatible
	  changes to the data.  This problem must be corrected
	  manually by the system administrator.</para>

	<para>The administrator must
	  decide which node has more important changes (or merge them
	  manually) and let <acronym>HAST</acronym> perform
	  full synchronization of the node which has the broken
	  data.  To do this, issue these commands on the node
	  which needs to be resynchronized:</para>

        <screen>&prompt.root; <userinput>hastctl role init &lt;resource&gt;</userinput>

Return to bug 159897