Attachment #39920 for bug #62834

View | Details | Raw Unified | Return to bug 62834
Collapse All | Expand All




    <indexterm><primary>diskless operation</primary></indexterm>

    <para>A FreeBSD machine can boot over the network and operate without a
      local disk, using filesystems mounted from an NFS server.  No
      system modification is necessary, beyond standard configuration
      files.  Such a system is relatively easy to set up because all
      the necessary elements are readily available:</para>
    <itemizedlist>
      <listitem>
	<para>There are at least two possible methods to load the kernel over


    <para>There are many ways to set up diskless workstations.  Many
      elements are involved, and most can be customized to suit local
      taste.  The following will describe variations on the setup of a
      complete system, emphasizing simplicity and compatibility with
      the standard FreeBSD startup scripts.  The system described has
      the following characteristics:</para>

    <itemizedlist>
      <listitem>

	  configuration files overridden by ones specific to diskless
	  operation or, possibly, to the workstation they belong to.</para>
	<para>The parts of the <filename>root</filename> which have to be
	  writable are overlaid with &man.mfs.8; (FreeBSD 4.X) or
	  &man.md.4; (FreeBSD 5.X) filesystems.  Any changes will be lost
	  when the system reboots.</para>
      </listitem>
      <listitem>
	<para>The kernel is transferred and loaded either with
 	  <application>etherboot</application>, or <emphasis>PXE</emphasis>,
      as some situations may mandate the use of either method.</para>
      </listitem>
    </itemizedlist>


	other hosts.</para>
    </caution>

    <para>All the information in this section has been tested
      using FreeBSD releases 4.9-RELEASE-p2 and 5.2-RELEASE.  The text is
      primarily structured for 4.X usage.  Notes have been inserted where
      appropriate to indicate 5.X changes.</para>
    


    <sect2>
      <title>Background information</title>

      <para>Setting up diskless workstations is both relatively
        straightforward and prone to errors.  These are sometimes
        difficult to diagnose for a number of reasons. For example:
        <itemizedlist>
          <listitem>
        <para>Compile time options may determine the possible values
              or syntax of runtime parameters.</para>
          </listitem>
          <listitem>
        <para>Error messages are often cryptic or totally
           absent.</para>
          </listitem>
        </itemizedlist>

      <para>In this context, having some knowledge of the background
        mechanisms involved is very useful to solve the problems that
        may arise.</para>

      <para>Several operations need to be performed for a successful
        bootstrap:</para>
        
        <itemizedlist>
          <listitem>
        <para>The machine needs to obtain initial
            parameters such as its IP address, executable file name,
            server name, root path.  This is done using the DHCP or
            BOOTP protocols. DHCP is a compatible extension of BOOTP,
            and uses the same port numbers and basic packet
            format.</para>
        <para>It is possible to configure a system to use only BOOTP.
            The &man.bootpd.8; server program is included in the
            base FreeBSD system.</para>

        <para>However, DHCP has a number of advantages over BOOTP (nicer
            configuration files, possibility of using PXE, plus many
            others not directly related to diskless operation), and
            we shall describe mainly a DHCP configuration, with
            equivalent exemples using &man.bootpd.8; when possible.
            The sample configuration will use the ISC DHCP software
            package (release 3.0.1.r11 was installed on the test
            server: ).</para>
           </listitem>

          <listitem>
        <para>The machine needs to transfer one or several
            programs to local memory. Either TFTP or NFS are used. The
            choice between TFTP and NFS is a compile time option in
            several places. A common source of error is to specify file
            names for the wrong protocol: TFTP typically transfers all
            files from a single directory on the server, and would
            expect simple file names relative to this directory. NFS
            needs absolute file paths.</para>
          </listitem>

          <listitem>
        <para>The possible intermediate bootstrap programs and the
            kernel need to be initialized and executed. There are
            several important variations in this area:</para>
          <itemizedlist>
            <listitem>
          <para>When using PXE, the machine will load &man.pxeboot.8;,
            which is a modified version of the FreeBSD third stage
            loader. The loader will obtain most parameters necessary
            to system startup, and leave them in the kernel
            environment before transferring control. It is possible to
            use a GENERIC kernel in this case.</para>
            </listitem>

              <listitem>
            <para><application>etherboot</application>, will directly
                load the kernel, with less preparation.  A kernel with
                specific options will need to be built.</para>
              </listitem>
          </itemizedlist>

          <para>PXE and <application>etherboot</application> work
            equally well with 4.X systems. Because 5.X kernels
            normally let the loader do more work for them, PXE is
            preferred for 5.X systems.</para>
          <para>If your BIOS and network
            cards support PXE, you should probably use it.  However,
            it is still possible to start a 5.2 system with
            <application>etherboot</application></para>

          </listitem>

        <listitem>
      <para>Finally, the machine needs to access its filesystems. NFS
       is used in all cases.</para>
        </listitem>
      </itemizedlist>

      <para>See also &man.diskless.8;</para>

    </sect2>

    <sect2>
      <title>Setup Instructions</title>

	  <secondary>booting</secondary>
	</indexterm>

              
	  workstation that retrieves its configuration over the network: BOOTP
	  and DHCP.  They are used at several points in the workstation
	  bootstrap:</para>
	<itemizedlist>
	  <listitem><para><application>etherboot</application> uses
	      DHCP (by default) or BOOTP (needs a configuration option) to
	      find the kernel.  (PXE uses DHCP).</para>
	  </listitem>
	  <listitem><para>The kernel uses BOOTP to locate the NFS
	      root.</para>
	  </listitem>
	</itemizedlist>

	<para>It is possible to configure a system to use only BOOTP.
	  The &man.bootpd.8; server program is included in the
	  base FreeBSD system.</para>

	<para>However, DHCP has a number of advantages over BOOTP (nicer
	  configuration files, possibility of using PXE, plus many others
	  not directly related to diskless operation), and we shall describe
	  both a pure BOOTP, and a BOOTP+DHCP configuration, with an
	  emphasis on the latter, which will use the ISC DHCP software
	  package.</para>

	<sect4>
	  <title>Configuration Using ISC DHCP</title>
	  <indexterm>

	  <para>The <application>isc-dhcp</application> server can answer
	    both BOOTP and DHCP requests.</para>

	  <para>As of release 4.9, <application>isc-dhcp
  	    3.0</application> is not part of the base
	    system.  You will first need to install the
	    <filename role="package">net/isc-dhcp3-server</filename> port or the

	  <para>Once <application>isc-dhcp</application> is installed, it
	    needs a configuration file to run, (normally named
	    <filename>/usr/local/etc/dhcpd.conf</filename>).  Here follows
	    a commented example, where host <literal>margaux</literal>
	    uses <application>etherboot</application> and host
	    <literal>corbieres</literal> uses PXE.</para>

          <programlisting>
          default-lease-time 600;

              hardware ethernet 01:23:45:67:89:ab;
              fixed-address margaux.example.com;
              next-server 192.168.4.4;<co id="co-dhcp-next-server">
              filename "/data/misc/kernel.diskless";<co id="co-dhcp-filename">
              option root-path "192.168.4.4:/data/misc/diskless";<co id="co-dhcp-root-path">
            }
            host corbieres {
              hardware ethernet 00:02:b3:27:62:df;
              fixed-address corbieres.example.com;
              filename "pxeboot";<co id="co-dhcp-filenamePxe">
              next-server 192.168.4.4;
              option root-path "192.168.4.4:/data/misc/diskless";
            }
          }
          </programlisting>


	    </callout>

	    <callout arearefs="co-dhcp-next-server"><para>The
		<literal>next-server</literal> directive designates the TFTP
		or NFS server to use for loading loader or kernel file (the
		default is to use the same host as the DHCP server).</para>
	    </callout>

        <callout arearefs="co-dhcp-filename"><para>The
        <literal>filename</literal> directive defines the file that
        <application>etherboot</application> or PXE will load for the
        next execution step. It must be specified according to the
        transfer method used. <application>etherboot</application> can
        be compiled to use NFS or TFTP. The FreeBSD port
        configures NFS by default.</para>
         </callout>

	    <callout arearefs="co-dhcp-filenamePxe"><para>PXE uses TFTP, 
          which is why a relative file name is used here (this may
          depend on the TFTP 
          server configuration, but would be fairly typical). Also, PXE
          loads <filename>pxeboot</filename>, not the kernel.</para>
	    </callout>

	    <callout arearefs="co-dhcp-root-path"><para>The

	  </indexterm>

	  <para>Here follows an equivalent <command>bootpd</command>
	    configuration (reduced to one client).  This would be found in
	    <filename>/etc/bootptab</filename>.</para>

	  <para>Please note that <application>etherboot</application>

	  of everything.  Else refer to <xref linkend="ports"> for
	  information about ports and packages.</para>

    <para>You can change the <application>etherboot</application>
    configuration (ie: to use TFTP instead of NFS) by editing the
    Config file in the etherboot source directory.</para>

	<para>For our setup, we shall use a boot floppy.  For other methods
	  (PROM, or dos program), please refer to the
	  <application>etherboot</application> documentation.</para>


      </sect3>

      <sect3>
	<title>Booting with PXE</title>

        <para>By default, the &man.pxeboot.8; loader loads the kernel via NFS.
        It can be compiled to use TFTP instead by specifying the
        LOADER_TFTP_SUPPORT option in
        <filename>/etc/make.conf</filename>. See the comments in
        <filename>/etc/defaults/make.conf</filename> for
        instructions.</para> 
        
        <para>To use PXE when the machine starts, you will usually
        need to select the <literal>Boot from network</literal> option
        in your BIOS setup, or type a function key during the PC
        initialization.</para> 

     </sect3>

      <sect3>
	<title>Configuring the TFTP and NFS Servers</title>

	  <secondary>diskless operation</secondary>
	</indexterm>

	<para>If any of the boostrap programs (PXE, pxeboot, or etherboot)
	 are configured to use TFTP, you need to enable
	 <command>tftpd</command> on the file server:</para>

        <procedure>
          <step>
            <para>Create a directory from which <command>tftpd</command>

            <para>Add this line to your
	      <filename>/etc/inetd.conf</filename>:</para>

	    <programlisting>tftp  dgram  udp  wait  root  /usr/libexec/tftpd  tftpd -s /tftpboot</programlisting>

	    <note><para>It appears that at least some PXE versions want
		the TCP version of TFTP.  In this case, add a second line,

	  location is set in both <filename>inetd.conf</filename> and
	  <filename>dhcpd.conf</filename>.</para>

	<para>In all cases, you also need to enable NFS and export the
	  appropriate filesystem on the NFS server.</para>

        <procedure>

	      point and replace <replaceable>margaux</replaceable>
	      with the name of the diskless workstation):</para>

	    <programlisting><replaceable>/data/misc</replaceable> -alldirs -ro <replaceable>margaux corbieres</replaceable></programlisting>
          </step>
	  <step>
	    <para>Tell <command>mountd</command> to reread its configuration

	  <secondary>kernel configuration</secondary>
	</indexterm>

	<para>If using <application>etherboot</application>, you need to
      create a kernel configuration file for the diskless client
      with the following options (in addition to the usual ones):</para>

	<programlisting>
      options     BOOTP          # Use BOOTP to obtain IP address/hostname
      options     BOOTP_NFSROOT  # NFS mount root filesystem using BOOTP info
          options     BOOTP_COMPAT   # Workaround for broken bootp daemons.
	</programlisting>

	<para>You may also want to use <literal>BOOTP_NFSV3</literal>,
          <literal>BOOT_COMPAT</literal> and
          <literal>BOOTP_WIRED_TO</literal> (refer to
          <filename>LINT</filename>).</para> 

    <para>These option names are historical and slightly
      misleading as they actually enable indifferent use of DHCP or BOOTP
      inside the kernel (it is also possible to force strict BOOTP or
      DHCP use).</para>
        
	<para>Build the kernel (See <xref linkend="kernelconfig">),
	  and copy it to the place specified in
	  <filename>dhcpd.conf</filename>.</para> 

        <note><para>When using PXE, building a kernel with the above
            options is not strictly necessary. Enabling them will
            cause more DHCP requests to be issued during kernel
            startup, with a small risk of inconsistency between the
            new values and those retrieved by pxeboot in some special
            cases. The advantage of using them is that the host name
            will be set as a side effect. Else you will need to set
            the host name by another method, for example in a
            client-specific <filename>rc.conf</filename> file.</para>
        </note>

        <note><para>In order to be loadable with
            <application>etherboot</application>, a 5.x kernel needs
            to have the device hints compiled in. You would typically
            set the following option in the configuration file (see
            the NOTES configuration comments file).
            <programlisting>hints         "GENERIC.hints" </programlisting>
            </para>
        </note>

      </sect3>


	  the place where the filesystem will be created (the
	  <literal>DEST</literal> variable).</para>

    <note><para>The <filename>clone_root</filename> script distributed
            with FreeBSD 5.2 is out of date relative to the 4.x version, and
            also needs a few specific 5.x adjustments. Please refer to PR
            <ulink url="http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/62417">
              misc/62417</ulink>.</para>
        </note>

	<para>Refer to the comments at the top of the script for
	    instructions.  They explain how the base filesystem is built,
	    and how files may be selectively overridden by versions specific

	    confusing.  Use them for reference only, except if you prefer
	    the method that they describe, in which case you will need
	    customized <filename>rc</filename> scripts.</para>

     <note><para>When using PXE, it may happen that the loader will
            get a wrong idea of the root filesystem to be used by the
            kernel (this will be the case if the cloned root
            <filename>/etc/fstab</filename> is a copy of the server's,
            as the client-specific version will not be visible by the
            loader, which looks for hints there).</para>
          <para>This will cause
            error messages at the end of the kernel initialization,
            when the kernel tries to mount the wrong root and fails,
            or even actual errors if it succeeds because a local disk
            happens to be present. </para>

          <para>It is possible to prevent this problem from happening
            by inserting the following  line in
            <filename>/boot/loader.conf</filename>:</para>
            <programlisting>
         vfs.root.mountfrom=""
            </programlisting>
        </note>


      </sect3>

      <sect3>
	<title>Configuring Swap</title>

	<para>If needed, a swap file located on the server can be
	  accessed via NFS.  One of the methods commonly used to do this
	  has been discontinued in release 5.X.</para>

        <sect4>
      <title>NFS swap with FreeBSD 4.x</title>

     <para>The swap file location and size can be specified with
     BOOTP/DHCP FreeBSD-specific options 128 and 129. Examples of
     configuration files for isc-dhcp 3 or bootpd follow.</para>

	<procedure>
	  <step><para>Add the following lines to
	  <filename>dhcpd.conf</filename>:</para>
	    <programlisting>
        # Global section
        option swap-path code 128 = string;
        option swap-size code 129 = integer 32;

        host margaux {
            ... # Standard lines, see above
            option swap-path <replaceable>"192.168.4.4:/netswapvolume/netswap"</replaceable>;
            option swap-size <replaceable>64000</replaceable>;
        }
	    </programlisting>

        <para><literal>swap-path</literal> is the path to a directory
        where swap files will be located. Each file will be named 
        <filename>swap.<replaceable>clientIp</replaceable></filename>.</para>
   
	    <para>Older versions of <command>dhcpd</command> used a syntax of
	      <literal>option option-128 "...</literal>, which is no
	      longer supported.</para>
	    <para><filename>/etc/bootptab</filename> would use the

	  <step>
	    <para>On the NFS swap file server, create the swap
	    file(s)</para>
      <screen>
      &prompt.root; <userinput>mkdir <replaceable>/netswapvolume/netswap</replaceable></userinput>
      &prompt.root; <userinput>cd <replaceable>/netswapvolume/netswap</replaceable></userinput>
      &prompt.root; <userinput>dd if=/dev/zero bs=1024 count=<replaceable>64000</replaceable> of=swap.<replaceable>192.168.4.6</replaceable></userinput>
      &prompt.root; <userinput>chmod 0600 swap.<replaceable>192.168.4.6</replaceable></userinput>
        </screen>
	    <para><replaceable>192.168.4.6</replaceable> is the IP address
	      for the diskless client.</para>
	  </step>

	    <para>On the NFS swap file server, add the following line to
	    <filename>/etc/exports</filename>:</para>
	    <programlisting>
<replaceable>/netswapvolume</replaceable>  -maproot=0:10 -alldirs <replaceable>margaux</replaceable>
	    </programlisting>
	    <para>Then tell <application>mountd</application> to reread the
		exports file, as above.</para>
	  </step>
	</procedure>
        </sect4>

        <sect4>
      <title>NFS swap with FreeBSD 5.x</title>

          <para>The kernel does not support enabling NFS swap at boot
          time. Swap must be enabled by the startup scripts, by
          mounting a writeable file system and creating and enabling a
          swap file. This can be done, for example, in
          <filename>/etc/rc.local</filename>. Example:</para>

          <screen>
    swapfile=/writableVolume/swap/swap.<replaceable>MyIP</replaceable>
	dd if=/dev/zero of=$swapfile bs=1k count=1 oseek=100000
	swapon $swapfile
	rm -rf $swapfile
          </screen>

        </sect4>
      </sect3>

      <sect3>

Return to bug 62834