Attachment #21295 for bug #37353

View | Details | Raw Unified | Return to bug 37353
Collapse All | Expand All




<article>
  <articleinfo>
    <title>Mirroring FreeBSD</title>
    <pubdate>$FreeBSD: doc/en_US.ISO8859-1/articles/hubs/article.sgml,v 1.13 2002/04/01 01:45:14 keramida Exp $</pubdate>
    <authorgroup>
      <author>
        <firstname>Jun</firstname>

        Depending on the set of releases, architectures,
        and degree of completeness you want to mirror, a huge
        amount of disk space may be consumed. Also keep in mind,
        that <emphasis>official</emphasis> mirrors are probably required to be
        complete. The CVS repository and the web pages should
        always be mirrored completely. Also note, that the
        numbers stated here, are reflecting the current

        estimates here:
      </para>
      <itemizedlist>
         <listitem><para>Local site, no public access: basically no minimum,
           but &lt; 2 Mbps could make syncing a pain.</para></listitem>
         <listitem><para>Unofficial public site: 34 Mbps is probably a good start.</para></listitem>
         <listitem><para>Official site: &gt; 100 Mbps is recommended, also your host
           should be connected as close as possible to your border router.</para></listitem>
      </itemizedlist>

        indeed consume some CPU. For AnonCVS it might
        be a nice idea to set up a memory resident filesystem (MFS) of at least
        300 MB, so you need to take this into account
        for your memory requirements. The following
        are just examples to give you a very rough hint.
      </para>
      <para>
        For a moderately visited site, that offers
        <application>rsync</application>, you might
        consider a current CPU with around 800Mhz - 1 GHz,
        and at least 512MB RAM. This is probably the
        minimum you want for an <emphasis>official</emphasis>
        site.
      </para>
      <para>
        For a frequently used site you need definitely
        more RAM (consider 2GB as a good start),
        and possibly more CPU, which could also mean,
        that you need to go for a SMP system.
      </para>
      <para>
        You also want to consider a fast disk subsystem.

    <sect2 id="mirror-services">
      <title>Services to offer</title>
      <para>
        Every mirror site is required to have a set of core services
        available. In addition to these basic services, which mirrors are
        required to provide, there is a number of optional services that
        server administrators may choose to offer.  This section explains
        which services you can provide and how to go about implementing them.
      </para>
      <sect3 id="mirror-serv-ftp">
        <title>FTP (required for FTP fileset)</title>

          the path <filename>/pub/FreeBSD</filename>.
        </para>
        <para>
          There is a lot of software available which
          can be set up to allow anonymous FTP
          (in alphabetical order).
          <itemizedlist>
            <listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd
              can be used. Be sure to read &man.ftpd.8;.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/ncftpd</filename>: A commercial package,
              free for educational use.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/oftpd</filename>: An ftpd designed with
              security as a main focus.</para>
            </listitem>
            <listitem>

            <listitem><para><filename role="package">ftp/vsftpd</filename>: The <quote>very secure</quote> ftpd.</para></listitem>
            <listitem>
              <para><filename role="package">ftp/wu-ftpd</filename>: The ftpd from Washington
                University. It has become infamous, because of the huge
                amount of security issues that have been found in it.
                If you do choose to use this software be sure to
                keep it up to date.
              </para>
            </listitem>
          </itemizedlist>
          FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
          <application>wu-ftpd</application> and maybe <application>ncftpd</application>
          are among the most commonly ones.
          The others do not have a large userbase among mirror sites.
        </para>
      </sect3>
      <sect3 id="mirror-serv-rsync">
        <title>RSYNC (optional for FTP fileset)</title>
        <para>
          <application>Rsync</application> is often also offered for convenience, for the
          contents of the FTP area of FreeBSD. The 
          protocol is different from FTP in many ways, and
          overall, it can be stated, that it is much more
          bandwidth friendly, as only differences between files
          are transferred, not whole files. 
          <application>Rsync</application> does require significant amount of memory for
          each instance. The size depends on the size of
          the synced module in terms of number of directories and 
          files. <application>Rsync</application> can use <command>rsh</command> and
          <command>ssh</command> (now default) as a transport,
          or use it's own protocol for stand-alone access
          (this is the preferred method for public rsync servers).

        <title>HTTP (required for webpages, optional for FTP fileset)</title>
        <para>
          If you want to offer the FreeBSD webpages, you need
          to install a webserver a.k.a <application>httpd</application>.
          You may optionally offer the FTP fileset via HTTP.
          The choice of Webserver software is left up to the mirror administrator.
          Some of the most popular choices are:


      <sect3 id="mirror-serv-cvsup">
        <title>CVSup (desired for CVS repository)</title>
        <para>
          <application>CVSup</application> is a very efficient way of distributing files.
          It works similar as rsync, but was specially designed for
          the use with CVS repositories. If you want to offer the
          FreeBSD CVS repository, you really want to consider
          offering it via <application>CVSup</application>. Still it is possible to offer
          the CVS repository via <application>AnonCVS</application>, FTP,
          <application>Rsync</application> or HTTP, but
          people would benefit much more from <application>CVSup</application> access. 
          <application>CVSup</application> was developed by &a.jdp;.
          It is a bit tricky to install on non-FreeBSD platforms, 
          since it is written in Modula-3 and therefore requires
          a Modula-3 environment. John Polstra has built a 
          stripped down version of M3, that is sufficient to
          run <application>CVSup</application>, and can be installed much easier. 
          See <ulink url="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</ulink>
          for details. Related ports are:


          CVSup is known to eat some CPU on the server as on the 
          client, since it needs to compare lots of files.
        </para>
        <note>
          <para>
            Please have look at the <application>CVSup</application>
            like &man.cvsup.1; and consider using the <option>-s</option>
            option, as it can reduce the amount of work to be done
            a lot.
          </para>
        </note>
      </sect3>
      <sect3 id="mirror-anoncvs">
        <title>AnonCVS (optional for CVS repository)</title>
        <para>
          If you have the CVS repository, you may want to offer
          anonymous CVS access. A short warning first:
          There is not that much demand for it,
          and it requires some experience and you need to know,
          what you are doing.
        </para>
        <para>
          Generally there are two ways, how
          to access a CVS repository remotely: via
          <emphasis>pserver</emphasis> or via <command>ssh</command>
          (we don't consider <command>rsh</command>).
          For anonymous access, <emphasis>pserver</emphasis> is
          very well suited, but some still offer <command>ssh</command>
          access as well. There is a custom crafted
          <ulink url="ftp://ftp.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</ulink>
          in the CVS repository, to be used as a login-shell for the
          anonymous ssh account. It does a chroot, and therefore
          requires the CVS repository to be available under the
          anonymous user's home-directory, which may not be possible

cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
          </programlisting>
          See the manpage for details of the options. See also the cvs <emphasis>info</emphasis>
          page, about additional ways to make sure, access is read-only.
          It is advisable, that you create an unprivileged account, 
          preferably called <username>anoncvs</username>.
          Also you need to create a file <filename>passwd</filename>

          </itemizedlist>

          <filename role="package">ftp/mirror</filename> was very popular, but seemed
          to have some drawbacks, as it is written in &man.perl.1;,
          and did have real problems on mirroring large
          directories like a FreeBSD site. There are rumors, that
          the current version has fixed this, by allowing

          <application>rsync</application> is already mentioned
          in <xref linkend="mirror-serv-rsync">.
          Since <application>rsync</application> access is not 
          required, your preferred upstream site may not allow it.
          Since it is quite common, though, chances are small, that
          you cannot use it. You can always consider using an
          upstream server, that offers it, just for the benefits
          of rsync. 

              machine, most admins impose limitations on their
              server. For a mirror, you should ask the site maintainer
              you are syncing from, about their policy, and maybe
              an exception for your host (since you are a mirror).
            </para>
          </note>
          A command line to mirror FreeBSD could look like that:

          way to mirror the archive, in terms of efficiency, but
          it is only available from few sites. In fact I just know
          <hostid>ftp-master.freebsd.org</hostid> for sure.
          <note id="mirror-cvsup-s-option">
            <para>
              Please have look at the <application>CVSup</application> documentation
              like &man.cvsup.1; and consider using the <option>-s</option>
              option, as it can reduce the amount of work to be done
              a lot.
            </para>
          </note>
        </para>
      </sect3>
    </sect2>


          You should also have a look at <filename>/usr/share/examples/cvsup</filename>
        </para>
        <note>
          <para>
            Please don't forget to consider the hint,
            mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
            above.
          </para>
        </note>
      </sect3>
      <sect3 id="mirror-cvs-other">
        <title>Using other methods</title>

          anyway. Since most sites offer the CVS repository as
          part of the FTP fileset under the path
          <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
          the following methods could be used.
          <itemizedlist>
            <listitem><para><application>FTP</application></para></listitem>
            <listitem><para><application>RSYNC</application></para></listitem>


          <important>
            <para>
              You can <emphasis>NOT</emphasis> use AnonCVS to 
              mirror the CVS repository, since CVS does not allow
              you to access the repository itself, but only checked
              out versions of the modules.

         <important>
           <para>
             The building of the documentation, as well as lots
             of side issues, is documented itself in:
             <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/">fdp-primer</ulink>.
             Please read this piece of documentation, especially if you
             have problems, building the documentation.

       </para>
     </sect3>
   </sect2>
   <sect2 id="mirror-how-often">
     <title>How often should I mirror?</title>
     <para>
       Every mirror should be updated on a regular
       basis. You will certainly need some script
       framework for it, that will be called by 
       &man.cron.8;. Since nearly every admin
       does this his own way, we cannot give
       specific instructions. It could work
       like this:
     </para>
     <procedure>
       <step>
         <para>
           Put the command to run your mirroring application
           in a script. Use of a plain <command>/bin/sh</command>
           script is recommended.
         </para>
       </step>
       <step>
         <para>
           Add some output redirections, so diagnostic
           messages are logged to a file.
         </para>
       </step>
       <step>
         <para>
           Test, if your script works. Check the logs.
         </para>
       </step>
       <step>
         <para>
           Use &man.crontab.1; to add the script the
           appropriate user's &man.crontab.5;.
         </para>
       </step>
     </procedure>
     <para>
       Here are some recommended schedules:
       <itemizedlist>
         <listitem><para>FTP fileset: daily</para></listitem>
         <listitem><para>CVS repository: daily to hourly</para></listitem>
         <listitem><para>WWW pages: daily</para></listitem>
       </itemizedlist>
     </para>
   </sect2>
  </sect1>
  <sect1 id="mirror-where">
    <title>Where to mirror from</title>

      <para>
        Mirrors are organized by country. All
        official mirrors have a DNS entry of the form
        <hostid>ftpN.CC.freebsd.org</hostid>. Whith 
        <emphasis>CC</emphasis> (i.e. country code) beeing the
        <emphasis>top level domain</emphasis>
        of the country, where this mirror is located;
        and <emphasis>N</emphasis> is a number,
        telling that the host would be the <emphasis>Nth</emphasis>
        mirror in that country.
        (Same applies to <hostid>cvsupN.CC.freebsd.org</hostid>,
        <hostid>wwwN.CC.freebsd.org</hostid>, etc.)
        There are mirrors with no <emphasis>CC</emphasis> part.
        These are usually located in the US, but don't need to.
        <hostid>ftp.freebsd.org</hostid> is currently

          for access. If you are already an <emphasis>official</emphasis>
          mirror, this certainly helps you getting access.
          In any other case make sure your country really needs another mirror.
          If it already has three or more, ask the &a.hubs; first.
          the &a.hubs; first.
        </para>
        <para>
          There are just two master sites, one for the FTP fileset and
          one for the CVS repository (the webpages and docs are
          obtained from CVS, so there is no need for it).
        </para>
        <sect4 id="mirror-where-master-ftp">
          <title>ftp-master.freebsd.org</title>

            Set up the required authentication by following
            <ulink url="http://people.freebsd.org/~jdp/cvpasswd/">these
            instructions</ulink>. Make sure you specify the server as
            <hostid>freefall.freebsd.org</hostid> on the <command>cvpasswd</command>
            command line, as described in this document,
            even when you are contacting
            <hostid>cvsup-master.freebsd.org</hostid>

        official mirrors, since the project is sort of
        tolerant here. It is more easy to say,
        what <emphasis>official tier-1 mirrors</emphasis>
        are required to. All other official mirrors
        can consider this a big <emphasis>should</emphasis>.
        <note>
          <para>

    <sect2 id="mirror-official-become">
      <title>How to become official then?</title>
      <para>
        An interesting question, especially, since the state
        of being official comes with some benefits, like a much
        higher bill from your ISP, as more people will be using
        your site. Also it may be a key requirement, to get access

        </step>
        <step>
          <para>
            Then you can ask the &a.doc; or the &a.hubs;  
            to add your mirror site to the mirror list in the
            <ulink url="http://www.freebsd.org/handbook">FreeBSD
            Handbook</ulink>. Make sure you tell them the email address,
            to list as the maintainer of the site.

Return to bug 37353