<title>Vendor imports with <acronym>SVN</acronym></title>

	<important>

	  <para>Please read this entire section before starting a vendor

	    import.</para>

	</important>

	<note>

	  <para>Patches to vendor code fall into two categories:</para>

	  <itemizedlist>

	    <listitem>

	      <para>Vendor patches: these are patches that have been

		issued by the vendor, or that have been extracted from

		the vendor's version control system, which address

		issues which in your opinion can't wait until the next

		vendor release.</para>

	    </listitem>

	    <listitem>

	      <para>&os; patches: these are patches that modify the

		vendor code to address &os;-specific issues.</para>

	    </listitem>

	  </itemizedlist>

	  <para>The nature of a patch dictates where it should be

	    committed:</para>

	  <itemizedlist>

	    <listitem>

	      <para>Vendor patches should be committed to the vendor

		branch, and merged from there to head.  If the patch

		addresses an issue in a new release that is currently

		being imported, it <emphasis>must not</emphasis> be

		committed along with the new release: the release must

		be imported and tagged first, then the patch can be

		applied and committed.  There is no need to re-tag the

		vendor sources after committing the patch.</para>

	    </listitem>

	    <listitem>

	      <para>&os; patches should be committed directly to

		head.</para>

	    </listitem>

	  </itemizedlist>

	</note>

	<sect4>

	  <title>Preparing the tree</title>

	  <para>If importing for the first time after the switch to

	    Subversion, flattening and cleaning up the vendor tree is

	    necessary, as well as bootstrapping the merge history in

	    the main tree.</para>

	  <sect5>

	    <title>Flattening</title>

	    <para>During the conversion from <acronym>CVS</acronym> to

	      Subversion, vendor branches were imported with the same

	      layout as the main tree.  This means that the

	      <literal>pf</literal> vendor sources ended up in

	      <filename>vendor/pf/dist/contrib/pf</filename>.  The

	      vendor source is best directly in

	      <filename>vendor/pf/dist</filename>.</para>

	    <para>To flatten the <literal>pf</literal> tree:</para>

	    <screen>&prompt.user; <userinput>cd <replaceable>vendor/pf/dist/contrib/pf</replaceable></userinput>

&prompt.user; <userinput>svn mv $(svn list) ../..</userinput>

&prompt.user; <userinput>cd ../..</userinput>

&prompt.user; <userinput>svn rm contrib</userinput>

&prompt.user; <userinput>svn propdel -R svn:mergeinfo .</userinput>

&prompt.user; <userinput>svn commit</userinput></screen>

	    <para>The <literal>propdel</literal> bit is necessary

	      because starting with 1.5, Subversion will automatically

	      add <literal>svn:mergeinfo</literal> to any directory

	      that is copied or moved.  In this case, as nothing is

	      being merged from the deleted tree, they just get in the

	      way.</para>

	    <para>Tags may be flattened as well (3, 4, 3.5 etc.); the

	      procedure is exactly the same, only changing

	      <literal>dist</literal> to <literal>3.5</literal> or

	      similar, and putting the <command>svn commit</command>

	      off until the end of the process.</para>

	  </sect5>

	  <sect5>

	    <title>Cleaning up</title>

	    <para>The <literal>dist</literal> tree can be cleaned up

	      as necessary.  Disabling keyword expansion is

	      recommended, as it makes no sense on unmodified vendor

	      code and in some cases it can even be harmful.

	      <application>OpenSSH</application>, for example, includes

	      two files that originated with &os; and still contain the

	      original version tags.  To do this:</para>

	    <screen>&prompt.user; <userinput>svn propdel svn:keywords -R .</userinput>

&prompt.root; <userinput>svn commit</userinput></screen>

	  </sect5>

	  <sect5>

	    <title>Bootstrapping merge history</title>

	    <para>If importing for the first time after the switch to

	      Subversion, bootstrapping

	      <literal>svn:mergeinfo</literal> on the target directory

	      in the main tree to the to the revision that corresponds

	      to the last related change to the vendor tree, prior to

	      importing new sources:</para>

	    <screen>&prompt.user; <userinput>cd <replaceable>head/contrib/pf</replaceable></userinput>

&prompt.user; <userinput>svn merge --record-only svn+ssh://svn.freebsd.org/base/<replaceable>vendor/pf/dist@180876</replaceable> .</userinput>

&prompt.user; <userinput>svn commit</userinput></screen>

	  </sect5>

	</sect4>

	<sect4>

	  <title>Importing new sources</title>

	  <para>With two commits&mdash;one for the import itself and

	    one for the tag—this step can optionally be repeated

	    for every upstream release between the last import and the

	    current import.</para>

	  <sect5>

	    <title>Preparing the vendor sources</title>

	    <para>Unlike in <acronym>CVS</acronym> where only the needed

	      parts were imported into the vendor tree to avoid bloating

	      the main tree, Subversion is able to store a full

	      distribution in the vendor tree.  So, import everything,

	      but merge only what is required.</para>

	    <para>A <command>svn add</command> is required to add any

	      files that were added since the last vendor import, and

	      <command>svn rm</command> is required to remove any that

	      were removed since.  Preparing sorted lists of the

	      contents of the vendor tree and of the sources that are

	      about to be imported is recommended, to facilitate the

	      process.</para>

	    <screen>&prompt.user; <userinput>cd <replaceable>vendor/pf/dist</replaceable></userinput>

&prompt.user; <userinput>svn list -R | grep -v '/$' | sort >../old</userinput>

&prompt.user; <userinput>cd <replaceable>../pf-4.3</replaceable></userinput>

&prompt.user; <userinput>find . -type f | cut -c 3- | sort >../new</userinput></screen>

	    <para>With these two files, <command>comm -23 ../old ../new</command>

	      will list removed files (files only in

	      <filename>old</filename>), while <command>comm -13 ../old ../new</command>

	      will list added files only in <filename>new</filename>.</para>

	  </sect5>

	  <sect5>

	    <title>Importing into the vendor tree</title>

	    <para>Now, the sources must be copied into

	      <filename><replaceable>dist</replaceable></filename> and

	      the <command>svn add</command> and

	      <command>svn rm</command> commands should be used as

	      needed:</para>

	    <screen>&prompt.user; <userinput>cd <replaceable>vendor/pf/pf-4.3</replaceable></userinput>

&prompt.user; <userinput>tar cf - . | tar xf - -C ../dist</userinput>

&prompt.user; <userinput>cd <replaceable>../dist</replaceable></userinput>

&prompt.user; <userinput>comm -23 ../old ../new | xargs svn rm</userinput>

&prompt.user; <userinput>comm -13 ../old ../new | xargs svn --parents add</userinput></screen>

	    <para>If any directories were removed, they will have to be

	      <command>svn rm</command>ed manually.  Nothing will break

	      if they are not, but they will remain in the tree.</para>

	    <para>Check properties on any new files. All text files

	      should have <literal>svn:eol-style</literal> set to

	      <literal>native</literal>.  All binary files should have

	      <literal>svn:mime-type</literal> set to

	      <literal>application/octet-stream</literal> unless there

	      is a more appropriate media type.  Executable files should

	      have <literal>svn:executable</literal> set to

	      <literal>*</literal>.  No other properties should exist

	      on any file in the tree.</para>

	    <para>Committing is now possible, however it is good

	      practice to make sure that everything is OK by using the

	      <command>svn stat</command> and

	      <command>svn diff</command> commands.</para>

	  </sect5>

	  <sect5>

	    <title>Tagging</title>

	    <para>Once committed, vendor releases should be tagged for

	      future reference.  The best and quickest way to do this

	      is directly in the repository:</para>

	    <screen>&prompt.user; <userinput>svn cp svn+ssh://svn.freebsd.org/base/<replaceable>vendor/pf/dist</replaceable> svn+ssh://svn.freebsd.org/base/<replaceable>vendor/pf/4.3</replaceable></userinput></screen>

	    <para>Once that is complete, <command>svn up</command> the

	      working copy of

	      <filename><replaceable>vendor/pf</replaceable></filename>

	      to get the new tag, although this is rarely

	      needed.</para>

	    <para>If creating the tag in the working copy of the tree,

	      <command>svn:mergeinfo</command> results must be removed:</para>

	    <screen>&prompt.user; <userinput>cd	<replaceable>vendor/pf</replaceable></userinput>

&prompt.user; <userinput>svn cp dist 4.3</userinput>

&prompt.user; <userinput>svn propdel svn:mergeinfo -R 4.3</userinput></screen>

	  </sect5>

	</sect4>

	<sect4>

	  <title>Merging to head</title>

	  <screen>&prompt.user; <userinput>cd <replaceable>head/contrib/pf</replaceable></userinput>

&prompt.user; <userinput>svn up</userinput>

&prompt.user; <userinput>svn merge --accept=postpone svn+ssh://svn.freebsd.org/base/<replaceable>vendor/pf/dist</replaceable> .</userinput></screen>

	  <para>The <literal>--accept=postpone</literal> tells

	    Subversion that it shouldn't complain because merge conflicts

	    will be taken care of manually.</para>

	  <para>It is necessary to resolve any merge conflicts.

	    This process is the same in <acronym>SVN</acronym> as in

	    <acronym>CVS</acronym>.</para>

	  <para>Make sure that any files that were added or removed in

	    the vendor tree have been properly added or removed in the

	    main tree.  To check diffs against the vendor branch:</para>

	  <screen>&prompt.user; <userinput>svn diff --no-diff-deleted --old=svn+ssh://svn.freebsd.org/base/<replaceable>vendor/pf/dist</replaceable> --new=.</userinput></screen>

	  <para>The <literal>--no-diff-deleted</literal> tells

	    Subversion not to complain about files that are in the

	    vendor tree but not in the main tree, i.e. things that

	    would have previously been removed before the vendor

	    import, like for example the like the vendor's makefiles

	    and configure scripts.</para>

	  <para>Using <acronym>CVS</acronym>, once a file was off the

	    vendor branch, it was not able to be put back.  With

	    Subversion, there is no concept of on or off the vendor

	    branch.  If a file that previously had local

	    modifications, to make it not show up in diffs in the

	    vendor tree, all that has to be done is remove any left-over

	    cruft like &os; version tags, which is much easier.</para>

	  <para>If any changes are required for the world to build

	    with the new sources, make them now, and keep testing

	    until everything builds and runs perfectly.</para>

	</sect4>

	<sect4>

	  <title>Committing the vendor import</title>

	  <para>Committing is now possible!  Everything must be

	    committed in one go.  If done properly, the tree will move

	    from a consistent state with old code, to a consistent

	    state with new code.</para>

	</sect4>

      </sect3>

      <sect3>