%ping aa.
  yp_match: clnt_call: RPC: Timed out

Fix: 

NIS DNS first try,  search hostname is "aa.".  it's fail.
   NIS DNS second try, serach hostname is "aa..subdomain.domain". 
                                         ^^  two dot include.

   This append is line 396 in /usr/src/usr.sbin/ypserv/yp_dnslookup.c

        if (hent == NULL) {
                if ((h_errno == TRY_AGAIN || h_errno == NO_RECOVERY)
                                        && q->domain && *q->domain) {
   domain append --->   snprintf(retrybuf, sizeof(retrybuf), "%s.%s",
                                                q->name, *q->domain);
                        if (debug)
                                yp_error("retrying with: %s", retrybuf);
                        q->id = yp_send_dns_query(retrybuf, q->type);
                        q->ttl = DEF_TTL;
                        q->domain++;
                        return;
		}

   then DNS hostname search call "res_mkquery()" is fail.
   and set h_error = NO_RECOVERY. 
   therefore ypserv is host search looping, Not answer YP error.
How-To-Repeat:   /etc/host.conf : hosts
                   bind
                   nis
        
  /etc/rc.conf: nis_client_enable="YES" 
                nisdomainname="test-dom"
                nis_server_enable="YES"
                nis_yppasswdd_enable="YES"
                nis_server_flags="" or nis_server_flags="-n"

  /var/yp/Makefile.dist
		B=-b  # not comment out
   
 This is OK.
  %ping aa
   ping: cannot resolve aa: Unknown host

 Not terminate
  %ping aa.
   yp_match: clnt_call: RPC: Timed out
   yp_match: clnt_call: RPC: Timed out