Bug 272818 - hv_kvp_daemon high CPU usage on Internet router
Summary: hv_kvp_daemon high CPU usage on Internet router
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-07-30 18:12 UTC by Wencey Wang
Modified: 2023-07-31 14:03 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Wencey Wang 2023-07-30 18:12:58 UTC
The `hv_kvp_daemon` runs scripts such as:

```
sh -c "netstat -rn | grep hn1 | awk '/default/ {print $2 }'"
```

to collect the gateway configuration.

However, on systems used as an Internet router that contain millions of routes, this approach drains the CPU, causing Hyper-V Manager to freeze. Additionally, the script itself is not multi-fib aware.
Comment 1 Greg Becker 2023-07-30 23:04:41 UTC
Presuming "hn1" is an interface (?), there's only one "default" route, and the default route is always printed within the first few lines, then having awk stop on the first match should kill the netstat command before it generates very much output, which I would think would greatly reduce the CPU overhead.

But I'd have to see the output of your 'netstat -rn' command to be sure.  Worst case, netstat does a whole lot of CPU intensive work before it starts emitting routes, in which case curbing the output may not help much.

I didn't see any sysctl to print the default route, which would have been nice and probably result in much less overhead than launching into netstat.  Perhaps there is some other light-weight way in which to get the default route?

Off the top of my head, something like the following might improve things:

netstat -rn | awk -v interface=hn1 '$1 == "default" && $4 == interface {print $2; rc=1; exit; } END{exit !rc}'
Comment 2 Yuri Pankov freebsd_committer freebsd_triage 2023-07-31 04:27:31 UTC
(In reply to Greg Becker from comment #1)
> I didn't see any sysctl to print the default route, which would have been nice and probably result in much less overhead than launching into netstat.

route -n get default
Comment 3 Wencey Wang 2023-07-31 06:52:54 UTC
(In reply to Greg Becker from comment #1)
I don't think having awk stop on the first match is a fix. As there's no default route on an Internet router. Instead, there are routes for every routable CIDR. (https://en.wikipedia.org/wiki/Default-free_zone)

Here goes my example, which contains several lines of IPv6 route in the `netstat -rn` output, I have to hide the next-hop and ifname but you can easily get the idea.

```
Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           URS         lo0
::1                               link#1                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           URS         lo0
2001::/32                         hidden next-hop               UG1         hidden ifname
2001:4:112::/48                   hidden next-hop               UG1         hidden ifname
2001:200::/32                     hidden next-hop               UG1         hidden ifname
2001:200:900::/40                 hidden next-hop               UG1         hidden ifname
2001:200:e00::/40                 hidden next-hop               UG1         hidden ifname
2001:200:c000::/35                hidden next-hop               UG1         hidden ifname
2001:200:e000::/35                hidden next-hop               UG1         hidden ifname
2001:218::/32                     hidden next-hop               UG1         hidden ifname
2001:218:2002::/48                hidden next-hop               UG1         hidden ifname
2001:218:2200::/40                hidden next-hop               UG1         hidden ifname
2001:218:3004::/48                hidden next-hop               UG1         hidden ifname
```

As comment #2 suggested, `route(8)` is much better than parsing the output of `netstat -rn`. 
I also think `route(4)` can be used as a good data source instead of parsing text.
Comment 4 Greg Becker 2023-07-31 14:03:02 UTC
Thanks Wencey, that's exactly what I wanted to see, which invalidates all my original presumptions.

So, now presuming that there are contexts in which these hv_kvp_daemon scripts run where there is a default route, then the pipeline should return that route.  

So I deleted the default route on one of my machines to see what 'route -n get default' (thanks Yuri!) would return, and curiously it prints a "No error: 0" error message to stderr and exits zero.  However, my new awk command should give the same error as from the grep in the original pipeline when there is no default route.


root@sm1:~ # route delete default
delete net default

root@sm1:~ # route -n get default
route: route has not been found: No error: 0
root@sm1:~ # echo $?
0

root@sm1:~ # route -n get default | awk -v ifce=ix0 '$1 ~ /gateway:/ || ($1 ~ /interface:/ && $2 == ifce) {val[n++] = $2} END{ if (n == 2) print val[0]; exit (n != 2)}'
route: route has not been found: No error: 0
root@sm1:~ # echo $?
1


And after I restore my default route it seems to do the right thing:

root@sm1:~ # route add default 172.16.1.1
root@sm1:~ # route -n get default | awk -v ifce=ix0 '$1 ~ /gateway:/ || ($1 ~ /interface:/ && $2 ==
ifce) {val[n++] = $2} END{ if (n == 2) print val[0]; exit (n != 2)}'
172.16.1.1

root@sm1:~ # echo $?
0


So, it appears to me that the following pipeline would work.  Is there any way you can test this out?  Presumably the interface is a positional parameter to the script, so you'd need to replace the "ix0" in my awk command with something like ifce="$1" or whatever...  Note that this also presumes the output of 'route -n get default' is fairly rigid.


route -n get default | awk -v ifce=ix0 '$1 ~ /gateway:/ || ($1 ~ /interface:/ && $2 == ifce) {val[n++] = $2} END{ if (n == 2) print val[0]; exit (n != 2)}'


I'm not the hp_kvp_daemon maintainer, so I wont speculate as to why they are doing this directly via route(4).

Hope this helps!
Greg