Bug 218922

Summary:	Download of ports source not being identified as coming from FreeBSD OS.
Product:	Ports & Packages	Reporter:	Joe Barbish <qjail1>
Component:	Ports Framework	Assignee:	Port Management Team <portmgr>
Status:	Closed Works As Intended
Severity:	Affects Many People	CC:	fk, portmgr, ports-bugs
Priority:	---
Version:	Latest
Hardware:	Any
OS:	Any

Description Joe Barbish 2017-04-28 00:17:12 UTC

It has come to my attention that the downloading of a ports source by our ports framework is not identifying FreeBSD as the requesting operating system which results in the download request being marked as "unknown" by the source-hosting site.

Here is an example of what I am talking about.

https://sourceforge.net/projects/qjail/files/qjail-5.1.tar.bz2/stats/os?dates=2017-01-01%20to%202017-04-20

In discussion with sourceforge tech support, they said:
“We unfortunately would not be able to determine the OS utilized if the download request lacks the browser user agent information.

My port has its source hosted on sourceforge.net. That site has a function that shows the count of times the port source file has been downloaded along with the country and the type of operating system of the computer requesting the download. 99.9% of the time the operating system is shown as "unknown". I can only assume those are downloads originating from FreeBSD systems using the port "make install" method. I believe whatever utility employed by the port system framework to download the port source is not populating the HTTP_USER_AGENT or the HTTP_REFERER environment variable, which is the root cause of this problem.

I did this command on my host
/usr/bin/grep -r HTTP_REFERER /usr/ports/Tools
/usr/bin/grep -r HTTP_REFERER /usr/ports/Mk
/usr/bin/grep -r HTTP_ USER_AGENT /usr/ports/Tools
/usr/bin/grep -r HTTP_ USER_AGENT /usr/ports/Mk
Got no hits, but
/usr/bin/grep -r fetch /usr/ports/Tools
/usr/bin/grep -r fetch /usr/ports/Mk
Got a lot of hits.

I would say the ports system does not use HTTP_REFERER or HTTP_USER_AGENT and the fetch utility is being employed to perform the physical download.

Here is my problem; I tried to read and follow the logic in some of those scripts, but the coding style is foreign to me and I do not posses the technical ability to manipulate the port system scripts to make changes.

I think here is an untapped opportunity to let the world port source-hosting servers take notice that FreeBSD is a widely used operating system. In its own way it builds FreeBSD awareness and it’s not a big deal or time-consuming task to accomplish.

Selecting the more correct download utility and populating the correct environment variable with the value of “FreeBSD” would correct this problem. A far better solution would be to inquire the host system for the version of the host system OS and use that information, IE: FreeBSD 11.0-RELASE, which would result in more meaningfully information being captured at the source-hosting sites.

I hope the ports powers-to-be can appreciate the benefits of this change.

Comment 1 Fabian Keil 2017-04-28 14:50:12 UTC

As a user of a FreeBSD deviate (ElectroBSD) I consider the
fact that the User-Agent currently doesn't leak the uname
when fetching distfiles a feature and do not "appreciate the
benefits of this change".

The more information you add to the User-Agent the easier it
gets to track individual users without having to rely on other
information.

Currently distfiles are fetched with the default User-Agent
used by fetch(1) which looks like "fetch libfetch/2.0".

If SF cared about it, they could already separate clients that
use the fetch(1) default User-Agent from the "unknown" section
and you could then conclude that most of these requests were
made using vanilla FreeBSD.

If the ports system would default to adding the uname etc. to the
User-Agent, some FreeBSD deviates would probably simply default
to adding "FreeBSD" as well instead of leaking the real uname
information without the users consent. Therefore it's unlikely to
result in more reliable statistics anyway.

Comment 2 Joe Barbish 2017-04-29 19:59:25 UTC

Previous stated "Currently distfiles are fetched with the default User-Agent
used by fetch(1) which looks like "fetch libfetch/2.0"."

I take it that the default User-Agent information is hard codded in the fetch utility source or maybe more correctly in the libfetch source.

Changing the hard codded User-Agent information from "fetch libfetch/2.0" to "FreeBSD" would accomplish the same goal as making changes to the ports framework to override fetch's default User-Agent information.

I have no preference about how the goal gets accomplished. All I want to see is that all port fetches get marked as coming from a FreeBSD operating system.

Comment 3 Baptiste Daroussin freebsd_committer

2022-08-26 08:45:47 UTC

if a change should happen here, it should not be done in the ports tree but in the fetch command, let's close this ticket as it is regarding port I will let you create a new ticket against the fetch(1) command.