Bug 296516 - atom feed is behind bot protection
Summary: atom feed is behind bot protection
Status: Closed FIXED
Alias: None
Product: Services
Classification: Unclassified
Component: Core Infrastructure (show other bugs)
Version: unspecified
Hardware: Any Any
: --- Affects Only Me
Assignee: Cluster Admin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2026-07-04 15:28 UTC by Benjamin Takacs
Modified: 2026-07-07 23:37 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Benjamin Takacs 2026-07-04 15:28:37 UTC
The atom feed for the ports repo is behind bot protection, which makes it useless as it is for consumation of feed readers and not web browsers. I guess other feeds are behind bot protection too, but I haven't checked.
Comment 1 Benjamin Takacs 2026-07-04 15:50:09 UTC
For some reason the bot protection is ignored when curl is send as User-Agent string, so as a workaround that can be used till bot protection is disabled for resources intended for bots.

I also checked the  src  feed, that is also behind bot protection, but the SA&EN feed is not.
Comment 2 Philip Paeps freebsd_committer freebsd_triage 2026-07-04 20:58:30 UTC
We have a couple of local Anubis bypass policies, including one for curl.

Please let us know the User-Agent: or another identifying marker for your feed reader so we can add it to the allowlist.

As far as I can tell we don't have an allowlist for paths.  Maybe we should have one.  I'll try to figure out how to do that.
Comment 3 Philip Paeps freebsd_committer freebsd_triage 2026-07-04 21:14:22 UTC
What's the URL of the feeds you're trying to access?  I just checked in more detail: the SA&EN feed is on www.FreeBSD.org, which doesn't go via Anubis.  So that's good.  We should easily be able to sort you out too. :)
Comment 4 Benjamin Takacs 2026-07-05 02:14:21 UTC
https://cgit.freebsd.org/ports/atom/?h=main and https://cgit.freebsd.org/src/atom/?h=main are behind bot protection. I'm using RSS Guard on my FreeBSD system, it seems like it uses "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0 RSS Guard/4.8.6" as User-Agent. But a path-based approach seems like a better idea, than adding bypasses for every feed reader in existence and other consumers of those feeds.

Additionally feed readers can autodiscover feeds via <link rel="alternate" type="application/atom+xml" …> (or type="application/rss+xml" for rss feeds) so it would be nice to allow a single request to https://cgit.freebsd.org/ports/ (and /src/), so that well-behaved feed readers can perform autodiscovery, before treating them as unwanted for that page. But no idea how complex Anubis can be configured or maybe that page is also fine without bot protection.
Comment 5 Philip Paeps freebsd_committer freebsd_triage 2026-07-06 02:08:03 UTC
I added Anubis rules to permit /{doc,ports,src}/atom/ without being challenged.  I believe our rules are reloaded every ten minutes.  Please let me know whether you see any improvement.  Thanks!
Comment 6 Benjamin Takacs 2026-07-07 22:10:36 UTC
(In reply to Philip Paeps from comment #5)
It works fine now, thanks
Comment 7 Philip Paeps freebsd_committer freebsd_triage 2026-07-07 23:37:27 UTC
Thanks for confirming!