Bug 254645

Draft of script: https://github.com/freebsd/freebsd-ci/pull/126/files

Comment 2 Luca Pizzamiglio 2021-04-09 20:49:38 UTC

Hi Mateusz, thanks a lot for your work on this area.

I tried to play a bit with your scripts and the tools that you kindly patched to support FreeBSD.

I have a couple of questions:
* I'm using skopeo to download docker images and convert them to OCI format. How did you configured the storage driver in etc/containers/storage.conf
AFAIK, zfs is supported (it would be ideal), but it would work only for root
* one important information that FreeBSD OCI images should have is the os.version, to enforce proper check on jails and host messages. Do you think os.version (in the manifest field of index.json) is a suitable place or it would be better to adopt a custom annotation?

(In reply to Luca Pizzamiglio from comment #2)
Hi Luca,

Thank you for questions.

> * I'm using skopeo to download docker images and convert them to OCI format. How did you configured the storage driver in etc/containers/storage.conf
> AFAIK, zfs is supported (it would be ideal), but it would work only for root

This storage configuration matters in runtime context. I only briefly played with containerd with nullfs snapshotter. For more runtime details I'd refer to great Samuel's work: https://github.com/samuelkarp/runj

> * one important information that FreeBSD OCI images should have is the os.version, to enforce proper check on jails and host messages. Do you think os.version (in the manifest field of index.json) is a suitable place or it would be better to adopt a custom annotation?

Yes, we can put whatever is needed for runtime to validate images. I created minimal draft of runtime spec for FreeBSD for my needs: https://github.com/runhyve/umoci/commit/759dccf4cbbe47a54ca2ef3d44d9c3eb83d7a157
And Samuel made his own draft: https://github.com/samuelkarp/runj/blob/main/runtimespec/config.go

I think that these specs would grow and stabilize over time and in the future they should be submitted as official OCI spec: https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go

os.version in particular could land in FreeBSD specific subtree of schema. Here's example for Solaris: https://github.com/opencontainers/runtime-spec/blob/master/schema/config-solaris.json

In the meantime I published images for supported FreeBSD releases: https://hub.docker.com/r/kwiat/freebsd/tags

(In reply to Mateusz Kwiatkowski from comment #0)
Hi Mateusz!

> I wanted to start discussion about providing official OCI images by FreeBSD project and publishing them in one of public registries (eg. Docker HUB).

I think this is a great idea!  While runj (which you referenced) is a personal project, my day job is very container-centric and I have a few suggestions here.

Docker, Inc. sponsors an "Official Images" program that publishes images to Docker Hub.  The program is managed on GitHub [1] and would enable a FreeBSD developer to be in control of the images.  The advantage of going through the "Official Images" program is the use of a short name that most of the container ecosystem will recognize (for example, "docker pull debian" implicitly pulls the "Official Image" of Debian located at docker.io/library/debian [2]; FreeBSD could have the "freebsd" short name).

Docker Hub also supports "organizations" (similar to GitHub organizations); the FreeBSD project could potentially get the "freebsd" organization, though it appears to be already taken by an inactive account [3].

In my day job at Amazon I work alongside the Amazon ECR team, who maintains a public registry [4].  Like Docker Hub, Amazon ECR Public allows for friendly names associated with an AWS account and for verified publishers.  I use a friendly alias with the image I published [5] and the FreeBSD project could have the "freebsd" alias.

(In reply to Luca Pizzamiglio from comment #2)

> AFAIK, zfs is supported (it would be ideal)

Hi Luca! containerd has a zfs snapshotter [6] which works on Linux, but I have not yet tried it on FreeBSD.

> one important information that FreeBSD OCI images should have is the os.version, to enforce proper check on jails and host messages.

I'm still new to FreeBSD, so my apologies for asking a stupid question.  Is there a requirement that a FreeBSD userland program is built to run on a particular version of FreeBSD?  Do the kernel or syscall interfaces change between versions?

The Windows container images use os.version to indicate compatibility as Windows does require the container images to correspond with the underlying host.

(In reply to Mateusz Kwiatkowski from comment #3)

> Yes, we can put whatever is needed for runtime to validate images. I created minimal draft of runtime spec for FreeBSD for my needs

I believe Luca was referring to the image spec [7], which already includes os.version in the index platform object.  I'm generating OCI images [8], but not currently including os.version.

> FreeBSD specific subtree of schema

For the runtime config, agreed!  I'd be happy to collaborate on this with you.

Thanks!
Sam

[1] https://github.com/docker-library/official-images/
[2] https://hub.docker.com/_/debian
[3] https://hub.docker.com/u/freebsd
[4] https://docs.aws.amazon.com/AmazonECR/latest/public/public-repositories.html
[5] https://gallery.ecr.aws/samuelkarp/freebsd
[6] https://github.com/containerd/zfs
[7] https://github.com/opencontainers/image-spec
[8] https://github.com/samuelkarp/runj/blob/main/demo/rootfs.go#L102-L152

(In reply to Samuel Karp from comment #5)
> Docker, Inc. sponsors an "Official Images" program that publishes images to Docker Hub.  The program is managed on GitHub [1] and would enable a FreeBSD developer to be in control of the images.  The advantage of going through the "Official Images" program is the use of a short name that most of the container ecosystem will recognize (for example, "docker pull debian" implicitly pulls the "Official Image" of Debian located at docker.io/library/debian [2]; FreeBSD could have the "freebsd" short name).

That's what I'm thinking about as a long term goal of OCI@FreeBSD project. I think that for that matter core team should be engaged. Ed is already subscribing this issue so let's wait for his input.


> I'm still new to FreeBSD, so my apologies for asking a stupid question.  Is there a requirement that a FreeBSD userland program is built to run on a particular version of FreeBSD?  Do the kernel or syscall interfaces change between versions?

I have been running jails with base systems lagging several major versions behind host system and 99% of software was working fine. I only recall problems with network related apps from base like netstat, sockstat - there were probably some struct changes. Backwards compatibility is one of FreeBSD's selling points but I think that spec should include os.version and by default refuse to launch old images in sake of security and to encourage users to run supported versions.


> I believe Luca was referring to the image spec [7], which already includes os.version in the index platform object.  I'm generating OCI images [8], but not currently including os.version.

Ok, I agree. I'll add it to my images.


> For the runtime config, agreed!  I'd be happy to collaborate on this with you.

I think I already asked on Github, but I'll repeat. :-) What do you think on creating #freebsd channel in OCI Slack? We can also have #freebsd-oci on freenode if you prefer IRC.

(In reply to Mateusz Kwiatkowski from comment #6)

> I think that spec should include os.version

I think that makes sense.  Would the version be something like 13-RELEASE or a different value?

> by default refuse to launch old images in sake of security and to encourage users to run supported versions

I don't think the spec should mandate this, but it would be possible to encode such a default behavior in a runtime implementation.  I don't know that I have a firm opinion yet on whether or not that's a reasonable default.

> What do you think on creating #freebsd channel in OCI Slack?

It's late in my time zone now so I'm not available to chat synchronously, but I've gone ahead and created a #freebsd channel in the OCI Slack.

(In reply to Samuel Karp from comment #7)

> I think that makes sense.  Would the version be something like 13-RELEASE or a different value?

13.0 or 13.0-RELEASE should be fine to start with. 

> I don't think the spec should mandate this, but it would be possible to encode such a default behavior in a runtime implementation.  I don't know that I have a firm opinion yet on whether or not that's a reasonable default.

Yes, it should be runtime behavior. In my opinion at least warning would be desirable when running outdated image.

> It's late in my time zone now so I'm not available to chat synchronously, but I've gone ahead and created a #freebsd channel in the OCI Slack.

Thank you! I assume that all written communication is async. :-)

Comment 9 Luca Pizzamiglio 2021-04-15 08:22:43 UTC

(In reply to Samuel Karp from comment #5)
FreeBSD versioning follows (or at least tries to) semantic versioning at system-call level.

A binary built for FreeBSD 12.1 will run on FreeBSD 12.1+, but there are no guarantees that it can work on FreeBSD 12.0 or 11.x

In FreeBSD 12.2 new system calls can be added, but no breaking changes are introduced.
In FreeBSD 13.0 new system calls can be added, but also potential breaking changes, like struct definitions passed in system calls and so on.
However, in FreeBSD 13.0 there are compatibility layers to support FreeBSD 12.x ABI (and all the previous version of FreeBSD, you can run binaries from FreeBSD 4 on FreeBSD 13 if you want).

That's why it's extremely important for a OCI image to have its FreeBSD version, to allow the runtime to chose the correct/compatible image to download/run.

A FreeBSD 12.2 host won't use a FreeBSD 13.x image, but it can use a 11.4 or  a 12.1 one (packages are build following a similar approach, one repo per major release).

(In reply to Luca Pizzamiglio from comment #9)

Thanks, that makes a lot of sense.  Including the version in the index would allow a runtime to choose the correct image when pulling.  I assume that the incentive for an image creator to publish multiple versions of the image in an index would be to both achieve compatibility across FreeBSD systems that are in the wild and provide images that can take advantage of newer FreeBSD features or security enhancements?

Comment 11 David Chisnall 2021-04-17 11:19:09 UTC

There's a related question of what should go in the image.  Most containers don't need the toolchain, vi, and a load of other bits of the standard base system install.  With pkgbase, it's easy to select the minimal subset.

Comment 12 Li-Wen Hsu 2021-04-17 11:57:51 UTC

(In reply to David Chisnall from comment #11)
This is a very good point and minimizing the image size is important.
I suggest before pkgbase is mature, we can start from setting src.conf to remove things not needed in a container image.