eldevelopment &mdash; qi.bookwar.info https://qi.bookwar.info/tag:eldevelopment Notes on Release, Build and CI Engineering Tue, 26 May 2026 01:49:48 +0000 https://i.snap.as/B5tGWZ1X.ico eldevelopment &mdash; qi.bookwar.info https://qi.bookwar.info/tag:eldevelopment dist-git and exploded SRPMS - demystified https://qi.bookwar.info/dist-git-and-exploded-srpms-demystified?pk_campaign=rss-feed <![CDATA[In this article we address another topic which appeared in multiple discussions recently. We take a look at the difference between the SRPM and the so called dist-git repository of a package. And why do we indeed prefer the dist-git. !--more-- How RPM packages work? In simple words RPM packages need three things: archive of the original sources of the upstream application; set of patches which needs to be applied to the original sources; recipe (RPM spec file), which describes how to apply the patches, how to build the code and how to install it on a target system. When developing an RPM package you treat the upstream sources as a read-only object. You can not change the upstream sources, they should match the exact content upstream has released. To diverge from upstream, for example to backport a fix or to integrate the software better in the system, you create and maintain patches as separate files next to your upstream sources. Then to build a package the build system needs to fetch the archive with original sources, unpack it, apply patches as described in the spec, run the build scripts again as described in the spec, arrange the resulting files in a specific way and pack them into archive together with the installation recipe. This archive is the final "binary RPM" which you can install on your system using rpm or dnf commands. As we build software for multiple architectures, we can produce several binary RPMs from the same source data by building them on different workers with different architectures (one for x8664, one for aarch64 and so on). What is dist-git dist-git is a git repository with a specific layout, which Fedora, CentOS Stream and RHEL use to develop RPM packages. The very minimal dist-git repo would look like this: . ├── my-app.spec // spec file ├── sources // reference to the sources └── patch-for-some-feature.patch // patch to apply to the sources The important feature of the dist-git is that it doesn't store the unpacked sources of the application. It only stores a reference to the tarball of original upstream sources in a so-called lookaside cache. This reference is stored in the file which is called ./sources in the root of the git repository. See for example sources of a glibc package in Fedora Rawhide The lookaside cache of Fedora and CentOS (Stream or not Stream) is public and you can download any of its content. Now, since dist-git is the main repository where package development is happening, package maintainers often use it to store all sorts of additional things (scripts, readme files, infra configurations..) which can help them to do the work. There is also a recommended way to write tests in dist-git (see TMT). These integration tests are not part of the RPM package, but they are used in CI workflows and we recommend to put them in the dist-git repository, so that people can contribute to the package and the test development via the same interface. Example - keepalived dist-git Let's take a random package build, for example keepalived-2.2.4-6.el9. dist-git for the package has the following structure: . ├── bz2028351-fix-dbus-policy-restrictions.patch // patches ├── bz2102493-fix-variable-substitution.patch ├── bz2134749-fix-memory-leak-https-checks.patch ├── gating.yaml // CI configuration ├── .gitignore // standard gitignore ├── keepalived.init // additional sources ├── keepalived.service // additional sources ├── keepalived.spec // spec file ├── rpminspect.yaml // rpminspect checks configuration ├── sources // reference to the lookaside cache └── tests // dist-git test scenarios, run on every merge request ├── keepalived.conf.in ├── runtests.sh └── tests.yml Here I marked with asterisk the files which are not relevant to the RPM package build. What is SRPM As explained above, RPM package build requires multiple inputs. While the inputs are stored in dist-git and lookaside cache, you need to fetch them and carry around the build system to the build workers. Instead of fetching data from the internet during the build process (no build systems should ever do this!), we fetch all of the sources at the beginning, pack them in a tarball (SRPM file) and then use that self-contained tarball to run the builds in the isolated build environment. The SRPM then serves as a record of what build system got as input to produce the binary files. Example - keepalived SRPM SRPM for the package contains the following data: bz2028351-fix-dbus-policy-restrictions.patch 1.58 KB bz2102493-fix-variable-substitution.patch 929.00 B bz2134749-fix-memory-leak-https-checks.patch 1.87 KB keepalived-2.2.4.tar.gz 1.10 MB keepalived.service 392.00 B keepalived.spec 20.47 KB You can see how the SRPM was produced by the build system together with binary RPMs via the Koji build task https://kojihub.stream.centos.org/koji/buildinfo?buildID=27965 The build task used dist-git commit as the input: Source: git+https://gitlab.com/redhat/centos-stream/rpms/keepalived#fc07f81c047dca49df2fc9d20513a7f52005a54d Note how the SRPM contains full tarball of the original upstream sources (1.10 MB of it). This tarball was fetched from the dist-git lookaside cache during the SRPM build step. What is exploded SRPM Fedora and RHEL use dist-git repositories for a very long time. Fedora dist-git has always been public, while RHEL dist-git repositories were internal and not available for people outside of Red Hat. So the only way for CentOS Project to rebuild RHEL code was to take the SRPM files and use them as the source of the rebuild. Since CentOS Project needed to rebrand or adjust certain packages, they didn't take RHEL SRPMs as is, rather they unpacked them and put the unpacked sources in git repository. This way they got access to at least some history of the changes, were able to apply their own patches and generally increased the visibility of the content. Example - keepalived exploded SRPM "Exploded SRPM" at git.centos.org for this package looks like: . ├── .gitignore ├── .keepalived.metadata // same as ./sources in dist-git ├── SOURCES │   ├── bz2028351-fix-dbus-policy-restrictions.patch // patches │   ├── bz2102493-fix-variable-substitution.patch │   ├── bz2134749-fix-memory-leak-https-checks.patch │   └── keepalived.service // additional sources └── SPECS └── keepalived.spec // spec file Exploded SRPM git again doesn't store the upstream tarball in the repository and references the lookaside cache via .keepalived.metadata file. You can see the same files as included in the SRPM, though they are put into a different directory structure. And none of the additional files (tests, scripts, configs) are available. Take away dist-git repository is the original source of an RPM package build. Fedora, CentOS Stream and RHEL packages are all built directly from dist-git repositories. SRPM is an artifact of the build process. It is produced from the commit in dist-git and then stored alongside the binary RPM. Exploded SRPM is an attempt to recover the original git structure from the SRPM in case there is no access to the dist-git repository. It does contain the same source files and spec as in dist-git, but it is not able to recover additional non-packaged data, like configuration files, tests and so on. We recommend to use dist-git for any collaboration and development purposes. ---- P.S. You can also take a look at the Source Git initiative which aims to change the approach to RPM sources to make upstream source code more accessible. ELdevelopment ]]> In this article we address another topic which appeared in multiple discussions recently. We take a look at the difference between the SRPM and the so called dist-git repository of a package. And why do we indeed prefer the dist-git.

How RPM packages work?

In simple words RPM packages need three things:

  • archive of the original sources of the upstream application;
  • set of patches which needs to be applied to the original sources;
  • recipe (RPM spec file), which describes how to apply the patches, how to build the code and how to install it on a target system.

When developing an RPM package you treat the upstream sources as a read-only object. You can not change the upstream sources, they should match the exact content upstream has released.

To diverge from upstream, for example to backport a fix or to integrate the software better in the system, you create and maintain patches as separate files next to your upstream sources.

Then to build a package the build system needs to fetch the archive with original sources, unpack it, apply patches as described in the spec, run the build scripts again as described in the spec, arrange the resulting files in a specific way and pack them into archive together with the installation recipe.

This archive is the final “binary RPM” which you can install on your system using rpm or dnf commands.

As we build software for multiple architectures, we can produce several binary RPMs from the same source data by building them on different workers with different architectures (one for x86_64, one for aarch64 and so on).

What is dist-git

dist-git is a git repository with a specific layout, which Fedora, CentOS Stream and RHEL use to develop RPM packages.

The very minimal dist-git repo would look like this:

.
├── my-app.spec                      // spec file
├── sources                          // reference to the sources
└── patch-for-some-feature.patch     // patch to apply to the sources

The important feature of the dist-git is that it doesn't store the unpacked sources of the application. It only stores a reference to the tarball of original upstream sources in a so-called lookaside cache.

This reference is stored in the file which is called ./sources in the root of the git repository. See for example sources of a glibc package in Fedora Rawhide

The lookaside cache of Fedora and CentOS (Stream or not Stream) is public and you can download any of its content.

Now, since dist-git is the main repository where package development is happening, package maintainers often use it to store all sorts of additional things (scripts, readme files, infra configurations..) which can help them to do the work.

There is also a recommended way to write tests in dist-git (see TMT). These integration tests are not part of the RPM package, but they are used in CI workflows and we recommend to put them in the dist-git repository, so that people can contribute to the package and the test development via the same interface.

Example – keepalived dist-git

Let's take a random package build, for example keepalived-2.2.4-6.el9.

dist-git for the package has the following structure:

.
├── bz2028351-fix-dbus-policy-restrictions.patch  // patches
├── bz2102493-fix-variable-substitution.patch
├── bz2134749-fix-memory-leak-https-checks.patch
├── gating.yaml         // * CI configuration
├── .gitignore          // * standard gitignore
├── keepalived.init     // additional sources
├── keepalived.service  // additional sources
├── keepalived.spec     // spec file
├── rpminspect.yaml     // * rpminspect checks configuration  
├── sources             // reference to the lookaside cache 
└── tests               // * dist-git test scenarios, run on every merge request
    ├── keepalived.conf.in
    ├── run_tests.sh
    └── tests.yml

Here I marked with asterisk the files which are not relevant to the RPM package build.

What is SRPM

As explained above, RPM package build requires multiple inputs. While the inputs are stored in dist-git and lookaside cache, you need to fetch them and carry around the build system to the build workers.

Instead of fetching data from the internet during the build process (no build systems should ever do this!), we fetch all of the sources at the beginning, pack them in a tarball (SRPM file) and then use that self-contained tarball to run the builds in the isolated build environment.

The SRPM then serves as a record of what build system got as input to produce the binary files.

Example – keepalived SRPM

SRPM for the package contains the following data:

bz2028351-fix-dbus-policy-restrictions.patch	1.58 KB
bz2102493-fix-variable-substitution.patch	929.00 B
bz2134749-fix-memory-leak-https-checks.patch	1.87 KB
keepalived-2.2.4.tar.gz	1.10 MB
keepalived.service	392.00 B
keepalived.spec	20.47 KB

You can see how the SRPM was produced by the build system together with binary RPMs via the Koji build task https://kojihub.stream.centos.org/koji/buildinfo?buildID=27965

The build task used dist-git commit as the input:

Source:  git+https://gitlab.com/redhat/centos-stream/rpms/keepalived#fc07f81c047dca49df2fc9d20513a7f52005a54d

Note how the SRPM contains full tarball of the original upstream sources (1.10 MB of it). This tarball was fetched from the dist-git lookaside cache during the SRPM build step.

What is exploded SRPM

Fedora and RHEL use dist-git repositories for a very long time. Fedora dist-git has always been public, while RHEL dist-git repositories were internal and not available for people outside of Red Hat.

So the only way for CentOS Project to rebuild RHEL code was to take the SRPM files and use them as the source of the rebuild.

Since CentOS Project needed to rebrand or adjust certain packages, they didn't take RHEL SRPMs as is, rather they unpacked them and put the unpacked sources in git repository. This way they got access to at least some history of the changes, were able to apply their own patches and generally increased the visibility of the content.

Example – keepalived exploded SRPM

“Exploded SRPM” at git.centos.org for this package looks like:

.
├── .gitignore
├── .keepalived.metadata  // same as ./sources in dist-git
├── SOURCES
│   ├── bz2028351-fix-dbus-policy-restrictions.patch  // patches
│   ├── bz2102493-fix-variable-substitution.patch
│   ├── bz2134749-fix-memory-leak-https-checks.patch
│   └── keepalived.service  // additional sources
└── SPECS
    └── keepalived.spec  // spec file

Exploded SRPM git again doesn't store the upstream tarball in the repository and references the lookaside cache via .keepalived.metadata file.

You can see the same files as included in the SRPM, though they are put into a different directory structure. And none of the additional files (tests, scripts, configs) are available.

Take away

dist-git repository is the original source of an RPM package build. Fedora, CentOS Stream and RHEL packages are all built directly from dist-git repositories.

SRPM is an artifact of the build process. It is produced from the commit in dist-git and then stored alongside the binary RPM.

Exploded SRPM is an attempt to recover the original git structure from the SRPM in case there is no access to the dist-git repository. It does contain the same source files and spec as in dist-git, but it is not able to recover additional non-packaged data, like configuration files, tests and so on.

We recommend to use dist-git for any collaboration and development purposes.

P.S. You can also take a look at the Source Git initiative which aims to change the approach to RPM sources to make upstream source code more accessible.

#ELdevelopment

]]> https://qi.bookwar.info/dist-git-and-exploded-srpms-demystified Sun, 23 Jul 2023 20:08:58 +0000 Continuity of Linux distributions https://qi.bookwar.info/continuously-built-linux-distributions?pk_campaign=rss-feed <![CDATA[I know people who imagine distribution development as the process of piling up the code in the git repository for 6 months and then building it all in one go at the end of those 6 months, so that it can finally be shipped. This is very far from reality. And it is impossible to explain things like CentOS Stream without addressing this confusion. !--more-- Linux distributions are not just developed continuously, they are built continuously. When you have your generic application, you have your sources. You contribute changes to sources, integrating them into the main branch. And then you decide to build the tip of the branch and you get an artifact - a binary. When later you make changes to the sources, you throw away the previous binary, and build yourself a new one. Linux distributions are different. When you change a distribution, you apply a change to a one part of the sources, then you build those sources into a package and add the package to a shared pool of latest packages. And then this shared pool of packages (we call it buildroot) is used to build a next change in the distribution. Linux distribution is "self-hosted"), it grows by updating its buildroot and using its new state to build its next updates. And updates are applied individually per package. Why am I focusing on this? Because it has practical consequences. Packaged Linux distribution is not a single binary, it is a “compound” artifact, where different parts of it (packages) are built at different times using different states of the buildroot. Imagine we have two packages A and B in the distribution. Package A is more static, it doesn’t get that many updates, so in four weeks it got updated once. While package B is more actively developed and gets updates every week. This will look somewhat like this: Time | Week 1 | Week 2 | Week 3 | Week 4 | Package A | v1.0.0 | v1.0.1 | --- A-v1.0.1, B-v18 Package B | v15 | v16 | v17 | v18 | So if you look at the result of the 4-weeks development, you see that package A has been updated from v1.0.0 to v1.0.1 and package B has been updated from v15 to v18. Yet package A was not built on week 4. It was built on week 2 using the state of the buildroot available at that week 2. If that package A has a build dependency on Package B, then it was using that v16 version of the dependency. And it was not updated after the change to v17 in B. There are several points to take from here. First point: packaged distributions do not appear out of nowhere. Linux distributions are either continuously developed from some origin, like Fedora Rawhide is developed for 20 years from its original Fedora 1 state (I'll ask people more knowledgeable than me to explain how that original state was created). Or they are branched (aka forked) from another distribution. Or they are bootstrapped (forked, but with much more work) using another distribution. Second point: packaged distribution is not fully defined by the static snapshot of its sources. If on week 4 you use the latest git sources of the distribution, check them out and build packages from them, you will get package A of version v1.0.1 built using a build dependency on the package B of v18. Which may or may not lead to a different result. And if this doesn’t scare you, please read again. You can not reproduce the exact state of the Linux distribution from a snapshot of its sources. And not because something is hidden from the sources (dist-git repository state has more information about the package than SRPM of that package does, and we'll talk about it another day). The issue is that distribution is not just its sources. Distribution is its sources and its buildroot with all its complex history. In my opinion it is a big fail for the entire industry to think about a Linux distribution as a fixed set of RPMs and SRPMs on a DVD(or the iso image). I understand where it comes from, but it is a fail anyway. When I think about a distribution, I think about the buildroot of a distribution as a sort of git repository: it has history, it has merge requests_. When we update a package, we add a new binary package (RPM) to the buildroot. In other words, we make a new commit to the buildroot state. And it is not just an abstraction, I think we can literally implement the changes to the distribution buildroot as merge requests to a git repository with the list of packages (And if you are interested and want to try and help make it happen - let's talk about it). Third point: branching of a Linux distribution is not just branching of the sources, it is branching of its binaries. Again this is something that application developers won't expect. When we create a branch of the distribution we don't just create a branch in every git source of every package included in the distribution. We also create a "branch" of the buildroot. All binary packages built till a certain day in the mainline of a distribution are copied into a branch. They form the buildroot of the branch, which is then can be updated via a standard update procedure. We do not rebuild a package after branching, unless there is a new change which we want to land in this specific package. --- This is a heavy-weight article, and thank you for getting this far. But let me reiterate the main message: Packaged Linux distribution is not built from scratch from the snapshot of its sources. We accumulate changes as they happen in different packages, and we inherit, merge and branch the pool of binary packages the same way we inherit, merge and branch their sources. We will look into how this applies to RHEL and CentOS conversation in next articles. --- As folks pointed out, the process described in this article applies to packaged Linux distributions like Fedora, Debian or RHEL. There are other ways to build a distribution and you can check the article by Colin Walters, where he discusses the alternatives. ELdevelopment]]> I know people who imagine distribution development as the process of piling up the code in the git repository for 6 months and then building it all in one go at the end of those 6 months, so that it can finally be shipped. This is very far from reality. And it is impossible to explain things like CentOS Stream without addressing this confusion.

Linux distributions are not just developed continuously, they are built continuously.

When you have your generic application, you have your sources. You contribute changes to sources, integrating them into the main branch. And then you decide to build the tip of the branch and you get an artifact – a binary. When later you make changes to the sources, you throw away the previous binary, and build yourself a new one.

Linux distributions are different.

When you change a distribution, you apply a change to a one part of the sources, then you build those sources into a package and add the package to a shared pool of latest packages. And then this shared pool of packages (we call it buildroot) is used to build a next change in the distribution.

Linux distribution is “self-hosted”, it grows by updating its buildroot and using its new state to build its next updates. And updates are applied individually per package.

Why am I focusing on this? Because it has practical consequences.

Packaged Linux distribution is not a single binary, it is a “compound” artifact, where different parts of it (packages) are built at different times using different states of the buildroot.

Imagine we have two packages A and B in the distribution. Package A is more static, it doesn’t get that many updates, so in four weeks it got updated once. While package B is more actively developed and gets updates every week.

This will look somewhat like this:

Time      | Week 1 | Week 2 | Week 3 | Week 4 |
Package A | v1.0.0 | v1.0.1                   |  --->    A-v1.0.1, B-v18
Package B | v15    | v16    | v17    | v18    |

So if you look at the result of the 4-weeks development, you see that package A has been updated from v1.0.0 to v1.0.1 and package B has been updated from v15 to v18. Yet package A was not built on week 4. It was built on week 2 using the state of the buildroot available at that week 2. If that package A has a build dependency on Package B, then it was using that v16 version of the dependency. And it was not updated after the change to v17 in B.

There are several points to take from here.

First point: packaged distributions do not appear out of nowhere.

Linux distributions are either continuously developed from some origin, like Fedora Rawhide is developed for 20 years from its original Fedora 1 state (I'll ask people more knowledgeable than me to explain how that original state was created). Or they are branched (aka forked) from another distribution. Or they are bootstrapped (forked, but with much more work) using another distribution.

Second point: packaged distribution is not fully defined by the static snapshot of its sources.

If on week 4 you use the latest git sources of the distribution, check them out and build packages from them, you will get package A of version v1.0.1 built using a build dependency on the package B of v18. Which may or may not lead to a different result.

And if this doesn’t scare you, please read again.

You can not reproduce the exact state of the Linux distribution from a snapshot of its sources. And not because something is hidden from the sources (dist-git repository state has more information about the package than SRPM of that package does, and we'll talk about it another day). The issue is that distribution is not just its sources. Distribution is its sources and its buildroot with all its complex history.

In my opinion it is a big fail for the entire industry to think about a Linux distribution as a fixed set of RPMs and SRPMs on a DVD(or the iso image). I understand where it comes from, but it is a fail anyway.

When I think about a distribution, I think about the buildroot of a distribution as a sort of git repository: it has history, it has merge requests. When we update a package, we add a new binary package (RPM) to the buildroot. In other words, we make a new commit to the buildroot state.

And it is not just an abstraction, I think we can literally implement the changes to the distribution buildroot as merge requests to a git repository with the list of packages (And if you are interested and want to try and help make it happen – let's talk about it).

Third point: branching of a Linux distribution is not just branching of the sources, it is branching of its binaries.

Again this is something that application developers won't expect. When we create a branch of the distribution we don't just create a branch in every git source of every package included in the distribution. We also create a “branch” of the buildroot. All binary packages built till a certain day in the mainline of a distribution are copied into a branch. They form the buildroot of the branch, which is then can be updated via a standard update procedure.

We do not rebuild a package after branching, unless there is a new change which we want to land in this specific package.

This is a heavy-weight article, and thank you for getting this far.

But let me reiterate the main message:

Packaged Linux distribution is not built from scratch from the snapshot of its sources.

We accumulate changes as they happen in different packages, and we inherit, merge and branch the pool of binary packages the same way we inherit, merge and branch their sources.

We will look into how this applies to RHEL and CentOS conversation in next articles.

As folks pointed out, the process described in this article applies to packaged Linux distributions like Fedora, Debian or RHEL. There are other ways to build a distribution and you can check the article by Colin Walters, where he discusses the alternatives.

#ELdevelopment

]]> https://qi.bookwar.info/continuously-built-linux-distributions Sat, 22 Jul 2023 16:10:49 +0000 EL-development https://qi.bookwar.info/el-development?pk_campaign=rss-feed <![CDATA[When you live in the world of ever changing development tools and frameworks, disrupting technologies and freshly generated buzzwords it is tempting to look down at a stable Linux distribution as a static, boring thing, which you install and forget. Hopefully you don’t forget it completely and at least try to be up to date with the latest updates. But let’s be honest, everyone working in this field has an example of a machine which worked long past the official End Of Life of its operating system. And that’s OK. Many people can live a happy and exciting life without worrying too much about the depths of the Enterprise Linux-development. Yet if you are in the business of long-term planning, in the business of development of services which are built to last and in the business of supporting environments too sensitive to change, you have to accept that the stable Linux distribution is a living thing. It has its own development, own lifecycle, its own dependencies and even its own stable branches. It also has some unique challenges people don’t meet while carelessly sliding on the surface. Also I personally find it interesting and exciting too, but I don’t expect everyone to share the interest. Anyways, whatever your reasons to come here, welcome on board, buckle up and let’s dive in. As I am unable to write everything I want to write in one go, I am going to use this page to collect the notes which I (or maybe someone else too, send me links) have written on the topic. You can also send me (@bookwar@fosstodon.org) questions, opinions, corrections. And I will mention you in ~my will~ the credits of the book if I ever write it. We will see how far it goes. Local articles dist-git and exploded SRPMS – demystified SRPMs, exploded SRPMs and dist-git - what is it all about? Continuity of Linux distributions I know people who imagine distribution development as the process of piling up the code in the git repository for 6 months and then building it all in one go at the end of those 6 months, so that it can finally be shipped. This is very far from reality. And it is impossible to explain things like CentOS Stream without addressing this confusion. The Curse Of Bug To Bug Compatibility The chase for “bug-to-bug compatibility” hurts community, hurts RHEL customers and hurts the industry as a whole. The real innovation behind the CentOS Stream is the attempt to change it. Links to externals sites Joe Brockmeier https://dissociatedpress.net/2023/06/24/red-hat-and-the-clone-wars/ "Red Hat and the Clone Wars" - saga in many chapters. Not necessarily required for technical understanding, but provides historical context. Gordon Messmer https://medium.com/@gordon.messmer/in-favor-of-centos-stream-e5a8a43bdcf8 On CentOS Stream vs minor releases of RHEL https://medium.com/@gordon.messmer/if-you-dont-understand-its-purpose-you-can-t-improve-the-process-4e48260c3887 On CentOS Stream vs CentOS Linux ELdevelopment ]]> When you live in the world of ever changing development tools and frameworks, disrupting technologies and freshly generated buzzwords it is tempting to look down at a stable Linux distribution as a static, boring thing, which you install and forget.

Hopefully you don’t forget it completely and at least try to be up to date with the latest updates. But let’s be honest, everyone working in this field has an example of a machine which worked long past the official End Of Life of its operating system.

And that’s OK. Many people can live a happy and exciting life without worrying too much about the depths of the Enterprise Linux-development.

Yet if you are in the business of long-term planning, in the business of development of services which are built to last and in the business of supporting environments too sensitive to change, you have to accept that the stable Linux distribution is a living thing. It has its own development, own lifecycle, its own dependencies and even its own stable branches. It also has some unique challenges people don’t meet while carelessly sliding on the surface.

Also I personally find it interesting and exciting too, but I don’t expect everyone to share the interest.

Anyways, whatever your reasons to come here, welcome on board, buckle up and let’s dive in.

As I am unable to write everything I want to write in one go, I am going to use this page to collect the notes which I (or maybe someone else too, send me links) have written on the topic. You can also send me (@bookwar@fosstodon.org) questions, opinions, corrections. And I will mention you in ~my will~ the credits of the book if I ever write it.

We will see how far it goes.

Local articles

  • dist-git and exploded SRPMS – demystified SRPMs, exploded SRPMs and dist-git – what is it all about?

  • Continuity of Linux distributions I know people who imagine distribution development as the process of piling up the code in the git repository for 6 months and then building it all in one go at the end of those 6 months, so that it can finally be shipped. This is very far from reality. And it is impossible to explain things like CentOS Stream without addressing this confusion.

  • The Curse Of Bug To Bug Compatibility The chase for “bug-to-bug compatibility” hurts community, hurts RHEL customers and hurts the industry as a whole. The real innovation behind the CentOS Stream is the attempt to change it.

Joe Brockmeier

Gordon Messmer

#ELdevelopment

]]> https://qi.bookwar.info/el-development Sat, 22 Jul 2023 13:23:14 +0000 The curse of bug-to-bug compatibility https://qi.bookwar.info/the-curse-of-bug-to-bug-compatibility?pk_campaign=rss-feed <![CDATA[Disclaimer: I am a Senior Principal Engineer in Red Hat. I was a member of the RHEL 9/CentOS Stream 9 Bootstrap team. Opinions are my own. Tl;dr The chase for "bug-to-bug compatibility" hurts community, hurts RHEL customers and hurts the industry as a whole. The real innovation behind the CentOS Stream is the attempt to change it. !--more-- What is ABI compatibility? ABI compatibility is a requirement for certain interfaces to not change for a certain length of time, so that you can safely rely on the availability of a certain function and certain library which behaves in a predictable way. It is important to know that RHEL ABI compatibility doesn't say "all ABIs and APIs are stable forever". The real ABI compatibility of RHEL is described in details in the official ABI Compatibility guidelines: https://access.redhat.com/articles/rhel9-abi-compatibility I recommend to take a look and check which compatibility level is assigned to your favorite library according to this guide. What is bug to bug compatibility? The term bug-to-bug compatibility when applied in isolation means that for this specific issue when we implement a new system we carry the bug over to a new implementation. Your users rely on the broken behavior so much that they require it on the new system, see xkcd/Workflow. The way the term is applied to RHEL conversation is very generic, and therefore has much less sense. It implies that one Linux distribution has the same bugs as the another. Linux distribution is not a static thing though. Linux distribution is a pool of package builds, each versioned on its own, and updated on its own, which are then combined into different subsets based on different rules. It is both the power and the weakness of a Linux distribution. The power as the ability to combine and mix and match packages allows you to create solutions to a large range of tasks. The weakness as different combinations of components may lead to different behavior. Add the branching structure of RHEL to it, with all of its powerful minor-stream complexity, and you'll realize that there is simply no single state of RHEL, which you can be bug-to-bug compatible to. Thus, at the distribution level the "bug-to-bug compatibility" concept does not exist. It is an overhyped buzzword people use without putting too much thought into it. Why does it hurt ABI compatibility guidelines is the open formal standard for the RHEL-compatible ecosystem. It is the .odt kind of specification for Linux distributions. On the other hand, the mythical bug-to-bug compatibility is the .docx. You chase the always moving target which you can not control or predict. The huge amount of issues coming to RHEL support, and the demand from RHEL customers for longer and more extensive and never-ending support cycles, come from the fact that the ecosystem doesn't follow the standard, and relies on "undefined behavior" of the specific implementation of it. Think about it: Even a single RHEL minor release, as the most stable and most restricted flow of updates we have at hand, is not bug-to-bug compatible to itself. Yes, we change things and we fix bugs. That's what updates do. More to that, even a static snapshot of a RHEL minor release is not a good reference for anything. Paying RHEL customers never have a system deployed exactly in the same state how RHEL engineers test it. Every single customer changes the system so that they cherry-pick certain updates, freeze some other and install custom compatible versions of certain things. And that is generally OK. Until it isn't and generates the issue and a support case. The ABI-compatibility standard, which we enforce, gives us both - the limits which we shouldn't cross, but also the flexibility to adjust within those limits. And any kind of "pinning" to the undefined behavior of a specific shapshot of RHEL at a specific point in time implemented by a third-party or ourselves creates an issue for future updates not knowing about that hidden requirement. So yes, relying on undefined behavior is bad for business. For all businesses. As well as for community. It is simply bad for all people on Earth who use whatever those businesses and non-businesses create. What is CentOS Stream really? CentOS Stream is sort of "RHEL Stable Proposed Updates". Yes, Red Hat Marketing and Branding folks do not want it to be explained this way, and probably cringe at this very moment. Thus, instead of looking at labels, look at the tech side: It is an ABI-stable and continuous Linux distribution, the mainline of RHEL, from which we branch RHEL minor releases. But CentOS Stream also represents an open reference implementation of the ABI compatibility standard of the RHEL-compatible ecosystem. If your RHEL-compatible application or service doesn't work on CentOS Stream, you do it wrong. And I'll rephrase it: people think that they need a RHEL "clone", because it gives them the access to vendors who develop and test for RHEL. The point though is that to get access to RHEL ecosystem community needs vendors to start develop and test for CentOS Stream. And this is where Red Hat's interest and community interest overlap. ---- Now some may ask, but what if my requirements are not covered by that standard? Then let's adjust the standard. Bring those requirements in. Do not assume that the requirements, which do not work on CentOS Stream, will somehow magically be fullfilled by RHEL. Because they won't. And some may say, ok, it is interesting, but all this talk about standards and requirements looks really complicated and time-consuming. Then bring your tests. The easy way to write a standard is to turn it into a distribution-agnostic test. If you are worried that CentOS Stream will break a certain behavior, write a test and let's gate all CentOS Stream updates with it (And while we are at it, we can also gate all Fedora updates and even upstream updates using the same test, see Packit ) And then some may say, but what if my requirements are so special and custom that they can not be a standard for all. Then you don't need bug-to-bug compatibility. You need the power of remixing. Do your own customization via a Special Interest Group SIG, see for example Hyperscale SIG. Make a version which fits your goals, your schedule, your workflow and your quality requirement, but based on the open shared standard of the ecosystem. Eldevelopment]]> Disclaimer: I am a Senior Principal Engineer in Red Hat. I was a member of the RHEL 9/CentOS Stream 9 Bootstrap team. Opinions are my own.

Tl;dr

The chase for “bug-to-bug compatibility” hurts community, hurts RHEL customers and hurts the industry as a whole. The real innovation behind the CentOS Stream is the attempt to change it.

What is ABI compatibility?

ABI compatibility is a requirement for certain interfaces to not change for a certain length of time, so that you can safely rely on the availability of a certain function and certain library which behaves in a predictable way.

It is important to know that RHEL ABI compatibility doesn't say “all ABIs and APIs are stable forever”. The real ABI compatibility of RHEL is described in details in the official ABI Compatibility guidelines:

https://access.redhat.com/articles/rhel9-abi-compatibility

I recommend to take a look and check which compatibility level is assigned to your favorite library according to this guide.

What is bug to bug compatibility?

The term bug-to-bug compatibility when applied in isolation means that for this specific issue when we implement a new system we carry the bug over to a new implementation. Your users rely on the broken behavior so much that they require it on the new system, see xkcd/Workflow.

The way the term is applied to RHEL conversation is very generic, and therefore has much less sense. It implies that one Linux distribution has the same bugs as the another.

Linux distribution is not a static thing though. Linux distribution is a pool of package builds, each versioned on its own, and updated on its own, which are then combined into different subsets based on different rules. It is both the power and the weakness of a Linux distribution. The power as the ability to combine and mix and match packages allows you to create solutions to a large range of tasks. The weakness as different combinations of components may lead to different behavior.

Add the branching structure of RHEL to it, with all of its powerful minor-stream complexity, and you'll realize that there is simply no single state of RHEL, which you can be bug-to-bug compatible to.

Thus, at the distribution level the “bug-to-bug compatibility” concept does not exist. It is an overhyped buzzword people use without putting too much thought into it.

Why does it hurt

ABI compatibility guidelines is the open formal standard for the RHEL-compatible ecosystem. It is the .odt kind of specification for Linux distributions. On the other hand, the mythical bug-to-bug compatibility is the .docx. You chase the always moving target which you can not control or predict.

The huge amount of issues coming to RHEL support, and the demand from RHEL customers for longer and more extensive and never-ending support cycles, come from the fact that the ecosystem doesn't follow the standard, and relies on “undefined behavior” of the specific implementation of it.

Think about it: Even a single RHEL minor release, as the most stable and most restricted flow of updates we have at hand, is not bug-to-bug compatible to itself. Yes, we change things and we fix bugs. That's what updates do.

More to that, even a static snapshot of a RHEL minor release is not a good reference for anything. Paying RHEL customers never have a system deployed exactly in the same state how RHEL engineers test it. Every single customer changes the system so that they cherry-pick certain updates, freeze some other and install custom compatible versions of certain things. And that is generally OK. Until it isn't and generates the issue and a support case.

The ABI-compatibility standard, which we enforce, gives us both – the limits which we shouldn't cross, but also the flexibility to adjust within those limits. And any kind of “pinning” to the undefined behavior of a specific shapshot of RHEL at a specific point in time implemented by a third-party or ourselves creates an issue for future updates not knowing about that hidden requirement.

So yes, relying on undefined behavior is bad for business. For all businesses. As well as for community. It is simply bad for all people on Earth who use whatever those businesses and non-businesses create.

What is CentOS Stream really?

CentOS Stream is sort of “RHEL Stable Proposed Updates”. Yes, Red Hat Marketing and Branding folks do not want it to be explained this way, and probably cringe at this very moment. Thus, instead of looking at labels, look at the tech side: It is an ABI-stable and continuous Linux distribution, the mainline of RHEL, from which we branch RHEL minor releases.

But CentOS Stream also represents an open reference implementation of the ABI compatibility standard of the RHEL-compatible ecosystem.

If your RHEL-compatible application or service doesn't work on CentOS Stream, you do it wrong.

And I'll rephrase it: people think that they need a RHEL “clone”, because it gives them the access to vendors who develop and test for RHEL. The point though is that to get access to RHEL ecosystem community needs vendors to start develop and test for CentOS Stream. And this is where Red Hat's interest and community interest overlap.

Now some may ask,

  • but what if my requirements are not covered by that standard?

Then let's adjust the standard. Bring those requirements in. Do not assume that the requirements, which do not work on CentOS Stream, will somehow magically be fullfilled by RHEL. Because they won't.

And some may say,

  • ok, it is interesting, but all this talk about standards and requirements looks really complicated and time-consuming.

Then bring your tests. The easy way to write a standard is to turn it into a distribution-agnostic test.

If you are worried that CentOS Stream will break a certain behavior, write a test and let's gate all CentOS Stream updates with it (And while we are at it, we can also gate all Fedora updates and even upstream updates using the same test, see Packit )

And then some may say,

  • but what if my requirements are so special and custom that they can not be a standard for all.

Then you don't need bug-to-bug compatibility. You need the power of remixing.

Do your own customization via a Special Interest Group SIG, see for example Hyperscale SIG. Make a version which fits your goals, your schedule, your workflow and your quality requirement, but based on the open shared standard of the ecosystem.

#Eldevelopment

]]> https://qi.bookwar.info/the-curse-of-bug-to-bug-compatibility Tue, 11 Jul 2023 14:14:06 +0000