[coreboot] A word on releases

[Stefan Reinauer]

Hi people,

we have had a great first year with regular coreboot releases. Exactly
12 months ago, we released coreboot 4.1, the first release after the 4.0
release in 2011 and our "classic" branch in 2014.

Since then we have had 4 successful releases, both Patrick and Martin
went through the release process, and with this process we were able to
move two corporations working with coreboot (Intel and Google)
significantly closer to working with upstream coreboot.

However, releases take a lot of time, and we were not able to tighten
the release process with each release, as we had originally hoped, and
so, after I talked with Patrick and Martin, we have decided to move to
a slightly slower paced release cycle, creating only two releases per
year. In turn, we will try to improve the overall quality of the
releases in the future. This switch will mean, that the coreboot 4.5
release will happen in fall 2016, rather than this month.

I have written up a few questions that I came across, and some answers
to them. 

If you are using coreboot releases, I would like to hear from you, so
that we can continue to improve the release process and tune it towards
its users, while we spend the majority of the time on making coreboot
better.

We're continually improving the release process and raising the quality
bar and look for developers interested in helping on that topic.

Questions and comments, flames and praise are, as always, welcome!

Stefan



FAQ
---

1. What kind of releases do we do?

The coreboot project is now creating a new release every six months.

Before this cycle, we created a new release every quarter, but every
second release has basically been unused.


2. Why do we need releases at all?

Lots of people work "offline". You check out a given revision of
coreboot, and do your work. When you are done, you will have to fix up
your patch to still work with the now latest revision, before you can
submit it to the upstream tree. Depending on how complex your problem
is, or how busy you are with other stuff, a lot of time can pass between
the start and the finish line.

Releases are a simple mechanism to make sure everybody is on the same
page. A way of documenting what has changed between the start of your
effort and it's availability to the community.


3. Isn't this just psychological?

In the past we have seen groups of people working together on a
coreboot port to a new platform, with slightly different trees, being 1
or 2 weeks apart. This can cause all kinds of funny effects, that take
away from the efficiency of the development process, and that, lacking
up to date documentation, every engineer has to go through again.

With over 100 people contributing over 3000 patches per year to our code
base, these little changes sum up to a significant amount.


4. Who are these releases for?

If you are a developer that wants to put coreboot on your board at home,
you probably don't care much, yet, because chances are, we didn't even
test your board before making the release.

If you are setting up an internal coreboot tree for your fellow
coworkers to colaborate with you, and you can't work on coreboot.org
directly, please use the latest released version to base your work off,
rather than top of tree. This will make it much easier to support your
work.


5. What about testing releases?

Right now, we are hand testing releases on a few selected machines
before every release by boot testing on a given setup. We try to cover
as much diversity as we can (e.g. Intel, AMD, ARM, ARM64). However, this
part of the release process is less formalized as the coreboot project
lacks a large scale test system setup at the moment. Mostly, what the
release manager for a release has available at the time is what will be
tested.


6. What's with that dropped board thing?

Short story: It's not really worth bringing forward all boards all the
time, so we're trying to find a reasonable cut-off line.

Long story:
Coreboot is a fast moving project with a lot (a whole lot) of boards,
and even more features.

In the very early days (1999 - 2003), most boards wouldn't compile or
work shortly after they've been merged, unless someone went ahead and
fixed it. When we added abuild to the development process, that didn't
happen anymore and so we attempted to move all boards forward to the
latest feature set all of the time.

Since coreboot itself has no user interface and barely any attack
surface, there are barely good reasons to update your firmware very
often, especially if work on your board has generally stopped. On the
same hand, a lot of design decisions made 10 or 15 years ago can only
be overcome with significant effort that is not justified for obsolete
hardware.
In order to keep the code healthy and clean, we are retiring old boards
in release branches. For example, if a board was removed right after the
release of 4.4, you can still check out the "4.4" branch and build your
board with that branch. If the board is broken on that branch, you can
check in fixes to the branch to get it working again.

There is no hard cut-off line, but typically boards that have been in
the tree for more than 10 years will be retired at some point.


7. What (else) happens before a release?

- Stabilization of the tree, review and check-in of critical fixes
- Boot testing on a handful boards
- Creation of a change log and release notes. This will contain some
  useful metrics about the last release, including the amount of new
  code integrated, old code removed, new contributors in the project,
  significant high level changes, and what needs to be done to unmerged
  ports to be adapted to these changes.
- A new branch is cut, named after the release (e.g. "4.4").
  This branch can be used to push critical fixes for a given release,
  albeit this has not happened very often in the past (Thanks to Patrick
  Rudolph and Nico Huber for being an exception)
- A new tag is created, named after the release (e.g. "4.4")
- Announcements on the blog


8. Why don't we have monthly releases?

Our release process, while, as Patrick called it, follows a "minimum
viable" approach, the testing, reviews and documentation are a lot of
manual work going into each release. At this point, it is a fair
assumption that it is a 1 week (40+h) effort to create a new release,
for someone who has done a coreboot release before. We would like to
keep the project overhead (and the frustrations that come with that)
adequately low. As right there are only a handful of entities relying on
coreboot releases right now, a bi-annual release cycle seems reasonable.


9. What about board status?

The board status script (and the attached web page
https://www.coreboot.org/Supported_Motherboards ) is currently the best
method to determine when a board was last tested and working, other than
getting in touch with the original developer.
Unfortunately there is currenlty no real documentation about the
process, and reports rely on community members willing to test new
versions on their boards.


10. Long term plans

The longer term plan is to provide infrastructure that will allow the
coreboot project to have 100% of the boards tested regularly (e.g. on
every commit, but at the minimum for every release).

As we gather more metrics for the quality of releases, we will keep
increasing the bar for each release.