FUDcon Lawrence Secondary Arch Team for Power notes

•February 25, 2013 • Leave a Comment

As the Fedora Secondary Arch Team for Power met again this year at the North  American FUDCon, this year in Lawrence we’ve put together a pretty long list
of topics we wanted to cover and discuss during those days. Details were
available here
which was linked from the main secondary arch wiki for Power on fp.org the
last few months.

The team meeting at Lawrence were:

Phil Knirsch
Karsten Hopp
David Aquilina
Brent Baude
Mark Hamzy
Gustavo Luiz Duarte

+ special guests appearances from the Fedora release engineering,
infrastructure and the ARM teams

Lets dive into the various topics we dicussed and what came out of it.

F18 retrospective

The main point here was to list the several good and bad things we noticed
during the F18 development so we could repeate the good ones and fix the bad
ones. Here the final list:

– Managed to hit almost all target dates
– Kept schedule in sync with primary
– Almost passed final RC checklist for our RC4 (just 2 bugs)
– Overall only 2 power specific bugs

– Signing was/is still an issue:
o Reliability
o Process
– Detected Custom paritioning bug specific for Power very late during beta
testing, resulting in a delay there
– Multipath needed lots of cycles and attention to get working properly

Regarding the bad things we’ve had several discussions:

For the signing issues we had we’ve already started to address some of the
know issues during the development for F18. The main things that still bugged
us were the reliability and the overall process. For reliability we’ll be
working with Miloslav Trmac to provide network dumps and error messages.
Several from the team will also be at the DevConf in Brno in February to work
with him on this.
Regarding the process issues we’ve discussed several options and came up with
the following flow:

1) Sign packages for tag X. Returns list of all signed packages for tag X
2) Using this list run mesh to compose tree
3) Depending on tree (updates or release) create images and push bits

This will ensure that all packages that get pushed will always be signed on
the first go. It will run the chance that a push on a certain day might miss a
few packages that were just built, but we think the benefit from always only
pushing signed bits more than outweights this.

Next up the late discovery of a critical bug for Beta. I really helped to
begin with that we stuck to the primary schedule as closely as possible. That
allowed us to get composes and images really early on. But in this specific
instance this didn’t really help as it took us several days until we realized
that this was actually a Power specific issue (with Prep partitions), which
meant the fix for it would be late for Beta. On the other hand it allowed us
to test the updates.img functionality even on the install media. To ensure we
cover the most important scenarios we’ll be moving towards using the testing
templates as primary already does, of course with slight modifications and
additions. Additionally providing a one-stop pointer on the arch wiki to learn
about the current state of builds, signing, composes and install tests using a
dashboard style will help keep everyone informed about all important stages.

Last up was multipath. We’ve hit that during the late Alpha testing and
noticed that multipath wasn’t on any of the criterias during Beta testing.
We’ll propose to add those to the criteria also for primary for upcoming
Fedora releases.

Supported archs/hardware for F19

We discussed this for a bit and at the end came to the conculsion not to
do to much changing there compared to Fedora 18:
– Compiler flags just as before for ppc64, generic 64bit optimization
– For ppc we only provide Everything and Fedora trees, no more install media
or images
– Following primary make ppc64 pure, meaning installs won’t contain ppc
packages anymore

grub2 as bootloader on install media (DVD)

The driving factor here was that with switching to grub2 for Fedora 18 we now
have a modern bootloader for Power that allows lots of cool things, but for
the media we’ve still used yaboot (similar to primary using syslinux as the
media bootloader). As it turns out we already are doing that on primary
for UEFI images, so this has already gotten some testing and the necessary
code is already available. The plan is now to create experimental boot ISOs
with grub2 to see if they work properly and write a proposal post Fedora 19 to
switch to grub2 and touch base with primary releng & anaconda in the time in
between to coordinate the efforts.

Discuss use of test templates as used on primary:

This went pretty quickly as we all agreed that this was a really good idea.
The list will certainly need some review and weeding/additions based on what
we’ve seen (e.g. multipath install test case). Additionally the final goal is
to link the release criteria with specific test cases where applicable. This
will make testing and checking if we meet the release criteria much easier.

Ideas on how to improve the visibility of the whole process (koji-shadow logs,
signing progress, updates, etc.) aka Dashboard

This topic was related to some of the issues we faced during Alpha and Beta
testing. The main goal for the discussion about this was to improve information
availability and visibility to everyone. So the idea is to create a dashboard
online that contains all following current information:
– Compose Status (last ran, logs)
– koji-shadow status (last ran, logs)
– Signing Status (last run, signed vs. unsigned #s)
– buildbranched status
– Failed builds
– Testing Status (IBM QA? AutoQA? )

Work on that has been started and David Aquilina has some of it already
available in text form.

Infrstructure review

As we’ve had quite a few infrastructure changes over the last year we wanted
to take the time and review the current state and if there are any areas where
we need changes or improvements. Kevin Fenzi from the infrastructure team came
over to talk with us about it in an overall fashion. Main points coming out if
it were:
– EPEL needs action (new machine), especially to support RHEL-7 in the future
– Overall no issues otherwise, capacity and setup works really well
– Storage might become an issue again, but NAS upgrade planed anyway
– Backups for important boxes (koji & composer)

Continued optimization opportunities

Just a brief look at what packages we currently compile as ppc64p7 and
brainstormed about a possible switch to ppc64p8 at some point.

SDK tutorial/demo – discussion on applicability for Fedora

Unfortunately had to skip that due to lack of time.

Collaberative articles (brainstorm session)

Talked about more articles for the next releases, e.g. followup articles for
mock: What are repositories? among others things.

Project page rewrite/re-do/redesign

As the main wiki for Power in fp.org got a bit cluttered over the past 3
releases we thought it was time to clean that up. Basic idea is to just have a
bit of immediate information on the main page and have separate sub pages for
specific details.

EPEL discussion

Related to Infrstructure review there seems to be the wish to drop Power
support for EPEL at some point. Discussed again with Kevin what we’d need to
do to keep that working. Mainly need a current HW for the buildsystem for it.

Releng discussion w/ primary (process & schedule, checklist items)

We discussed various topics around release engineering with Dennis Gilmore
from rel-eng and Brendan Conoboy from the ARM teams.

Here the summary of the subtopics we had:

– Checklist/release criteria: Which does primary care about, which not?
– As long as it’s sane, primary doesn’t care about our checklist
– Scheduling – what are the milestones we need to sync on? how long does it take for content to hit the mirrors?
– Content has to be staged for mirrors on Friday for a Tuesday release
– Must release on a Tuesday (preferrably announcing at 10a Eastern)
– Signing – messy & unreliable. What can we do to help debug?
– Packet trace between client & bridge needed
– bodhi messier than we thought (requires multiple sign & mash
attempts even on primary)
– just deal with it.
– koji garbage collection?
– script still needs to be written to sync stages of gc b/w primary &
– koji-shadow strategy?
– shadowing stable and -updates-testing simultaneously is still sound
– -override tag sync between primary and secondary?
– no complaints if we run our own version of the sync tag script for
the override tags, but listening on the fedmsg message bus might be
better (but would need new tooling)
– (infra?) how does primary monitor koji, storage, buildbranched, etc?
– infra might be able to hook up basic system monitoring for us
– cronjob output goes to dgilmore, should probably set up an
arch-releng list.

Bug review

Started doing that early Sunday but only did a brief overview and some initial
pruning of old bugs or bugs that we knew were already fixed or weren’t
relevant anymore. Will do more individual pruning in the upcoming weeks via
irc meetings.

Statememt of Direction

The Fedora Secondary Arch for Power team intends to focus and deliver the
following items for Fedora 19:
– Deliver a release for Power systems installable on typical currently
available 64bit hardware
– All packages will be built with general 64bit Power support
– Higher optimized builds for a select list of components available as
additional ppc64p7 arch packages
– Deliver 32bit compatibility packages for all components
– No 32bit install trees and DVD ISOs
– Sync development with the primary schedule
– Ensure release criteria are met for all milestones

Thanks & regards, Phil

PS: I’ve also added this info the the wiki at https://fedoraproject.org/wiki/Architectures/PowerPC/Meetings/FUDCon_Lawrence_2013