DevOpsDays Antwerp, 2024

This was an extraordinary edition, considering we were celebrating 15 years of DevOpsDays Ghent (the OG DevOpsDays), albeit in Antwerp this time (Ghent-east if we want to make it sound more canonical). There were a lot of familiar faces, both in the crowd and on stage. As Kris Buytaert mentioned, seeing people wearing 12-year-old t-shirts of conferences you've attended is special. These are our people; this is our tribe.

The usual formula — why change a winning team; in devops, we do small iterations and not big bangs. In the AM, there are single-track talks, and in the PM, there is the meat of these (un)conferences: Open Space.

Day 1

Talks

On day one, we saw a talk on Wiring for Fast Flow based on a paper the speaker co-authored with Cisco's John Rauser. In this talk, we discussed how the Internet enabled extreme flow at scale and how we can learn from this in our organizations. We also discussed how siloed teams can do their own thing but still "interface" through some lingua franca using the Internet's hourglass model.

Interesting stuff. If you're into this, he wrote a book, Flow Engineering.

Next up, on the topic of flow, we had Fast Flow: Blockers and Pitfalls by Manuel Pais, one of the co-authors of Team Topologies, ranting on common blockers to fast flow in tech organizations. Some misinterpretations stem from how people read his book "team topologies". He'd also like to amend some of the things he wrote.

With A Decade of DORA, Nathan Harvey took us through the history of Devops, where we started in 2009, to what will now be the tenth DORA report. One interesting tidbit I found was how he said, "Most people stop reading accelerate after page 17, but Appendix A is the thing to remember" (see picture). It's not just DORA metrics or trying to measure engineering efficiency (hint: you can't). We also explored ROI that can be realized with devops transformations, based on a whitepaper from the DORA institute and Google.

In DevOps is not a Factory, Brit Myers from System Initiative discussed how our guiding flow philosophies have roots in factory and process optimization. She broke it down in her talk and showed how, rather than framing product teams as factories, we can more easily build high-performing organizations if we treat them like professional sports teams.

In DevSecOops: 15 years of DevOps and patching is still horrible (but should you care?), Michael Ducy wanted to update a well-known devsecops meme. Explaining how initially Sec felt left out, but over time as we shifted left, it meant "more shit from left". It was an amusing story about how Vendor X's security tool dumped a CSV file containing 7 million 'detected' CVEs onto the operations team without any real consideration for actual defence strategies or the relevance of the severity rankings. Ironically, 95% were false positives, rendering the severity scores almost meaningless. It was a very recognizable story on how we should do more to explore effective collaboration when it comes to SecOps.

I think the gist was, "Security should be integrated with operations and understand modern architectures & practices, how to build effective CVE mitigation strategies". Preaching to the choir, of course, but insightful and ringing true to the core in our ears.

Before lunch there was a talk on Platform Engineering Is Not About Tech on some key aspects often overlooked in implementing a platform and how it's possible to address them along the way. This felt less recognizable for us, as we feel like skipping the platform team Kool-aid for now (read: we're not at that scale — especially compared to others later down this write-up).

The few hands that went up on the question, "Who's building or planning to build a platform?" also clarified that this was not what most people considered a concern. This was very different at KubeCon Paris, where the topic had its dedicated track.

Open Space

We had insightful talks with industry peers on how to do DORA metrics and especially what to use them for (Goodhart's Law et al.). This talk made me put the Apache Devlake on our "to check" list.

In choosing career paths, a lot of focus was on the split between the technical and managerial tracks and why our disciplines generally have so much distrust when it comes to management. The consensus was that it's "just another silo", as was dev and ops. We have different scopes (ops: days, hours, maybe even minutes; manager: months, quarters, a year) and other incentives (ops: stability is good; management: no changes mean risk). To people on the cusp of "What track should I go into?" we suggested the excellent blog series The Engineer/Manager Pendulum. Some participants noted that having a lot of technical experience can make landing a technical job harder, as more prominent companies tend to steer you toward a more managerial function.

With us wanting to own infrastructure of at least 8 Python apps from our AITeam, I thought it fitting to join a space on Python Deployments. It was in an interesting open space on Python deployments and how it's still a mess after 12 years. It was very interesting to hear someone say they do what we plan on doing (Debian LTS with default Python, albeit on VMs instead of containers), and we continued elaborating on that idea in the group; it piqued a lot of interest. The consensus seemed to be that it sounded like a good idea. Moreover, there are few to no authoritative resources online, and once we have something that we're confident works, we should give back to the community in the form of blog posts or similar.

Someone proposed to talk about Documentation, which predictably got much attention. Both tools and processes revolving around documentation were discussed. Alas, it seems to be an unsolved problem in our industry. Some companies have their fresh hires do complete proofreading; others consider their unit tests to be enough documentation. Some even go so far as to hire technical writers for this. We polled for infrastructure diagrams-as-code, but no one succeeded at these either. We conclude that there is still a HUGE gap in the ecosystem, ready for disruption.

From the Technical Interviews topic, it was clear that people struggled with the amount of rounds some companies seemed to have. Also, the sometimes weird psycho-analytical questionnaires are considered a nuisance. Some experienced people had shared their trajectory going from doing white-board sessions, pair-programming, take-home-assignment, presenting hard algorithmic questions, etc... and they concluded that they always mainly went with the attitude in the first place. Hard skills (instead of soft skills) were preferred only in cases where extreme specialization was required.

Day 2

In AI, You Complete Me, Patrick Debois, the Godfather of DevOps, showed how genAI is making an impact in engineering workflows, bringing realism to where we are in practice and what is coming up next.

Besides showing what's new in editor land (e.g., https://www.trycursor.com), the thought-provoking thing is where AI Agents will be in a couple of years, how we can have generalist and specialist agents, and how to use them.

He asked the audience, "How many of you need a way to produce lines of code faster?" There was no real show of hands here; writing code is only a marginal part of our jobs, so why optimize for it? He continued to state that this is not the main power of AI. The main power is allowing people to do the one thing that AI is bad at: augment humans thinking about complex problems and reviewing solutions.

In Green Code on the Cloud: Integrating Sustainability into DevOps Practices, Paulina Dubas repeated how code efficiency, right-sizing, and caring about dark data can help achieve sustainability in software development and operations. This not only contributes to environmental goals but also enhances operational efficiency and reduces overall costs.

Platform Engineering XXL was a whole different beast. These people need a platform! SAP has 100k+ technical employees, so a platform team of 400 people created "Hyperspace," a "Platform as a Product," to guide a highly fragmented organization into Golden Paths. Even if not something we'd be doing right now (let alone at that scale), it was fascinating to see how you could. We'll be happy with base images and shared CI templates for now. 😄

In Evolving DevOps Teams and Flexible Organizational Culture, Ikuo Odanaka discussed the journey they've been on while digitalizing Japan's pharmaceutical sector. This was his first talk outside of the Japanese region. His energy was contagious, and there was a lot of shouting and laughing. Unittesteru! DEVOOOPS!

He also shared their AARRR Pirate Metrics Framework, which reminded us of our Town halls for some reason.

According to Mandi Walls, Culture Is Still A Challenge, which explains why, just as our community has been saying these past 15 years, it still boils down to culture and why it is so hard to change. There are no new takeaways, per se, but it’s very much worth it to keep repeating why it’s so important and challenging.

Open Space

In DevOps Best Practices, I hoped to discuss some actual practices, but instead, it was more about “how would you feel about a community site that tries to have them."

In Reproducible Builds, we explored the why and how of reproducible builds. Do we mean reproducibility or retention? On the topic of reproducibility for the sake of security auditing, what even is that? Shouldn’t our Docker build hosts, the container host, and all its libraries be frozen in time? The topic went philosophical quickly (with references to Reflections On Trusting Trust) and harked back to Ducy’s talk with the updated unicorn meme.

In post-mortems, we heard an interesting idea of a monthly or bi-monthly "Failure café." The idea is to review new post-mortems and discuss them. We might take this one home with us.

Last but not least, in Balance between management & devops (bottom-up versus top-down), we reviewed mistrust and misalignment with management (a recurring theme). And how we should consider “yet another silo” and that it boils down to communication (doesn't it always?). “We want to do faster deploys" doesn't resonate, but “we can do better at lead retention but we’re blocked by x, but no worries, we have the solution” can be two different messages that mean the same thing. The latter is likely more successful.

In the How to Open space, which had only six people turn up, we went over what worked for us and what did not. We went quickly into how some more introverted people are harder to reach or how tenure or level can influence the conversation, especially during regular meetings. This led to mentioning Liberating Structures, which seemed interesting as it proposes alternative methods to conclude more inclusively without requiring specialized training. There was also an agreement that each meeting should have one person who was not afraid of being fired to speak up and dispute any unreasonable or mediocre idea, no matter who coined it. Or one should have psychological safety.

Our very own Mike started the (Semi) Public Speaking topic. Upon arrival in the room, he was greeted by a room full of the brightest presenters at the conference, coaches, and even an (ex) EU parliament member. 😄 Next, there were enough other people interested in the topic that we ran out of seats to accommodate everyone.

When we asked how they started public speaking, they shared how they joined local drama/theatre groups or just started blogging and writing a lot. In the end, a good presenter is someone who is good at storytelling.

As for tips on preparing a presentation, they told us to build up experience with the tooling. How do you hold a microphone? How do you get comfortable with its weight or breathe with a boom around your ear? They rehearse by filming themselves and do several trial runs of the talk. Some have annotations inside their slide deck that indicate remaining time or when to take a sip of water. On dealing with ‘stage fright,’ they elaborated that getting training or coaching is a good help. They even, almost in tandem, concurred on going to a shrink to overcome the sickly feeling before/during presenting.

Mike got recommended to attend a local Toastmasters gathering, and people shared their lists of books on the subject.

In Dev Envs, the discussion landed on who owns maintaining things and who should monitor what. Some were of the idea that developers should be able to tweak anything and that the ops side of things should monitor and guide where possible. Others have experienced developers wanting to do just that: develop. Whether writing CI files, docker images, and thinking about these things belongs to their core value stream is debatable. We did not conclude anything, however.

Closing words

Seeing so many familiar faces and talking to so many people was excellent. We had many takeaways that made the trip well worth it. We came home energized with many new ideas and things to write down. Thank you, DevOpsDays organizers, and thank you, Silverfin, for making this possible. 🙌