Tech Debt Burndown Podcast Series 1 E6: The Flow

Tech Debt Burndown Podcast

What Does Your Pipeline Look Like?

Recording date: May 1, 2021

Download at Apple Podcasts, Google Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

‘People often escape from a role before decisions they have made have a chance to catch up to them.’ - Chris

Let’s talk about pipeline. Lots of things get improved by building continuous delivery pipelines and then going back to them and iteratively making improvements. But Chris thinks that pipelines themselves can be a place where we accrue tech debt, even at the entry point to those pipelines.

So, Chris has been looking at branching strategies.

If you look at open source development, people agree that trunk based strategies are the way to go, so we’re taking short terms branches or forks off a trunk, but coming back to it quickly. But engineers from an enterprise background have a range of versions across different customers. This quickly runs to vast differences between versions, and lots of tech debt.

What suffers? Updates like security patches, and the engineers, who have to work across a multitude of branches as opposed to being on a constant edge of the trunk version, is pure tech debt.

Nick mentioned that this seems to point back to engineer flow - if your engineers can’t get into the groove because they’ve got no institutional memory and they have to constantly figure out where they are in order to fix things, they’re actually creating tech debt while thy are addressing old tech debt!

It’s interesting to think how we got here, but it’s true that, because people move around the industry, people, Chris notes, often leave a role before decisions they have made have a chance to catch up to them. So, decision logging becomes super important. But outside the code itself, the log and commit log that people add are critical pieces of context for the next people.

All too often, though, they say things like, “Merge to master” or the dreaded, “Fixes”.

After that, the readme and changelogs that go with things should tell a story, but there’s a more macro scale look at decisions people make: what were the tradeoffs that people had in mind when they made their decisions? This is absolutely critical to know. Because when we make decisions, thinking we can back out of them< we sometimes find ourselves in architectural cul de sacs.

The conversation turns to Type 1-Type 2 decisions which Jeff Bezos began to discuss back in 1997. Nick gives as an example of what’s intended to be a reversible decision the log pipeline that his teams are working on. Chris Mentioned that pipeline engineering may seem to be less worthy work done, but Chris and Nick both feel that pipeline enabling work is among the most important in an organization.

In carrying on the new tradition of references related to what we’re discussing, Chris comes up with two important areas of research: James Governor’s work on RedMonk about Progressive Delivery, and Netflix’s work on Chaos Engineering.

Nick mentioned also the SLSA: Supply-chain Levels for Software Artifacts proposal, which is an amazing project available now on GitHub. The SLSA (pronounced ‘Salsa") project is “inspired” by how Google does things (and has been at least updated by at least some people who work at Google), and ultimately will have a full set of standards, a catalog of technical controls, and a process for accreditation - but for now it’s a diabolically useful set of principles and definitions that are really worth a lot of time.

One major point of the project is to be able to know and be certain about the provenance of code at every stage of the pipeline - at the very least to be able to attest to the sanctity of code, but also to, say, stop unauthorized people from being able to insert code into codebases (Heloooooo, Codecov Hackers!). Nick and his colleagues have begun to use SLSA as a great way - at a very minimum - to agree on a taxonomy and lexicon, and as milestones on their journey. Chris and Nick agreed that SLSA should be an episode in its own right, hopefully with a guest involved in its management to discuss it (hint, nudge, wink).

The episode concludes with a pivot to a blog post by Signal’s Moxie Marlinspike who took on the mobile device forensics company Cellebrite. Cellebrite had earlier claimed it had “broken” Signal encryption, which was utter balderdash. In the information security community, the document quickly was called the Moxie Manifesto but it should be read as nothing short of a declaration of war against Cellebrite. Cellebrite customers (like law enforcement) are casualties of the war.

Moxie shot back with what may be mistaken for a lighthearted report but which is both ruthless and credible based on his demonstrated exploitation of a Cellebrite box:

• Cellebrite uses very old software components as it assembles its tools, and has been lax about security patches for years;
• This is very plausible because the threat model faced by a Cellebrite box is one of someone with physical access to the box
• Cellebrite uses unlicensed components, with specific example of Apple Dynamic Link Library files; and
• Moxie claims that “Industry-standard exploit mitigation defenses are missing, and many opportunities for exploitation are present”

Chris and Nick were both very interested by the manifesto and think it will be have a long tail.