Transcript

Mandi Walls:
Welcome to Page it to the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve the system reliability and the lives of the people supporting those systems. I’m your host, Mandi Walls, find me at @lnxchk on Twitter.
All right, welcome back, this week I have with me Mike Fiedler. I’ve known Mike for a long time, and we used to be sort of neighbors in New York, he lived in a building across the street from me. Other than that too, he was part of the Chef community when I worked at Chef, and now he’s working for Python. So Mike, welcome to the show.

Mike Fiedler:
Thanks, I’m really excited to be here.

Mandi Walls:
So tell us about what you’re currently doing and the kinds of things you’ve gotten yourself into these days.

Mike Fiedler:
Oh, my goodness. So what I’m doing now, so as of August this year I am employed by the Python Software Foundation, a nonprofit that is the force behind the Python language or the stewards of the Python language, and the Python Package Index, PyPI. And the work we do is supporting the Python community at large, which is all over the planet and a little bit beyond.

Mandi Walls:
What do you mean a little bit beyond?

Mike Fiedler:
Well, if you haven’t noticed, the Mars rover and the little helicopter are powered by Python. So we’ve left the planet.

Mandi Walls:
Oh, very nice.

Mike Fiedler:
Which is very cool.

Mandi Walls:
That is super cool. Years ago I went to a presentation by Vint Cerf, he used to do the interplanetary internet talk about that stuff. That’s super cool, so now you guys are actually deployed into the universe.

Mike Fiedler:
Yeah. It’s a little bit of a weird thing to think about, of just working at something as core as the Python language or Python Package Index, where I focus. Where our users are literally anyone and everyone, because there’s almost nothing digital that anybody interacts with that isn’t related in some fashion to Python these days.

Mandi Walls:
Yeah. So your personas are then infinite, thinking about what people need and what they’re doing, and taking in all those various requirements, must be a crazy challenge.

Mike Fiedler:
It really is. And Python is about 30 years old now, that’s one of the older popular languages today. When you think about the other software programming languages, many of them are much younger and newer, and very often they’ll look to Python to see kind of what mistakes we may have made along the way to try and avoid them. Which is cool for them, but it’s hard for us as we kind of look at our legacy and our backwards compatibility requirements, and how we want to provide different user use cases.
Python is heavily used in web and all other kind of user facing applications, but it’s also heavily used in the scientific community that have vastly different concerns, constraints, programming styles, but they all want to use this language that they find very approachable. One of the biggest strengths of Python is not just the language and the simplicity of using it, but the very large standard library that the language has. This is where Python often has its batteries included, where you can accomplish a large amount of things with just the language and the standard library. But also the package community, the Python Package Index, P-Y-P-I, Pypi.org. Which open source maintainers around the world are pretty much producing packages that do core functionality, that are kind of like picking up, not Lego blocks off the floor and putting them into your applications, but more fully formed pieces of Lego blocks, to kind of mash together and use in order to accomplish the goals that you’re trying to set out.

Mandi Walls:
Yeah. And how many are there now? How many packages are there?

Mike Fiedler:
I mean, we’ve got a stat that updates once a day, because it’s a computationally expensive query. But there are just under half a million projects, 490,000 projects with over 5 million releases.

Mandi Walls:
Wow.

Mike Fiedler:
Yeah. Does that mean every project has 10 releases? No, and some projects only have one, but some have a lot more, so on average. But yeah, a given project over its lifecycle can have many releases, and people are releasing new versions all the time. And I work on the package index that hosts all of these different packages and provides them to the different client side installers. We talked about different personas or use cases. The way Python has historically worked is through the Python Enhancement Proposal process or PEP, some places would call it an RFC or those kinds of processes, in order to publish the ideas that they want to change, kind of provide their research or motivation, what other things they’ve thought of. And there’s even a section in there called How to Teach This, which is a good prompt to say, “When this rolls out to the wider public, how do you actually talk about this change? And what do we want people to walk away when they hear about this change?” Which I find a great kind of part of the PEP process.
The reason I bring up the PEP process, is the package index is 20 years old, and it’s gone through a couple of different iterations of its existence. The most recent big rewrite happened around 2015 to 2018, and launched in 2018, and that’s what we have today, it’s called the Warehouse. It provides some interfaces and metadata about packages for client side installers, but there’s a lot of client side installers. The most common one people are familiar with is PIP, but there’s others. There’s PDM, there’s Poetry, there’s Pipenv, there’s Conda, and all of these tools are operating on a contract as defined by a PEP. So it’s not the case that, oh, there’s too many installers. No, there’s installer contracts. There’s one basically that we all follow, kind of like the RFC for TCP. You don’t have to have the same TCP library as I do, but because we’ve agreed on a contract, we can communicate effectively.
And that’s how a lot of the Python packaging universe works, is it’s a bit of a slower approach, because we have to accommodate all of the different use cases and client side installers, because we promised a contract with them. And that alleviates a lot of concerns when it comes to, how do you mature a given tool? Because we’ve defined a contract, so we just have to stay close to that contract, or very loud and slowly change it over time, so that way that gives all of the other maintainers of those client side projects a chance to update and move forward.

Mandi Walls:
Yeah. The install base of Python worldwide seems for a human brain mostly infinite. And thinking about how long it took some folks to get from older versions of Python onto newer versions of Python, and we’re still occasionally talking to folks that are like, “Oh, is this library going to be updated for the latest couple of versions?” And all those kinds of things like that. That ecosystem sprawl is immense for a language like Python. As like you say, when I worked at NIH for a while, the folks there are on a much different schedule for how they develop scientific libraries and those kinds of things, versus what folks are doing for interacting with a PagerDuty API with our library. The timescales are vastly different for how often they’re going to update anything.

Mike Fiedler:
Yeah. I think it’s a very valid kind of observation call out, because it’s very easy to fall into the trap of, well, just update your libraries and push a new version. And coming from a web operations background, it’s like, “Okay, that’s relatively easy. Update your packages, run through your tests, make sure you’re happy with it, and then push it.”
But when you’re shipping code to a satellite, or a rocket, or somewhere that is not as easily updateable, say a Large Hadron Collider, if you will, they’re going to approach the update or process with probably some different constraints or… I’m not going to say with more rigor, but they will be different than what others think they are going to be presented with. Like the ability to push code to GitHub publicly and then have GitHub actions run a bunch of steps for me, and then I cut a release, and then I don’t have to worry about past releases, that’s great as an open source maintainer of a project. But commercial companies, they may have some policy that says they have to support three versions back, so how are they going to maintain different library compositions? So it becomes a fun but challenging landscape to consider when making any kinds of drastic changes.

Mandi Walls:
So how did you get into this last one? You said you’ve been there since August, so how did you find this particular job? How’d you sneak into this?

Mike Fiedler:
Well, this is a bit of a weird one for me. You mentioned that we met through Chef. I came up on the operation side of the house, I wasn’t a software engineer or a software developer, I didn’t study that, I came up kind of helping keep things running fast and scale. That’s where I got my first taste of Python back in about 2007, 2008, when as part of a team we were migrating some of our Nagios checks, which we used to monitor our systems and architecture. And the language that Nagios was very happy to use was Perl. And if you remember, you had to walk through Perl and deal with all of the complexities of that language, and kind of do your thing in Perl. And then a version of Nagios came out that supported Python, and that was like, “Oh, all right, I can actually try and write readable code now.”

Mandi Walls:
Hey, now.

Mike Fiedler:
My Perl was very much unreadable, but the Python was much better. So I got to step into this foray of I can start writing some of these checks in Python, in a language that more people on the team can read. And then as I kind of moved throughout my career since then, I drifted away from Python and got into Chef. Chef is very much a Ruby based language, DSL on top of Ruby, but an excellent tool set. And where I found it most helpful for me, was it started putting me in the mindset of software development, as opposed to purely understanding the systems, and the operating system, and the hardware constraints. I started to understand how to compose software through systems tooling. So it was a really cool bridge, if you will, between these two worlds for me.
And becoming a maintainer of a bunch of cookbooks was great, but then delving into some of the tooling, I wrote some Knife plugins, I contributed back to Chef Core once, maybe twice. But it was a fun challenge to understand how these things work, puzzle apart all the pieces, and then figure out, what can I put in here that I wanted to make happen for whatever challenge I was faced with?

And that approach to dynamic programming languages, Ruby, helped me come back to Python a couple years later and say, “Ah, all this makes a ton more sense right now.” Even to the point where the Pragmatic Object-Oriented Design in Ruby book, the POODR book, P-O-O-D-R, by Sandi Metz, excellent book, was like, “This is a great understanding of how to write software in Ruby.” And it’s like, “All right, it’s not a huge jump to understand the same principles in Python.” So when I was working through different jobs in my career, I fluctuated between individual contributor and manager. And as a manager, you very often don’t get to code as much as you used to. And it’s super rare, unless you’re a really tiny startup and you’re very much in it with the team, a very small team. But then you’re probably shirking some of your management responsibilities.
So I had to focus on management, so I did that. But as a way to kind of continue my learning, my passion, I kept going with open source, and I’ve now been contributing to my GitHub account for over a decade. And it’s been fun to kind of build up different tools, different processes, different things over time, and a lot of them are in Python now. I started contributing to the Warehouse code base a few years ago in JavaScript, there’s a little bit of JavaScript in there. And I wanted one feature on the website, and JavaScript was the way to implement it. So I came into Python through JavaScript, the Python Package Index through JavaScript. I met some of the maintainers through GitHub Issues, and they were really awesome and welcoming. And after contributing a variety of different patches, and changes, and fixes, I delved heavier into the Python code base and cleared up deprecation warnings. Upgraded us from I think it was Python 3.8.0 to 3.10.0, and then quickly 3.11.0. Saw massive CPU decrease, which was excellent, just by upgrading from Python 3.10.0 to 3.11.0.
So that’s just a big shout-out to the core Python team and the Faster Python Project, that are continuing to push the envelope of, how fast can we make this thing? Which is really cool to see over time. I had become a volunteer maintainer, just working on it as part of a passion project. And I had taken a career break in January, and spent a lot of time just working on the Warehouse. And a job role opened up, and they published it widely, other folks interviewed as well, and I landed the gig. And it’s been really cool to work on specifically security and safety for the code base and the ecosystem, because that’s what I’m focused on. The role would not be possible without the generosity of Amazon Web Services, who’s funding a large part the role for a year, and hopefully they’ll do it again.

Mandi Walls:
Oh, that’s excellent, super. So let’s talk about the safety and security part for a bit. Python traditionally doesn’t have the… I hate to pick on Node, but I’m going to pick on Node. Because holy heck, every now and then you get these wild stories out of the Node community. And they’re like, “Something’s been poisoned in the well over there.” And you’re just like, “Could you not fix this?” But the methodology over there seems to not fall in that direction. But as we’ve talked about with Python, it’s used so many different places that are mission-critical. I know it’s used by medical researchers, it’s used by folks where their main mission is life critical kinds of systems. So I’m assuming that safety and security are a whole lot more important than maybe some front end code might be when things kind of go bad other places.

Mike Fiedler:
So I think you’re absolutely right. The use cases for Python are wide and varied, and there’s no way we can enumerate all of them. I can’t say for certain what software medical places are using, but I’m pretty sure Python’s everywhere. Even if it’s part of their build process, it’s involved in some fashion. And the hard part with a lot of open source stuff is this notion of trust, who do you trust? When and why, why should you trust the Python language? Why use it? Well, you can trust that it is built and published in the transparency, in the open. My colleague, the security developer in residence, Seth Larson, also at the Python Software Foundation, recently published a blog on the Python build process, where visualized. Where he kind of takes apart the process of publishing the Python language runtimes and the core CPython runtime, not the language, but the CPython runtime.
And he calls out, there’s a couple of weaknesses in the process, and here’s how he’s thinking about addressing those and making those less of a risk. Because security is never done, it’s never, “Okay, we are secure now.” I wish that was a thing. Even bank vaults, they lock them, they have their alarm, but those cadre of hooligans are going to get into that bank vault. So it’s a matter of an ongoing investigation always of, what is out there, what could we do, what should we do? And treading the line between, okay, we could wall all of this off in six feet of concrete, but then no one can get to it, no one can use it, it’s useless. So if that is the most secure, and the most insecure is nothing, it’s a spectrum. Somewhere in the middle we have to figure out what we’re willing to trade off, what we’re willing to compromise, and when?
So when it comes to security of the runtime, Seth has done a great job of detailing what we do there. But again, Seth is just pointing it out, it’s been built up by tons of volunteers and the core developers over time. So kudos to all of them for working on this. And then beyond that, we started getting into the packages. And we start to ask ourselves, “Why would we use a package? When do we use a package? Which packages do I trust, and why?” So very often you’ll get a, “Well, I trust this package because it has lots of stars, lots of other people like it.” And it’s like, “Okay, that’s one way to go about it.” You could go read the code and say, “All right, it does what it says on the tin, and it solves my problem so I don’t have to, so I’m going to therefore use it.”
And that’s usually a really good one-time touchpoint of trust. So if you were to take Package Foo and try it out, and you love Package Foo version 1.0, great, you’ve read all the code in Package Foo. Now version 2.0 comes out, and you don’t know what’s in there, you could read the release notes, or you could read all the lines of code. It becomes very expensive very quickly. So what a lot of folks do is they’ll look at other signals and say, “Well, was it published by the same publisher? If Mandi published Package Foo version 1, did Mandi also publish package version 2?” And there’s a variety of different signals folks use to determine if they are secure or not. And we partner with a bunch of different security research teams that will follow the stream of releases from PyPI, check out those packages, and tell us if something is wrong with them.
So we have basically malware searchers, or I don’t know if they want to be called malware searchers, vulnerability finders, if you will, sounds nicer. And they’re doing an excellent job of finding stuff, reporting it. And we are trying, as always, again, as my role, of how to make that easier for them to report and faster for us to react, because those malware packages have the potential to hurt something.
The good news is that with some of the features that we have implemented over the past couple of years, getting malware into a very popular or previously trustworthy package is increasingly harder. So you can create Package Foo, a random package and put it up there with some malware in it. If no one’s using it, then it’s not as bad a problem. It’s still a problem, because you don’t want ticking time bombs just sitting around, so we do definitely take those down. But for the bigger or more established projects, it’s finding out, how is this project being published? If it’s being published from my laptop, then my laptop becomes the security weakness point. And that’s where we introduced trusted publishers, which was developed thanks to a grant, which I don’t remember who funded it. But it’s a mechanism by which tools like GitHub Actions can use the OpenID Connect protocol to request a short-lived token to publish from them to PyPI. So there’s no long-lived token just sitting around that someone could leverage.

Mandi Walls:
Sure. So that’s a benefit for folks worried about their software bill of materials, and all that kind of stuff as it [inaudible 00:21:12].

Mike Fiedler:
Yeah. It starts to unlock a lot of those fun supply chain concerns, where if we can provide methodologies, tools, approaches for software builders to easily supply software bills of materials, attestations, signatories that are verified through public transaction logs, then that increases the overall trust chain. That what the people who put something in a package hasn’t been changed by the time it gets to you.
That still doesn’t account for what they put in that package to begin with, but again, that’s that first round of due diligence that most folks need to do. Of is this the package that I want? Does it do what I think it’s supposed to do? Has it been kind of vetted by others? And if it has, then sure, I’ll give it a try. And I think providing these mechanisms, and tools, and approaches for teams of teams, and companies, and individual users across the landscape helps mitigate that concern of open source dependencies being just splat out by everyone. They are still splat out by everyone, but if Bob who created some package decides they don’t want to do it anymore, what happens after that? And how do we mitigate risks of Bob’s popular package going away?

Mandi Walls:
Yeah, succession planning for open source projects. I mean, we’ve all been there. They have things that get written, and life happens, and people either get bored, or they have to use something else at work and they’re no longer knee-deep in this particular kind of environment, so they don’t feel particularly drawn to it anymore, and they just move on. And when your enterprise is dependent upon the kindness of strangers to maintain your software, there’s some things that get lost in the process.

Mike Fiedler:
Yeah. I mean, for enterprises I can recommend that any enterprises or CTOs listening to this podcast, go check out Tidelift, Tidelift.com. It’s an excellent program that helps support financially open source maintainers through corporations who subscribe to their subscription. It helps mitigate some risk, it’s not perfect, nothing is perfect, but it helps kind of push that message further, so that way anyone using open source packages that are covered by a Tidelift subscription have some degree of confidence that a security incident will be handled within a certain amount of reasonable time.

Mandi Walls:
Yeah, absolutely. Tidelift is super cool. They were on… And I think we’ve got a friend over there. But they were on a Gartner cool products list, I think last year. Yeah, super interesting on that one. Oh yeah, Jeremy Katz is over there at Tidelift, so excellent.

Mike Fiedler:
Yeah. I think open source and security is something that is coming even more in the forefront into the United States government right now.

Mandi Walls:
Absolutely, yeah.

Mike Fiedler:
The Cyber Incident Security Administration. I don’t remember exactly what CISA stands for, we can google it, CISA. But they recently, two months ago, had put out a request for information, to understand for folks like us to respond to the government on, “Here’s what we think are important parts of the open source universe.” How to reduce classes of vulnerabilities at scale. They just want to know what we’re thinking about, and what we think they could help with. So we, and as well as many other organizations, responded to this request for information recently. In hopes that these will become topics that governments, regulators and funders will understand, “Here are some things that are important to fund and kind of put some effort behind.” So that way the public or common goods of open source remain public, and common, and helpful to everybody.

Mandi Walls:
Yeah, it’s come a long way in the last couple of decades.

Mike Fiedler:
It really has.

Mandi Walls:
That we’re at the place where the US government is soliciting these kinds of comments. When before that in the early two thousands, that seemed impossible, an impossible dream that we’d ever get to the point where there would be the desire or the impetus to have an actual kind of program like that in the government.

Mike Fiedler:
Yeah. I can’t figure out when we kind of hit the inflection point in open source, because you remember enterprises would buy software solutions from companies, big enterprises, from other big… IBM, and Red Hat, and those kinds of folks. And open source was kind of a secondary, an afterthought, to get some inspiration from. And then engineers inside companies would be writing all of the code for that company, and none of it would ever see the light of day outside of that organization.

Mandi Walls:
No, and we’d even talked to teams when I was at Chef in the earlier days, they’re like, “Oh, no, we’re not actually allowed to use open source software.” And you’re like, “What kind of crazy business are you running where you’re paying for absolutely everything? That’s just absolutely wild.”

Mike Fiedler:
It came with a different set of trade-offs. They were trading off money for support contracts, not necessarily better software.

Mandi Walls:
No, in a lot of ways, not at all.

Mike Fiedler:
What I think happened was many… And again, this is my perspective. I’m by no means a software historian, or know how the world has evolved. But it seems like the evolution happened when folks were starting to see that there were some really good open source tools out there. I’m going to go after the Apache HTTPD web server, it was an awesome free to use web server that did a whole lot for you that you then didn’t have to do. You were back when AOL had the AOL web server, right?

Mandi Walls:
Oh, yeah, it’s in TCL, right? Yeah, I ran AOL’s server for several years before we switched over to Apache and Tomcat.

Mike Fiedler:
Yeah. It was great for AOL, it did everything that AOL wanted, but if you wanted anything that AOL hadn’t written, you weren’t going to get it. You would have to engage the web server engineering team, and file a feature request, and it probably is not going to get prioritized, because nobody’s got time for your crazy thing. But with the Apache HTTPD web server project, it was open. If you wanted to add a feature, you could read through their contribution policy and submit a feature request, and then submit the code that implemented the feature request. That was unheard of.

Mandi Walls:
Yeah. And the modules too, they’re an easier on-ramp, even if you weren’t going to get into the core code. You add your module, whatever it ends up being, and all of a sudden there’s this new set of functionality in this free to use open source software. And you’re like, “Why would you buy a web server at that point?” You’re just not going to.

Mike Fiedler:
Right. And it started this kind of wave of companies bundling some open source software, some custom written code, to be a solution. That’s where I think this wave of open source technologies has really boosted. I think one of the things that stood out to me was back when I was at Datadog, very early on we were asking people to install an agent on their servers. And it was very important to us, and still is at Datadog… I’m not there anymore, I haven’t been for some years.
But the agent, that any of the code that we were asking customers to run, had to be open source and auditable by them. They needed to be able to see every single line of code that we were asking them to put on their machines, because what if their security policy said, “We don’t allow.” Well, we wanted to be able to get into those companies by saying, “The things you see are open source, the things that we’ll be running. We’re not asking you to install a keylogger or a sniffer on your machine, we’re asking you to install observability tools.” Which are kind of sniffers. “But specifically targeting your application stack and stuff like that.”
And then that was just an excellent conversation starter, say, “You know what? This other BigCo enterprise is asking you to install something, you don’t know what’s in that, I don’t know what’s in that. I could sit there and reverse engineer it and poke at it, but I don’t have that kind of time.” But this commitment to open source really helped a lot of companies bridge that gap to say, “I’m willing to try this because of how you’re going about it.”

Mandi Walls:
Yeah, excellent. Well, sir, this has been a fascinating conversation, and a nice trip down memory lane a little bit too. One other thing that we mentioned at the beginning, in addition to your current job, you’re also an AWS hero. And we’re recording this the week before re:Invent, I think it will go out just after re:Invent. So what’s it like being an AWS hero?

Mike Fiedler:
Yeah, I get that occasional question of, “What is a hero? How do you become a hero?” So for context, Amazon Web Services or AWS, hopefully all of your listeners know what AWS is, but if they don’t, it’s an excellent cloud provider. And they have a internal program which recognizes people out in the world, in the community, not employees of AWS. You literally can’t be an employee of AWS and still be a hero. They’ve hired out of the hero program, and that’s how you lose your hero status. There’s about 250 of us worldwide in all sorts of disciplines, it’s not a very large group. But it’s recognizing individuals in areas of expertise about AWS or AWS technologies. And the best way to become a hero is to not want to be one, it’s one of those things you kind of get tapped on the shoulder, of like, “Hey, want into the club?” If you want into the club, it’s very hard to get in. But if you just do what you’re doing, you get tapped, and it’s like, “That’s kind of cool.”
It provides us this really clear access to service teams when they are looking for feedback on particular features, ideas, or if we have some critique that we’d like to provide them outside of the public eye, we can kind of fast channel that and be a little more candid and honest than what we would probably do publicly. But then on the outside of that, very often we can speak knowledgeably about a lot of the AWS technologies that we interact with, and provide our customers, our partners, our friends with suggestions and ideas. Like I hang out in the Hangout Slack, and there’s a very active AWS channel, and occasionally I’ll have the answer for somebody looking for, “How do I do this?” A recent one was, “How do I delete all of my versions of S3 files?” And you can do that about three to four different ways, but there’s one that doesn’t cost you a whole lot and is pretty easy. So I suggested using an S3 lifecycle policy, and hopefully that works out for them and they don’t end up with footing a bill for deleting millions of objects.
But the hero program is really cool, at re:Invent there’s going to be a fair amount of heroes speaking at different talks, in different venues, there’s a developer lounge in the expo hall. Again, if you’re listening to this, it’s probably after re:Invent, and maybe we hung out there. So if we did, high-five from the past into the future.

Mandi Walls:
Yeah, right. It’s time travel, is what it all is.

Mike Fiedler:
Yeah.

Mandi Walls:
Awesome. Well, excellent. It has been great to talk to you, and catch up, and learn all the insides and outsides of Python, and all that great stuff there. So where can folks find you now? Where are you in the social media landscape these days?

Mike Fiedler:
Oh, it’s getting crazier. The easiest way is to go to Mike.Fiedler.me, that is my website, it will link you to all the other ones. But if you’re on Mastodon, I’m [email protected]. If you’re on GitHub, I’m Mike the Man on GitHub. Generally, I try to be Mike the Man everywhere, but that’s where you can find me. I’m on HangOps, and a bunch of other slacks that are out there.

Mandi Walls:
Sweet, all right. Well sir, thank you so much. I’ll put a bunch of the things that we talked about in the show notes for folks too so they can find them, and can find you wherever you are. And thanks for being on the show this week, Mike, this has been great.

Mike Fiedler:
Thanks Mandi, it’s been awesome.

Mandi Walls:
Awesome. All right folks, we’ll be back again in a couple of weeks with another show. In the meantime, we’ll wish you an uneventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor, PagerDuty, for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at Pageittothelimit.com, and you can reach us on Twitter @pageit2thelimit, using the number 2. Thank you so much for joining us, and remember, uneventful days are beautiful days.