I wanted to do some REST client stuff in PHP with cURL today. It has the neat CURLOPT_PINNEDPUBLICKEY option that one can use to set the certificate fingerprint. Unfortunately its a bit cumbersome, as one needs to enter the fingerprint in base64, instead of just the format the browser displays if you look at the cert directly (e.g.: 529728d7c43746c0bb02ac4a4c3bffb7028ac1591ecb08b6a0721a660aa1d3ce). Luckily we can just get the correct fingerprint format using this one command: openssl x509 -pubkey < cert.crt | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 You can download the cert using Chrome or Firefox and then use the openssl command above to get the correct base64 formatted string (e.g.: WcHR5sNo7T3ZDL6b/dLgFFETh4tdkm8SAHXfqKLo9wo=). The same works for the Symfony HTTP Client library that also expects the same format (that is further passed on to cURL) for the peer_fingerprint options entry. The official docs unfortunately do not mention this at all and just replace the formatted value by ‘…’. Oh well.
Silos and Knowledge Sharing
This blog post is a reworked chapter from my master thesis with the title “Site Reliability Engineering in a Transformative IT Landscape – Transitioning from Monolith to Microservices” at TUM submitted on the 15.12.2018. I wrote this thesis while working as an SRE at eGym in Munich, observing their technological transformation and how they applied SRE in practice. Introducing a microservice landscape to an existing system is not only a challenge on the technical side. It especially requires a change of mind in the way our development processes works. eGym already utilized agile processes for quite some time. There are small teams with a maximum of 6-8 people that are focused around certain aspects of the business. The way teams are set up, also affects the way the system architecture will eventually look like. Organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations Conway’s law That statement is from the late 60s and even today perfect fits the microservice architecture paradigm. If we as an organization are destined to develop architectures that coincide with our communication structure, then we should organize our teams in a way that allows them […]
Why software matters
Years ago I worked at a consulting company writing software for a German car manufacturer. This project was a multi-year endeavor and had all the bits and pieces in place to become the next generation legacy software that nobody wants to touch. We are talking about a multi-million investment and a 50 person team of engineers, business analysts, architects and managers. We were essentially building a Java EE monolith with a Javascript-backed frontend. Back then, in 2017, Kubernetes was starting to emerge as an interesting option and we experimented around with deployments, but it didn’t affect the architecture decisions too much. We continued to build a humongous monolith and the project was only set to go live after the core functionality had been built. About four years in, the team was finally ready to launch the first alpha to select customers. Yes, you read this right: we proceeded with development for a long period of time without ever testing that the architecture and resulting system could sustain the load. Internally we were doing Scrum, but only occasionally we did a hand over to fit the car manufacturer’s waterfall project planning. I left the consulting company before the system went live. […]
RestartOPs
This term describes the general operational practice of restarting backend services to overcome system failures. While popular with java deployments, this can be observed with other systems as well. I took the liberty to collect a few gems from the Internet: Restart One MongoDB Deployment (or also the manager app) DataStax OpsCenter IBM Tivoli Storage Manager Oracle SAP With all the above documentation, this obviously shows to be a widely spread industry practice that everyone should absolutely adopt right away! RestartOPs allows you to save precious time spent on debugging applications and understanding the actual problem that one is seeing. Who needs root cause analysis, when the problem simply goes away after a juicy shutdown -r now was issued. Why bother looking deeper, if one can setup that restart command in a crontab to restart your entire production stack weekly, no, daily. Hmm, why not do it hourly! This is obviously a joke! Please never do restart ops if you are serious about your production environment. The only time restarting should solve your issues when you have your parents on the phone and can’t take a closer look at the underlying problem. (Probably switched the keyboard to Japanese again) I’ve […]
The simplest CI setup ever
This weekend I started to go back and look at a few projects of mine. A few years ago I setup most of those using a mix of Travis CI for building/testing, docker hub for image building/hosting and then somehow wiring this all together with a custom stack. Today this feels outdated and overly complex. As an SRE I’m allergic to complexity. So I spent a few hours checking out the new Github packages and actions. Pricing Both packages and actions come with a unlimited option for open source projects and a nice free tier for private repositories. For actions you get 2000 build minutes per month. For packages 500MB in storage is included. There is more fine print for data transfer but that seems really generous and will mostly suffice for any small to medium size project. (Don’t mine bitcoin in the Github actions runner please :-D) Setup Github has a somewhat unfair advantage here: their stuff is build right into the UI. But that is exactly why its nice. Github has been amazing at creating clear and understandable user interfaces. Compare to the mess that docker hub is, I can only applaud them for a job well done. […]
Operating your own mail server is a pain
tl:dr; I’m going to tell you all the reasons you shouldn’t run your own mail server. I actually wanted to name this post “Thinking about running your own mail server? Here is why this is a bad idea” but I’d like for everyone to opt out of click bait titles. I stopped clicking on anything that sounds like click bait (Youtube, Twitter, News) as my brain got trained to see those as low quality time stealers. Anyway, I wanted to talk about my experience of running a mail server. Over the last 10+ years I’ve operated my own mail server on VMs hosted by netcup.de. (Not a paid ad, I promise) Their servers have decent performance and for just a few euros they are a steal. So I went and setup ispconfig – a management interface for basic web, mail and dns servers – to have a more simpler interface for configuration. Till Brehm (the maintainer of ispconfig) is doing an amazing job at developing that piece of software over the last decade. The interface is clean, neat and gives your customers basic settings to do self service if they want to. Configuring a mail server is complex Configuring your […]
Where it all started – goodbye Cybton
In my previous post I’ve mentioned my entry to programming was first via C++ and afterwards I wandered of into the wide fields of web development. A fundamental problem back then for me was that I did not have any money and that properly running a website required you to have a static IP with a Webserver that ideally could not only deliver static content like HTML, CSS and images, but also included PHP to have dynamic content. One of those hosters, that offered free webspace was cybton.com. As it did not have any ads on the pages offered through the webspace itself and had a cool community with a bonus system, that was the one I’ve opted for. The bonus system allowed you to upgrade your webspace to get more disk space and other cool features. You could earn points to “buy” those perks by actively engaging in the community. Imagine those fake internet points from stackoverflow would allow you to get a free server. It was super successful and at its peek, the community was super active and newcomers were quickly helped with getting their project started. One thing I remember quite clearly back then was the mostly […]
Chase your Dreams
At age 10, I first got to know computers by a book from my brother called “C++ for game developers” even though I didn’t have the slightest idea of coding then and the book started out pretty steep with all kinds of things, over two-three years I managed to get through to the last page. After I discovered PHP and web programming for me. It was totally contrary to what C++ had to offer: get things done, easily, quickly and without too much complexity. There I developed my first few complex web 2.0 applications. Already then I realized that I’m somewhat not a loser at this whole programming thing. And let me it tell you: it was a lot of fun! So I decided quite early at the age of 14 to go to a technical high school, that would encourage me to dive deeper into computer science. At the HTL I also learned a lot about how a computers electronic circuits work and why things make sense the way they were designed. Back then I already set my goals high to become a professional in this field. Especially my professors in Highschool encouraged me to shoot for the stars. […]
Kubernetes: proxy requests without additional pods
Sometimes you need to provide a legacy access to various downloads or proxy some requests to a different endpoint, that might not be running in your cluster. One can natively redirect such requests with having to add additional deployments / containers to your Kubernetes cluster. There is a special type of Kubernete’s service object that simply points any traffic to that external DNS name. This isn’t really document all too well but eventually you will find enough issues and pointers to frankenstein a solution together. For anybody else looking on how to do this correctly, here is run down with nginx-ingress-controller:0.19.0 that worked for me. First we create a normal ingress object, that allows us to terminate the SSL and look into the path of the HTTP request and decide if this is a request that is relevant to be proxied. Lets quickly take a look at what is going on here. We first configure our Ingress Controller to use nginx and enable automatic provisioning via our cluster ACME/cert-manager. Then we instruct nginx to rewrite the target URL to add the bucket. This way, we can swap buckets without the user facing URL having to change. (Notice to not include […]
Kubernetes: get service account secret for dashboard login
To login to the Kubernetes dashboard, you need a token. This token can be retrieved with the following command: kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep <admin-user> | awk ‘{print $1}’)