Why software matters

Years ago I worked at a consulting company writing software for a German car manufacturer. This project was a multi-year endeavor and had all the bits and pieces in place to become the next generation legacy software that nobody wants to touch. We are talking about a multi-million investment and a 50 person team of engineers, business analysts, architects and managers. We were essentially building a Java EE monolith with a Javascript-backed frontend. Back then, in 2017, Kubernetes was starting to emerge as an interesting option and we experimented around with deployments, but it didn’t affect the architecture decisions too much. We continued to build a humongous monolith and the project was only set to go live after the core functionality had been built. About four years in, the team was finally ready to launch the first alpha to select customers. Yes, you read this right: we proceeded with development for a long period of time without ever testing that the architecture and resulting system could sustain the load. Internally we were doing Scrum, but only occasionally we did a hand over to fit the car manufacturer’s waterfall project planning. I left the consulting company before the system went live. […]

RestartOPs

This term describes the general operational practice of restarting backend services to overcome system failures. While popular with java deployments, this can be observed with other systems as well. I took the liberty to collect a few gems from the Internet: Restart One MongoDB Deployment (or also the manager app) DataStax OpsCenter IBM Tivoli Storage Manager Oracle SAP With all the above documentation, this obviously shows to be a widely spread industry practice that everyone should absolutely adopt right away! RestartOPs allows you to save precious time spent on debugging applications and understanding the actual problem that one is seeing. Who needs root cause analysis, when the problem simply goes away after a juicy shutdown -r now was issued. Why bother looking deeper, if one can setup that restart command in a crontab to restart your entire production stack weekly, no, daily. Hmm, why not do it hourly! This is obviously a joke! Please never do restart ops if you are serious about your production environment. The only time restarting should solve your issues when you have your parents on the phone and can’t take a closer look at the underlying problem. (Probably switched the keyboard to Japanese again) I’ve […]

The simplest CI setup ever

This weekend I started to go back and look at a few projects of mine. A few years ago I setup most of those using a mix of Travis CI for building/testing, docker hub for image building/hosting and then somehow wiring this all together with a custom stack. Today this feels outdated and overly complex. As an SRE I’m allergic to complexity. So I spent a few hours checking out the new Github packages and actions. Pricing Both packages and actions come with a unlimited option for open source projects and a nice free tier for private repositories. For actions you get 2000 build minutes per month. For packages 500MB in storage is included. There is more fine print for data transfer but that seems really generous and will mostly suffice for any small to medium size project. (Don’t mine bitcoin in the Github actions runner please :-D) Setup Github has a somewhat unfair advantage here: their stuff is build right into the UI. But that is exactly why its nice. Github has been amazing at creating clear and understandable user interfaces. Compare to the mess that docker hub is, I can only applaud them for a job well done. […]

Operating your own mail server is a pain

tl:dr; I’m going to tell you all the reasons you shouldn’t run your own mail server. I actually wanted to name this post “Thinking about running your own mail server? Here is why this is a bad idea” but I’d like for everyone to opt out of click bait titles. I stopped clicking on anything that sounds like click bait (Youtube, Twitter, News) as my brain got trained to see those as low quality time stealers. Anyway, I wanted to talk about my experience of running a mail server. Over the last 10+ years I’ve operated my own mail server on VMs hosted by netcup.de. (Not a paid ad, I promise) Their servers have decent performance and for just a few euros they are a steal. So I went and setup ispconfig – a management interface for basic web, mail and dns servers – to have a more simpler interface for configuration. Till Brehm (the maintainer of ispconfig) is doing an amazing job at developing that piece of software over the last decade. The interface is clean, neat and gives your customers basic settings to do self service if they want to. Configuring a mail server is complex Configuring your […]

Where it all started – goodbye Cybton

In my previous post I’ve mentioned my entry to programming was first via C++ and afterwards I wandered of into the wide fields of web development. A fundamental problem back then for me was that I did not have any money and that properly running a website required you to have a static IP with a Webserver that ideally could not only deliver static content like HTML, CSS and images, but also included PHP to have dynamic content. One of those hosters, that offered free webspace was cybton.com. As it did not have any ads on the pages offered through the webspace itself and had a cool community with a bonus system, that was the one I’ve opted for. The bonus system allowed you to upgrade your webspace to get more disk space and other cool features. You could earn points to “buy” those perks by actively engaging in the community. Imagine those fake internet points from stackoverflow would allow you to get a free server. It was super successful and at its peek, the community was super active and newcomers were quickly helped with getting their project started. One thing I remember quite clearly back then was the mostly […]

Chase your Dreams

At age 10, I first got to know computers by a book from my brother called “C++ for game developers” even though I didn’t have the slightest idea of coding then and the book started out pretty steep with all kinds of things, over two-three years I managed to get through to the last page. After I discovered PHP and web programming for me. It was totally contrary to what C++ had to offer: get things done, easily, quickly and without too much complexity. There I developed my first few complex web 2.0 applications. Already then I realized that I’m somewhat not a loser at this whole programming thing. And let me it tell you: it was a lot of fun! So I decided quite early at the age of 14 to go to a technical high school, that would encourage me to dive deeper into computer science. At the HTL I also learned a lot about how a computers electronic circuits work and why things make sense the way they were designed. Back then I already set my goals high to become a professional in this field. Especially my professors in Highschool encouraged me to shoot for the stars. […]

Kubernetes: proxy requests without additional pods

Sometimes you need to provide a legacy access to various downloads or proxy some requests to a different endpoint, that might not be running in your cluster. One can natively redirect such requests with having to add additional deployments / containers to your Kubernetes cluster. There is a special type of Kubernete’s service object that simply points any traffic to that external DNS name. This isn’t really document all too well but eventually you will find enough issues and pointers to frankenstein a solution together. For anybody else looking on how to do this correctly, here is run down with nginx-ingress-controller:0.19.0 that worked for me. First we create a normal ingress object, that allows us to terminate the SSL and look into the path of the HTTP request and decide if this is a request that is relevant to be proxied.  Lets quickly take a look at what is going on here. We first configure our Ingress Controller to use nginx and enable automatic provisioning via our cluster ACME/cert-manager. Then we instruct nginx to rewrite the target URL to add the bucket. This way, we can swap buckets without the user facing URL having to change. (Notice to not include […]

ZSH shortcut: quickly commit using a function

I have many private git repos to manage all the configuration files, ansible playbooks, source code and other stuff you would want to version. Most of them are just for me to view and often I want to quickly ‘save’ my current process by creating a commit I can go back to. I just recently switched to zsh after years of using bash – never had I thought that my life would change that much. So many neat little things that are really helpful and make my life easier. Themes, plugins and the oh so amazing oh-my-zsh really make it a pleasure to do stuff on the console. (I always have to bite my tongue once I SSH into a server that only has sh or bash now. So in order to commit quicker without having to write two to three commands like add, commit & push one can create a function that does all that and takes some arguments in case you need them. During my research, as any scientist should do properly before reinventing the wheel, I found Hasan has thought of the same thing a couple years back. Here is my version of it: gitc() { if […]

The road to Kubernetes: hosting your own cluster on VMs

I finally wanted to setup my own kubernetes cluster as everyone I talk to, said its the hottest shit. I’m using three VMs, hosted at Netcup running the latest Debian 9 Stretch build. I’ve installed most basic tools for me and also already set up docker using this amazing ansible role. Make sure to disable any swap you have configured – kubelet will not start otherwise. The documentation on how to install things is pretty good, but I’ve missed some details, that I banged my head on, so I will copy most snippets over for future reference. Keep in mind, that this might have already changed and is no longer working at the time you read this. First install all needed CLI tools on each of the three hosts: apt-get update && apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add – cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl Start the systemd Service for kubelet, our kubernetes manager – also on every node. systemctl enable kubelet && systemctl start kubelet Now the docs are a bit unspecific, but here’s a command to set the correct cgroup on Debian 9.   […]