Silos and Knowledge Sharing

This blog post is a reworked chapter from my master thesis with the title “Site Reliability Engineering in a Transformative IT Landscape – Transitioning from Monolith to Microservices” at TUM submitted on the 15.12.2018. I wrote this thesis while working as an SRE at eGym in Munich, observing their technological transformation and how they applied SRE in practice. Introducing a microservice landscape to an existing system is not only a challenge on the technical side. It especially requires a change of mind in the way our development processes works. eGym already utilized agile processes for quite some time. There are small teams with a maximum of 6-8 people that are focused around certain aspects of the business. The way teams are set up, also affects the way the system architecture will eventually look like. Organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations Conway’s law That statement is from the late 60s and even today perfect fits the microservice architecture paradigm. If we as an organization are destined to develop architectures that coincide with our communication structure, then we should organize our teams in a way that allows them […]

RestartOPs

This term describes the general operational practice of restarting backend services to overcome system failures. While popular with java deployments, this can be observed with other systems as well. I took the liberty to collect a few gems from the Internet: Restart One MongoDB Deployment (or also the manager app) DataStax OpsCenter IBM Tivoli Storage Manager Oracle SAP With all the above documentation, this obviously shows to be a widely spread industry practice that everyone should absolutely adopt right away! RestartOPs allows you to save precious time spent on debugging applications and understanding the actual problem that one is seeing. Who needs root cause analysis, when the problem simply goes away after a juicy shutdown -r now was issued. Why bother looking deeper, if one can setup that restart command in a crontab to restart your entire production stack weekly, no, daily. Hmm, why not do it hourly! This is obviously a joke! Please never do restart ops if you are serious about your production environment. The only time restarting should solve your issues when you have your parents on the phone and can’t take a closer look at the underlying problem. (Probably switched the keyboard to Japanese again) I’ve […]

The simplest CI setup ever

This weekend I started to go back and look at a few projects of mine. A few years ago I setup most of those using a mix of Travis CI for building/testing, docker hub for image building/hosting and then somehow wiring this all together with a custom stack. Today this feels outdated and overly complex. As an SRE I’m allergic to complexity. So I spent a few hours checking out the new Github packages and actions. Pricing Both packages and actions come with a unlimited option for open source projects and a nice free tier for private repositories. For actions you get 2000 build minutes per month. For packages 500MB in storage is included. There is more fine print for data transfer but that seems really generous and will mostly suffice for any small to medium size project. (Don’t mine bitcoin in the Github actions runner please :-D) Setup Github has a somewhat unfair advantage here: their stuff is build right into the UI. But that is exactly why its nice. Github has been amazing at creating clear and understandable user interfaces. Compare to the mess that docker hub is, I can only applaud them for a job well done. […]

Operating your own mail server is a pain

tl:dr; I’m going to tell you all the reasons you shouldn’t run your own mail server. I actually wanted to name this post “Thinking about running your own mail server? Here is why this is a bad idea” but I’d like for everyone to opt out of click bait titles. I stopped clicking on anything that sounds like click bait (Youtube, Twitter, News) as my brain got trained to see those as low quality time stealers. Anyway, I wanted to talk about my experience of running a mail server. Over the last 10+ years I’ve operated my own mail server on VMs hosted by netcup.de. (Not a paid ad, I promise) Their servers have decent performance and for just a few euros they are a steal. So I went and setup ispconfig – a management interface for basic web, mail and dns servers – to have a more simpler interface for configuration. Till Brehm (the maintainer of ispconfig) is doing an amazing job at developing that piece of software over the last decade. The interface is clean, neat and gives your customers basic settings to do self service if they want to. Configuring a mail server is complex Configuring your […]

Where it all started – goodbye Cybton

In my previous post I’ve mentioned my entry to programming was first via C++ and afterwards I wandered of into the wide fields of web development. A fundamental problem back then for me was that I did not have any money and that properly running a website required you to have a static IP with a Webserver that ideally could not only deliver static content like HTML, CSS and images, but also included PHP to have dynamic content. One of those hosters, that offered free webspace was cybton.com. As it did not have any ads on the pages offered through the webspace itself and had a cool community with a bonus system, that was the one I’ve opted for. The bonus system allowed you to upgrade your webspace to get more disk space and other cool features. You could earn points to “buy” those perks by actively engaging in the community. Imagine those fake internet points from stackoverflow would allow you to get a free server. It was super successful and at its peek, the community was super active and newcomers were quickly helped with getting their project started. One thing I remember quite clearly back then was the mostly […]

LetsEncrypt now used everywhere

So today I decided to move all my certificates from StartSSL to LetsEncrypt. Not only is StartSSL really a bad CA with recent problems, but also they limit you with several obstacles that just don’t make any sense. They really just want you to sign up for one of their “great” extended validation thingys. Over the last couple months they have really improved the web interface but still this is not enough to deal with today’s challenges of delivering secure connections to users easily. Their new APIs and StartEncrypt service are merely a late effort, trying to outbid LE with a worse service. Not worth the time or effort. The biggest problem is, that today I run multiple domains on my server and I need to provide one single certificate with all domains via Dovecot / Postfix. StartSSL allows you to have up to five domain names in the certificates they sign. (For example www.bruck.me and bruck.me would be a total of two domain names) So I’ve ran out of the possibility of using all my domains with one StartSSL certificate. Well, LE offers up to 100 domain names in one certificate. Of course wildcard certificates would be nicer, but […]

Properly setup proguard for an Android project

When dealing with an Android project you want to use Proguard to minify, shrink and possibly even obfuscate the code. The gains from this are huge and many smart minds have put a lot of thought into Proguard. We encountered that the TUM Campus App shrinked from 20 Megabytes to just 9 Megabytes with all the optimization in place – huge savings if you deploy it to 10k+ clients! Really if you are not using this in your project currently you must be insane! Anyways if you rely on external Libraries like Retrofit (Which is totally awesome, use it!) then you need to add some proguard rules in order to tell it what not to remove from those libs because it is really required but maybe not directly used. Mostly that is some models which get serialized and you might encounter some warnings but those don’t really are not interesting to you as a lib user. This repository has a great collection on proguard files for various libs. Use it, don’t reinvent the wheel!

PHP: File uploads fail without any error

When handling uploads with PHP often it can happen, that the $_FILES array is simply empty. This can occur when one of the following things is true: Check php.ini for file_uploads = On, post_max_size, and upload_max_file_size. Make sure you’re editing the correct php.ini – use phpinfo() to verify your settings. Make sure your FORM tag has the enctype=”multipart/form-data” attribute. Do not use javascript to disable your form file input field on form submission! Make sure your directory has read+write permissions set for the tmp and upload directories. Make sure your FORM tag has method=”POST”. GET requests do not support multipart/form-data uploads. Make sure your file destination and tmp/upload directories do not have spaces in them. Make sure all FORMs on your page have /FORM close tags. Make sure your file input tag has a NAME attribute. An ID attribute is NOT sufficient! ID attributes are for use in the DOM, not for POST payloads. Your /tmp folder is full Hope this helps!

Umzug zu Netcup: umstieg auf einen vServer

Leider ist das alte Angebote, welches ich bei http://www.webspace-verkauf.de/ hatte nicht nur total überteuert, sonder hatte ich auch in den beinahe 10 Jahren die ich dort war, einige unangenehme Erlebnisse. Von zufälligen Sperrungen meines ganzen Accounts, wegen irgendwelchen Kleinigkeit, bis hin zu das der Kundenservice teilweise über drei Tage braucht um eine zweizeilige Email zu beantworten. Naja, jedenfalls wars einfach Zeit die Lager zu wechseln 😀 An dieser Stelle auch einen Dank an Christian Blechert, dem ich schon lange auf seinem Blog folge, für die Empfehlung von Netcup – wirklich ein toller Anbieter zum Kampfpreis. Umzug von 5 Domains ohne Probleme durchgeführt und vServer promt nach der Verifizierung durch nen Telefonanruf verfügbar gewesen. Der virtuelle Server lauft sehr stabil und hat auch einiges an power und das zum gleichen Preis wie für ein doofes Webspacepaket beim alten Anbieter 🙂