Running servers (and services) well is not trivial
June 2019
It all started with a Reddit comment thread called The “mass exodus “from Github to GitLab: 10 . In it, someone commented that they did not understand why there was a need for cloud Git services in the first place, since running your own Git server for your company was easy enough. I think that part of this view is due to the difference between ‘on premise’ and ‘off premise’ approaches to environments , but as a sysadmin I had a twitchy reaction to the ‘it’s easy’ part .
These days, it’s often relatively easy to ‘just set up a server’ or a service, especially if you already work in the cloud. Spin up a VM or a Docker image, install some stuff, done, right? Well, not if you want this to be reliable infrastructure. So let’s run down what you or I would have to do to set up a general equivalent of Github for internal company use:
If you can’t outsource all authentication to some other system in your company, you’ve just taken on an ongoing maintenance task of adding, removing, and updating users and groups (and any other things that have to authenticate to the Git server). Failure to do this well, or failure to be hooked into other aspects of the company’s authentication and HR systems, can result in fun things like fired employees still retaining high-privilege access to your Git server (ie the company’s Git server). You probably don’t want that.
(Non-integrated authentication causes sufficiently many problems that it’s featured in a sysadmin test .)
If you’re using Docker or a VPS in the cloud, don’t forget to figure out how you’re going to get persistent storage.
Once you have backups set up, make sure that you’re monitoring them on an ongoing basis so that you can find out if they break.
I’m going to generously assume that the system never has performance problems (which you’d have to troubleshoot) and never experiences software issues with your chosen Git server and any databases and the like that it may depend on (which you’d have to troubleshoot too). Once set up, it stays quietly running in the corner and does actively require your attention. This is, shall we say, not always the experience that you actually get.
(I’m also assuming that you can just get TLS certificates from Let’s Encrypt and that you know how to do this. Or perhaps the Git server stuff you picked does this all for you.)
Unfortunately we’re not done, because the world is not a nice place. Even with the service just working, we still have more things to worry about:
New versions may require changes to other bits of the system, any local customizations, or to things you integrated the old version with. Updates are often a pain but at the same time you have to do them sooner or later.
If this is something that matters to the company and the company is more than a few people, it’s also not just ‘you’ (a single person) who will be looking after the server. The company needs at least a Few people involved so that you can go on vacation or even just get sick without the company running the risk of the central Git server That all the developers use just falling over and no one knowing How to bring it back.
In some environments this Git server will either be exposed to the Internet (even if it’s only used by company people) or at least available across multiple ‘internal network’ locations because, say, your developers are not really on the same network as your production servers in the cloud. This will likely raise additional network security issues and perhaps authentication issues. This is very especially the case if you have to expose this Git service to the Internet. Mere obscurity and use only by company insiders is not enough any more these days; There are systems that mass scan the entire IPv4 Internet and keep world-accessible databases of what they find. If you have open ports or services, you have problems, and that means you’re going to have to do the work to close things down.
Basically all of this applies to pretty much any service, not just Git. If you want to run a mailer, or an IMAP server, or a web server hosting a blog or CMS, or a DNS server, or whatever, you will have to face all of these issues and often more. Under the right circumstances, the initial setup can appear extremely trivial; you grab a Docker image, you start it somewhere, you make your internal DNS point a name to it, and you can call it up in your browser or otherwise start poking it. But that’s just the start of a long and tiresome journey to a reliable, secure service that can be sustained Over the long haul and won’t blow up in your face someday.
I’m naturally inclined toward the ‘on premise’ view of things where we do stuff internally. But all of this is why, if someone approached me at work about setting up a Github-like service locally, I would Immediately ask ‘is there some reason we can’t pay Github to do this for us? ‘ I’m pretty confident that Github’ll do it better, and if staff time isn’t free they’ll probably do it cheaper.
PS: I’ve probably missed some things you’d need to think about and tackle, which just goes to show how non-trivial this really is.
GIPHY App Key not set. Please check settings