Upgrade alone complex – Updating glibc without the pain, misery, and the laughing man
So, there’s a 12 year old bug in glibc, and it allows RCE with permissions of the running process, how wonderful.
Oh yeah, and a simple DNS Lookup can cause it if you use a particular function from the 80’s.
And yes, it’s now our problem.
Here’s a link to the write-up provided by Qualys: http://www.openwall.com/lists/oss-security/2015/01/27/9
And here’s a post by Rob at Errata Security called You shouldn’t be using gethostbyname() anyway good advice.
I’m starting to get the hang of these named vulnerabilities, just not the media frenzy that comes after, so, I thought i’d do something about it.
Anyway, time to bring some calm to an otherwise very distressing situation for some sysadmins, and show how you can ensure everything is patched and ok, with MINIMAL downtime.
let’s begin
Check if you are vulnerable!
The people at Qualys who found this bug provided a nice little PoC written in C that you can run to check if your glibc is vulnerable.
Just copypasta the code below into your text editor of choice and save as ‘GHOST.c’ or something along those lines.
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#define CANARY "in_the_coal_mine"
struct {
char buffer[1024];
char canary[sizeof(CANARY)];
} temp = { "buffer", CANARY };
int main(void) {
struct hostent resbuf;
struct hostent *result;
int herrno;
int retval;
/*** strlen (name) = size_needed - sizeof (*host_addr) - sizeof (*h_addr_ptrs) - 1; ***/
size_t len = sizeof(temp.buffer) - 16*sizeof(unsigned char) - 2*sizeof(char *) - 1;
char name[sizeof(temp.buffer)];
memset(name, '0', len);
name[len] = '\0';
retval = gethostbyname_r(name, &resbuf, temp.buffer, sizeof(temp.buffer), &result, &herrno);
if (strcmp(temp.canary, CANARY) != 0) {
puts("vulnerable");
exit(EXIT_SUCCESS);
}
if (retval == ERANGE) {
puts("not vulnerable");
exit(EXIT_SUCCESS);
}
puts("should not happen");
exit(EXIT_FAILURE);
}
Then, once that’s done do the following:
gcc GHOST.c -o GHOST
Replace GHOST.C with whatever you called the thing above.
Preferrably not as root run your new binary like thus
./GHOST
If you get “Vulnerable” as the return, you need to upgrade, keep following this guide, otherwise, you’re set! But best to check below where i’m trying to find stuff running using older libs.
Upgrade time!
Upgrade your packages, be it with aptitude, apt, pacman, yum or emerge. If your upstream has updated the package, an ‘upgrade all’ should suffice and bring a new glibc into your system.
If you only want to install the new glibc package, there is a way to do it for each package manager, it should be documented or discussed, google will help with update single package <packagemanager> as the query.
Here’s some popular distro advisories and notifications where the packages have been upgraded:
Debian’s Announcement: https://lists.debian.org/debian-security-announce/2015/msg00025.html
Redhat’s Announcement: https://access.redhat.com/articles/1332213
Your init should do a warm restart and pull in the new glibc, this behaviour has been observed on Debian Wheezy with sysvinit
Finding things still running the old glibc
Let’s face it, sometimes we cannot simply restart. Until we reach the nirvana of everything being clustered, we’ll have to live with this.
Here’s a way to find services and processes that are still using the old glibc. For this example, I shall be using a Debian Wheezy machine.
The command is:
sudo lsof -n | grep libc-
output will be something like this:
init 1 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so udevd 374 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rpcbind 1787 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rpc.statd 1818 statd DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rpc.idmap 1832 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rsyslogd 2187 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rs:main 2187 2192 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rsyslogd 2187 2194 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so rsyslogd 2187 2195 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so acpid 2240 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so atd 2259 daemon DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nmbd 2277 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so dbus-daem 2311 messagebus DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so smbd 2329 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lldpd 2352 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lldpd 2354 _lldpd DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so smbd 2372 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so avahi-dae 2386 avahi DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so avahi-dae 2387 avahi DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nginx 2484 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nginx 2485 www-data DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nginx 2486 www-data DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nginx 2487 www-data DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so nginx 2488 www-data DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2519 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2520 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2521 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2522 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2523 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2524 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2525 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2526 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2527 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so libvirtd 2518 2528 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so cron 2548 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so dhclient 2582 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so gmain 2655 2741 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so gdbus 2655 2836 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so polkitd 2841 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so gdbus 2841 2968 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so exim4 3083 Debian-exim DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so sshd 3108 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so minissdpd 3133 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3143 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3144 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3145 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3146 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3147 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so getty 3148 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so smbd 11793 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so udevd 27806 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so sshd 28095 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so bash 28100 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so lsof 28160 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so grep 28161 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so lsof 28162 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so lwresd 28213 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lwresd 28213 28214 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lwresd 28213 28215 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lwresd 28213 28216 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so lwresd 28213 28217 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so named 28322 bind DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so named 28322 28323 bind DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so named 28322 28324 bind DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so named 28322 28325 bind DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so named 28322 28326 bind DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so
Ok, let me point out a few things here off the bat.
init 1 root mem REG 254,0 1603600 7340685 /lib/x86_64-linux-gnu/libc-2.13.so
init here is running the updated version of glibc, notice in the third argument after ‘root’ it has ‘mem’? This is a good thing. Its memory-mapped but has an existing file on disk behind it.
However, if we look at libvirtd:
libvirtd 2518 2528 root DEL REG 254,0 7340558 /lib/x86_64-linux-gnu/libc-2.13.so
the ‘mem’ is now ‘DEL’, this means its mapped in memory, but the file behind it no longer exists. This is still running a vulnerable glibc.
Update!
A follower of mine @aptcat noticed something different with their lsof output with a 32-bit kernel:
syslogd 1435 root mem REG 253,3 74591600 (deleted)/lib/i386-linux-gnu/libc-2.13.so (stat: No such file or directory)
You will note that lsof still has it in memory, but notes the file itself is deleted.
Thanks for that @aptcat!
The solution? Restart services!
You can get some easy wins with daemons like sshd, libvirtd, postfix/exim4, etc since they can restart without showing any major visible disruption. Nginx/apache is in the list of these, but it depends on how much config it has to parse and how much traffic you get at any one moment.
Other services like mysqld, redis, etc that will either kill important data or take a noticible time to start up again might need to be scheduled in with your user for when an appropriate time to restart these is.
I would still highly recommend a full reboot anyway, as this will ensure everything comes back with a fresh glibc, and you wont get nasty surprises.
Using sysvinit, the way to restart these as root/sudo is as follows:
sudo service THING restart
Though, note that some things have themselves named differently than you would think. For example sshd under debian sysvinit is ‘ssh’ under the service control script, use tab-complete (mash TAB) if you are unsure. (Likewise with apache2 on RHEL/Centos, it can sometimes be httpd)
Systemd users will be able to use sudo systemctl THING restart, however i’d imagine systems that use systemd would have already patched glibc… Oh well, might be different for everyone.
After that, assuming the service has restarted without issue and everything looks peachy, the ‘DEL’ should change to ‘mem’.
Now it’s just a matter of hunting down other services or programs to restart.
Here’s a quick one getting a list of processes and also looking for ‘DEL’
sudo lsof -n | grep libc- | grep DEL
That should be it.
If you have any suggestions on how to do any of this better, please don’t hesitate to mention me on twitter (@yinettesys), I will RT and converse with those who provide much better methodology than mine.
Now, let’s hope the media scrum isn’t as large as it has been in the past.