Yinette's Webshite

A collection of security stuff and all sorts of other random shit.

GHOST in the Library

Upgrade alone complex – Updating glibc without the pain, misery, and the laughing man

So, there’s a 12 year old bug in glibc, and it allows RCE with permissions of the running process, how wonderful.

Oh yeah, and a simple DNS Lookup can cause it if you use a particular function from the 80’s.

And yes, it’s now our problem.

data

Here’s a link to the write-up provided by Qualys: http://www.openwall.com/lists/oss-security/2015/01/27/9

And here’s a post by Rob at Errata Security called You shouldn’t be using gethostbyname() anyway good advice.

I’m starting to get the hang of these named vulnerabilities, just not the media frenzy that comes after, so, I thought i’d do something about it.

Anyway, time to bring some calm to an otherwise very distressing situation for some sysadmins, and show how you can ensure everything is patched and ok, with MINIMAL downtime.

let’s begin

Check if you are vulnerable!

The people at Qualys who found this bug provided a nice little PoC written in C that you can run to check if your glibc is vulnerable.

Just copypasta the code below into your text editor of choice and save as ‘GHOST.c’ or something along those lines.

#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

#define CANARY "in_the_coal_mine"

struct {
  char buffer[1024];
  char canary[sizeof(CANARY)];
} temp = { "buffer", CANARY };

int main(void) {
  struct hostent resbuf;
  struct hostent *result;
  int herrno;
  int retval;

  /*** strlen (name) = size_needed - sizeof (*host_addr) - sizeof (*h_addr_ptrs) - 1; ***/
  size_t len = sizeof(temp.buffer) - 16*sizeof(unsigned char) - 2*sizeof(char *) - 1;
  char name[sizeof(temp.buffer)];
  memset(name, '0', len);
  name[len] = '\0';

  retval = gethostbyname_r(name, &resbuf, temp.buffer, sizeof(temp.buffer), &result, &herrno);

  if (strcmp(temp.canary, CANARY) != 0) {
    puts("vulnerable");
    exit(EXIT_SUCCESS);
  }
  if (retval == ERANGE) {
    puts("not vulnerable");
    exit(EXIT_SUCCESS);
  }
  puts("should not happen");
  exit(EXIT_FAILURE);
}

Then, once that’s done do the following:

gcc GHOST.c -o GHOST

Replace GHOST.C with whatever you called the thing above.

Preferrably not as root run your new binary like thus

./GHOST

If you get “Vulnerable” as the return, you need to upgrade, keep following this guide, otherwise, you’re set! But best to check below where i’m trying to find stuff running using older libs.

Upgrade time!

Upgrade your packages, be it with aptitude, apt, pacman, yum or emerge. If your upstream has updated the package, an ‘upgrade all’ should suffice and bring a new glibc into your system.

If you only want to install the new glibc package, there is a way to do it for each package manager, it should be documented or discussed, google will help with update single package <packagemanager> as the query.

Here’s some popular distro advisories and notifications where the packages have been upgraded:

Debian’s Announcement: https://lists.debian.org/debian-security-announce/2015/msg00025.html

Redhat’s Announcement: https://access.redhat.com/articles/1332213

Your init should do a warm restart and pull in the new glibc, this behaviour has been observed on Debian Wheezy with sysvinit

Finding things still running the old glibc

Let’s face it, sometimes we cannot simply restart. Until we reach the nirvana of everything being clustered, we’ll have to live with this.

Here’s a way to find services and processes that are still using the old glibc. For this example, I shall be using a Debian Wheezy machine.

The command is:

sudo lsof -n | grep libc-

output will be something like this:

init          1              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
udevd       374              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rpcbind    1787              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rpc.statd  1818             statd  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rpc.idmap  1832              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rsyslogd   2187              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rs:main    2187  2192        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rsyslogd   2187  2194        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
rsyslogd   2187  2195        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
acpid      2240              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
atd        2259            daemon  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nmbd       2277              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
dbus-daem  2311        messagebus  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
smbd       2329              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lldpd      2352              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lldpd      2354            _lldpd  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
smbd       2372              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
avahi-dae  2386             avahi  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
avahi-dae  2387             avahi  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nginx      2484              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nginx      2485          www-data  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nginx      2486          www-data  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nginx      2487          www-data  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
nginx      2488          www-data  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2519        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2520        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2521        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2522        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2523        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2524        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2525        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2526        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2527        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
libvirtd   2518  2528        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
cron       2548              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
dhclient   2582              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
gmain      2655  2741        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
gdbus      2655  2836        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
polkitd    2841              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
gdbus      2841  2968        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
exim4      3083       Debian-exim  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
sshd       3108              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
minissdpd  3133              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3143              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3144              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3145              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3146              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3147              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
getty      3148              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
smbd      11793              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
udevd     27806              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
sshd      28095              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
bash      28100              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
lsof      28160              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
grep      28161              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
lsof      28162              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so
lwresd    28213              root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lwresd    28213 28214        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lwresd    28213 28215        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lwresd    28213 28216        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
lwresd    28213 28217        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
named     28322              bind  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
named     28322 28323        bind  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
named     28322 28324        bind  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
named     28322 28325        bind  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so
named     28322 28326        bind  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so

Ok, let me point out a few things here off the bat.

init          1              root  mem       REG              254,0  1603600    7340685 /lib/x86_64-linux-gnu/libc-2.13.so

init here is running the updated version of glibc, notice in the third argument after ‘root’ it has ‘mem’? This is a good thing. Its memory-mapped but has an existing file on disk behind it.

However, if we look at libvirtd:

libvirtd   2518  2528        root  DEL       REG              254,0             7340558 /lib/x86_64-linux-gnu/libc-2.13.so

the ‘mem’ is now ‘DEL’, this means its mapped in memory, but the file behind it no longer exists. This is still running a vulnerable glibc.

Update!

A follower of mine @aptcat noticed something different with their lsof output with a 32-bit kernel:

syslogd    1435               root  mem       REG              253,3            74591600 (deleted)/lib/i386-linux-gnu/libc-2.13.so (stat: No such file or directory)

You will note that lsof still has it in memory, but notes the file itself is deleted.

Thanks for that @aptcat!

The solution? Restart services!

You can get some easy wins with daemons like sshd, libvirtd, postfix/exim4, etc since they can restart without showing any major visible disruption. Nginx/apache is in the list of these, but it depends on how much config it has to parse and how much traffic you get at any one moment.

Other services like mysqld, redis, etc that will either kill important data or take a noticible time to start up again might need to be scheduled in with your user for when an appropriate time to restart these is.

I would still highly recommend a full reboot anyway, as this will ensure everything comes back with a fresh glibc, and you wont get nasty surprises.

Using sysvinit, the way to restart these as root/sudo is as follows:

sudo service THING restart

Though, note that some things have themselves named differently than you would think. For example sshd under debian sysvinit is ‘ssh’ under the service control script, use tab-complete (mash TAB) if you are unsure. (Likewise with apache2 on RHEL/Centos, it can sometimes be httpd)

Systemd users will be able to use sudo systemctl THING restart, however i’d imagine systems that use systemd would have already patched glibc… Oh well, might be different for everyone.

After that, assuming the service has restarted without issue and everything looks peachy, the ‘DEL’ should change to ‘mem’.

Now it’s just a matter of hunting down other services or programs to restart.

Here’s a quick one getting a list of processes and also looking for ‘DEL’

sudo lsof -n | grep libc- | grep DEL

That should be it.

If you have any suggestions on how to do any of this better, please don’t hesitate to mention me on twitter (@yinettesys), I will RT and converse with those who provide much better methodology than mine.

Now, let’s hope the media scrum isn’t as large as it has been in the past.