Using SaltStack to recover from CVE-2015–0235 (Qualys Security Advisory, GHOST: glibc gethostbyname buffer overflow)
Most of us sysadmin types were pounded with this announcement this morning. The GHOST vulnerability is worth patching against—most Linux distros have already released patches—but it’s useful to know if your machines are vulnerable, or if after patching, the patch was successful.
The canonical way to test for the vulnerability is with a short C program:
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#define CANARY "in_the_coal_mine"
struct {
char buffer[1024];
char canary[sizeof(CANARY)];
} temp = { "buffer", CANARY };
int main(void) {
struct hostent resbuf;
struct hostent *result;
int herrno;
int retval;
size_t len = sizeof(temp.buffer)
- 16*sizeof(unsigned char)
— 2*sizeof(char *) — 1;
char name[sizeof(temp.buffer)];
memset(name, '0', len);
name[len] = '\0';
retval = gethostbyname_r(name, &resbuf, temp.buffer,
sizeof(temp.buffer), &result, &herrno);
if (strcmp(temp.canary, CANARY) != 0) {
puts("vulnerable");
exit(EXIT_FAILURE);
}
if (retval == ERANGE) {
puts("not vulnerable");
exit(EXIT_SUCCESS);
}
puts("test aborted: should not happen");
exit(EXIT_FAILURE);
}
Which can then be saved to a file “ghost.c” and compiled on most Linux machines with
gcc ghost.c -o ghost
Running it with ‘./ghost’ should produce either “not vulnerable” with an exit code of 0, or “vulnerable” with an exit code of 1.
But let’s say you have 1000 machines, all with running salt-minions. How can we test for this on all of them?
We’ll assume first that they are all the same distro as your Salt master. Yes, I know that’s a degenerate case, but to start with let’s just consider the easy route.
First, save ghost.c to a directory on your master and compile it as describe above. Then put the executable in your /srv/salt directory (or wherever your file_roots points). Put this sls file in the same directory:
/tmp/ghost:
file.managed:
- source: salt://ghost
- owner: root
- mode: '0644'
runghost:
cmd.run:
- name: /tmp/ghost
Now you can fire off this on all your minions with
salt \* state.sls ghosttest
Because Salt will treat the result of cmd.run as a failure if the executed command returns a non-zero exit status, all vulnerable minions will show “FAILED”. Successfully patched minions will show “SUCCESS.”
Note that all vulnerable services will need to be restarted after a patch (or the affected system will need to be rebooted). Salt can help with this if, in fact, you need to restart individual services rather than restart an entire box.
There are a couple of odd results you can get back from this. First, on one of my machines I got
w01:
—————
ID: /tmp/ghost
Function: file.managed
Result: True
Comment: File /tmp/ghost is in the correct state
Started: 11:06:30.632664
Duration: 779.398 ms
Changes:
—————
ID: runghost
Function: cmd.run
Name: /tmp/ghost
Result: False
Comment: Command “/tmp/ghost” run
Started: 11:06:31.412444
Duration: 60.247 ms
Changes:
—————
pid:
28508
retcode:
127
stderr:
/bin/bash: /tmp/ghost: No such file or directory
stdout:
Salt told me the file was present and in the correct state, but bash said “No such file or directory.” Bug in Salt, right? I mean, that’s happened before.
No, not today! If I logged into the machine and ran the executable by hand I got the same message. In this case it was because all my other machines are 64-bit, but this one is 32-bit, and the test executable was linked against the 64-bit glibc. So the message was correct, but confusing since the missing file is not the executable but the library.
Let’s fix this. I happen to have development tools installed on that box, so let’s build a 32-bit compiled version there, put it back on the master, and also modify the sls file so the correct executable will get copied to 64 or 32 bit machines.
/tmp/ghost.c:
file.managed:
- source: salt://ghost.c
gcc ghost.c -o ghost:
cmd.run:
- user: root
- cwd: /tmp
cp.push:
module.run:
- path: /tmp/ghost
Then, run this sls and copy the file out of the cache directory (see cp.push documentation)
# salt <32bitminion> state.sls ghostbuild
# cp /var/cache/salt/master/minions/<32bitminion> /tmp/ghost \
/srv/salt/ghost32
(replace 32bitminion with the minion_id where you did the build)
Now change your ghostcheck.sls to look like this
/tmp/ghost:
file.managed:
{% if grains['osarch'] == 'i386' %}
— source: salt://ghost32
{% else %}
— source: salt://ghost
{% endif %}
— owner: root
— mode: '0700'
runghost:
cmd.run:
— name: /tmp/ghost
— cwd: /tmp
— user: root
— require:
— file: /tmp/ghost
Now I get accurate results from all my minions, 32-bit or 64-bit.
Obviously the simpler way to do this would be to build and run ghost.c on all minions, but many folks don’t keep gcc and friends on things like webservers.
Finally, if you don’t want to reboot all your machines, you just want to restart affected services, you can do the following (props to the hackernews discussion for this snippet)
salt \* cmd.run 'netstat -lnp | grep -e "\(tcp.*LISTEN\|udp\)" | cut -d / -f 2- | sort -u'
which will tell you which services on which machines need to be restarted. Then for each of these services and machines you can say
salt <affectedminion> service.restart <affectedservice>
Finally, shameless plug for the awesome company I work for—if you want to learn more about Salt, **SaltConf** would be a great place to do it! March 3–5, 2015, Grand America Hotel, Salt Lake City.
