On the morning of September 24th 2026, Endcrawl had a brief outage. I'd just typed:

sudo kill -TERM -1490947

And instantly all services went down.

Right away I blamed myself. Maybe I mistyped that number! Maybe my code had a signal handling bug. Maybe I don't understand Unix as well as I think I do. Maybe I'm an idiot!

As it turned out, the outage was caused by a bug in

kill(1)

.

This is really surprising. The

kill

program is 50 years old. Its spec is simple and stable. It's probably exercised a billion times a day.

A bug in

kill

was so surprising that, once I'd found it, I couldn't stop there. I needed to know how this bug came to be, and what had led the programmers so badly astray. When I eventually found the root cause, there was an even bigger surprise. It had led to not just one bug, but a whole series of

kill

bugs that shipped in Ubuntu 14, 16, 18, 22 and 24 LTS.

This post documents a decade of

kill

bugs, then makes some recommendations.

An Outage Not Understood Happens Again

Back to the outage. I scrambled to put the fire out. After that came a surprise second fire. By the end of all this I was pretty demoralized. I was ready to accept that I'd made a dumb mistake and ready to get on with the work day. There was this nagging feeling though…

Remember that final scene in Burn After Reading?

"What did we learn Palmer?"

"I don't know sir."

"I don't f**king know either. I guess we learned not to do it again."

"Yes sir."

"Although I'm f**ked if I know what we did..."

What had just happened? Was I about to be hit with a third outage? How could I be sure?

An outage not understood is an outage that happens again. That nagging feeling I had was a call to understand. With a sigh, I put all my other work on hold and started a proper post mortem.

The Proximate Cause

Let's examine the command that caused the outage:

sudo kill -TERM -1490947

The negative number here is a process group identifier, or pgid for short. A pgid is like a pid, but identifies a group of related processes. By convention, killing a positive number sends a signal to a single process, while killing a negative number signals all processes in that process group.

I was attempting to cancel a big Endcrawl 7k test render. A normal person would have killed one of the processes involved and let failures cascade to the others. But killing the whole group simultaneously strikes me as cleaner.

Process group signaling isn't some exotic operation. You use it all the time. Every time you

^C

a command, the tty sends

SIGINT

to the foreground process group. Interactive shells run all but the simplest commands in a new process group, which among other things makes job control possible.

Process group signaling is in fact an awesome Unix innovation. I love it so much I added it to daemontools, where

svc -+h

now avoids orphans in a multiprocess service bounce.

At any rate, imagine my surprise when the command didn't just kill those render processes, but killed all processes on the machine save one!

The Bug

Here's the relevant snippet from procps 4.0.4, the provider of

/bin/kill

in Ubuntu 24.04 LTS. Can you spot the bug?

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, &optindex)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        print_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = (long)('0' - optopt);
        if (!execute_kill((pid_t) pid, signo, use_sigqueue, sigval))
          exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
    /* ... */
  }

My initial guess was wrong. There's no underflow or overflow bug with

-1490947

.

Notice we're in the unrecognized option switch case. Here's what

getopt(3)

says about unrecognized options:

By default,

getopt()

prints an error message on standard error, places the erroneous option character in optopt, and returns

'?'

as the function result.

Aha. The

optopt

variable only holds the first character after the dash, not the whole negative pid. When character

1

is subtracted from character

0

we get a pid of

-1

. What happens when you kill pid

-1

? Bad things:

If pid equals -1, then sig is sent to every process for which the calling process has permission to send signals, except for process 1 (init).

This ran as root, so the misparsing of

-1490947

as

-1

killed every process but init. Bug found, mystery solved.

Wait But Why

Except for me this just raised more questions. Why is the code assuming the unrecognized option is a "signal [sic] digit negative PID"? Why is the single digit negative pid case special, and what handles the general case of a multidigit negative pid?

Moreover, how could a bug like this exist in 2026, and in a Ubuntu LTS release no less? Has it always been there?

No, that couldn't be. I've killed process groups in the past. And the

kill -TERM -12345

thing is an ancient Unix incantation. This used to work dammit!

Thinking back, I could vaguely remember some negative pid kills gone awry. Never in an "oops this killed everything in sight" way. More in a "weird, nothing happened" way, or a "poof there goes my shell" way. Minor ways more easily shrugged off as user error.

It turns out I wasn't imagining these. The last decade has been a rough one for negative pid killers. There have been no less than FOUR distinct bugs around negative pids. And they all shipped in Ubuntu LTS versions.

The History

Before I tell the tale of four bugs, let's go even farther back in Unix history.

In the beginning Unix had processes. By 1973 it had the

kill(1)

program. But it took a decade to evolve process groups as we now know them. It wasn't until Unix V9 in 1986 that you could kill a process group via negative pid. Near as I can tell, negative pid killing has worked fine in the BSDs all these years.

In the early 90s this thing called Linux came along. Although Linux initially borrowed a lot of BSD code for its userland, by the late 90s this approach was in legal jeopardy. Everything had to be rewritten, every wheel reinvented. And quickly. It was an exciting time to be a unix programmer.

And so it came to pass that Linux's

kill

was reinvented written by Albert Calahan. After Albert DoSed himself by forking 5 children, Debian downstream maintainer Craig Small took over poor old neglected procps.

Negative pid killing seems to have worked fine until 2013. Then the trouble began.

Bug 1: Ubuntu 14 Can't Kill -1

Ubuntu 14.04 LTS included procps 3.3.9. Prior to this, the

kill

code did what the BSDs do and used pointer arithmetic to walk and parse

argv

.

Personally I'm fond of that approach. Stallman isn't so fond of it though. In GNU world, you're supposed to standardize command line parsing via

getopt

. This is probably a good idea in the general case, but as we'll see, it was probably the wrong choice in the specific case of

kill(1)

.

Here's the relevant code after the conversion to

getopt

:

opterr=0; /* suppress errors on -123 */
while (loop == 1 && (i = getopt_long(argc, argv, "l::Ls:hV", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      }
      loop=0;
      break;
    /* ... */
  }

argc -= optind + sigopt;
argv += optind;

for (i = 0; i < argc; i++) {
  pid = strtol_or_err(argv[i], _("failed to parse argument"));
  if (!kill((pid_t) pid, signo))
    continue;
  exitvalue = EXIT_FAILURE;
  continue;
}

The code worked fine for multidigit negative pids. However, if you were one of those freaks creative sysadmins who delighted in a

kill -HUP -1

now and again to restart all your services, you noticed a bug: nothing happened.

The bug is due to option bundling.

• Multidigit case: With

  kill -HUP -123

  ,

  getopt

  acts as if

  -1

  ,

  -2

  , and

  -3

  were all separately passed. The unrecognized option case will be hit three times, and only at

  -3

  will the

  -123

  argument be consumed by incrementing

  optind

  . Because the code breaks out of the

  getopt

  loop early at

  -1

  , an unconsumed

  -123

  argument correctly makes it to the pid killer.

• Single digit case: With

  kill -HUP -1

  , however,

  getopt

  has fully consumed that argument and incremented

  optind

  before we break out of the loop. The pid killer sees no arguments and nothing happens.

Bug 2: Ubuntu 16 Accidentally Kills -1

A cure was in order. Unfortunately it was worse than the disease.

Ubuntu 16.04 LTS shipped with procps 3.3.10. The

kill -HUP -1

bug was fixed at the expense of multidigit negative pid kills: a

kill -TERM -12345

now killed everything.

This is the same bug that caused the outage under Ubuntu 24. We reviewed this above. The only mystery here is why procps circa Ubuntu 24 resurrected the known-buggy code circa Ubuntu 16. My best guess is that the following bugs were deemed even worse.

Bug 3: Ubuntu 18 Segfaults Killing -1

Ubuntu 18.04 LTS shipped with procps 3.3.12.

This attempted to fix multidigit negative pid killing. Instead of parsing

optopt

, which only corresponds to a single bundled character like

-1

when

-123

is passed, the new code stops and tries to parse the whole argument:

opterr=0; /* suppress errors on -123 */
while (loop == 1 && (i = getopt_long(argc, argv, "l::Ls:hV", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = atoi(argv[optind]);
        if (kill((pid_t)pid, signo) != 0)
        exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
      loop=0;
      break;
    /* ... */
  }

This works correctly for multidigit negative pids. But if you passed a single digit negative pid, as in a

kill -TERM -1

, the pid argument was fully consumed, and

optind

would run off the end of

argv[]

. Segfault!

Bug 4: Ubuntu 22 Kills Your Shell

Ubuntu 22.04 LTS shipped with procps 3.3.17.

This fixed the segfault by ensuring

optind

never ran off the end of

argv[]

:

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = atoi(argv[optind-1]);
        if (kill((pid_t)pid, signo) != 0)
        exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
      xerrx(EXIT_FAILURE, _("internal error"));
    /* ... */
  }

Once again, it seems the contributor didn't understood how

getopt

option bundling interacts with

optind

, and hopefully by this point you're developing some sympathy for them. With this patch a

kill -TERM -1

works correctly and no longer segfaults. But multidigit negative pid killing was broken for a second time, in a brand new way.

Suppose we try to

kill -TERM -123

. Thanks to option bundling,

optind

isn't incremented yet when we hit the unrecognized option case. Per some quirky behavior noted below,

argv[optind-1]

is now the string

kill

.

How does

atoi(3)

handle a non-numeric string like

"kill"

? It returns

0

.

And what does the kill system call do with zero?

If pid equals 0, then sig is sent to every process in the process group of the calling process.

Because it's a simple command, your shell won't isolate

kill -TERM -123

in a new process group. So this bug will unexpectedly kill your shell. Poof!

What Did We Learn, Palmer?

There are several takeaways.

The Narrow Learning

For one, the way

getopt

only advances

optind

with the final bundled option is deeply non-obvious. I didn't find evidence that any contributor understood this. I didn't understand it myself at first. That took careful study.

It's a real surprise. Surprising behavior like this should be avoided.

I feel this learning is too myopic though. Can one even avoid this surprising

getopt

behavior? You can't disable option bundling.

The Broader Learning

Getopt Considered Harmful.

Well okay, at least in this specific case. The

kill(1)

program is just an unusual beast. It accept