Crash A Xen Virtual Machine, Crash Xend

When I migrated my colo server to new drives and Xen I kept the old server drives as a separate virtual machine. The old drives were corrupt and causing all sorts of bizarre crashes but only intermittently.

Yesterday xend disappeared and would crash soon after starting. After a fair bit of searching I found this post which matches what I was seeing. Though they ran into the problem trying to restore an image, it had the same symptom, a crashed virtual machine.

The comment in this post says it all:

Restarting xend was a bit of a brave move. Although its intended to be restartable, there are definitely bugs in the restart code, particularly if you have domains in unusual states at the time (e.g. crashed, awaiting reaping). Error case code is always the least tested...

So if you can crash a virtual machine you can effectively neuter the whole machine. Without xend you can't start or stop any virtual machines in domain0. Even the workaround mentioned later in the thread doesn't really help as you still can't stop anything.

Luckily one of my virtual machines was still accessible via ssh and I was able to shut that down before rebooting the box. The others just got nuked.

Considering I'm about to move to the USA this makes me somewhat nervous....