Dealing with system consoles from hell
We've all been there. It's midnight, the maintenance window for work on a few remote systems is quickly closing, and some critical infrastructure component is down hard. It might be a router, a switch, a storage array, or a server -- it doesn't really matter. There's trouble in River City.
To the casual observer, it might seem that the hardest part of this scenario is finding and fixing the actual problem causing the outage. But in many cases, that's the least of it. The hard part is trying to do something absurdly simple -- such as properly generating a break sequence during the boot cycle.
A good example of this is a recent situation I fell into when a Oracle storage array was throwing critical errors on boot following an update. The serial console was rife with error messages and prompts for further action. Each option was in the form of ESC-2, ESC-4, and so on. That was all well and good when I was SSH'd into the serial console server and driving the console directly, but when I connected to the service processor directly through SSH using the Sun Oracle Java-based shared shell applet, there was no way to generate the required Escape key. I wound up switching the console between the shared SSH session and my serial connection just to hit the Escape key. It wasn't just for this function -- try using vi without a functional Escape key.
