We had one of those issues that just sneaks up on you. Just when you think you're covered from a monitoring perspective, an eye-opening issue reminds you that good monitoring is an ever-evolving process. So, what happened?...
System A sends a trade with an identifier. System B receives that trade and does several things with it, including creating a new account if one doesn't already exist. System A in an in-house application. System B is a vended application. System B has a limit in terms of how many characters the identifier can contain (7-characters). System A has been sending 7-characters or less until, of course, it reached 10,000,000...Boom!
Why didn't the monitoring catch it? Monitoring is typically built on a per-application basis. Support teams typically monitor things like space, memory, CPU, etc. Folks might even set up alerts to monitor business transactions and how well they're progressing. What I see very little of is monitoring that ensures that interfaces are being respected and that situations like the ones above won't occur.
In this case, we had been monitoring to make sure System B wasn't overflowing the number field in the DB. However, we hadn't been monitoring from the application standpoint to make sure that some other application wasn't overflowing the application limits.
Monitoring is something that should be build in a phased approach: hardware, processes, business transactions, middleware, etc. Don't forget "interfaces." If you do, things will likely go BOOM sooner rather than later!
