Highlights
- Knight’s Electronic Trading Group (ETG) managed an average daily trading volume of more than 3.3 billion trades daily, trading over 21 billion dollars…daily. That’s no joke!
- On July 31, 2012 Knight had approximately $365 million in cash and equivalents.
- One of the core functions of SMARS is to receive orders from other components of Knights trading platform (“parent” orders) and then send one or more “child” orders out for execution.
- The larger the parent order, the more child orders would be generated.
- one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers.
- Orders sent to the eighth server triggered the supposable repurposed flag and brought back from the dead the old Power Peg code.
- Basically, Power Peg would keep track of the child orders and stop them once the parent order was completed.
- When the Power Peg flag on the eighth server was activated the Power Peg functionality began routing child orders for execution, but wasn’t tracking the amount of shares against the parent order – somewhat like an endless loop.
- Knight’s executions constituted more than 50% of the trading volume, driving certain stocks up over 10% of their value.
- There was no kill-switch (and no documented procedures for how to react) so they were left trying to diagnose the issue in a live trading environment where 8 million shares were being traded every minute .
- they reacted by uninstalling the new code from the servers it was deployed to correctly.
- In other words, they removed the working code and left the broken code.
- In 45-minutes Knight went from being the largest trader in US equities and a major market maker in the NYSE and NASDAQ to bankrupt.
- It is not enough to build great software and test it; you also have to ensure it is delivered to market correctly so that your customers get the value you are delivering
- Any time your deployment process relies on humans reading and following instructions you are exposing yourself to risk.
- Deployments need to be automated and repeatable and as free from potential human error as possible.
- principles for Continuous Delivery apply here
- • Releasing software should be a repeatable, reliable process. • Automate as much as is reasonable.