Innovation in Quality Assurance – What is the Impact on Trading technology?

Editor's Notes

• #2: Good afternoon.   Thanks a lot everyone for making it that late in the day. I've heard that event managers try to put some fictional speakers near the end of the conference, just to say that they were not able to make it, so everyone is on time for the cocktails. I am real and my name is Iosif Itkin.
• #3: I am Co-Founder and Joint CEO of Exactpro. We have 310 engineers focused on quality assurance and are mostly based in Russia. Since May we are part of London Stock Exchange Group.  In New York, you know how to make good things. In Russia, we know how to break things. That is why we are so good at software testing.
• #4: Today, I would like to share some ideas on how changes in quality assurance affect trading technology. These are QA trends: - FrAgile process - Crowd-sourced testing - Formal methodologies and - Cognitive technology
• #5: The test is dead since Google Test automation conference in 2011. Reid Hoffman said - If you are not embarrassed by the first version of your product, you've launched too late. Agile methods do not necessarily result in better products, but they can make software developers happier. Testing ideas first, to find the right product seems to be appealing and fun.
• #6: Why not have many users by the time our competitors have a well tested product? Crowd-testing is relying on a large user pool to collect immediate feedback on your software. To be sustainable, crowd-sourcing had to rely on the right code instrumentation. What is the point of using it if your are not able to efficiently process the output?   But can we really use it in finance? Can we allow an algo going berserk to burn money and face prosecution?   Maybe what we need is the very opposite to agile?
• #7: There are areas like transport, medicine, nuclear power that have more rigorous controls. Quality assurance methods evolve there. Advances in model checking allowed PROVING as opposed to ASSUMING that something will work. There was a post in the TABB Forum a few months ago questioning why are we not using mathematics and computer science similar to what NASA uses to design safe autopilot?   Is not it embarrassing that we have outages in the financial markets at the same time when spacecrafts are gliding over the outskirts of the solar system? We can invest heavily into formal methods to have more stable financial markets?  Well we can. But, the thing to remember is that despite of all the formal methods the New Horizons probe had an outage that knocked it out of contact with Earth for three days.
• #8: To expect that your systems will break is the only way to improve the quality of your software. So what's next? Can we learn something from other industries? Be both faster and safer. To understand how to do it lets look into the Doomsday scenario. Lets talk about people who had severe problems with technology.
• #9: Battlestar Galactica and Sarah Connor. The only reason the characters were able to make it until the end of the second season is that they were relying on the very same technology that was trying to kill them. They were protected by systems having the same level of sophistication as those they were struggling with.   This is the mentality that we should adapt. When we use a complex platform, we need to build software to test our software.   And you do not want your risk control and test instruments to be inferior to what will hit you. Having a good robot on your side is the only way to survive the robot apocalypse.
• #10: So, what's the impact on the trading technology?  Developing testing instruments and monitoring tools should has the same priority as the trading platform itself.   But, can we use agile process and continuous integration? Yes, it is possible, when your system is designed with testability in mind.  You can deploy, start, restart and configure it automatically.   Software engineer in test approach is just building software to test software. Do not expect formal artefacts from the agile process. Have a parallel stream instead to develop the necessary test harness.   We can use formal verification. But, the software will break anyway. The absence of adequate monitoring and kill switches is what turns a problem into a disaster.  
• #11: What about audit and regulatory requirements? Develop passive testing tools. They do not generate any messages themselves. They capture the traffic and store it for analysis.   Passive testing tools can gather all the evidence you need. Ensure that your trading system can co-exist with the tools.   Passive testing tools can also serve as both surveillance and client on-boarding systems. You can use them for the crowd-sourcing as well.   If your test environment is opened to others and you have a strong passive test capability you'll be able to get a lot of valuable feedback without even asking for it.
• #12: LSEG's Technology services division incorporates MillenniumIT, GATElab and Exactpro. As a result, we have access to a unique set of software from all three companies and can use it to build test infrastructure for any system.
• #13: Lets consider testing trading algos. We can deploy one or several scalable matching engines to serve as market simulators. We can deploy GATElab’s algopath or Exactpro tool called Mini-Robots to act as market participants. They can both replay some data and at the same time mimic a realistic market impact. In addition, we can have latency and front-running proxies for better tests diversity.   It is possible to take the next version of the algo and put it to the test. However, what is the point of running it once? Markets are not deterministic. Good simulated markets are not deterministic either. We will get a slightly different result every time we run the tests. You need to repeat your tests multiple times to assess the quality and efficiency of your software.   This is where everything I mentioned earlier comes into place: 1) Efficient restart and testability to enable unattended test execution 2) Machine-readable specifications to synchronize all the tools and formally iterate the possible tests 3) Passive testing capabilities and potentially market surveillance to capture and store all the relevant metrics (both P&L and system related ones)
• #14: You can run as many tests as you like and pass your system through a wide range of scenarios. But, we all have limited time and also limited hardware.   It is worth mentioning now two more quality assurance trends. You need to efficiently select a limited subset of tests from an infinite number of possibilities. Cognitive technology and mutation testing are there to help. It is possible to use them to find the optimal test scenario subset for your trading platform. You can constantly enhance the test set
• #15: Thank you very much.   Please do come to St Petersburg, Russia for ExTENT Conference in November if you would like to learn more about trading technology trends and quality assurance. I will take your questions now if there are any