In this Case Study we look at a number of changes in a key system. The client collected much of their income towards the end of the month when their customers got paid; these ‘Payment Processors’ were mission-critical. By concentrating on current needs, we were able to help improve the processors in a number of ways, and deliver the required changes quickly.
The problem 2008
Payment runs were taking so long that human intervention during the run could remove the need to take a payment at all. Locking data for the whole run was not an option, but we were able to increase resilience and reliability by double-checking all data before attempting a payment.
The result of these changes was that fewer customers were inconvenienced, and in addition, less time was spent by Call Center and Technical staff in resolving the issues that arose.
The problem 2009
Business was growing fast and the month-end payment runs were taking hours. 42 IT Solutions staff removed expensive parts of the processing that did not need to be done at the same time as the payment attempt. We then converted the processor to run in a multi-threaded way, with the option to change the number of threads in the command. Throughput was raised from 20 or less payments per minute, and now rises in excess of 800 per minute (while still allowing for future growth). Additionally, making different runs work nicely side-by-side means that the client can now attempt several thousand payments per minute. Towards the end of the month, collections now peak around £250,000 per minute.
The business needed estimates of when processor runs would complete to enable decisions to be taken about when other runs and processes could be run. We provided on-screen estimates of when various threads would complete, improving their ability to plan when they would next run a process.
The problem 2012
Implementation of new Object Relational Mapper (ORM) software in the system started to change the performance, although this was not an anticipated consequence of the change. Monitoring implemented by 42 IT Solutions brought these issues to light, and demonstrated a slow-down in throughput of several batch processes and most importantly in the Payment Processors. However, even with the basic evidence provided by graphs of throughput over time, it was not a trivial process to get the business and other technical teams to acknowledge the problem – in a sense, as everything appeared to work (albeit more slowly), many technicians and much of the business considered this ‘normal operation’.
After repeatedly highlighting throughput problems in many processes and situations, we finally got technical staff to investigate, and eventually they found several configuration issues in the new ORM. When these problems were fixed, we saw marked improvements in performance in many areas, including the payment processors.