On Monday, 16 September 2019, PPO experienced a service disruption of two minutes. During this period, users would have seen an Internal Server Error page. The failures were due to our application servers and database servers being out of sync as a result of a manual deployment mistake.
At 08:35 AM (UTC+0), a deployment was started and soon thereafter, at 08:36, we received notifications of requests failing. Upon realising the problem, corrective measures were immediately taken and the service was restored at 08:38.
We would like to apologize for any inconvenience that resulted from this incident and will certainly make sure that we learn from this and use it to drive improvements across our services.