On Monday, 16 September 2019, PPO experienced a service disruption of two minutes. During this period, users would have seen an Internal Server Error page. The failures were due to our application servers and database servers being out of sync as a result of a manual deployment mistake. At 08:35 AM (UTC+0), a deployment was … Continue reading “2019-09-16: PPO service outage”
On Monday, 11 September, a change to our Single Sign-On (SSO) functionality containing a bug was deployed to production which prevented unauthenticated SSO users from signing in. Herewith the details of incident, the root cause and our mitigating actions. At 13:58 PM (UTC+2), a deployment was done to our application servers containing new SSO features. … Continue reading “2017-09-11: PPO service outage”
On Friday, 7 October, a bug was introduced into our production environment that resulted in some users not being able to authenticate and necessitated us to take the service off-line to perform a rollback. Below is a detailed explanation of the events surrounding the incident, our investigation into the root cause, as well as the … Continue reading “2016-10-07: PPO Service Outage”
On Tuesday, 17 May 2016, we suffered a partial outage which resulted in some users not being able to access PPO or experiencing severely degraded response. The outage started at 12:04 PM (UTC+0) and was fully resolved at 12:18 PM (UTC+0). Below is a detailed explanation of the events surrounding the incident, our investigation into … Continue reading “2016-05-17: Partial PPO service outage”