Category: Incidents

2019-09-16: PPO service outage

On Monday, 16 September 2019, PPO experienced a service disruption of two minutes. During this period, users would have seen an Internal Server Error page. The failures were due to our application servers and database servers being out of sync as a result of a manual deployment mistake. At 08:35 AM (UTC+0), a deployment was … Continue reading “2019-09-16: PPO service outage”

2017-09-11: PPO service outage

On Monday, 11 September, a change to our Single Sign-On (SSO) functionality containing a bug was deployed to production which prevented unauthenticated SSO users from signing in. Herewith the details of incident, the root cause and our mitigating actions. At 13:58 PM (UTC+2), a deployment was done to our application servers containing new SSO features. … Continue reading “2017-09-11: PPO service outage”

2016-10-07: PPO Service Outage

On Friday, 7 October, a bug was introduced into our production environment that resulted in some users not being able to authenticate and necessitated us to take the service off-line to perform a rollback. Below is a detailed explanation of the events surrounding the incident, our investigation into the root cause, as well as the … Continue reading “2016-10-07: PPO Service Outage”