Category: Incidents

2017-09-11: PPO service outage

On Monday, 11 September, a change to our Single Sign-On (SSO) functionality containing a bug was deployed to production which prevented unauthenticated SSO users from signing in. Herewith the details of incident, the root cause and our mitigating actions. At 13:58 PM (UTC+2), a deployment was done to our application servers containing new SSO features. … Continue reading “2017-09-11: PPO service outage”

2016-10-07: PPO Service Outage

On Friday, 7 October, a bug was introduced into our production environment that resulted in some users not being able to authenticate and necessitated us to take the service off-line to perform a rollback. Below is a detailed explanation of the events surrounding the incident, our investigation into the root cause, as well as the … Continue reading “2016-10-07: PPO Service Outage”