Server Instability & Capacity Issues
Incident Report for Ikonik Digital Agency
Postmortem

We apologize for the downtime experienced as we know how valuable your web assets are to your businesses. We have investigated the problem which occurred thoroughly.

The Root Cause

The root of the problem came from an update to our PHP modules on Sept 28 2022 at 2:00 AM CDT. The update included two additional modules which would have enhanced the hosting environment and allow more integrated and advanced web development. Upon enabling these modules, there was no evidence of incorrect implementation. However, as the day progressed our monitoring bells went off, first at 6:30 AM CDT. We also had reports from a few WordPress clients which brought us closer to the issues at hand and helped us in rectifying the issues more speedily. We made a decision to disable all modules enabled in the update until we make a full assessment on their impacts on our server.

What have we learnt?

In moving forward from this incident, we have learnt to thoroughly test each implementation and update before pushing it to production, no matter how small or big. Testing should include all major assets like websites, scripts and apps which exist on our servers after any update to the core server functionality and modules. The results from a bad implementation can lead to very impactful and costly downtimes as seen with this incident, with full acknowledgement of the incident we hope to mitigate these occurrences.

Posted Sep 30, 2022 - 17:04 CDT

Resolved
We are currently investigating an issue where our server was experiencing some inconsistencies, problems of instability and capacity.

We were notified of the problem at 6:30 AM CDT. Further details were received on the extent to which the WordPress sites were affected which helped us to locate the problem and put a fix in place. The server operations were monitored closely. The issues were fully resolved at 4:30 PM CDT.
Posted Sep 28, 2022 - 04:30 CDT