What products were affected and what was the impact?
Shopify Duty & Tax
Impact: CRITICAL
What timeframe did this issue occur?
| Date | Time | |
|---|---|---|
| From: | May 20th, 2023 | 09:30 MST |
| To: | May 20th, 2023 | 10:30 MST |
How was the issue detected?
At 9:31 AM MST, we began receiving alerts of increased latency with some Shopify Duty & Tax quote requests.
What functionality was affected?
Shopify Duty & Tax quotes for some customers.
What problems did this cause?
From approximately 9:30 am to 10:30 am MST, latency was sufficiently elevated to cause some quote requests from our Shopify Duty & Tax plugin to fail.
What was the resolution of the problem and steps that are being taken for continued follow-up?
We identified that the issue was caused by a scheduled maintenance job that obtained a lock on the database. The database lock significantly increased latency.
What mitigation solutions will we put in place to prevent this issue from occurring in the future?
The database lock was not necessary, and the locking code has been removed. This will prevent this issue from happening again in the future.
Also, we are educating our engineers on the proper usage of database locking strategies, and implementing protection measures during the code review process. Additionally, we are improving our monitoring and on-call coverage to ensure faster response times to issues that impact shoppers.