Webhook functionality has been completely restored as of October 22 around 7 PM PT.
The impact was that some webhooks that should have been sent between October 5 and 22 were delayed, and were instead sent on October 22nd between 7 and 8 PM PT.
The team is in the process of implementing monitoring/alerting to prevent this situation from occurring again.
While working to implement a permanent fix to the webhook communication issue, the backlog of pending webhook messages that were stuck were released and sent out. For the customers who use our webhook service, you may see a temporary increase in webhook events being posted to your order or inventory management system that are subscribed to our webhook service. Some of these messages may be duplicate purchaseorder.complete or order.complete events you received earlier.
Upon review, we found that the workaround we implemented at 8:32 PM PT on Tuesday (10/19) restored automatic, real-time webhook sending to customers. This means our manual sending of Purchase Order and Fulfillment Order completed webhook events on an hourly basis was unnecessary, so we have stopped doing that.
A few customers have reported that some webhooks are including X-Shipwire-Signature headers that cannot be validated on their end. While we look into this, we request that you disable Shipwire webhook secret validation.
Our Engineering team continues to investigate this webhook issue and the root cause of it, so we can implement a permanent solution and provide assurance that this will not occur again.
Our Engineering team continues to investigate this webhook issue and the root cause of it. While we continue to investigate, we have come up with a work around that will allow us to trigger Purchase Order and Fulfillment Order completed webhook events to be sent out to our customers on an hourly basis. While this work around is in use, any changes to your webhook topic subscriptions will not take effect in real time and will experience some delay before it reflects in the Shipwire platform. This includes creating new webhook subscriptions, editing current subscriptions, or deleting a subscription. Once our Engineering team identifies and resolve the root cause, real time webhook updates will be restored as well as you're subscriptions updates reflecting in real time.
We are currently looking into an issue where some customers may not be getting events sent to them via webhooks. If you are a webhook subscriber and need to lookup order information while we are addressing this issue, please use our merchant dashboard. The dashboard can be accessed at https://merchant.shipwire.com/