webhooks unstable

[Yıldırım]

Hello, 

we're currently investigating your report. We'll get back to you shortly with information.

[SKAU AS]

The webhooks are deliverd slow today. Is there problems with webhooks today?

Looks like there is a delay of 10-12 minutes.

Example: External action triggered in visma

Date :8/24/2020 11:30:28 AM

{"EntityId":"b1686b5e-4fa4-4eb3-ac90-7e1a45ce2533","NotificationId":"836ae5ea-b5a9-407a-b35a-d5c45a9dead9","Event":"SalesOrder_Action_Triggered","ActionCommand":"wallmob","TimeStamp":"637338654280630000","EntityLocation":"api/v1/salesorder/004016"}

Webhook recived.

AUGUST 24, 2020 11:42 AM

[
    {
        "notificationId": "836ae5ea-b5a9-407a-b35a-d5c45a9dead9",
        "contextId": "1089046",
        "event": "salesorder_changed",
        "relativeResourceUri": "api/v1/salesorder/004016",
        "resourceUri": "https://integration.visma.net/API/controller/api/v1/salesorder/004016",
        "action": "wallmob",
        "timeStamp": 637338654280630000
    }
]

[Yıldırım]

Hello, 

we're currently investigating your report. We'll get back to you shortly with information.

[Jarno Korkola]

Hi,

Couple customers here in Finland have also reported that there is problems with integrations. Seems like there is a some issues with webhooks.

[Yıldırım]

---

@Jarno Korkola wrote:

Hi,

Couple customers here in Finland have also reported that there is problems with integrations. Seems like there is a some issues with webhooks.

---

Can you share Notification IDs / Timestamps of webhook notifications that have experienced the delay ? 

[SKAU AS]

NotificationId":"ae87d7c6-3ef1-45e5-a323-69017b98d712"

TimeStamp":"637338798617600000"

[Yıldırım]

Hello, 

Infrastructure team is currently working on the incident to determine the root cause and resolve this as soon as possible. Updates can be followed at https://status.visma.com/incidents/92pzgfclg866

[Yıldırım]

Hello, we received an update on the current status of the investigation,
The delay in sending the webhooks is caused by high load on the DB. This has improved now, but we still have a delay on this. Currently, it's under one minute. We will continue to monitor the situation.

[SKAU AS]

Today we are experiencing even slower/ non working. None of the webhooks sendt today is recived.

This is the latest webhook

{"EntityId":"b1ac5ae9-6ba6-417d-96d1-3f364cadfc3c","NotificationId":"d3a81e75-0a0b-41f0-9cb5-46671f09595f","Event":"InventoryItem_Updated","TimeStamp":"637339582457500000","EntityLocation":"api/v1/inventory/90002+++++++++++++++"}

This is todays first webhook sendt today.

{"EntityId":"b1ac5ae9-6ba6-417d-96d1-3f364cadfc3c","NotificationId":"63fbb15c-b4ea-454a-a9d9-21a156adaf05","Event":"InventoryItem_Updated","TimeStamp":"637339577746630000","EntityLocation":"api/v1/inventory/90002+++++++++++++++"}

[Yıldırım]

Thank you for your feedback. We've informed the team about this. Investigation is still ongoing.

[Yıldırım · Accepted solution]

Hello,

the fix has been deployed, please let us know whether you are still experiencing the delay with webhook notifications. We will continue to monitor the situation closely. 
https://status.visma.com/

[SKAU AS]

No problems today Good work! 

[Yıldırım]

Thanks, great to hear that John ! Please let us know in case of any issues.

[SKAU AS]

There is a delay on webhooks today.  About 3 minutes.  

Can you check this on? 

{"EntityId":"8c7e5425-7815-45d5-a7ff-f40160c2b0e9","NotificationId":"6d5569c9-9e31-44ec-884a-da922293d00a","Event":"SalesOrder_Action_Triggered","ActionCommand":"wallmob","TimeStamp":"637347320786830000","EntityLocation":"api/v1/salesorder/004088"}

[Yıldırım]

Hello, we've informed the team and this is being checked out now.

[Yıldırım]

More information will be provided as it becomes available.Updates at: https://status.visma.com/incidents/8dpkcy0qtyf5

[Yıldırım]

Hello, 

The team is currently working on a fix, it has been planned to be deployed today. This may vary depending on the process. More conclusive details about the deployment will be published at https://status.visma.com/incidents/8dpkcy0qtyf5

[Yıldırım]

Hello, 

Fix has been deployed, the team will continue monitoring. Please let us know if the problem persists.
https://status.visma.com/incidents/8dpkcy0qtyf5