URGENT! 500 error in salesorder api in GET request after 01 today.

[Gabriella Hennings]

Hi 

after approx. 01 today we only get internal server error from salesorder api, in the GET request. 

we get the errors when adding the modifiedSince parameter? it has been working fine until tonight. 

I have also tried to run the request in swagger, same there, 500 error. and no message. (I have also tried 2 different customers, same error)

have you made any updates??

This is really urgent since we don't receive any orders back. 

I see that the naming of the lastchanged parameter seems to be different in different requests, but I don't know if it has always been like that or if that has something to do with it.

[Gabriella Hennings]

Our integrations are also up and running again. 

But I would like to hear an explanation from Visma how this can happen. It causes severe issues for our customers.

[Cristian Gherghinescu]

Hi,

Getting back with more details about the problem from 21st of June.

Around the interval: 2023-06-21T02:10:00.406+03:00 and 2023-06-21T10:38:26.406+03:00 most of the calls to the Sales Order Service API returned 500.

We have restarted the cluster of the service and then calls got back to normal.

While investigating the cause of this we noticed that the calls our service makes to another internal service to retrieve information about a tenant are failing with an Unauthorized exception. From the logs it seemed that Authorization header was not included in those calls. Restarting the cluster fixed this problem but we created a task to further investigate why the Authorization header was not included and another case to improve the code to be more resilient when making calls to this internal service.

Another action point for us is to improve the monitoring. The current monitoring detected the errors but we did not have any alarms set for a high rate of them. We will implement a warning when the error rate reaches 20% and a critical alarm for Central Operations when the rate reaches 50%. This will allow us to respond faster when something similar might happen.

[Joni Wolff]

Hi! We are doing a post mortem and will publish it when ready

[Theis]

We are now able to process orders again.

[Cristian Gherghinescu]

There have been an increased number of 500 responses.

Now the service is stable again

[Gabriella Hennings]

we seem to have problem with all endpoints in the salesorderapi, same with the customer endpoint.

we have several integrations not working at the moment. this is critical and not acceptable!

[Steinar Møller]

This is Critical!
Same issue for us and reported from our customers using this endpoint,  please let us know when this will be fixed. 

[Theis]

Not sure if it is related, but we are unable to POST on Salesorders V3 as of this morning. Also getting error code 500.