Customer API: Get all invoice endpoint performance issue

[Anonymous]

Hello team, 

For one of our requirement, we would need to have all the documents of a customer with its latest payment date. Currently, to achieve this, we are hitting "api/v1/customer/{customerNumber}/document" endpoint to fetch all the documents and parallelly hitting "api/v1/customer/{customerNumber}/invoice" to get all invoices (by setting "expandApplications" true) and merging the "Applications" array from invoice result with the document result.

But the problem here is the performance, due to loads of data, invoice endpoint does not have very good response time and it is almost taking 10s minimum for a request with all the required filters. Without filters, sometimes it takes up to 40s (maximum). However, document endpoint works pretty much faster.
Here are the details observed during our test,
https://integration.visma.net/API-index/controller/api/v1/customer/{customerNumber}/invoice

Case1:

Request from no.of application at a time: 1
Average Response Time: 20-30s (without any filter), 10-15s (with Status filter),  6-11~(with paging of 100), 4-8~ (with paging and status)

No. of records returned: 220~

Case2:

Request from no.of application at a time: 1
Average Response Time: 10-15s (without any filter), 4-10s (with Status filter), 3-6~(with paging), 2-5~ (with paging and status)

No. of records returned: 78~

We indeed use paging to fetch only a maximum of 100 records at a time but we would need all the data of a customer to do some calculations so we have to wait until all records are fetched which depends on a number of invoices the customer holds and on total the aggregated response time gets higher. Also, we tested with one user at a time and if many users are using our application and hitting the endpoint at the same time causes still slower.

It would be much helpful if you could invest some investigation on this endpoint and make it much faster than now or any suggestions to fetch light weighed model instead of loading loads of data.

We also observed that setting "expandApplications" to true on document endpoint -> "/customer/{customerNumber}/document" doesn't fetch applications details in the result. Not sure if it is a bug or something shouldn't be a parameter at all for this endpoint. But it would be great if this is fixed and application details are being sent along with documents because we don't require additional request to invoice endpoint then.

Keen on knowing how the performances are throttled in a different environment, so any explanation on this will be helpful for us to understand it better.

Looking forward to your reaction on this.

Thanks,
Naveen S

[Magnus Johnsen]

Hi,

Just to clarify, the only reason you are using the customer/customerNr/documents endpoint is that it is faster that just getting customer/customerNr/invoice, but the only thing you want from the documents endpoint is the invoices?

To reduce the load, we would advise that you use more parameters to limit the amount of data that needs to be fetched, for example using the status and lastModified, depending on the information you need from the result, already paid documents could be stored on your end to limit the data needed to be fetched.

Also, the most optimal way to handle this is to have 10 requests running parallel when talking about time.

As of now, 100 is the default maxPageSize, so unless you specify a lower number, the pageSize will always be 100.

[Anonymous]

Hi Magnus, 

Thanks for coming back to me on this. I believe you meant the document type 'Invoice', in that case, no. We chose document endpoint as we needed all types of document and its basic details which we comfortably got it from it. But recently we had a requirement to include the latest "PaymentDate" of all closed document which is not available in document endpoint. That's where we chose invoice endpoint since the "Applications" details are enclosed here.

We actually don't store any data at all our side and we just provide a panel to display the data fetched from Visma. We try to use as many parameters we could but restricting by date won't be useful as we need all the records of a customer.

Sorry, I quite don't understand 10 requests running in parallel, do you mean triggering 10 requests to the same endpoint and combine results? In that case, I doubt we could not predict the end of the result.

[Magnus Johnsen]

Alright, so if I understand you correctly, up until now when you have not had the need of presenting the payment date, there has been no problems, but since you've been asked to include this, you have hit a "performance wall"?

Regarding the 10 requests parallel, according to tests made by one of our users, the best performance is reached by running 10 parallel requests, meaning make sure that you do not just have 1 request going at a time, but say that you have 12 pages you need to get, send the request for those first 10, whenever any of the requests are done, send of the next one and so on.

[Anonymous]

That's right. Since when we need to have some field from invoice endpoint, we hit the performance issue.

I can understand about running parallel requests, but how can we know the page size in upfront for a customer? Is there is any way in place already?

[Magnus Johnsen]

Yes, there is a default maxSize, this can be changed at any point, you can see the latest default if you have a look at the news page: Enforcement of pagination on API endpoints and change of maxPagesize - September release (8.25)

We will always try to make sure to share any changes to pagination settings on the news tab.

As of right now, the default maxSize has been changed to 500, as can be seen in the metaData in the response of the request.

This means that you now that unless you provide pageNumber & pageSize that is lower than 500, you will always get a response that contains up to 500 documents from page 1. 

So the way that you would know it upfront depends on your integration, are you displaying all documents in one list or do you use pages? If you have a pagination in your integration you would know by sending the parameter with that page size, and you know that if you do not set it, the default size will be 500 as of right now. 

[Magnus Johnsen]

Hi,

We can confirm that the customerDocument endpoint does not contain the "applications" field, we will open a case with the developers and get back to you in this thread when we have more information from the developers. 

[Anonymous]

Hi Magnus,

Thanks for taking up this 'Applications' field addition to "customerdocuments" endpoint. If this is done, I hope it will indeed solve our requirement easily. I'll also try to play around with page sizing and multiple requests to the API and understand how to achieve better performance results.

Looking forward to hear from you!