Issue with filtering of attributes

[Florian Haase]

Hi,

we just found an issue with checkbox - attributes and filtering in GET Methods:

We have customers in  Visma.Net and AFTER the customers where imported we  introduced a new checkbox attribute (feks Member). Here the GET with attributefilter: {"MEMBER":"0"} will not give any result even if all the checkboxes are unchecked.

If we now set the Checkbox to Checked for one of the customers and save, the filter {"MEMBER":"1"} will return this one customer.

When we now uncheck this customer again and save, the API on the GET ({"MEMBER":"0"}) will just return this one customer but not all the others which not are checked. I understand the datadesign back this issue and I think we can avoid this by run a mass-update on all the customers. But maybe good to be aware of this problem.

Regards,

Florian

[Magnus Johnsen]

Hi,

So if I understand you correctly, you have attributes set on customerClass, have a bunch of customers with that class created, add another attribute and then then try to find the customers where this has not been ticked in?

[Florian Haase]

Yes right.

After we have checked and unchecked all customers we get the right number of customers.

But now we got another problem, and thats actually serious:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22RESERVERT%22%3A%220%22...

That call returns the right customers and the right number of customers - all nice. BUT:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22RESERVERT%22%3A%220%22...
returns a blank list - nothing here - no metadata either.

if I browse to page 5 i get a list of customers and the metadata:
"metadata": { "totalcount": 378 }

but I have no idea how many customers there are and on which pages - thats not possible to handle in this way.

I should get the result on the first page in this case.

Florian

[Florian Haase]

Be aware of that we set filter here on 2 attributes at the same time. Maybe this causes the problems....

[Magnus Johnsen]

Hi,

We are testing this and will get back to you with our results. 

[Magnus Johnsen]

Hi,

We have tested this and the correct syntax for finding multiple attributes is as below:

customer?attributes={"DATETEST":"2019-01-01","TEST":"testvalue"}

When encoding, the equal signs should not be encoded, both examples below works:

customer?attributes=%7B"DATETEST":"2019-01-01","TEST":"testvalue"%7D

customer?attributes=%7B%22DATETEST%22%3A%222019-01-01%22%2C%22TEST%22%3A%22testvalue%22%7D

Regarding the pagination,

The returned metaData totalCount:

We have a case in the backlog for this, the issue is that the operation first gets all objects for the endpoint, and then skipping the objects that don't match the attributes. So the totalCount is set during the initial SQL search.

The solution for this is to create new V2 endpoints where the JOIN is done in the initial SQL query.

The result when using pagination is incorrectly returned:

We can confirm that this is happening and will open a case with the developers. Progress will be posted in this thread.

In our test:

customer?attributes={"TEST":"testvalue"}&pageNumber=1&pageSize=1 : Two objects match this criteria.

• pageSize=1:
  • pageNumber=1: Incorrectly returns empty array
  • pageNumber=2: Returns 1st object
  • pageNumber=3-4: Returns empty array
  • pageNumber=5: Returns 2nd object
• pageSize=2:
  • pageNumber=1: Returns 1st object
  • pageNumber=2: Returns empty array
  • pageNumber=3: Returns 2nd object
• pageSize=3 & 4
  • pageNumber=1: Returns 1st object
  • pageNumber=2: Returns 2nd object
• pageSize=5
  
  • pageNumber= 1: Returns both objects as expected

[Florian Haase]

Should we skip the pagination on the endpoint until V2 is ready?

[Magnus Johnsen · Accepted solution]

Hi,

It depends on if it will be enforced or not, for this endpoint. 

We'd say that until the bug with on what page the results shows up in is resolved, you won't be able to properly handle the requests.

For the case of the metaData, as long as the above is working as expected, GET ALL works for fetching all objects with pagination, termination of the query would be when you have increased pageNumber until a empty array is returned.

But as mentioned, for this to work, the first point needs to be working. 

[Trygve Storrønningen1]

Skipping pagination is not a solution for us; we are waiting for followup and final solution in this thread.

[Magnus Johnsen]

Hi,

We don't think that skipping pagination should be done in any case, all we are saying is that for now, when using pagination and filtering together it is not working.

We fully agree that this needs to be resolved and the case has been opened with high priority.

Thank you. 

[Trygve Storrønningen1]

Making totalCount available in the response header would be more reliable -- as it is now the empty response also give no totalCount metadata. 

One case; the customer entity has been alive for some time, and contains 3197 entities. 

Only recently the attribute signaling we should pick it up is introduced. 

The attribute is a checkbox, and we check it on five recent customers for test.

We use the URLs given above to GET the customers.

The five customers returned is now in the "far end" of the recordset.

Meaning; when GET with 

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22...

I get the empty json in response.

Only after reading 31 empty respones and not giving up, I get the five new customers.

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22%7D&pageNumber=32&pageSize=100

 If the totalCount is avaiable in response header, the json response can miss or be maleformed, we still know the size of the recordset. And can at least start try paginating to some page far, far away 😉

This was voted over here:

https://www.visma.com/software/support/portal/forum/showthread.php?12866

We should of course have been more awake when this was voted over; but we missed it.

With all respect, you as system owners also missed it - designing it by result of vote.

My conclusion:  "totalCount only in response body? no no no...."

So, maybe totalCount also in response header? 🙂

[Florian Haase]

Yes, 

I agree that it could be nice to have the totalCount in the header too. But it still would not help to know that you can expect 30 customers if they are spread over 50 pages where 30 pages are just blank. 

In our integrations we just fetch the next page until we get no result again (so far), and don't use the totalCount. But I'm actually not sure what the best way to do is. First this issue has to be fixed and the filter should not just delete the records from an existing result set as mentioned above. I hope that the V2 endpoint is coming soon.

Florian

[Trygve Storrønningen1]

I agree the issue have to be fixed - and wait for V2 -- but it applies for all endpoints with pagination so some work to do for Visma.

It would actually help with totalCount in header; because now I have no idea how many calls to do.

I get empty json and 200 when calling the first page.

In this case I also get empty json and 200 when calling the next 30 pages.

I also get empty json and 200 when calling non-existing page 55.

So how to know when to stopp trying?

At least if I knew there are 3197 records; I could call only 32 pages of 100.

totalCount in header would be a pain killer while waiting for the real treatment -- V2

[Trygve Storrønningen1]

We can also confirm this problem.

When we combine pagination and attributes filtering, we get empty response in testclient -- where we just recently added these attributes and started using them. In production this attribute has been present for a long time.

With production company 1555332;

Correct response with this URL without pagination:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22...

Correct response with this URL with pagination:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22...

With test company 1555345:

Correct response with this URL without pagination:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22...

Empty response with this URL with pagination:

https://integration.visma.net/API/controller/api/v1/customer?attributes=%7B%22DDLDKUNDE%22%3A%221%22...