Mijn Communities
Help

File API - Uploading files (Publisher)

21-04-2023 16:57 (Bijgewerkt op 21-04-2023)
  • 0 Antwoorden
  • 0 kudos
  • 2642 Weergaven

 

Uploading files - Publishers

Publisher is a Client Application that can upload files of authorized business types and authorized tenants. The uploaded files can also be listed/downloaded.

 

File upload

Recommended usage of uploading files:
  • Multipart upload (<100MB: The file is uploaded in a single request. (100MB max). The metadata that describes the file is also added in this request.
  • Resumable upload (>100MB): The file is uploaded by chunks. (9MB max each chunk) The metadata that describes the file is added in the first request.

Small files (<4 MB):

  • Multipart upload when possible.
  • Resumable upload and smaller chunks if you have strict response time restrictions or your network connection isn’t reliable.

Medium files (> 4 MB and <100 MB):

  • Resumable when possible and chunks of 4MB. If you have strict response time restrictions or your network connection isn’t reliable, you could use smaller chunks.

  • Multipart when you don’t expect your files to be bigger than 100 MB (consider the file growth over time) and most of your files are small (<4 MB)

Big files (>100 MB):

  • Resumable only. When possible, use chunks of 4MB. If you have strict response time restrictions or your network connection isn’t reliable, you could use smaller chunks.

 

Multipart upload

A multipart upload request allows you to send metadata along with the data to upload. Use this option if the data you send is small enough to upload again in its entirety if the connection fails.

To use multipart upload:

  • Create a POST request to the method’s /files URI with the query parameter uploadType=multipart.

Example - POST request multipart

POST https://api.youserve.nl/fileapi/v1.0/files?uploadType=multipart
  • Add the top-level HTTP header: Content-Type. Set to multipart/related, and include the boundary string you’re using to identify the different parts of the request

Example - Content-Type header

Content-Type: multipart/related; boundary=foo_bar_baz
  • Create the body of the request. Format the body according to the multipart/related content type [RFC 2387], which contains two parts:

a) Metadata part. Must come first, and must have a Content-Type header set to application/json; charset=UTF-8. Add the file’s metadata to this part in JSON format. The required metadata fields are:

Key Value
FileName [FileName]
BusinessTypeId [BusinessTypeId]

 

Note: For security reasons, file names have some restrictions:

  • Filenames should fulfil the following characters sets:
[a-z],[A-Z],[0-9],[-],[_],[.],[(],[)],[,],[$],[+],[`],[=],[']
  • Malicious file extensions are not allowed

b) Media part. Must come second, and should contain the bytes of the files. Identify each part with a boundary delimiter. [RFC 2046] The boundary delimiter line is defined as a line consisting entirely of two hyphen characters (“-“, decimal value 45) followed by the boundary parameter value from the Content-Type header field, optional linear white-space, and a terminating CRLF. The boundary delimiter MUST occur at the beginning of a line and it is terminated by either the header fields for the next part and two additional CRLFs, or by two additional CRLFs, in case there are no header fields for the next part. Boundary delimiters must not appear within the encapsulated material, and must be no longer than 70 characters, not counting the two leading hyphens.

Example - Boundary delimiter for Metadata part:

--foo_bar_baz[CRLF]
Content-Type: application/json; charset=UTF-8[CRLF]
[CRLF]

Example - Boundary delimiter for Media part:

--foo_bar_baz[CRLF]
[CRLF]

The boundary delimiter line following the last body part is a distinguished delimiter that indicates that no further body parts will follow. Such a delimiter line is identical to the previous delimiter lines, with the addition of two more hyphens after the boundary parameter value.

Example - Boundary delimiter for last body part:

--foo_bar_baz--
  • Send the request:

Example - Send a multipart upload request using the Sandbox File Type

POST/mft/v1.0/files
?uploadType=multipart HTTP/1.1
Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx
Content-Type: multipart/related;boundary=foo_bar_baz

--foo_bar_baz
Content-Type: application/json; charset=UTF-8

{
    "name":"TestFile.txt",
    "businesstypeid":"7100"
}
--foo_bar_baz

This is a test file
--foo_bar_baz--

If the request succeeds, the server returns the HTTP 201 Created status code along with the file’s metadata:

Example - Response successful request

HTTP/1.1 201
Content-Type: application/json

{
    "id": "7ff1b498-169d-4048-8459-78f7bd7905a3",
    "name": "TestFile.txt",
    "size": 19,
    "creationDate": "2020-02-27T11:03:49.2033291Z",
    "tenantId": "sandbox",
    "businessType": {
    "id": 7100,
    "name": "7100"
},
    "numChunks": 1
}

If the request fails, the server returns the HTTP 400, indicating Bad request.

Example - Response failed request


HTTP/1.1 400
Content-Type: application/json

{
    "CorrelationId": "rrt-06f2e934a3cbdb710-a-de-13067-4959526-1",
    "Message": "Error reading body of request. Please, check all the boundaries of the request",
    "ErrorCode": "400",
    "Exception": null
}

 

 

Resumable upload

This protocol allows you to resume an upload operation after a communication failure interrupts the flow of data. Use this option if you transfer large files or if the likelihood of a network interruption or some other transmission failure is high
 

Initiate a resumable upload session and upload the first chunk of file

To initiate a resumable upload session:

  • Create a POST request to the method’s /files URI and add the query parameter uploadType=resumable.

Example - Initial Post Request Resumable

POST https://api.youserve.nl/fileapi/v1.0/files?uploadType=resumable
  • Add the top-level HTTP header: Content-Type. Set to multipart/related, and include the boundary string you’re using to identify the different parts of the request.

Example - Content-Type header

Content-Type: multipart/related; boundary=foo_bar_baz
  • Create the body of the request. Format the body according to the multipart/related content type [RFC 2387], which contains two parts:

a) Metadata part. Must come first, and must have a Content-Type header set to application/json; charset=UTF-8. Add the file’s metadata to this part in JSON format. The required metadata fields are:

Key Value
FileName [FileName]
BusinessTypeId [BusinessTypeId]

Note: For security reasons, file names have some restrictions:

  • Filenames should fulfill the following characters sets:
[a-z],[A-Z],[0-9],[-],[_],[.],[(],[)],[,],[$],[+],[`],[=],[']
  • Malicius file extensions are not allowed

b) Media part. Must come second, and should contain the first chunk data of the file.

Create chunks of a maximum of 9MB. It’s recommended to have chunks with the same size, except for the final chunk that completes the upload. The best performance using this method is achieved with chunks of 4mb.

Identify each part with a boundary delimiter. [RFC 2046] The boundary delimiter line is defined as a line consisting entirely of two hyphen characters (“-“, decimal value 45) followed by the boundary parameter value from the Content-Type header field, optional linear white-space, and a terminating CRLF.

The boundary delimiter MUST occur at the beginning of a line and it is terminated by either the header fields for the next part and two additional CRLFs, or by two additional CRLFs, in case there are no header fields for the next part.

Boundary delimiters must not appear within the encapsulated material, and must be no longer than 70 characters, not counting the two leading hyphens.

Example - Boundary delimiter for Metadata part:

--foo_bar_baz[CRLF]
Content-Type: application/json; charset=UTF-8[CRLF]
[CRLF]

Example - Boundary delimiter for Media part:

--foo_bar_baz[CRLF]
[CRLF]

The boundary delimiter line following the last body part is a distinguished delimiter that indicates that no further body parts will follow. Such a delimiter line is identical to the previous delimiter lines, with the addition of two more hyphens after the boundary parameter value.

Example - Boundary delimiter for last body part:

--foo_bar_baz--
  • Send the request.

Example - Initiate a resumable upload session using the sandbox File Type

POST/mft/v1.0/files
?uploadType=resumable HTTP/1.1
Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx
Content-Type: multipart/related;boundary=foo_bar_baz

--foo_bar_baz
Content-Type: application/json; charset=UTF-8

{
  "name":"TestFile.txt",
  "businesstypeid":"7100"
}
--foo_bar_baz

This is the first chunk of the file
--foo_bar_baz--

If the session initiation request succeeds, the response includes a 206 Partial ContentHTTP status code, along with the upload token. Copy and save the upload token, so you can use it for uploading the next chunks of the file.

Example - Successful Response of initial request

HTTP/1.1 206
Content-Type: application/json

{
  "uploadToken": "4927b6ffb51445ee982237ab6e4b3b30"
}

Note: The upload token expires after one hour.

If the request fails, the server returns the HTTP 400, indicating Bad request.

Example - Response failed request

HTTP/1.1 400
Content-Type: application/json

{
  "CorrelationId": "rrt-06f2e934a3cbdb710-a-de-13067-4959526-1",
  "Message": "Error reading body of request. Please, check all the boundaries of the request",
  "ErrorCode": "400",
  "Exception": null
}

A few checks for mitigation:

  1. Setting the right boundary definition for multipart/form-data body. Be mindful of setting two consecutive dashes (for example, –foo_bar_baz–) to mark the end of the boundary within the body of the HTTP request and two additional CRLFs after the boundary delimiters.
  2. If you are using Azure Logic Apps, go to the Logic App Code View. Within the HTTP body definition, replace every \n with \r\n. This is the key for successful file upload with multipart/form-data HTTP request using Azure Logic Apps.

Upload next chunks of the file

To upload the file in multiple chunks follow the steps:

  • Create a PUT request to the method’s /files URI.
    • Add the query parameter uploadType=resumable
    • Add the query parameter uploadToken=[uploadToken]
    • Add the query parameter position=[position]

Example - Request to upload next chunks of the file

Put v1.0/files
?uploadType=resumable
&uploadToken=[uploadToken]
&position=[position]
  • Add the chunk’s data to the request body. Create chunks of a maximum of 9MB. It’s recommended to have chunks with the same size, except for the final chunk that completes the upload.

  • Add the top-level HTTP header Content-Type. Set to application/octet-stream

  • Send the request, and process the response. If the upload request is interrupted, or if you receive a 5xx response, send the request again. If the request succeeds, the server responds with 206 Partial Content, along with the upload token.

  • Repeat steps 1 through 7 for each chunk that remains in the file. It’s possible to parallel the upload of the intermediate chunks. The last chunk needs to be sent at the end of the requests, to close the session once that you have received confirmation that the previous chunks have been stored successfully.

Example - Upload next chunks of the file. This example shows a request that sends the next bytes of the file, and uses the upload token obtained in the previous step:

PUT/mft/v1.0/files
?uploadType=resumable
&uploadToken=4927b6ffb51445ee982237ab6e4b3b30
&position=1 HTTP/1.1
Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx
Content-Type: application/octet-stream

And this is the second chunk of the file

If the request succeeds, the server responds with 206 Partial Content

Example - Successful response of intermediate request

HTTP/1.1 206

Complete a resumable upload session

There are two ways to complete a resumable upload session:

a) Add the query parameter close=true in the last PUT request.

b) Send additional Post request without bytes.

Complete a resumable upload session in the last PUT request

To complete a resumable upload session in the last upload request follow the steps described in section: “Upload next chunks of the file” and

  • Add the query parameter close=true
  • Add the top-level HTTP header: Content-Type. Set to application/octet-stream

Example - Put Request to complete a resumable upload session

Put https://api.youserve.nl/fileapi/v1.0/files
?uploadType=resumable
&uploadToken=[uploadToken]
&position=[position]
&close=true

If the request succeeds, the server responds with 201 created, along with the metadata of the file.

Example - Complete a resumable upload session in the last upload request

PUT/fileapi/v1.0/files
?uploadType=resumable
&uploadToken=4927b6ffb51445ee982237ab6e4b3b30
&position=2
&close=true HTTP/1.1
Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx
Content-Type: application/octet-stream

And this is the last chunk of the file

This example shows a request that sends the last bytes of the file, and complete the upload of the file, using the upload token obtained in the previous step:

If the request succeeds, the server responds with 201 Created, along with the metadata of the file.

Example - Successful response of final request

HTTP/1.1 201
Content-Type: application/json

{
  "id": "7ff1b498-169d-4048-8459-78f7bd7905a3",
  "name": "TestFile.txt",
  "size": 78,
  "creationDate": "2020-02-27T11:03:49.2033291Z",
  "tenantId": "sandbox",
  "businessType": {
    "id": 7100,
    "name": "7100"
},
  "numChunks": 2
}

Complete a resumable upload session sending an additional Post Request

To complete a resumable upload session sending an additional Post request without bytes, follow the steps:

  • Create a POST request to the method’s /files URI.
    • Add the query parameter uploadType=resumable
    • Add the query parameter uploadToken=[uploadToken]
    • Do not add the top-level HTTP header: Content-Type.

Example - Post Request to complete a resumable upload session

Post https://api.youserve.nl/fileapi/v1.0/files
?uploadType=resumable
&uploadToken=[uploadToken]

If the request succeeds, the server responds with 201 created, along with the metadata of the file.

Example - Complete a resumable upload session sending an additional Post request. This example shows a request that sends and additional Post request to complete the upload of the file, using the upload token obtained in the previous step:

Post/fileapi/v1.0/files
?uploadType=resumable
&uploadToken=4927b6ffb51445ee982237ab6e4b3b30 HTTP/1.1
Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx

If the request succeeds, the server responds with 201 Created, along with the metadata of the file.

Example - Successful response of final request

HTTP/1.1 201
Content-Type: application/json

{
  "id": "7ff1b498-169d-4048-8459-78f7bd7905a3",
  "name": "TestFile.txt",
  "size": 78,
  "creationDate": "2020-02-27T11:03:49.2033291Z",
  "tenantId": "sandbox",
  "businessType": {
    "id": 7100,
    "name": "7100"
},
  "numChunks": 2
}

 

File list of uploaded files

This endpoint allows you to retrieve a paginated collection of the files that have been uploaded by you.

In order to use the list endpoint:

  • Create a Get request to the method’s /files URI and add the query parameter role=publisher

Example - List Uploaded Files

GET https://api.youserve.nl/fileapi/v1.0/files?role=publisher

Optional Parameters

Parameter Name Value Description
pageIndex integer The number of the page. (Default:0)
pageSize integer The maximum number of files to return per page. Acceptable values are 1 to 1000, inclusive. (Default: 20)

Example - List Uploaded Files. This example shows how to list the files that have been uploaded by you (role=publisher)

GET/fileapi/v1.0/files
?role=publisher
&pageSize=10
HTTP/1.1

Host: api.youserve.nl
x-raet-tenant-id: sandbox
Authorization: Bearer xxxxxxxxx

If the request succeeds, the response includes a 200 OK HTTP status code, along with the paginated collection of the files that were uploaded by you.

By default, the paginated collection of files will be ordered by date descending.

Example - Response of List Uploaded Files

HTTP/1.1 200
Content-Type: application/json

{
    "data": [
    {
        "fileId": "4a31f004-aab8-4419-9072-077c35074553",
        "fileName": "TestFile1.txt",
        "fileSize": 209715222,
        "tenantId": "sandbox",
        "businessType": {
            "id": 7100,
            "name": "7100"
        },
        "publisherId": "1b604f7e-d40f-466d-b9b0-ddeb3945df14",
        "uploadDate": "2021-01-10T21:05:41.937"
    },
    {
        "fileId": "62777c81-e83f-4888-8202-cf35ea2ca38b",
        "fileName": "TestFile2.txt",
        "fileSize": 2048,
        "tenantId": "sandbox",
        "businessType": {
            "id": 7100,
            "name": "7100"
        },
        "publisherId": "1b604f7e-d40f-466d-b9b0-ddeb3945df14",
        "uploadDate": "2021-01-10T08:07:41.113"
    }
    ],
    "pageIndex": 0,
    "pageSize": 20,
    "count": 2
}

 

 

Search for uploaded files

To search for a specific set of files, use the query string $filter to filter the files to return.
 

The format of the query string is: field operator values

Where:

  • field specifies the field to search upon.

  • operator specifies the condition for the field.

  • values are the specific values you want to use to filter your search results.

Field Description Values
uploadDate Upload Date according to DateTime OData format <DateTime>
businessType Business Type ID <BusinessTypeId>
fileName Name of the file <FileName>

 

Operator Description
gt Great than
lt Less than
ge Great than or equal
le Less than or equal
eq Equal
ne Not equal
and And
or Or
( Open Parenthesis
) Close Parenthesis
startsWith parameter starts with the following substring (only for fileName)
endsWith parameter ends with the following substring (only for fileName)
contains parameter contains the following substring (only for fileName)

Query string examples

What you want to query Query String
Files greater than 2020-05-19T08:42:47.400Z uploadDate gt 2020-05-19T08:42:47.400Z
Files greater than 2020-05-19T08:42:47.400Z and lesser than 2020-05-19T16:42:47.400Z uploadDate gt 2020-05-19T08:42:47.400Z and uploadDate lt 2020-05-19T16:42:47.400Z
Files related to BusinessTypeID 7100 businessType eq 7100
Files related to BusinessTypeID 999998 or 999999 businessType eq 999998 or businessType eq 999999
All Files greater than 2020-05-19T08:42:47.400Z related to BusinessTypeID 7101 uploadDate gt 2020-05-19T08:42:47.400Z and businessType eq 7101
All files with a name that starts with “payroll” startsWith(FileName, 'payroll')
All files with a name that ends with “holidays“ endsWith(FileName, 'holidays')
All files with a name that contains “test“ contains(fileName, 'test')
All files with a name that starts with “payroll“ and a BusinessTypeId 7101 startsWith(FileName, 'payroll') and businessType eq 7101

 

 

Sort uploaded files

To sort the files in an specific order, use the query string $orderBy
 

The format of the query string is: field modifier

Where:

  • field specifies the key to sort. Valid keys are ‘uploadDate’, ‘businessType’, ‘fileName’.
  • modifier specifies the order of the sorting (‘asc’ and ‘desc’)

Example usage:

?$orderBy=uploadDate asc

By default, files are sorted by UploadDate desc.

Note: The order in which files need to be downloaded may depend on the type of data and the way the receiving system needs to process it. See examples below:

Query string examples

What you want query Query String
Retrieve ‘RST Functie’ files (BT: 101002. Definition of all functions/positions available in Beaufort/YouForce). You would only need the latest file so you can ignore the rest, as each file contains the complete list of functions/positions. $filter=businessType eq 101002.
In this case, the $OrderBy query string is not needed, as files are sorted by UploadDate desc by default.
Retrieve employee mutations in Bint Harmony Files (BT: 101100). Based on the default settings that Beaufort/YouForce only exports the mutations instead of the full employee dataSet, it is important that you get all the new files and download them from old to new. $filter=businessType eq 101100&$orderBy=uploadDate asc

 

 

File download uploaded files

This endpoint allows you to download the files that have been uploaded by you.

In order to use this download endpoint:

  • Create a Get request to the method’s /files/[fileId] URI and add the query parameter role=publisher
  • Add the HTTP header Accept=application/octet-stream

Example - Download Request

GET https://api.youserve.nl/fileapi/v1.0/files/[fileId]?role=publisher

Example - Download uploaded file. This example shows how to download a file that was uploaded by you using the Sandbox Tenant

GET/fileapi/v1.0/files/d92ffb21-7938-488b-a405-bcbc9e0a9696

?role=publisher

HTTP/1.1

Host: api.youserve.nl

x-raet-tenant-id: sandbox

Authorization: Bearer xxxxxxxxx

Accept: application/octet-stream

If the request succeeds, the response includes a 200 OK HTTP status code, along with the bytes of the file.

Example - Successful response of Download uploaded File

HTTP/1.1 200

<root>

<company>Visma YouServe</company>

<product>File API</product>

<content>Sample File</content>

</root> 

 

Troubleshooting

A few checks for mitigation

  • Setting the right boundary definition for multipart/form-data body. Be mindful of setting two consecutive dashes (for example, –foo_bar_baz–) to mark the end of the boundary within the body of the HTTP request and two additional CRLFs after the boundary delimiters.
  • If you are using Azure Logic Apps, go to the Logic App Code View. Within the HTTP body definition, replace every \n with \r\n. This is the key for successful file upload with multipart/form-data HTTP request using Azure Logic Apps.

 

 

Medewerkers