Data Store overview
The Qlik Application Automation for OEM Data Store is a database that can be used to store any type of items, e.g. Contacts, Products from a webshop or Orders. It can be used as permanent data storage or as a temporary data cache (*).
A Data Store is especially useful if you want to process items, and you want to keep track of the status for each item. This is called the exactly once processing
pattern.
There are other ways of processing items exactly once
inQlik Application Automation for OEM. Most connectors have blocks to retrieve data incrementally, by using an internal pointer. The Data Store however, is more advanced, since it keeps a status of each individual item and allows retrying individual items.
Note on data caching It is also possible to configure caching on individual blocks in an automation. The Data Store, however, gives you more control since you choose which items you add to the Data Store etc.
Creating a Data Store
InQlik Application Automation for OEM, go to the menu in the top right corner and select My Data Stores
. Now you can add a Data Store, change the settings of an existing Data Store, and see the items inside a Data Store.
Click the database icon to view items in a Data Store. Click the pencil icon to change the settings of a Data Store.
Once you have created a Data Store, you can add items to it in an automation, using the Data Store block Add item
.
Creating a Data Store in an automation or Template
If you build a Template that uses a Data Store, you have to create the Data Store inside the Blend/Template. You can do this with the block Create Datastore
. Simply put the block at the start of the Blend, under the Start block:
This block will be executed on each run of the Blend, if the Data Store already exists (based on the name), it will do nothing and return the details of the existing Data Store in the response. This way you can reference the id of the Data Store and use it throughout your Blend:
Notes:
- Do not put the block
Create Datastore
in the Setup flow of an automation, because in that case you will be unable to access the id of the Data Store when your automation is executed. - Make sure the name of your Data Store is unique, especially if multiple instances of a Template are used inside one customer account. Add e.g. a unique identifier in the name (e.g. bundle guid) to make sure that two Template instances will each have their own Data Store.
Status of items
Each item in a Data Store has a status:
- Ready to process: a new item, not yet processed by an automation
- Processed: an item that has been processed by an automation
- Error: an item that has been processed but the processing failed. It will be retried on the next run of the Blend.
- Failed: an item that will not be retried for processing
Avoiding duplicates
The Data Store is a great way to avoid duplicates. Each item in the Data Store has a unique key. The unique key can be anything, but typically it is the id of the item, e.g. an order id when you are processing orders from a web shop.
For each Data Store, you can configure how the Data Store handles adding new items with the same unique key. See setting Action to perform on duplicate
. You can choose between 3 settings:
- Update: when you add an item with a unique key that already exists in the Data Store, the existing item will be updated, and its status will NOT change. So if the existing item has status
Processed
, it will NOT be processed again after the update. - Replace: when you add an item with a unique key that already exists in the Data Store, the existing item will be replaced with this new item, and the new item will have a new status
Ready to process
. - Do nothing: when you add an item with a unique key that already exists in the Data Store, the new item will be ignored.
Example scenario: process orders exactly once
A typical scenario is that you have an automation that reads items from a connector, and writes them to the Data Store. For example, you may read orders from your webshop every day and store them in a Data Store for further processing.
The Data Store will automatically avoid creating duplicates, so in the automation you could simply retrieve ALL orders every time and add them to the Data Store over and over. There's no need to read them incrementally. In the Data Store, every order will only exist once.
For this scenario, make sure to set Action to perform on duplicate
to "Do nothing".
A second automation can read orders from the Data Store, process them and set the status of each order to Processed
.
Add items to a Data Store
Use the Data Store block Add item
to add an item to a Data Store. Here's an example that retrieves attendees of an event from Eventbrite, and adds the attendees one by one to a Data Store called Event attendees
. Note that the whole item (object) of each item is sent to the Data Store (using the input field Data
), and that the attendee id from Eventbrite is used as unique key to avoid duplicates (input field Uniquekey
).
Add items to a Data Store in bulk
Use the block Add items
(as opposed to Add item
) to add items in Bulk. This will dramatically increase the speed of inserting/updating a high number of records.
Use a block Loop batch
to create batches from a list of records. Example:
The batch from the loop is the input of the block Add items
:
Make sure that each item (object) in the list has a key with the name as defined under Uniquekey
. Note that in the block Add items
, the input Uniquekey
is a fieldname, not a value. You can use dot-notation for nested properties, e.g. contact.id
.
Existing records in the Data Store will be updated based on this field. In the above example, items will be updated in the Data Store based on the id
field from each contact from Salesforce.
Viewing items in the Data Store
The above automation could be scheduled to run every hour, and it will not create duplicates thanks to the unique key. You can view the items in the Data Store by going to My Data Stores
, selecting the Data Store Event attendees
, and clicking on the Database icon:
You can click on one item to see the item in full:
Processing items from a Data Store
You can create an automation to process items from a Data Store. Use the block List items to process
to get items from a Data Store with status Ready to process
and items with status Error
.
Note that the block List items to process
will not include items in status Processed
and Failed
, but it will include items in status Error
in order to retry processing the item.
In the following example, we are processing items and adding them to a Google Sheet. If the insert succeeded, we set the status of the item to Processed
using the block Update item status
:
Automatic update if item status to Processed
has been deprecated
The behaviour where items are set to Processed
automatically has been deprecated (Q4 2019). You have to set an item to processed explicitly using the block Update item status
.
On the next run of the Blend, the block List items to Process
will output all items in the Data Store with status Ready to process
AND status Error
. This allows you to retry processing items.
You could implement a counter in the Data Store for each retry, and said the status of the item to Failed
after a number of retries.
Keywords for searching: data store, datastore, data queue, dataqueue.
Did this page help you?
If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!