Determining the required number of database connections
As a rule of thumb, the higher the number of database connections opened for Compose, the more tables Compose will be able to load in parallel. It is therefore recommended to open as many database connections as possible for Compose. However, if the number of database connections that can be opened for Compose is limited, you can calculate the minimum number of required connections as described below.
To determine the number of required connections:
- For each task, determine the number of connections it can use during runtime. This value should be specified in the Advanced tab in the Manage Data Warehouse Tasks Settings window (Data Warehouse projects) or in the Manage Storage Tasks Settings window (Data Lake projects). When determining the number of required connections, various factors need to be taken into account including the number of tables, the size of the tables, and the volume of data. It is therefore recommended to determine the required number of connections in a Test environment.
-
Calculate the number of connections needed by all tasks that run in parallel. For example, in a Data Lake project, if three data storage tasks run in parallel, and each task requires 5 connections, then the number of required connections will be 15.
Similarly, in a Data Warehouse project, if a workflow contains two data warehouse tasks that run in parallel and each task requires 5 connections, then the minimum number of required connections will be 10. However, if the same workflow also contains two data mart tasks (that run in parallel) and the sum of their connections is 20, then the minimum number of required connections will be 20.
-
Factor in the connections required by the Compose Console. To do this, multiply the maximum number of concurrent Compose users by three and then add to the sum of Step 2 above. So, if the number of required connections is 20 and the number of concurrent Compose users is 4, then the total would be:
20 + 12 = 32