Managing lakehouse clusters

As a tenant or data admin, you can create and manage lakehouse clusters from the Lakehouse clusters section of the Administration or Data Integration activity centers. Users without an admin account can view and manage clusters from the Data Integration activity center according to granted permissions.

Monitoring a cluster

A lakehouse cluster is created during the network integration deployment using basic configuration. In the Administration activity center, click Lakehouse clusters, and select the Lakehouse clusters tab. The existing clusters within your tenant are displayed as follows:

Cluster status

Each cluster will be in one of the following states:

Managing a cluster

To manage an existing cluster, do the following:

1. In the Lakehouse clusters list, select the cluster you want to manage, and click More actions.

2. In the cluster properties pane, click More actions.

3. Select one of the following actions, which are visible depending on the current state of the cluster:

• Start: A cluster with a status of Stopping or Stopped can be started. All entities configured to run on the cluster will be resumed when the cluster is running. 
• Stop: You can pause a cluster with a status of Deploying, Disconnected, Running, Running (under provisioned), or Pending resources. All entities configured to run on the cluster will be stopped.
• View: The cluster details pane provides an overview of the cluster settings, including status, number of instances, family type, and scaling strategy.
• Edit: Cluster properties can be edited regardless of status.
• Roll: Rolling a cluster gradually replaces the cluster instances with new instances, and can be performed when the cluster status is Disconnected, Running, or Running (under provisioned). This action applies cluster configuration changes with immediate effect. Choose between:
  • Gradual: Ensures there is no downtime as the instances continue to run while updates are applied.
  • Immediate: All instances shut down immediately, causing a delay until they are updated and restarted.
• Scale now: Change the scaling strategy for a cluster in a Running state. This is available when Spot Instances have been configured. Click Scale now to change the number of instances.
• Delete: A cluster can be deleted if no entities are currently associated with it. Otherwise, tasks must first be deleted or moved to another cluster. The cluster state must be Deployment failed, or Stopped before it can be deleted.
• Redeploy: Visible if the cluster creation in AWS failed due to an unrecoverable error that requires user intervention, resulting in a status of Deployment failed. This action enables you to manually deploy the cluster.

Editing a cluster

To edit the properties for an existing cluster, do the following:

1. In the Lakehouse clusters list, select the cluster you want to manage.

2. In the cluster properties pane, click More actions, and select Edit.

3. The following cluster properties can be changed:

• Name: You can change the name of the cluster.

• Family type: The instance family type can be scaled up or down. Any changes require the cluster to be rolled.

4. Edit the instances in the cluster

• AWS On-Demand Instances: You can change the minimum number of guaranteed On-Demand Instances for the cluster.

• AWS Spot Instances: Update the Minimum and Maximum number of Spot Instances.

5. If required, change the scaling strategy for the ephemeral instances handling your workload. The following options are available:

• Low cost: Optimizes for low cost, though may lead to occasional periods of high latency.

• Low latency: Strives to maintain low latency, while allowing brief, necessary spikes.

• Consistent low latency: Proactively scales up to ensure latency remains low.

• Manual scaling: Retains a static number of instances with no automatic scaling.

6. Tags

   Add, edit, or delete associated Tags. Any changes require the cluster to be rolled.