Check Databricks Cluster AWS Attribute Best Practices

• Query id: b0749c53-e3ff-4d09-bbe4-dca94e2e7a38
• Query name: Check Databricks Cluster AWS Attribute Best Practices
• Platform: Terraform
• Severity: Medium
• Category: Best Practices
• URL: Github

Description

One or some Databricks Cluster AWS Attribute Best Practices are not respected
Documentation

Code samples

Code samples with security vulnerabilities

Positive test num. 1 - tf file

resource "databricks_cluster" "positive1" {
  cluster_name            = "data"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  aws_attributes {
    availability           = "SPOT"
    zone_id                = "auto"
    first_on_demand        = 1
    spot_bid_price_percent = 100
  }
}

Positive test num. 2 - tf file

resource "databricks_cluster" "positive2" {
  cluster_name            = "data"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  aws_attributes {
    availability           = "SPOT_WITH_FALLBACK"
    zone_id                = "auto"
    first_on_demand        = 0
    spot_bid_price_percent = 100
  }
}

Positive test num. 3 - tf file

resource "databricks_cluster" "positive3" {
  cluster_name            = "data"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  aws_attributes {
    availability           = "SPOT_WITH_FALLBACK"
    zone_id                = "auto"
    spot_bid_price_percent = 100
  }
}

Positive test num. 4 - tf file

resource "databricks_cluster" "positive4" {
  cluster_name            = "data"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  aws_attributes {
    availability           = "SPOT_WITH_FALLBACK"
    zone_id                = "us-west-2a"
    first_on_demand        = 1
    spot_bid_price_percent = 100
  }
}

Code samples without security vulnerabilities

Negative test num. 1 - tf file

resource "databricks_cluster" "negative" {
  cluster_name            = "Shared Autoscaling"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  aws_attributes {
    availability           = "SPOT_WITH_FALLBACK"
    zone_id                = "auto"
    first_on_demand        = 1
    spot_bid_price_percent = 100
  }
}