Environment

Env specific variables

Starlake allows you to use variables almost everywhere in the domain and job files. For example, you may need to set the folder name to watch to a different value in development and production environments. This is where variables may help. They are enclosed inside ${} or {{}}

Assuming we have a sales domain as follows:

load:
  name: "sales"
  directory: "{{root_path}}/sales"
  ack: "ack"

We create a file env.DEV.sl.yml in the metadata folder

env:
  root_path: "/tmp/userguide"

and the file env.PROD.sl.yml in the metadata folder

env:
  root_path: "/cluster/userguide"

To apply the substitution in the DEV env set the SL_ENV variable before running Starlake as follows:

export SL_ENV=DEV

In Production set it rather to:

export SL_ENV=PROD

Global Variables

To define variables across environments, declare them in the env.sl.yml file in the metadata folder.

Global variables definitions may be superseded by the env specific variables files.

Predefined variables

The following variables are preset to the current date/time and may be used anywhere:

Date Time Variable	Format
sl_date	yyyyMMdd
sl_datetime	yyyyMMddHHmmss
sl_year	yy
sl_month	MM
sl_day	dd
sl_hour	HH
sl_minute	mm
sl_second	ss
sl_milli	SSS
sl_epoch_second	Number of seconds since 1/1/1970
sl_epoch_milli	Number of milliseconds since 1/1/1970