What is it about?

This is an Action Element, in a pipeline, which consumes a single given input (which is a table in our platform) and uploads the data to a customer-provided Snowflake table. This means that each time this Action is executed, the whole contents of a table in the platform are dumped (inserted) into the customer-provided table.

What are the prerequisites to use this action?

The prerequisites to use this pipeline action are:

• The customer must be authorized into an already existing Snowflake account of their own. They will have access to:
  • The "account" (this field consists of the first two parts of the connection url's host).
  • The username and password.
  • The warehouse name and the database name.
• A desired table, with a desired structure and fields, must exist inside that instance, being such a table the target to dump the (in-platform) data into. It must have the appropriately desired columns. This will be determined as per customer's requirement, from one of their tables. This typically involves:
  • The table name and the table schema. They are typically chosen by the customer's development team and the schema is typically "public" (by default, unless the customer deliberately chose another one).
  • The fields. While this is also explicitly determined by the development team, client tools (e.g. DBeaver Community Edition) can be used to list the involved columns (given all the other data).
• A user with permissions to write in the desired table(s) must exist in that account. That user will be authorized for the customer to give their credentials to our platform. This user must have write permissions but, as a security recommendation, it must only have permissions into that desired table (and/or other tables that are meant to be populated by other usages of this action).
• A workspace, with a project, and a pipeline in that project. Such a pipeline must have a preceding integration node (or even better: any node that produces an SQL table in our platform). The SnowflakeSQL action must take one of those nodes as its only input. This will be described in the next section.

Example layout and configuration

A minimal sample pipeline layout would look like this:

Components:

1. A source integration, to draw data from.
2. A middle node, would sub-select or convert the input data to a new format or set of columns. The format would be chosen carefully since it will serve for the next step and must be a compatible set of columns to upload.
3. The Snowflake node will have the task of uploading the data in the incoming format from the middle node.

By compatible, this must be understood:

1. The target dump table will have many fields. Some of them may be optional (with a default value or automatic setting on absence), and some of them may be required (with no default value or automatic setting on absence).
2. The middle node will have a specific set of columns in its generated table.
3. The Snowflake action node will be configured to map fields from the middle node to fill all the required fields and, perhaps, some optional fields. In each case, however, the type of the fields on both sides of the mapping must match (e.g. mapping a string field in the middle to a string field in the dump table, or a DateTime field in the middle to a DateTime field in the dump table, ...).

For example, the credentials are set like this: