What is it about?

This is an Action Element, in a pipeline, which consumes a single given input (which is a table in our platform) and uploads the data to a customer-provided Redshift table. This means that, each time this Action is executed, the whole contents of a table in the platform are dumped (inserted) into the customer-provided table.

What are the prerequisites to use this action?

The prerequisites to use this pipeline action are:

• The customer must be authorized into an already existing Redshift instance of their own. They will have access to:
  • The "host" (typically an IP address, or a network name).
  • The "port" (it is a number, typically 5439).
  • A database name. The name is typically chosen by the customer's development team.
• A desired table, with a desired structure and fields, must exist inside that instance, being such a table the target to dump the (in-platform) data into. It must have the appropriately desired columns. This will be determined as per customer's requirement, from one of their tables. This typically involves:
  • The table name and the table schema. They are typically chosen by the customer's development team and the schema is typically "public" (by default, unless the customer deliberately chose another one).
  • The fields. While this is also explicitly determined by the development team, client tools (e.g. DBeaver Community Edition) can be used to list the involved columns (given all the other data).
• A user with permissions to write in the desired table(s) must exist in that server instance. That user will be authorized for the customer to give their credentials to our platform. This user must have write permissions but, as a security recommendation, it must only have permissions into that desired table (and/or other tables that are meant to be populated by other usages of this action).
• A workspace, with a project, and a pipeline in that project. Such a pipeline must have a preceding integration node (or even better: any node that produces an SQL table in our platform). The Redshift action must take one of those nodes as its only input. This will be described in the next section.

While all the data is known by the development team, in order to get the involved host and port the service's dashboards can be used:

https://lh3.googleusercontent.com/AqbD4Oe5eeWFjLSnbKrfljrQe2gjie0UCTKFm4Vieh1ckxJCLcj0195lRQX1D2NzlID_k4MJUTIiMiAUgzw_VV-fE0QB1vWJ-LRkbXYgAFuLJr5_xqbPejkGwV4_6SK5SYJlsuxY

Example layout and configuration

A minimal sample pipeline layout would look like this:

https://lh6.googleusercontent.com/xYI_7GcSgfvy2-QC6lqbcAwO4snjLIj_s7LyrtAFeNmixmF1zXZZDN5buL6u2pr9SzDFf7FgilESYfX4Mg-0TXSjIY_kQ343Gq1LkIpi7JojC4NCjYZmCA5qeN-a1Lr9JzOjDGof

Components:

1. A source integration, to draw data from.
2. A middle node, which would sub-select or convert the input data to a new format or set of columns. The format would be chosen carefully, since it will serve for the next step and must be a compatible set of columns to upload.
3. The Redshift node, which will have the task of uploading the data in the incoming format from the middle node.

By compatible, this must be understood:

1. The target dump table will have many fields. Some of them may be optional (with a default value or automatic setting on absence), and some of them may be required (with no default value or automatic setting on absence).
2. The middle node will have a specific set of columns in its generated table.