Skip to main content
Version: Next

AWS Redshift

Overview​

The AWS Redshift exporter inserts feature flag evaluation and tracking events directly into Redshift tables using the Redshift Data API.
info

AWS Redshift is a bulk exporter. Events are collected in memory and inserted in Redshift when FlushInterval or MaxEventInMemory is reached.

Configure the relay proxy​

To configure your relay proxy to use the AWS Redshift exporter, you need to add the following configuration to your relay proxy configuration file:

goff-proxy.yaml
# ...
exporters:
- kind: redshift
clusterIdentifier: "my-cluster"
database: "analytics"
dbUser: "goff_writer"
tableName: "feature_flag_evaluations"
awsRegion: "us-east-1"
autoMigrate: true
eventType: "feature"
# ...
info

In the relay proxy each exporter is wired to a single eventType (defaulting to feature). To export both evaluation and tracking events, declare two Redshift exporters, one per eventType:

goff-proxy.yaml
# ...
exporters:
- kind: redshift
workgroupName: "my-workgroup"
database: "analytics"
tableName: "feature_flag_evaluations"
autoMigrate: true
eventType: "feature"
- kind: redshift
workgroupName: "my-workgroup"
database: "analytics"
tableName: "tracking_events"
autoMigrate: true
eventType: "tracking"
# ...
Field nameMandatoryTypeDefaultDescription
kindstringnoneValue should be redshift.
This field is mandatory and describes which exporter you are using.
databasestringnoneName of the Redshift database receiving the events.
clusterIdentifierstringnoneIdentifier of a provisioned Redshift cluster. Set either clusterIdentifier or workgroupName.
workgroupNamestringnoneName or ARN of a Redshift Serverless workgroup. Set either clusterIdentifier or workgroupName.
dbUserstringnoneDatabase user used with clusterIdentifier to get temporary credentials. It cannot be used with workgroupName or secretArn.
secretArnstringnoneARN of an AWS Secrets Manager secret containing the database credentials. It cannot be used with dbUser.
tableNamestringfeature_flag_evaluations (feature), tracking_events (tracking)Table used by this exporter instance, it can be prefixed by a schema (my_schema.my_table).
awsRegionstringnoneAWS region of your Redshift cluster or workgroup. If empty, the default AWS configuration is used.
autoMigratebooleanfalseWhen true, the exporter creates the table if it does not exist.
eventTypestringfeatureEvent type to export. Use feature for evaluation events or tracking for tracking events.
flushIntervalint60000The interval in millisecond between 2 calls to the exporter (if the maxEventInMemory is reached before the flushInterval we will call the exporter before).
maxEventInMemoryint100000If we hit that limit we will call the exporter.

Configure the GO Module​

To configure your GO module to use the AWS Redshift exporter, you need to add the following configuration to your ffclient.Config{} object:

example.go
awsConfig, _ := config.LoadDefaultConfig(context.Background())

ffConfig := ffclient.Config{
// ...
DataExporters: []ffclient.DataExporter{
{
ExporterEventType: ffclient.FeatureEventExporter,
Exporter: &redshiftexporter.Exporter{
ClusterIdentifier: "my-cluster",
Database: "analytics",
DbUser: "goff_writer",
TableName: "feature_flag_evaluations",
AwsConfig: &awsConfig,
AutoMigrate: true,
},
},
{
ExporterEventType: ffclient.TrackingEventExporter,
Exporter: &redshiftexporter.Exporter{
ClusterIdentifier: "my-cluster",
Database: "analytics",
DbUser: "goff_writer",
TableName: "tracking_events",
AwsConfig: &awsConfig,
AutoMigrate: true,
},
},
},
// ...
}
err := ffclient.Init(ffConfig)
defer ffclient.Close()
FieldMandatoryDefaultDescription
DatabasenoneName of the Redshift database receiving the events.
ClusterIdentifiernoneIdentifier of a provisioned Redshift cluster. Set either ClusterIdentifier or WorkgroupName.
WorkgroupNamenoneName or ARN of a Redshift Serverless workgroup. Set either ClusterIdentifier or WorkgroupName.
DbUsernoneDatabase user used with ClusterIdentifier to get temporary credentials. It cannot be used with WorkgroupName or SecretArn.
SecretArnnoneARN of an AWS Secrets Manager secret containing the database credentials. It cannot be used with DbUser.
TableNamefeature_flag_evaluations (feature), tracking_events (tracking)Table used by this exporter instance, it can be prefixed by a schema.
AwsConfignoneAWS SDK configuration. If empty, the default AWS configuration is loaded.
AutoMigratefalseWhen true, the exporter creates the table if it does not exist.

FlushInterval and MaxEventInMemory are fields of ffclient.DataExporter (next to Exporter), not of redshiftexporter.Exporter:

FieldMandatoryDefaultDescription
FlushInterval1 minuteThe interval (time.Duration) between 2 calls to the exporter (if the MaxEventInMemory is reached before the FlushInterval we will call the exporter before).
MaxEventInMemory100000If we hit that limit we will call the exporter.

Authentication​

The exporter uses the Redshift Data API, so no JDBC connection or database password is needed. Depending on the fields you set, it authenticates with:

Authentication methodFieldsPermission required by the method
IAM identityclusterIdentifier (or workgroupName) and databaseredshift:GetClusterCredentialsWithIAM for a provisioned cluster, redshift-serverless:GetCredentials for Serverless. The database user is derived from the IAM identity.
Temporary credentials for a database userclusterIdentifier, database and dbUser (provisioned clusters only)redshift:GetClusterCredentials
AWS Secrets ManagerclusterIdentifier (or workgroupName), database and secretArnsecretsmanager:GetSecretValue on the secret

In addition to the permission of the authentication method, the IAM identity used by GO Feature Flag needs these Redshift Data API permissions:

  • redshift-data:BatchExecuteStatement to run the INSERT (and CREATE TABLE) statements.
  • redshift-data:DescribeStatement to wait for the result of the statements. It is required: without it the INSERT still runs, but the exporter cannot know it succeeded, so the export fails and the events are inserted again on the next flush.
  • redshift-data:CancelStatement to cancel a transaction that takes too long.

How events are inserted​

Each flush is inserted with multi-rows INSERT statements. Every value is sent as a named parameter of the statement, and large flushes are split into several statements to stay under the Redshift Data API limits (at most 500 rows and about 100KB per statement).

The statements of a flush are sent with BatchExecuteStatement and run in a single transaction: if one of them fails, nothing from that transaction is inserted and the events are retried on the next flush. A transaction contains up to 40 statements and about 1MB; a bigger flush is split in several transactions and if one of them fails, the events are retried and the rows of the transactions already committed are inserted again (at-least-once delivery).

The Redshift Data API is asynchronous, the exporter waits for the transaction to complete before returning. A transaction that takes more than 5 minutes is cancelled and the events are retried on the next flush.

Values that do not fit in the table​

A flush that fails is retried with the same events, so the exporter avoids sending rows that Redshift would reject because of their size:

  • Text values longer than their column (see the schemas below, sizes are in bytes) are truncated, without splitting a multibyte character.
  • Events with a JSON value (value, metadata, tracking_details, evaluation_context) larger than 65,535 bytes, or too large to fit in a single statement, are skipped. A warning with the number of skipped events and their flag keys is logged.

AutoMigrate​

When autoMigrate / AutoMigrate is enabled, the exporter runs a CREATE TABLE IF NOT EXISTS for the table it is about to write, using the schemas below. It is done once per exporter instance.

warning

AutoMigrate only creates the table, not the schema. If you use a schema-qualified tableName (e.g. my_schema.my_table), create the schema first (CREATE SCHEMA IF NOT EXISTS my_schema), otherwise the CREATE TABLE fails.

Table schemas​

feature_flag_evaluations​

ColumnRedshift typeSource field
kindVARCHAR(50)Kind
context_kindVARCHAR(50)ContextKind
user_keyVARCHAR(512)UserKey
creation_dateBIGINTCreationDate
keyVARCHAR(256)Key
variationVARCHAR(256)Variation
valueSUPERValue
defaultBOOLEANDefault
versionVARCHAR(64)Version
sourceVARCHAR(64)Source
metadataSUPERMetadata

tracking_events​

ColumnRedshift typeSource field
kindVARCHAR(50)Kind
context_kindVARCHAR(50)ContextKind
user_keyVARCHAR(512)UserKey
creation_dateBIGINTCreationDate
keyVARCHAR(256)Key
tracking_detailsSUPERTrackingDetails
evaluation_contextSUPEREvaluationContext

SUPER columns are inserted with JSON_PARSE.