AWS S3
Overviewβ
The S3 exporter will collect the data and create a new file in a specific folder everytime we send the data.Everytime the FlushInterval
or MaxEventInMemory
is reached a new file will be added to AWS S3.
If for some reason the AWS S3 upload fails, we will keep the data in memory and retry to add the next time we reach FlushInterval
or MaxEventInMemory
.
Configure the relay proxyβ
To configure your relay proxy to use the AWS S3 exporter, you need to add the following configuration to your relay proxy configuration file:
# ...
exporters:
- kind: s3
bucket: evaluation-data-bucket
# ...
If you are using the S3 exporter, the easiest way to provide credentials is to set environment variables. It will be used by GO Feature Flag to identify to your S3 bucket.
export AWS_SECRET_ACCESS_KEY=xxxx
export AWS_ACCESS_KEY_ID=xxx
export AWS_DEFAULT_REGION=eu-west-1
Field name | Mandatory | Type | Default | Description |
---|---|---|---|---|
kind | string | none | (mandatory) Value should be s3 .This field is mandatory and describes which retriever you are using. | |
bucket | string | none | (mandatory) Name of your S3 Bucket. | |
flushInterval | int | 60000 | The interval in millisecond between 2 calls to the webhook (if the maxEventInMemory is reached before the flushInterval we will call the exporter before). | |
maxEventInMemory | int | 100000 | If we hit that limit we will call the exporter. | |
format | string | JSON | Format is the output format you want in your exported file. Available format: JSON , CSV , Parquet . | |
filename | string | flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}} | You can use a config template to define the name of your exported files. Available replacements are {{ .Hostname}} , {{ .Timestamp}} and {{ .Format} | |
csvTemplate | string | {{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}};{{ .Source}}\n | CsvTemplate is used if your output format is CSV. This field will be ignored if you are using format other than CSV. You can decide which fields you want in your CSV line with a go-template syntax, please check exporter/feature_event.go to see what are the fields available. | |
path | string | bucket root level | The location of the directory in S3. | |
parquetCompressionCodec | string | SNAPPY | ParquetCompressionCodec is the parquet compression codec for better space efficiency. Available options |
Compatibility with S3-Compatible Solutionsβ
The S3 retriever is compatible with S3-compatible on-premises solutions such as MinIO
.
To use an S3-compatible storage provider, ensure that you configure the following parameters:
- Endpoint URL: Provide the endpoint URL of your S3-compatible solution.
- Access Credentials: Use your S3-compatible solution's access key and secret key.
- Bucket and Key: Specify the bucket name and the key where your configuration file is stored.
The AWS SDK requires a region to be set, even if you are using an S3-compatible solution.
export AWS_DEFAULT_REGION=xxx
If you get spammed with
SDK 2025/04/25 13:07:36 WARN Response has no supported checksum.
Not validating response payload. you might want to set export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
as well.
Configure the GO Moduleβ
To configure your GO module to use the AWS S3 exporter, you need to add the following
configuration to your ffclient.Config{}
object:
The S3 Exporter v2 will use the aws-sdk-go-v2
to store your evaluation data in an S3 bucket.
The S3 Retriever v1 will use the deprecated aws-sdk-go
to access your flag in an S3 bucket.
AWS has announce end-of-support for AWS SDK for Go v1, and it's recommended to migrate from S3 Retriever v1 to v2.
awsConfig, _ := config.LoadDefaultConfig(context.Background())
config := ffclient.Config{
// ...
DataExporter: ffclient.DataExporter{
// ...
Exporter: &s3exporterv2.Exporter{
Format: "csv",
FileName: "flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}}",
CsvTemplate: "{{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}};{{ .Source}}\n",
Bucket: "my-bucket",
S3Path: "/go-feature-flag/variations/",
Filename: "flag-variation-{{ .Timestamp}}.{{ .Format}}",
AwsConfig: &awsConfig,
},
},
// ...
}
err := ffclient.Init(config)
defer ffclient.Close()
Field name | Mandatory | Default | Description |
---|---|---|---|
Bucket | none | Name of your S3 Bucket. | |
AwsConfig | none | An instance of aws.Config that configures your access to AWS (see this documentation for more info). | |
FlushInterval | 60000` | The interval in millisecond between 2 calls to the webhook (if the maxEventInMemory is reached before the flushInterval we will call the exporter before). | |
MaxEventInMemory | 100000 | If we hit that limit we will call the exporter. | |
Format | JSON | Format is the output format you want in your exported file. Available format: JSON , CSV , Parquet . | |
Filename | flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}} | You can use a config template to define the name of your exported files. Available replacements are {{ .Hostname}} , {{ .Timestamp}} and {{ .Format} | |
CsvTemplate | {{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}};{{ .Source}}\n | CsvTemplate is used if your output format is CSV. This field will be ignored if you are using format other than CSV. You can decide which fields you want in your CSV line with a go-template syntax, please check exporter/feature_event.go to see what are the fields available. | |
Path | bucket root level | The location of the directory in S3. | |
ParquetCompressionCodec | SNAPPY | ParquetCompressionCodec is the parquet compression codec for better space efficiency. Available options |