S3 Exporter

The S3 exporter will collect the data and create a new file in a specific folder everytime we send the data.

Everytime the FlushInterval or MaxEventInMemory is reached a new file will be added to S3.

info

If for some reason the S3 upload fails, we will keep the data in memory and retry to add the next time we reach FlushInterval or MaxEventInMemory.

Check this complete example to see how to export the data in S3.

Configuration example

awsConfig, _ := config.LoadDefaultConfig(context.Background())
ffclient.Config{ 
    // ...
   DataExporter: ffclient.DataExporter{
        // ...
        Exporter: &s3exporterv2.Exporter{
            Format: "csv",
            FileName: "flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}}",
            CsvTemplate: "{{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}};{{ .Source}}\n",
            Bucket:    "my-bucket",
            S3Path:    "/go-feature-flag/variations/",
            Filename:  "flag-variation-{{ .Timestamp}}.{{ .Format}}",
            AwsConfig: &awsConfig,
        },
    },
    // ...
}

Configuration fields

Field	Description
Bucket	Name of your S3 Bucket.
AwsConfig	An instance of aws.Config that configures your access to AWS (see this documentation for more info).
CsvTemplate	(optional) CsvTemplate is used if your output format is CSV. This field will be ignored if you are using format other than CSV. You can decide which fields you want in your CSV line with a go-template syntax, please check internal/exporter/feature_event.go to see what are the fields available. Default: {{ .Kind}};{{ .ContextKind}};{{ .UserKey}};{{ .CreationDate}};{{ .Key}};{{ .Variation}};{{ .Value}};{{ .Default}};{{ .Source}}\n
Filename	(optional) Filename is the name of your output file. You can use a templated config to define the name of your exported files. Available replacements are {{ .Hostname}}, {{ .Timestamp}} and {{ .Format}} Default: flag-variation-{{ .Hostname}}-{{ .Timestamp}}.{{ .Format}}
Format	(optional) Format is the output format you want in your exported file. Available formats are JSON, CSV, Parquet. (Default: JSON)
S3Path	(optional) The location of the directory in S3.
ParquetCompressionCodec	(optional) ParquetCompressionCodec is the parquet compression codec for better space efficiency. Available options (Default: SNAPPY)

Check the godoc for full details.