Experimentation rollout / A/B Testing

An experimentation rollout is when your flag is configured to be served only for a determined time.

1. It means that before the rollout start date, the default value is served to all users.
2. Between the dates the flag is evaluated.
3. After the end date the default value is served to all users.

Example

YAML

experimentation-flag:
  variations:
    variationA: A
    variationB: B
  defaultRule:
    percentage:
      variationA: 50
      variationB: 50
  experimentation:
    start: 2021-03-20T00:00:00.1-05:00
    end: 2021-03-21T00:00:00.1-05:00

JSON

{
  "experimentation-flag": {
    "variations": {
      "variationA": "A",
      "variationB": "B"
    },
    "defaultRule": {
      "percentage": {
        "variationA": 50,
        "variationB": 50
      }
    },
    "experimentation": {
      "start": "2021-03-20T05:00:00.100Z",
      "end": "2021-03-21T05:00:00.100Z"
    },
  }
}

TOML

[experimentation-flag.variations]
variationA = "A"
variationB = "B"

[experimentation-flag.defaultRule.percentage]
variationA = 50
variationB = 50

[experimentation-flag.experimentation]
start = 2021-03-20T05:00:00.100Z
end = 2021-03-21T05:00:00.100Z

Check this example to see how it works.

Configuration fields

info

The dates are in the format supported natively by your flag file format.

Field	Description
start	The date the flag will be started to be served.
end	The date the flag will be stopped to be served.

A/B testing

info

A/B test is the shorthand for a simple controlled experiment. As the name implies, two versions (A and B) of a single variable are compared, which are identical except for one variation that might affect a user's behavior. A/B tests are widely considered the simplest form of controlled experiment.

(source wikipedia)

To have a proper A/B testing solution with the module you should use the experimentation rollout combined with the export of your data.

This combination will allow to have your experimentation running for a dedicated time, and you will have the data to know exactly which user was on which version of the flag.

To setup the duration of your A/B test you can use a tool ab-test-duration-calculator from vwo, that will help you to set up the test duration correctly.