A/B testing is the best way to challenge your hypotheses. A/B testing is essentially an experiment where you show your users different variants of the app at random and then analyze the results to determine which variation performed better.
In devtodev, you can work with A/B testing in the ‘A/B Testing’ section on the app level. All tests are stored in one table. Besides basic information about each test, you can see its status. There are five types of status:
Draft - draft of an unexecuted test.
Stopped - the test was stopped before completion. It’s not possible to restart the test. If you want to restart it, make a copy of the test and launch it.
In progress - the test is currently in progress.
Finished: No winner - test results determined that there was no winner.
Finished: Success - test results determined a winner group.
Before creating a test in the devtodev interface, you need to set variables through the SDK and use the methods for launching A/B tests. Use certain classes to set the variables and their default values. In case the variables are involved with the test, their values will vary depending on the group defined by the server. If the app will be offline and won’t be able to present the test to the user, he will see default values and the app will continue to function correctly.
Here you can find more information about SDK configuration (about setting variables and methods for launching A/B tests).
To go to the test creation wizard, open the desired devtodev project, navigate to the ‘A/B Testing’ tab and click the ‘+ Add new A/B test’ button.
You have opened the segment creation wizard that consists of five steps:
Enter a unique name of the test and its description. Try to make the description of the test as detailed as possible: its hypothesis, audience, description of the test groups, target metric, desired outcome, etc. Or simply insert a link to the test description. To learn more about test planning, open this link.
In this section, you need to create test assignment rules and define the audience size. Use the ‘Filter your audience’ and ‘Triggering event’ sections to set the assignment rules.
Filter your audience - use this option to define user properties that are needed to be included into the test. The devtodev SDK which is integrated into the application, will use the filters to select the audience whose current properties match the test requirements.
In this example, all paying users will participate in the test.
If you use several filters at once, only users or devices (this depends on the selected user identification method) that meet all conditions will participate in the test.
Set a triggering event if you want your test to include only the users who performed a certain event.
In this example, the devtodev SDK will include the user or device (this depends on the selected user identification method) in the test when the SDK receives information that the said user or device reached the fourth level.
To each trigger event, you can add more parameters that are related to the event. Events have different lists of additional parameters (see Events). Please note that you can’t use an event that you sent to devtodev via API as a trigger event.
The selected filters and trigger events cannot be altered after the start of the experiment.
The filters and trigger events become available for audience configuration after at least one event/property is received via the SDK, processed and accounted for in the devtodev analytics.
When applying both filters and trigger events, all conditions have to be met for the user/device to be included into the test.
Audience fraction - use this option to define the percentage or an absolute number of users out of those selected by the filter and/or completed the trigger event who will participate in the test. If the initial audience size is not enough for drawing firm conclusions, you can change it even after the test begins.
Please note that one user can participate in only one test at a time.
If you need to run several tests in parallel, then:
You need to create non-overlapping test audiences
If your audience overlaps, you need to configure the audience fraction so that part of the audience will be included into each test (e.g. 50% of the overlapping audience gets included into each of the two tests).
If you know the number of users that you need to achieve a statistically significant result, insert it in the ‘Max number of observation’ cell.
In this example,100% of users who completed more than three levels at the time of a sign up will be included in the test.
A user can be excluded from the test only in two cases:
The test time is up.
The test is stopped.
In this section, you can define the goal of your A/B test - metrics for analysis, criteria for stopping the experiment for a group, and the duration of the experiment.
You can set up one ‘Primary metric’ and no more than five ‘Secondary metrics’ for each test. The ‘Primary metric’ is used to assess the test result and to calculate statistical significance of the obtained result. ‘Secondary metrics’ are optional and do not take part in the final result assessment. However, they can improve the quality of the analysis and prove that the implemented changes did not influence other key metrics of the app.
For example, you can select one of the following as a secondary metric:
One of the fundamental metrics (ARPU, Paying conversion, Day-N retention, etc.)
User engagement with an event.
Average number of times an event was completed (per user).
Below, you can set the ‘Estimated experiment duration’ (days). The test will be stopped after the set number of days. You can also automatically stop the test execution in case the winning group is defined - simply check the ‘Stop the experiment when there is a winning group’ box.
If you don’t see any sense in continuing the test or you want to change the group settings and restart it, you are free to change the duration of the test or even stop it anytime during its course.
To calculate the size of the test audience, use any of the Size Calculators. To use them, first estimate the current value of the Primary metric and then define the result that you expect to get from the tested changes. In addition, you can set up several user experience funnels and display their results in the test report. This will give you additional information about how successful the test has been.
The main goal of the test above is to improve conversion to payment. However you can use the same method to test the conversion to trial or to ARPU.
One of the most crucial steps is setting up test groups and variables. You create a config containing various groups and their parameters. After the devtodev SDK reports that the user has been successfully included in the experiment, one of the groups becomes available in the app via the SDK.
By default, there are two groups available to you: Group A and a control group. You can increase the number of groups to maximum 4 in a single test by clicking the ‘+Add group’ button.
The control group usually includes users as they currently are. This way, you can test other variants to see the change in their metrics, relative to the same metrics at the moment. For each group you need to define a set of variables that is composed of the name of the variable and its value. The variables have to be defined inside of your app - they are supposed to grant your users different experiences.
In the above example, you can see three groups: the control group (it has default parameters) and two more groups that have other parameters for the button_color and button_size variables. The test will be focused on defining the most favorable size and color of the button. If one of the groups wins, it may lead to the change of interface for all users of the app.
When the app is launching, the SDK defines the test that the user will participate in. Then he is randomly assigned to a test group and the SDK applies all the variables defined for this group.
To make sure that all the test groups are set up correctly and that the app is handling the selected variables the right way, we highly recommend you to test the current test settings. In this section, you can check how the A/B test configuration runs on test devices, manually determine relevant groups for them and also check how the design and app behavior change in different groups.
Click ‘+Add test device’ to add a test device using an Advertising ID / User ID / devtodev ID or select it from the list of test devices in the current devtodev Space. After that, select a user group that you want to test on and click ‘Start checking’.
The settings of the selected group will be applied to the selected test device. From this moment on, the test device will start the test for the selected group and they will not wait for meeting the entry conditions that you have specified above.
Test devices do not save the information about the active test or the group. After you successfully finish testing one group, select the next one for the same test device and click ‘Restart checking’.
To be able to access the active test and its group at a test device after restarting the app, use the DTDRemoteConfig.сacheTestExperiment() method before initializing the SDK.
After you check all group settings on the test devices, you can launch the test for the entire selected audience or save it as a draft.
A maximum of 8 A/B tests can be run simultaneously in one project.
The test can have several outcomes. Let's look at them in more detail.
Force stop and test delete
It may so happen that you’ve launched an incorrectly configured test and now you need to stop it. To do this, you can select the required test by clicking on it in the list of experiments.
To stop the test (full stop, no chance to resume) - open the A/B test report and click on the edit icon in the upper right corner. The test editing wizard will open. Click on the Stop Test button at the bottom of the wizard page.
To remove the A/B test from the list - open the test editing wizard. At the bottom of the wizard page, click the Delete Test button. Please note that you can delete only the tests that were stopped or completed. The created A/B tests stay in the project until the user deletes them.
The SDK updates the A/B test config only during initialization. If the deleted experiment was previously activated on any device, then when the SDK is initialized, it will be available for activation. After the config gets updated, the SDK will remove this test from the database but will not report it via external interfaces.
This is intended to avoid changing the app settings that were received from the experiment config at the beginning of the session (e.g, the UI that the user is currently interacting with). The next time the app starts, the test will be deleted during SDK initialization.
Do not update the app interface and behavior when the user interacts with it. Do not use the network to receive default parameters. It is better to define them in the app
Test completion using the specified criteria
If you check the ‘Stop the experiment when there is a winning group’ box at the third step of the A/B test creation process, the test will automatically stop if ‘Probability to be best’ of one of the groups is larger than 95%. This metric (Probability to be best) can be considered to be a Bayesian approach. This value is auto-calculated for each group based on the selected Primary metric. If you want to stop the test when reaching a higher number (e.g. 99%), you can change the test duration and continue with its execution until you reach the desired outcome.
The test can finish when it reaches the end of the time period specified at the ‘Goals’ step in the ‘Estimated experiment duration → duration days’ section which is responsible for the test duration. For example, you set ‘duration days’ as 3. This means that the entire audience has only three days since the test creation to be included into the test and participate in it. When the app is launched and the SDK is initialized, it will compare the current time with the end time of the test. If the experiment time is up, the SDK will erase the test from the database and the users will not be able to receive any data. If the test time has run out when the app has been used, the SDK will not respond.
This is intended to avoid changing the app settings that were received from the experiment config at the beginning of the session (e.g, the UI that the user is currently interacting with). The next time the app starts, the test will be deleted as described above.
During the test execution, a report on the test results will be built and updated in real time. In the upper block, you can find all the basic information about the current state:
Current test status
Name of the group with the highest ‘Probability to be best’ and its value
Number of groups, total number of users in the test, and number of users in each group
Experiment time frame
Below you can see a graph. Its horizontal axis represents calendar days starting from the test start date, while the vertical axis represents the value of the selected metric for each of the groups. You can select the displayed metric in the top left corner of the graph. These are:
The Primary and Secondary metrics selected in the wizard
Probability to be best
A/B test audience
Underneath you can find a table with aggregated values for each of the metrics in each test group and their fluctuations relative to the control group. The ‘Probability to be best’ is also calculated for each metric including the Secondary. This way you can make sure that all the tested changes do not influence other metrics in a negative way. After that you can see the funnels configured in the A/B test creation wizard. They contain data on the number of users at each funnel stage, conversion rate from one stage to another, and the ‘Probability to be best’ for conversion from the first to the last stage.
When a user gets included into a test group, he is automatically marked with the ID of this group.
If you want to drill down even more and understand a subtle difference in metrics and behavior of the users, you can use these user groups as filters in any devtodev reports. Simply go to the report filters, open the ‘Segments’ tab, and select the required segment.
A/B test segment values are saved as a separate user property (‘A/B test groups’) in the user card at the moment the user gets included into an A/B test group.
The user can not enroll into two A/B tests at the same time. He can be included into one test and after it completes, be included into another. In this case, the ‘ A/B test groups’ field will contain names of the two groups.
Some operations can take a while because the SDK operates asynchronously. These operations are:
A/B test configuration.
After you’ve created an A/B test in the web interface, the SDK will receive its configuration via the network. Use the remoteConfigWaiting property in the DTDRemoteConfig class to set the maximum time allowed for waiting for the A/B test configuration.
remoteConfigWaitingis set in seconds
By default, the remoteConfigWaiting value is equal to null. This means that the test configuration waiting time is unlimited. The limitation is important in case the test that you are about to execute can influence your app functioning.
Example: DTDRemoteConfig.remoteConfigWaiting = 10
In this case, the SDK will wait for the A/B test configuration for 10 seconds. If it won’t receive the config during the allocated time, it will notify the developer by using the onReceived(result: DTDRemoteConfigReceiveResult) method with Failure value and disable the A/B testing until the next SDK initialization.
After the SDK finds a test, it will wait for a suitable group to include the user into and start the experiment.
By default, waiting time (groupDefinitionWaiting) is 10 seconds.
You can change this value up or down.
Example: DTDRemoteConfig.groupDefinitionWaiting = 15
In this case, the SDK will wait for a group for no more than 15 seconds. If It can't find a suitable test for 15 seconds, it will be cancelled and its activation will be impossible. After the next SDK initialization (app restart), the user will have another chance to participate in the test.
The DTDRemoteConfig.remoteConfigWaiting and DTDRemoteConfig.groupDefinitionWaitingvalues can be set only prior to SDK initialization!
In the DTDRemoteConfig class you need to set the default variables and their values using the DTDRemoteConfig.defaults property.
The variable values in the defaults property do not change on the SDK side.
After setting up DTDRemoteConfig.defaults, you can receive the variable values by using the DTDRemoteConfig.config property.
When working with A/B tests, always use the config property to receive the actual variables and their values!
If for some reasons, the device can not be included in the test (no network or network problems, devtodev did not assign it to a group, incorrect SDK implementation, etc.) DTDRemoteConfig.config will return default values. Using this approach, you can always shape the desirable UI and the business logic of the app.
To work with A/B testing, use the DTDAnalytics.initializeWithAbTest initialization method.
If you used DTDAnalytics.initialize previously, you need to replace it with DTDAnalytics.initializeWithAbTest as the SDK initialization method.
The DTDRemoteConfigListener interface is added to the initializeWithAbTest method. It implements three methods:
onReceived(result: DTDRemoteConfigReceiveResult)
onPrepareToChange()
onChanged(result: DTDRemoteConfigChangeResult, error: Error?)
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the [DTDAnalytics applicationKey:* abConfigListener:*] initialization method.
If you used [DTDAnalytics applicationKey:*] previously, you need to replace it with [DTDAnalytics applicationKey:* abConfigListener:*] as the SDK initialization method.
The DTDRemoteConfigListener interface is added to the [DTDAnalytics applicationKey:* abConfigListener:*] method. It implements three methods:
-(void)onReceivedResult:(enum DTDRemoteConfigReceiveResult)result
-(void)onPrepareToChange
-(void)onChangedResult:(enum DTDRemoteConfigChangeResult)result error:(NSError *)error
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the DTDAnalytics.initializeWithAbTest initialization method.
If you used DTDAnalytics.initialize previously, you need to replace it with DTDAnalytics.initializeWithAbTest as the SDK initialization method.
The IRemoteConfigListener interface is added to the initializeWithAbTest method. It implements three methods:
onReceived(result: DTDRemoteConfigReceiveResult)
onPrepareToChange()
onChanged(result: DTDRemoteConfigChangeResult, ex: Exception?)
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the DTDAnalytics.INSTANCE.initializeWithAbTest initialization method.
If you used DTDAnalytics.INSTANCE.initialize previously, you need to replace it with DTDAnalytics.INSTANCE.initializeWithAbTest as the SDK initialization method.
The IRemoteConfigListener interface is added to the initializeWithAbTest method. It implements three methods:
void onReceived(@NonNull DTDRemoteConfigReceiveResult result)
void onPrepareToChange()
void onChanged(@NonNull DTDRemoteConfigChangeResult result, @Nullable Exception ex)
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the DTDAnalytics.InitializeWithAbTests initialization method.
If you used DTDAnalytics.Initialize previously, you need to replace it with DTDAnalytics.InitializeWithAbTests as the SDK initialization method.
The DTDRemoteConfigListener interface is added to the InitializeWithAbTests method. It implements three methods:
OnReceived(DTDRemoteConfigReceiveResult result)
OnPrepareToChange()
OnChanged(DTDRemoteConfigChangeResult result, string exceptionMessage = null)
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the DTDAnalytics.InitializeWithAbTests initialization method.
If you used DTDAnalytics.Initialize previously, you need to replace it with DTDAnalytics.InitializeWithAbTests as the SDK initialization method.
The IDTDRemoteConfigListener interface is added to the InitializeWithAbTests method. It implements three methods:
OnReceived(DTDRemoteConfigReceiveResult result)
OnPrepareToChange()
OnChanged(DTDRemoteConfigChangeResult result, string error)
The DTDRemoteConfigListenermethods are not called in the main thread - you need to keep this in mind when interacting with UI elements of the app. Also, do not execute operations that block the thread in the DTDRemoteConfigListener methods, because this will disable the SDK operation (see examples).
To work with A/B testing, use the UDTDAnalyticsBPLibrary::InitializeWithAbTest initialization method.
If you used UDTDAnalyticsBPLibrary::Initialize previously, you need to replace it with UDTDAnalyticsBPLibrary::InitializeWithAbTest as the SDK initialization method.
Implements three delegate methods:
onRemoteConfigReceive(EDTDRemoteConfigReceiveResult result)
onRemoteConfigPrepareToChange()
onRemoteConfigChange(EDTDRemoteConfigChangeResult result, FString error)
To work with A/B testing, use the DTDAnalytics.InitializeWithAbTest initialization method.
If you used DTDAnalytics.initialize previously, you need to replace it with DTDAnalytics.initializeWithAbTest as the SDK initialization method.
The InitializeWithAbTest method takes 3 Callable as arguments, to work with A/B tests it is necessary to implement:
onRemoteConfigReceive(result: GDDTDRemoteConfigReceiveResult.ReceiveResult)
onRemoteConfigPrepareToChange()
onRemoteConfigChange(result: GDDTDRemoteConfigChangeResult.ChangeResult, error: String)
Methods implementation for working with A/B tests. When suitable conditions for the test occur, the following chain of events gets initiated:
Calling onReceived method
Calling onPrepareToChange method
Calling onChanged method
The onReceived method is triggered every time when the SDK loads A/B test configuration. The result argument returns the following:
Success if the configuration was loaded in the time provided. Time is set by the user in DTDRemoteConfig.remoteConfigWaiting.
Failure if the config wasn’t received during the provided time defined by the user in DTDRemoteConfig.remoteConfigWaiting.
Empty if the configuration was loaded in the time provided, but the list of experiments is empty. Time is set by the user in DTDRemoteConfig.remoteConfigWaiting.
If the DTDRemoteConfig.remoteConfigWaiting’s default value is equal to null, the OnReceived method and DTDRemoteConfigReceiveResult.Success are called when the config is received. The time of the OnReceived call depends on the network speed.
The onPrepareToChange() method is designed to notify the developer that the configuration of the A/B tests will be changed (the test is found or canceled, switch to a user who is not participating in the experiment, etc. (see Some features in the behavior of interfaces)).
SDK asynchronous behavior causes a gap in time between the moment the suitable conditions for the test occur and its possible activation. You need to keep this in mind when changing the app’s UI and UX and pause the user’s interaction with the app until it comes into action:
onChanged (see onChanged description) and a decision on test participation with a new config is made (see applyConfig description).
A timer on the developer’s side. The timer defines the time allocated for new config waiting. After the time has run out, waiting for the config should be canceled.
See the implementation example (Scenario 1)
After calling onPrepareToChange, the SDK executes an asynchronous request for entering the test. That will result in calling onChanged.
In case of disruption of the Internet connection or loss of focus, the test will be considered canceled until the next app launch.
The onChanged method is triggered in one of the following cases:
The DevToDev server offers to enroll in a test.
onChanged() with DTDRemoteConfigChangeResult.success is called
The DevToDev server refuses to execute the test.
onChanged() with DTDRemoteConfigChangeResult.failure is called
The device has no internet connection.
onChanged() with DTDRemoteConfigChangeResult.failure is called
If the offer is received after the 30 second waiting time has expired
onChanged() with DTDRemoteConfigChangeResult.failure is called
The test was previously activated (during the initialization)
onChanged() with DTDRemoteConfigChangeResult.success is called
If the onChanged method is triggered with status == DTDRemoteConfigChangeResult.success value, the possible options are:
Accept the configuration by calling the DTDRemoteConfig.applyConfig() method
If the configuration is accepted, the default parameters and group parameters (issued by the server) will overlap. After that, the parameters of the new configuration will be available in DTDRemoteConfig.config (you can use them to change the behavior or the interface of the app).
Refuse enrollment in the test DTDRemoteConfig.resetConfig()
If the test enrollment is refused or the time for decision making is up, the SDK will temporarily flag the test and it will become unavailable until the next initialization. After the next initialization, the test will be available for participation when its requirements are fulfilled.
In the cases where status == DTDRemoteConfigChangeResult.failure, the DTDRemoteConfig.applyConfig() method will be triggered and the default configuration will stay in place.
Set the default values DTDRemoteConfig.defaults = ["button_color": "green", "button_size": "8"]
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig.configvalue variants:
Example of SDK initialization that includes A/B testing:
Set the default values DTDRemoteConfig.defaults = @{"button_color": @"green", @"button_size": @"8"}
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig.configvalue variants:
Example of SDK initialization that includes A/B testing:
Set the default values DTDRemoteConfig.defaults = mapOf("button_color" to "green", "button_size" to "8")
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig.configvalue variants:
Example of SDK initialization that includes A/B testing:
Set the default values:
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig.configvalue variants:
Example of SDK initialization that includes A/B testing:
Set the default values:
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.Defaults
Example of the DTDRemoteConfig.Config value variants:
Example of SDK initialization that includes A/B testing:
Set the default values:
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.Defaults
Example of the DTDRemoteConfig.Config value variants:
Example of SDK initialization that includes A/B testing:
Set the default values:
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig.configvalue variants:
Example of SDK initialization that includes A/B testing:
Set the default values DTDRemoteConfig.SetDefaults()
Variable names have to be the same as in the ‘Group settings’ section of the web interface (see ‘web’ above). When the SDK gets included in the A/B test, the devtodev server automatically assigns it to a group.
SDK will not use new variable values that are not set in DTDRemoteConfig.defaults
Example of the DTDRemoteConfig value variants:
Example of SDK initialization that includes A/B testing:
When the SDK finds a suitable test(s), it calls the onPrepareToChange method and reports it. Then the SDK starts a timer for 30 seconds and sends a request to the devtodev server. The server returns an experiment number and information about a group. Or the server answers that the user can not participate in the test. This process takes some time. The request and answer can take as little as milliseconds, or much longer (30 seconds maximum) if, for example, the device’s internet connection is bad. Below, you can find several use cases that may help you to solve this problem if your app’s interface is sensitive to waiting for the config from devtodev.
When working with this class, the SDK provides synchronization of threads that aims at providing safe operation taking into account the asynchrony of the SDK.
Wrapper for remote parameters collection. Enables access to configuration values by using subscripting syntax.
Wrapper for working with remote configuration variables. It represents a method for data source identification, as well as methods for presenting values in the form of various data types.
It implements methods that report the status of A/B tests.
IDTDRemoteConfigListener
IDTDRemoteConfigListener
FDTDRemoteConfigReceiveResultDelegate
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), OnReceived does not get called
FDTDRemoteConfigPrepareToChangeDelegate
It notifies the developer about coming changes in the A/B test configuration.
FDTDRemoteConfigChangeResultDelegate
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
It is triggered every time the SDK loads the configuration of A/B tests.
Notifies whether the configuration of the A/B test has been changed.
Data options returned by the configChanged method signature
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Key/Value | defaults | Control Group | Group A | Group B |
|---|---|---|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Property | Description |
|---|---|
| Method | |
|---|---|
| type | Description |
|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Property | type | Description |
|---|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| Method | Description |
|---|---|
| type | Description |
|---|---|
| type | Description |
|---|---|
| DTDRemoteConfigChangeResult | Exception message | Description |
|---|---|---|
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
button_color
green
green
blue
gray
button_size
8
8
12
14
remoteConfigWaiting: Double
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
groupDefinitionWaiting: Double
Wait time for test group.
Default value - 10.0 (measured in seconds)
defaults: [String, Any]
Variables and their default values
config: DTDRemoteConfigCollection
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
applyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
resetConfig()
It cancels a suitable or running test
cacheTestExperiment()
A debug method for saving a test experiment after restarting the application
remoteConfigWaiting: double
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
groupDefinitionWaiting: double
Wait time for test group.
Default value - 10.0 (measured in seconds)
defaults: NSDictionary<NSString *,id>
Variables and their default values
config: DTDRemoteConfigCollection
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
(void) applyConfig
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
(void) resetConfig
It cancels a suitable or running test
(void) cacheTestExperiment
A debug method for saving a test experiment after restarting the application
remoteConfigWaiting: Double
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
groupDefinitionWaiting: Double
Wait time for test group.
Default value - 10.0 (measured in seconds)
defaults: Map<String, Any>
Variables and their default values
config: DTDRemoteConfigCollection
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
applyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
resetConfig()
It cancels a suitable or running test
cacheTestExperiment()
A debug method for saving a test experiment after restarting the application
setRemoteConfigWaiting()
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
getRemoteConfigWaiting()
Wait time for test group.
setGroupDefinitionWaiting()
Wait time for test group.
Default value - 10.0 (measured in seconds)
getGroupDefinitionWaiting()
Get time for test group.
setDefaults(Map<String, ? extends Object>)
Set map of variables and their default values
Map<String,Object> getDefaults()
Get map of variables and their default values
DTDRemoteConfigCollection getConfig()
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
applyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
resetConfig()
It cancels a suitable or running test
cacheTestExperiment()
A debug method for saving a test experiment after restarting the application
double RemoteConfigWaiting {get;set;}
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
double GroupDefinitionWaiting {get;set}
Wait time for test group.
Default value - 10.0 (measured in seconds)
Dictionary<string, object> DTDRemoteConfig.Defaults {get;set}
Variables and their default values
DTDRemoteConfigCollection DTDRemoteConfig.Config
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
void ApplyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
void ResetConfig()
It cancels a suitable or running test
void CacheTestExperiment()
A debug method for saving a test experiment after restarting the application
RemoteConfigWaiting: double
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
GroupDefinitionWaiting: double
Wait time for test group.
Default value - 10.0 (measured in seconds)
Defaults: IDictionary<String, object>
Variables and their default values
Config: DTDRemoteConfigCollection
Wrapper for remote parameters in the form of a collection. It allows access to the configuration values by using the subscripting syntaxis.
Actual variables and their values for working with A/B tests
ApplyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
ResetConfig()
It cancels a suitable or running test
CacheTestExperiment()
A debug method for saving a test experiment after restarting the application
SetRemoteConfigWaiting(float value)
Wait time for A/B test configuration.
Default value - 0.0 (measured in seconds)
float GetRemoteConfigWaiting()
Wait time for test group.
SetGroupDefinitionWaiting(float value)
Wait time for test group.
Default value - 10.0 (measured in seconds)
GetGroupDefinitionWaiting()
Get time for test group.
SetDefaults(const FDTDRemoteConfigDefaults& defaults)
Set USTRUCT with variables and their default values
TMap<FString, FDTDRemoteConfigValue> GetConfig()
Actual variables and their values for working with A/B tests
FDTDRemoteConfigValue GetRemoteConfigValue(const FString& key)
Return actual value for variable key
bool HasKey(const FString& key)
Verify if there are any values associated with the variable key in the remote configuration.
ApplyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
ResetConfig()
It cancels a suitable or running test
CacheTestExperiment()
A debug method for saving a test experiment after restarting the application
void SetRemoteConfigWaiting(int value)
Wait time for A/B test configuration.
Default value - 0 (measured in seconds)
int GetRemoteConfigWaiting()
Wait time for test group.
void SetGroupDefinitionWaiting(int value)
Wait time for test group.
Default value - 10 (measured in seconds)
int GetGroupDefinitionWaiting()
Get time for test group.
void SetDefaults(defaults: GDDTDRemoteConfigDefaults)
Set GDDTDRemoteConfigDefaults with variables and their default values
GDDTDRemoteConfigValue GetRemoteConfigValue(key: String)
Return actual value for variable key
bool HasKey(key: String)
Verify if there are any values associated with the variable key in the remote configuration.
void ApplyConfig()
It applies the A/B testing configuration. After the call, the default parameters get matched with the group parameters.
void ResetConfig()
It cancels a suitable or running test
void CacheTestExperiment()
A debug method for saving a test experiment after restarting the application
hasKey(key: String)
Return true if current configuration has value for a key.
DTDRemoteConfigSource.Default
The variable is set by default.
DTDRemoteConfigSource.Remote
The variable is set by the test group.
DTDRemoteConfigSource.Empty
The variable is not found in the remote configuration.
stringValue
String?
Gets the value as an optional string.
floatValue
Float
Gets the value as a Float.
doubleValue
Double
Gets the value as a Double.
Int32Value
Int32
Gets the value as an Int32.
Int64Value
Int64
Gets the value as an Int64.
integerValue
Int
Gets the value as an Int.
boolValue
Bool
Gets the value as a Bool.
stringValue
NSString
Gets the value as an optional string.
floatValue
float
Gets the value as a Float.
doubleValue
double
Gets the value as a Double.
Int32Value
long
Gets the value as an long.
Int64Value
long long
Gets the value as an long long.
integerValue
NSInteger
Gets the value as an NSInteger.
boolValue
BOOL
Gets the value as a Bool.
stringValue
String?
Gets the value as an optional string.
floatValue
Float
Gets the value as a Float.
doubleValue
Double
Gets the value as a Double.
intValue
Int
Gets the value as an int.
longValue
Long
Gets the value as an long.
booleanValue
Boolean
Gets the value as a Bool.
getStringValue
String
Gets the value as an optional string.
getFloatValue
float
Gets the value as a Float.
getDoubleValue
double
Gets the value as a Double.
getIntValue
int
Gets the value as an Int.
getLongValue
long
Gets the value as an Long.
getBooleanValue
Boolean
Gets the value as a Boolean.
StringValue
string
Gets the value as an optional string.
FloatValue
float
Gets the value as a float.
DoubleValue
double
Gets the value as a double.
LongValue
long
Gets the value as an long.
IntValue
int
Gets the value as an int.
BoolValue
bool
Gets the value as a bool.
StringValue
String
Gets the value as an optional string.
FloatValue
float
Gets the value as a float.
DoubleValue
double
Gets the value as a double.
Int16Value
Int16
Gets the value as an Int16
Int32Value
Int32
Gets the value as an Int32
Int64Value
Int64
Gets the value as an Int64
BoolValue
bool
Gets the value as a bool.
StringValue
FString
Gets the value as an string.
FloatValue
float
Gets the value as a float.
LongValue
int64
Gets the value as an long.
IntValue
int32
Gets the value as an int.
BoolValue
bool
Gets the value as a bool.
Source
EDTDRemoteConfigSource
Gets source of the value.
GetStringValue
String
Gets the value as an string.
GetFloatValue
float
Gets the value as a float.
GetIntValue
int
Gets the value as an int.
GetBoolValue
bool
Gets the value as a bool.
GetSource
GDDTDRemoteConfigSource.Source
Gets source of the value.
onReceived(result: DTDRemoteConfigReceiveResult)
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), onReceived does not get called
onPrepareToChange()
It notifies the developer about coming changes in the A/B test configuration.
onChanged(result: DTDRemoteConfigChangeResult,
error: Error?)
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
-(void)onReceivedResult:
(enum DTDRemoteConfigReceiveResult)result
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), onReceived does not get called
-(void)onPrepareToChange
It notifies the developer about coming changes in the A/B test configuration.
- (void)onChangedResult:
(enum DTDRemoteConfigChangeResult)result
error:(NSError *)error
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
onReceived(result: DTDRemoteConfigReceiveResult)
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), onReceived does not get called
onPrepareToChange()
It notifies the developer about coming changes in the A/B test configuration.
onChanged(result: DTDRemoteConfigChangeResult, ex: Exception?)
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
onReceived(DTDRemoteConfigReceiveResult result)
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), onReceived does not get called
onPrepareToChange()
It notifies the developer about coming changes in the A/B test configuration.
onChanged( DTDRemoteConfigChangeResult result, Exception ex)
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
void OnReceived(DTDRemoteConfigReceiveResult result);
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), OnReceived does not get called
void OnPrepareToChange();
It notifies the developer about coming changes in the A/B test configuration.
void OnChanged(DTDRemoteConfigChangeResult result, string exceptionText = null);
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
OnReceived(DTDRemoteConfigReceiveResult result)
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), OnReceived does not get called
OnPrepareToChange()
It notifies the developer about coming changes in the A/B test configuration.
OnChanged(DTDRemoteConfigChangeResult result, string error)
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
onRemoteConfigReceive(result: GDDTDRemoteConfigReceiveResult.ReceiveResult):
It is triggered every time the SDK loads an A/B test configuration. If the remoteConfigWaiting has a default value (null), onReceived does not get called
onRemoteConfigPrepareToChange()
It notifies the developer about coming changes in the A/B test configuration.
onRemoteConfigChange(result: GDDTDRemoteConfigChangeResult.ChangeResult, error: String)
It notifies the developer that the configuration has changed. Or about the reason why the configuration could not change.
DTDRemoteConfigReceiveResult.Success
It is triggered when the configuration of the A/B test is received.
DTDRemoteConfigReceiveResult.Failure
It is triggered if the SDK did not manage to get the A/B test configurations for the time period set in remoteConfigWaiting.
DTDRemoteConfigReceiveResult.Empty
It is triggered when the configuration of the A/B test is received, but the list of experiments is empty.
DTDRemoteConfigReceiveResult.Success
Configuration is changed.
DTDRemoteConfigReceiveResult.Failure
Configuration change attempt failed.
DTDRemoteConfigChangeResult.Success
null
When receiving an offer from the backend.
DTDRemoteConfigChangeResult.Success
null
When launching the SDK with a previously started test.
DTDRemoteConfigChangeResult.Success
null
When replacing the user with another user who previously started the test.
DTDRemoteConfigChangeResult.Failure
[A/B-Test Module] The Server refused to conduct the test.
If the server refused to provide a test available for participation.
DTDRemoteConfigChangeResult.Failure
[A/B-Test Module] Offer from devtodev not received within the allotted N seconds.
The backend offer is returned after a deadline e.g. due to a bad internet connection.
DTDRemoteConfigChangeResult.Failure
[A/B-Test Module] Offer from devtodev not received within the allotted N seconds.
The SDK was unable to receive an offer from the backend within N seconds e.g, duea bad internet connection or network problems.
DTDRemoteConfigChangeResult.Failure
[A/B-Test Module] In the process of receiving an offer from the server, the user was changed. Offer has been canceled.
When replacing the user with another user at the time of receiving an offer.
DTDRemoteConfigChangeResult.Failure
[A/B-Test Module] Offer refused, because application went into the background.
If a suitable test is found but the user made the app go into the background before the offer was received.
Our analytics data show that most of the people make a purchase during their first session. However, only 70% of users open our in-app store during the first session. We hypothesize that if we add a purchase screen to the tutorial then 100% of users will face it and the number of purchases during the first session will increase.
The criteria for test inclusion is starting the tutorial. The DTDAnalytics.tutorial(step: Int) event with step = -1 is the trigger.
Control group: a 10 step tutorial, no purchase screens
Group А: an 11 step tutorial. At step number 5, we will offer a special offer.
Our analytics data show that N users installed the app more than half a year ago but still did not make a purchase. We hypothesize that if we offer a huge discount then we can get additional income.
The criteria for test inclusion is the app install date and the “payer” status.
Control group: current version with usual prices
Group А: it has a discount badge on the main page. After clicking on the badge, a purchase window pops up
Note
When receiving the configuration, the SDK always calls the onReceived(result: DTDRemoteConfigReceiveResult) method, and when enrolling in a test - the onPrepareToChange() and onChanged(result: DTDRemoteConfigChangeResult, error: Error?) method. However, you can take some additional precocious measures. You can add a timer as in the following example:
Note
When receiving the configuration, the SDK always calls the onReceivedResult method, and when enrolling in a test - the onPrepareToChange and onChangedResult method. However, you can take some additional precocious measures. You can add a timer as in the following example:
Note
When receiving the configuration, the SDK always calls the onReceivedResult method, and when enrolling in a test - the onPrepareToChange and onChangedResult method. However, you can take some additional precocious measures. You can add a timer as in the following example:
Note
When receiving the configuration, the SDK always calls the onReceivedResult method, and when enrolling in a test - the onPrepareToChange and onChangedResult method. However, you can take some additional precocious measures. You can add a timer as in the following example:
Note
When receiving the configuration, the SDK always calls the OnReceived(DTDRemoteConfigReceiveResult result) method, and when enrolling in a test - the OnPrepareToChange() and OnChanged(DTDRemoteConfigChangeResult result, string exceptionText = null) method. However, you can take some additional precocious measures. You can add a timer as in the following example:
Note
When receiving the configuration, the SDK always calls the OnReceived(DTDRemoteConfigReceiveResult result) method, and when enrolling in a test - the OnPrepareToChange() and OnChanged(DTDRemoteConfigChangeResult result, string error). However, you can take some additional precocious measures. You can add a timer as in the following example:
Our analytics data show that only 60% of our users complete the tutorial. We hypothesize that if we make it easier (group A) or reduce the number of seps (group B), we will increase the percentage of tutorial completion.
New users who have not started the tutorial yet.
Control group: current version with usual difficulty and usual number of steps
Group А: low difficulty, usual number of steps
Group B: usual difficulty, few steps
