Optimizely beginners tips & tricks

We recently started experimenting with A/B-testing at our company, and are currently checking out Optimizely. While getting started some problems have popped up that took me a little bit of time to iron out. I’m writing this in the hope of saving someone else going through the same hassle.

For instructions on performing the actual installation you’re better of following Optimizely’s own getting started guide.

Some experiments should not run simultaneously

Optimizely lets you run multiple experiments at the time. To get results that you can trust from your experiments you probably want to avoid them interfering with each other. If your experiments changes the same part of the page, or the goals of one experiment could be affected by the changes made by another, you better make sure a single user is not exposed to both experiments at the same time.

To accomplish this, you can add the following javascript snippet, borrowed from Optimizely’s learning center, in each experiment’s targeting setup (Options -> Targeting):

// Setup array of mutually exclusive experiment ids
expArray=[<<experiment_id1>>,<<experiment_id2>>,<<experiment_id3>>];
// Set the id for the current experiment being evaluated
curExperiment = <<experiment_id_current>>;
// Optional. Needed if excluding multiple groupings
groupName = "groupA";

/*--  Do not modify below this line --*/
/*  Internal comments below:
    Iterate over current bucket mappings and
    set the global variable to the experiment
    the user is already included in.
*/
groupName = groupName || "groupA";
// Safety
groupName = "__" + groupName;

for(var key = 0; key < expArray.length; key++) {
     if (document.cookie.indexOf(expArray[key]) > -1) {
          optimizely[groupName] = expArray[key];
          break; // we found what we're looking for so end loop
     }
}

// If the global variable hasn't been set, set it now at random.
optimizely[groupName] = optimizely[groupName] || expArray[Math.floor(Math.random()*expArray.length)];
// Check if the current experiment matches the global experiment.
optimizely[groupName] == curExperiment;

You need to add the ID of all experiments that are to be run exclusively to the expArray variable, and the ID of the current experiment to the curExperiment variable. These IDs can be found by navigating to Options -> Diagnostic Report in each experiment.

Make sure every single element gets changed

Some of the experiments we are running make changes to multiple elements of the same type, for example in a

eksperimentsearch result. I noticed that on some page loads all elements did not get changed properly, and I’d end up with something looking rather broken, like what you see on the right. I’m confident this would affect the results of the experiment.

The Optimizely snippet seems to start (and finish) doing its thing before the page is fully loaded. Optimizely support did not figure it our either, but they did give me a workaround.

The workaround is to make sure the variation code runs in a loop until all elements are changed. Add the following javascript snipped in the experiment code editor:

/* Don't touch this code */
function waitForDelayedContent(selector, experiment, timeout, keepAlive) {
var intervalTime = 50;
var timeout = timeout || 3000;
var keepAlive = keepAlive || false;
var maxAttempts = timeout / intervalTime;
var attempts = 0;
var elementsCount = $(selector).length;

experiment(); // Run once first, no matter what.

var interval = setInterval(function() {
if ($(selector).length > elementsCount) {
if (!keepAlive) {
clearInterval(interval);
}
experiment();
elementsCount = $(selector).length;
} else if (attempts > maxAttempts) {
clearInterval(interval);
}
attempts ++;
}, intervalTime);
}
/* --------------------------------------------- */

Then pass your variation code to this function:

waitForDelayedContent(<CSS SELECTOR FOR ELEMENT TO CHANGE>, function(){

// Your variation code here

}, 10000);

For larger changes, implement both versions in your own code

When making large changes to the page, I found it a hassle spending lots of time making changes to both HTML and CSS in Optimizely’s own editor(s) and then, after the experiment finishes, having to extract all the changes and re-implement them in my own code.

What I’ve done, whenever I’m making changes that are more substantial than one or two simple changes to text, colors, size etc, is to implement both versions in my own code and push this into production. I hide the experiment code by default, and flip between the two using the variation code in Optimizely.

Your applications source code could then be something like this:

<div class="original">
  // Current stuff here
</div>

<div class="experiment" style="display: none;">
  // Experiment here
</div>

When this code is put in production, your variation code in Optimizely can be as simple as:

$(".original").remove();
$(".experiment").show();

Leave a Reply

Your email address will not be published. Required fields are marked *