My A/B Test Journey

• Over Several Months
• Used Tools
• VWO
• Optimizely
• Google Experiments
• Plain A/B Test Framework

Beginnings

• Web
• Easy to deploy
• Easy to fix/adjust
• Easy to get traffic
• VWO
• Big html libraries
• Felt a bit old
• Optimizely
• Embraced Angular
• relies on Cookies
• heavy monkey patching for localStorage in hybrid app
• needs to know your event data

Web Frameworks with
iOS & Android: 🙅💩

• "But Hybrid Apps are just WebViews!"
• "But the libraries are just JavaScript?!"

Google Analytics Experiments

• Half Successful
• Users in Sample Size
• Reporting: Web 👍, Mobile 💔
• Tedious: Experiment creation * platform
• Hacky - you don't really follow their workflow
• Take the Experiment HTML Snippet
• Extract Experiment ID
• Create Exp, copy Exp ID, don't mess up, ...
• Very prone to errors
• Shoot your events at that Experiment ID

Plain A/B Test Framework

• Rather unpopular Github Project: AlephBet (84⭐️)
• pure-javascript A/B (multivariate) testing framework for developers
• Define Goals
• Define Experiments
• Attach Goals to Experiments
• Drawbacks
• No Shiny Dashboards
• You have to get the numbers yourself (e.g. SQL)

Implementation

Adaption

• Whatever tracker you use, easily to adapt

tracking_adapter: {
  experiment_start: (experimentName, variant) => {
    jwTrackingManager.eventTrack(variant, {
      category: `EXP_${experimentName}`,
      label: '__ACTIVATED__',
      nonInteraction: true
    });
  },
  goal_complete: (experimentName, variant, eventName) => {
    jwTrackingManager.eventTrack(variant, {
      category: `EXP_${experimentName}`,
      label: `GOAL_${eventName}`,
      nonInteraction: true
    });
  }
}
            

Goals

.run(($state, $rootScope, experimentRepository) => {
  const goalName = 'CLICKOUT';
  const goal = new window.AlephBet.Goal(goalName);
  $rootScope.$on('clickout.clicked', () => goal.complete());

  // experimentRepository = wrapper around AlephBet
  experimentRepository.addGoal(goal);
});
            

Experiments

.run((experimentRepository, urlDataRepository, $timeout) => {
  const experimentName = 'BUYBOX_ABOVE_SEASONS';
  const goals = ['NOT_BOUNCED', 'PAGEVIEWS', 'CLICKOUT'];

  $timeout(() => {
    urlDataRepository.getActiveUrlPromise()
      .then(urlData => {
        const experimentData = {
          name: experimentName,
          variants: ['control', 'above_seasons'],
          trigger: () => experimentRepository.isSeoTraffic() &&
            urlData.object_type === 'show'
        };
        experimentRepository.addExperiment(experimentData, goals);
      });
  });
});
            

Our Experiment Config

{
  "TRAILER_ZAPPER": {
    "sample": 1.0,
    "platforms": ["ios", "android"]
  },
  "ABOVE_SEASONS": {
    "sample": 0.1,
    "platforms": ["web"]
  }
}
            

Raw Numbers

• Right out of Google Analytics

Result

• Retrieve Data
• A: Control Group
• B: Above Seasons