Projects/A/B Testing & Product Experimentation Analysis
intermediate5 milestones · ~10 hours

A/B Testing &
Product Experimentation Analysis

A product team ran a 50,000-user A/B test on a new homepage design and reports a +13% conversion lift (p=0.015). The CPO asks: 'Should we ship this to 100% of users?' You'll audit the experiment end-to-end: check Sample Ratio Mismatch, measure 95% confidence bounds against MDE, uncover device segment anomalies (mobile friction), verify revenue guardrail stability with Welch's t-test, and author a definitive 1-page executive decision memo.

Full project workspace, live PostgreSQL DB, AI Project Mentor & verified credential

What you'll learn

  • —Deduplicate raw clickstream event logs and isolate step-by-step conversion drop-offs
  • —Construct triangular cohort retention matrices and visualize decay with Seaborn heatmaps
  • —Evaluate binary conversion rates with two-sample proportion Z-tests and 95% Wald CIs
  • —Verify revenue guardrail stability with Welch's two-sample T-test (equal_var=False)
  • —Audit experimental integrity using Chi-square goodness-of-fit for Sample Ratio Mismatch (SRM)
  • —Detect temporal novelty effect decay across multi-week experiment windows
  • —Structure and defend a 1-page executive decision memo using the Minto SCR framework

The 5 milestones

Each milestone is reviewed before you advance.

  1. 01

    Set up your project repo

    Milestone 1

    Before any analytical work: create a public GitHub repository, push the standard skeleton, and paste the URL into your workspace.

  2. 02

    Multi-Step Funnel & Friction Analysis

    Milestone 2

    Ingest user clickstream logs, deduplicate multi-session events, compute step-over-step conversions, and isolate mobile drop-off friction.

  3. 03

    Monthly Cohort Retention & Lifecycle Decay

    Milestone 3

    Construct triangular monthly cohort retention matrices and visualize user lifecycle stabilization using Seaborn heatmaps.

  4. 04

    A/B Test Statistical Evaluation & Diagnostics

    Milestone 4

    Ingest 50k experiment rows, compute two-sample proportion Z-tests, 95% Wald CIs, Welch's ARPU tests, Sample Ratio Mismatch checks, and novelty fade.

  5. 05

    Executive Decision Memo & Portfolio Deliverable

    Milestone 5

    Synthesize findings into a high-stakes 1-page CPO executive recommendation memo and assemble your polished GitHub portfolio repository.

Reading & references

A/B Testing Dataset (homepage_ab_test.csv)
Article
A/B Testing & Product Analytics Master Starter Notebook
Article
Ronny Kohavi - Trustworthy Online Controlled Experiments Guide
Article

Build your portfolio & get certified

Want more projects like this? Join our Data Analyst Work Experience Program to complete 5+ guided industry projects, gain real experience, and earn an internship certificate.

Join the Data Analyst Work Experience Program

Free A/B Testing & Product Experimentation Analysis portfolio project — with AI review

Build a real a/b testing & product experimentation analysis project for your data analyst portfolio in your own public GitHub repo — free brief, starter template, and milestone guides. When your code is ready, the ₹99 pass gets every milestone reviewed against a fixed rubric (ab testing product analytics walkthrough included) and issues a verifiable certificate on completion.

Is the A/B Testing & Product Experimentation Analysis project free?

The full project brief, milestone guides, and starter template are free to audit. The AI-powered review pass and verified certificate unlock at ₹99.

Can I add the A/B Testing & Product Experimentation Analysis project to my resume and GitHub?

Yes — that is the point. You build in your own public GitHub repo, every milestone is reviewed against a fixed rubric, and the certificate links to the repo employers can inspect. Only list what is visible in your repo.

How long does the A/B Testing & Product Experimentation Analysis project take?

Most students finish the 5 milestones in 2–4 weekends. Milestones unlock in order, and you can re-submit any milestone that needs work.

Rubric-reviewed · Plagiarism-checked · Verifiable certificate