Using Optimized A/B Tests to Improve Online Software Services
In a recent paper titled “Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective,” Zhao and his coauthor propose a new method called pigeonhole design for helping web-facing firms target their audience. The design addresses the challenges of heterogeneity in experiments. Heterogeneity is applying the same treatment to different covariates (user demographics in an online platform). The problem arises when the same treatment produces different effects across covariates.
The gold standard for addressing heterogeneity is a method called blocking. Blocking characterizes users with similar covariates into groups and then randomly assigns users from each block into the control and treatment groups.
However, this method cannot always be used for online field experiments since the covariates are not always known until the user arrives at the website. Zhao likened the problem of heterogeneity to comparing apples to oranges. For example, an “apple” could be a 20-year-old male who comes to visit the Walmart website. He may view the “buy one get one free” coupon more favorably compared to the 50% off coupon, whereas the “orange” might be a 60-year-old female who views the 50% coupon more favorably.
“Imagine yourself running a large-scale randomized control trial. The heart of experimental design is to make an apples to apples comparison. You want to compare two units that are considered similar or the same,” Zhao said.
Given this, websites must learn how to target users by showing them the coupon they favor. This is where the pigeonholes come into play. The pigeonhole design divides the covariate space into smaller spaces called pigeonholes. It matches users who arrive at a pigeonhole into pairs so there is a roughly even amount of users in each group. One pigeonhole could be the 20~25-year-old men who visit the site at around 8pm, another pigeonhole could be the 50~60-year-old women who visit the website at around 1pm. Each pigeonhole might show different preferences for the coupon. Based on the data, the web-facing firm can better target their coupons to different customers.
During the experiment, Zhao and his team compared the results of the pigeonhole design to the matched-pair design and completely randomized design – both of which can be used as benchmarks. The matched-pair design would happen in a perfect world where the web-facing firm could see the next user before they arrive, enabling the user to be placed in a certain pigeonhole and given the treatment or control. The treatment group might be giving a user a 50% off coupon, and the control group might be giving the user a “buy 1 get one free” coupon.
In contrast to the matched-pair design, a web-facing firm using a completely randomized design essentially ignores the covariance. 50% of users are randomly given the control and the other 50% are given the treatment.
Ideally, the web-facing firm would want performance as close to the matched-pair design. However, they cannot foresee the users that arrive at the site. Web-facing firms can only see the covariate data when the user arrives. Once the user arrives, they are placed into a pigeonhole that has covariates most similar to theirs. The treatment and control are given based on the previous user in that pigeonhole. So if a 20-year-old male arrives and is given the treatment, then the next person that fits into the same pigeonhole would be given the control instead.
The results of Zhao’s experiment show that the pigeonhole design closely matched the performance of matched-pair designs which is unachievable for online designs. It also outperforms the completely randomized design, enabling companies to target their audience with smaller sample sizes.
Zhao said the courses he took in his undergraduate years and while obtaining his Ph.D. influenced his interest in optimizing experimental design problems and led him to this type of work.
“I bring the approach of looking at experimental design questions through an optimization perspective,” Zhao said. “People don’t typically do this traditionally, which is why this area is so interesting.”
Zhao, Jinglong, and Zijie Zhou. “Stratifying Online Field Experiments Using the Pigeonhole Design.” arXiv preprint arXiv:2201.12936 (2022).