{"id":1480,"date":"2025-11-25T16:28:26","date_gmt":"2025-11-25T16:28:26","guid":{"rendered":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/"},"modified":"2025-11-25T16:28:26","modified_gmt":"2025-11-25T16:28:26","slug":"reducing-experiment-duration-with-predicted-control-variates","status":"publish","type":"post","link":"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/","title":{"rendered":"Reducing experiment duration with predicted control variates"},"content":{"rendered":"<div id=\"dev95-2624064334\" class=\"dev95-- dev95-entity-placement\"><script async=\"async\" data-cfasync=\"false\" src=\"https:\/\/pl27862732.profitableratecpmnetwork.com\/2ad7a50e0bbc23ac6801d7b77c501463\/invoke.js\"><\/script>\r\n<div id=\"container-2ad7a50e0bbc23ac6801d7b77c501463\"><\/div><\/div><div>\n<p>I<em>n 2021, we published a blog post titled \u201c<a href=\"https:\/\/www.etsy.com\/ca\/codeascraft\/increasing-experimentation-accuracy-and-speed-by-using-control-variates\">Increasing experimentation accuracy and speed by using control variates<\/a>,\u201d describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolved at Etsy since then. Spoiler \u2013 It\u2019s changed a lot, decreasing our average experiment duration by 3 days!<\/em><\/p>\n<p>Etsy\u2019s mission is to Keep Commerce Human. To achieve this, we need to understand the impact each change to our platform has on our buyers&#8217; and sellers&#8217; experience. Whether that involves changing the color of the \u201cBuy Now\u201d button on the Etsy app or updating elements of how our algorithms rank search results, we leverage large-scale online experimentation to iterate on and improve the things we build.<\/p><div id=\"dev95-3814825458\" class=\"dev95- dev95-entity-placement\"><center>\r\n<script>\r\n  atOptions = {\r\n    'key' : '4ba6b6513c00e0ba76511f798ae56401',\r\n    'format' : 'iframe',\r\n    'height' : 50,\r\n    'width' : 320,\r\n    'params' : {}\r\n  };\r\n<\/script>\r\n<script src=\"https:\/\/www.highrevenueformat.com\/4ba6b6513c00e0ba76511f798ae56401\/invoke.js\"><\/script>\r\n\t<\/center><\/div>\n<p>However, running an experiment can be a long process. From design and setup to running the experiment and analyzing results, the entire experimentation process can take weeks to months. Experiments must run long enough to collect sufficient data for the results to be statistically significant \u2013 ensuring we can confidently attribute observed changes to the treatment, rather than random chance. On the other hand, being able to learn from an experiment quickly is a crucial step in the product development lifecycle, enabling faster improvements to Etsy. Fortunately, there are tools to reduce experiment runtime. <a href=\"https:\/\/exp-platform.com\/Documents\/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf\">CUPED<\/a> is one of them! Variance reduction techniques like CUPED can help reduce the time to run an experiment, shortening the overall experimentation lifecycle and time to learning, as visualized below.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"Experimentation Lifecycle\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?w=1280&#038;ssl=1\" title=\"Experimentation Lifecycle\"><br \/>\n<\/figure>\n<h2>A recap of CUPED<\/h2>\n<p>CUPED is a variance reduction technique that estimates experiment outcomes with greater speed and accuracy compared to a direct comparison between control and treatment groups. In 2021, Etsy <a href=\"https:\/\/www.etsy.com\/ca\/codeascraft\/increasing-experimentation-accuracy-and-speed-by-using-control-variates\">implemented CUPED<\/a> (Controlled-Experiment Using Pre-Experiment Data) for key metrics like Conversion Rate (the percentage of visitors that make a purchase). <\/p>\n<p>CUPED leverages historical visitor data collected before the experiment begins \u2014 for example, the number of purchases in the week prior to the experiment \u2013 to explain some natural variation in the outcome metric. The pre-experiment factors are used as covariates in a linear regression model to remove some of the \u201cnoise\u201d that is not attributable to the treatment. By accounting for this variation, CUPED reduces the variance of the treatment effect estimator, increasing statistical power and improving sensitivity without introducing bias.<\/p>\n<p>The CUPED correction can be conceptualized as:<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPED formula\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/c3af38\/7484132339\/inv_fullxfull.7484132339_n7wzrht2.jpg?w=1280&#038;ssl=1\" title=\"CUPED formula\"><br \/>\n<\/figure>\n<p>The CUPED-adjusted metric will have a smaller variance than the original metric, as visualized below, providing more precise estimates of a mean or treatment effect.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPED variance chart\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/c804c6\/7436156294\/inv_fullxfull.7436156294_lknh7nir.jpg?w=1280&#038;ssl=1\" title=\"CUPED variance chart\"><br \/>\n<\/figure>\n<p>Sample size, power, and variance are all related. Holding everything else unchanged, the smaller the variance of a metric, the smaller the sample size required to reach a desired power. Since we can reduce the variance of our metric by applying CUPED, we can achieve the same amount of power with a smaller sample size. In practice, a smaller sample size corresponds to a shorter experiment duration. <\/p>\n<p>Etsy\u2019s initial <a href=\"https:\/\/www.etsy.com\/ca\/codeascraft\/increasing-experimentation-accuracy-and-speed-by-using-control-variates\">implementation of CUPED<\/a> yielded an average variance reduction of 7% across all experiments, with some experiments achieving up to 30% variance reduction. Experiments that used CUPED-adjusted metrics in decision-making yielded a decision about 1 day earlier, on average. However, we\u2019re always iterating to improve our buyers\u2019 and sellers\u2019 experience on Etsy, and we knew we could do even better. Enter: CUPAC.<\/p>\n<h2>Leveling up further with CUPAC<\/h2>\n<p>During our research and implementation of CUPED in 2020, scientists at DoorDash published a <a href=\"https:\/\/careersatdoordash.com\/blog\/improving-experimental-power-through-control-using-predictions-as-covariate-cupac\/\">blog post<\/a> describing a novel statistical method, building on CUPED, called Control Using Predictions as Covariate, or \u201cCUPAC.\u201d <\/p>\n<p>When performing CUPAC, the pre-experiment data is first input into a non-linear machine learning model that captures more complex relationships than a linear model. The non-linear model is trained to predict the outcome metric of interest \u2013 for example, if an experiment is measuring the observed Conversion Rate, the model would predict Conversion Rate. The prediction more effectively captures the impact of pre-experiment behaviors on our experimental outcomes than the raw pre-experiment data because it captures complex relationships in the data that linear regression alone cannot. The prediction is then used as an \u201cML-based covariate\u201d in a linear regression to perform the CUPED correction: <\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPAC formula\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/635c10\/7484133503\/inv_fullxfull.7484133503_juhuwigr.jpg?w=1280&#038;ssl=1\" title=\"CUPAC formula\"><br \/>\n<\/figure>\n<p>The CUPAC-adjusted outcome has an even smaller variance than the CUPED-adjusted outcome, as visualized below.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPAC variance chart\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/652fa0\/7484077983\/inv_fullxfull.7484077983_7p7csls3.jpg?w=1280&#038;ssl=1\" title=\"CUPAC variance chart\"><br \/>\n<\/figure>\n<p>Empirically, our CUPAC-adjusted metrics showed even lower variance than CUPED. Our initial prototype demonstrated that CUPAC produced an adjusted metric with an additional 10% smaller variance when compared to our original CUPED estimator. Despite the added complexity, these results justified incorporating CUPAC into our experimentation pipeline. We hypothesized it would cut average experiment duration by an additional day, enabling teams to run more experiments and ship changes to Etsy faster.<\/p>\n<h2>Training and implementation<\/h2>\n<p>The first step was to train the CUPAC models to predict the ML-based covariate. We identified over 100 pre-experiment features, increasing from 3 features in CUPED, to capture more behavior prior to the experiment. Using these features, we iteratively trained and tuned the models in Vertex AI. Hyperparameters were optimized on a validation dataset to maximize the median correlation between the model\u2019s predictions and the observed in-experiment metrics across experiments.<\/p>\n<p>Initially we trained <a href=\"https:\/\/xgboost.ai\/\">XGBoost<\/a>, a popular <a href=\"https:\/\/developers.google.com\/machine-learning\/decision-forests\/intro-to-gbdt\">gradient boosted tree<\/a> model, but then found <a href=\"https:\/\/lightgbm.readthedocs.io\/en\/stable\/\">LightGBM<\/a>, a similar non-linear, tree-based model, was better suited to predict the covariate. When testing the models at scale with billions of predictions, LightGBM demonstrated both rapid training and prediction times, along with strong validation results.<\/p>\n<p>Once the models were trained, our next challenge was to implement them at scale. Our experimentation pipeline runs batch jobs for hundreds of experiments each day. From our original implementation, we had an Airflow DAG (directed acyclic graph) to orchestrate the CUPED variance reduction pipeline, as visualized below:<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPED pipeline\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/baa64c\/7436088400\/inv_fullxfull.7436088400_c1h413ot.jpg?w=1280&#038;ssl=1\" title=\"CUPED pipeline\"><br \/>\n<\/figure>\n<p>We evolved this pipeline to support CUPAC by adding a batch prediction step to produce the ML-based covariate.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"CUPAC pipeline\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/755784\/7437734230\/inv_fullxfull.7437734230_dc480y5e.jpg?w=1280&#038;ssl=1\" title=\"CUPAC pipeline\"><br \/>\n<\/figure>\n<p>In the above CUPAC pipeline, we perform the following steps: <\/p>\n<ul>\n<li>Calculate pre-experiment features and in-experiment data using BigQuery SQL jobs.<\/li>\n<li>Predict ML-based covariates with our trained LightGBM models via parallel Dataflow jobs using the pre-experiment features.<\/li>\n<li>Perform variance reduction with a Spark job that fits a linear regression model between the ML-based covariates and in-experiment data, creating the CUPAC-adjusted metrics.<\/li>\n<li>Apply statistical t-tests using the CUPAC-adjusted metric to calculate the treatment effect, p-value, and power of the experiment.<\/li>\n<\/ul>\n<h2>Impact: Shortening average experiment duration by 3 days<\/h2>\n<p>We measured success through variance reduction. Variance reduction is the percent change between the:<\/p>\n<ul>\n<li>Variance of the metric <em>without<\/em> CUPAC, and<\/li>\n<li>Variance of the CUPAC-adjusted metric. <\/li>\n<\/ul>\n<p>The original CUPED implementation showed 7% variance reduction, reducing overall experiment duration by almost 1 day, on average. After implementing CUPAC, we observed an average of 27% variance reduction, nearly <strong>4x as much variance reduction<\/strong>, when compared to CUPED, exceeding our early research estimates.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"Impact chart\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/63d8c5\/7437853008\/inv_fullxfull.7437853008_6cpvb4kq.jpg?w=1280&#038;ssl=1\" title=\"Impact chart\"><br \/>\n<\/figure>\n<p>The additional variance reduction shortens our average experiment duration by almost 3 days. <strong>This means a 10-day experiment could conclude in only 7 days due to the ability to reach power on a smaller sample size with CUPAC<\/strong>. These marginal time savings allow many teams to run <strong>10 or more additional experiments each year<\/strong>. That translates to more opportunities to test and faster insights into how we can deliver the best experience for our community of millions of sellers and buyers. <\/p>\n<p>Notably, there was a substantial spread in variance reduction among different metrics and experiments, ranging from 2% to 77%. In the chart below, each blue bar displays the percent variance reduction for a sampled metric on an experiment. <\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"Percent chart\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/9bc046\/7485759507\/inv_fullxfull.7485759507_eog6ydtb.jpg?w=1280&#038;ssl=1\" title=\"Percent chart\"><br \/>\n<\/figure>\n<p>The large range is expected because variance reduction can be influenced by several factors, such as metric definition, data accessibility, experimental design, and market characteristics. These factors impact how predictive the pre-experiment data is of the outcome metric, resulting in the degree of variance reduction. For example, two common experimentation metrics are Mean Visits and Purchase Rate. In the e-commerce setting, an individual&#8217;s visit behavior will almost always be more stable over time than their purchasing behavior. This implies that pre-experiment data is more correlated with in-experiment data for a visit-related metric than for a purchase-related metric. Therefore, CUPAC is more effective at reducing variance in a metric like Mean Visits than in a metric like Purchase Rate.<\/p>\n<h2>What\u2019s next?<\/h2>\n<p>Aligned with Etsy\u2019s culture of experimentation, we\u2019ll continue to evolve our pipeline to be nimble and flexible based on the needs of the teams that use them. <\/p>\n<p>One challenge we face is that teams use metrics curated to specific parts of the Etsy experience \u2013 like search, recommendations, seller features, etc. \u2013 to make decisions on their experiment results. However, our CUPAC models take significant time to train and maintain for each metric, consequently limiting the number of CUPAC-adjusted metrics we can develop. While we continue to grow CUPAC use, we also encourage teams to continue to use CUPED, which is more scalable and has lower maintenance costs. To account for this, we plan to increase the flexibility of CUPED to more metrics by automatically collecting pre-experiment data based on the metric definition to reduce noise. In tandem with our work on CUPAC, this CUPED expansion will enable teams across Etsy to benefit from variance reduction across all their team-specific metrics, not just a select few.<\/p>\n<p>Despite the success of CUPED and CUPAC thus far, there remains a need to explore additional variance reduction techniques for the current metrics that leverage CUPAC. In 2024, we released research findings exploring a novel approach: <a href=\"https:\/\/arxiv.org\/abs\/2410.09027\">Variance reduction combining pre-experiment and in-experiment data<\/a>. As we look to generalize our variance reduction architecture, we expect that incorporating such techniques will continue to strengthen our experimentation platform and enable product teams to iterate more quickly.<\/p>\n<p>Lastly, it is important to recognize that applying variance reduction in practice can be a never-ending race to squeeze the most noise out of these estimators. In our experience, the craft lies in finding the sweet spot between variance reduction, implementation cost, and the impact on experimentation velocity. That intersection is context-dependent and what makes experimentation code as craft.<\/p>\n<p>We hope our experience inspires you to try out variance reduction techniques and determine which one is best suited to your needs!<\/p>\n<h2>Acknowledgements<\/h2>\n<p>Thank you to Alexander Tank and Stephane Shao for their work on initial research and implementation of CUPAC. Thanks to Pablo Crespo for his research into extending our CUPAC models with more predictive features. And, thanks to Julie Beckley, Kevin Gaan, and Mary Hu for supporting and prioritizing this project. <\/p>\n<h2>References<\/h2>\n<p>A. Deng, Y. Xu, R. Kohavi, T. Walker (2013). <a href=\"https:\/\/exp-platform.com\/Documents\/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf\">Improving the sensitivity of online controlled experiments by utilizing pre-experiment data<\/a>.<\/p>\n<p>J. Li (2020). <a href=\"https:\/\/careersatdoordash.com\/blog\/improving-experimental-power-through-control-using-predictions-as-covariate-cupac\/\">Improving Experimental Power through Control Using Predictions as Covariate (CUPAC)<\/a>.<\/p>\n<\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1480\" class=\"pvc_stats total_only\" data-element-id=\"1480\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewbox=\"0 0 502 315\" preserveaspectratio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/i0.wp.com\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif?resize=16%2C16&#038;ssl=1\" border=\"0\" \/><\/p>\n<div class=\"pvc_clear\"><\/div>","protected":false},"excerpt":{"rendered":"<p>In 2021, we published a blog post titled \u201cIncreasing experimentation accuracy and speed by using control variates,\u201d describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolved<\/p>\n<div class=\"hosteria-entry-more\"><a href=\"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/\" class=\"no-underline font-light  group-hover:text-primary-800 dark:group-hover:text-primary-300 py-1\">Read more &gt;&gt;&gt;<\/a><\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1480\" class=\"pvc_stats total_only\" data-element-id=\"1480\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewbox=\"0 0 502 315\" preserveaspectratio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=\"0\" \/><\/p>\n<div class=\"pvc_clear\"><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"fp_fajr_begins":"","fp_fajr_iqamah":"","fp_dhuhr_begins":"","fp_dhuhr_iqamah":"","fp_asr_begins":"","fp_asr_iqamah":"","fp_maghrib_begins":"","fp_maghrib_iqamah":"","fp_isha_begins":"","fp_isha_iqamah":"","fp_midnight":"","fp_midnight_name":"","fp_sunrise":"","fp_single_prayer_begins_title":"","fp_single_prayer_iqamah_title":"","fp_prayer_times_for_today":"","fp_hijra_date":"","fp_fajr_name":"","fp_dhuhr_name":"","fp_asr_name":"","fp_maghrib_name":"","fp_isha_name":"","fp_sunrise_name":"","fp_currentDate":"","fp_current_time":"","fp_current_title":"","fp_current_location":"","fp_masjid_name":"","fp_prayer_title":"","fp_next_prayer_iqamah_time":"","fp_next_prayer_iqamah_title":"","fp_next_prayer_begins_time":"","fp_next_prayer_begins_title":"","fp_next_prayer_title":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[37],"tags":[],"class_list":["post-1480","post","type-post","status-publish","format-standard","hentry","category-posts"],"a3_pvc":{"activated":true,"total_views":0,"today_views":0},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Reducing experiment duration with predicted control variates - Dev95<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/\" \/>\n<meta property=\"og:locale\" content=\"ar_AR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reducing experiment duration with predicted control variates - Dev95\" \/>\n<meta property=\"og:description\" content=\"In 2021, we published a blog post titled \u201cIncreasing experimentation accuracy and speed by using control variates,\u201d describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolvedRead more &gt;&gt;&gt;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/\" \/>\n<meta property=\"og:site_name\" content=\"Dev95\" \/>\n<meta property=\"article:published_time\" content=\"2025-11-25T16:28:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0\" \/>\n<meta name=\"author\" content=\"dev95\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629\" \/>\n\t<meta name=\"twitter:data1\" content=\"dev95\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 \u062f\u0642\u0627\u0626\u0642\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/\"},\"author\":{\"name\":\"dev95\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\"},\"headline\":\"Reducing experiment duration with predicted control variates\",\"datePublished\":\"2025-11-25T16:28:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/\"},\"wordCount\":1747,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/508bdd\\\/7485804299\\\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0\",\"articleSection\":[\"Posts\"],\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/\",\"url\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/\",\"name\":\"Reducing experiment duration with predicted control variates - Dev95\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/508bdd\\\/7485804299\\\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0\",\"datePublished\":\"2025-11-25T16:28:26+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#breadcrumb\"},\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/508bdd\\\/7485804299\\\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0\",\"contentUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/508bdd\\\/7485804299\\\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dev95.site\\\/reducing-experiment-duration-with-predicted-control-variates\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dev95.site\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Reducing experiment duration with predicted control variates\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"name\":\"Dev95\",\"description\":\"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.\",\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dev95.site\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ar\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\",\"name\":\"Dev95\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"width\":512,\"height\":512,\"caption\":\"Dev95\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\",\"name\":\"dev95\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"caption\":\"dev95\"},\"url\":\"https:\\\/\\\/dev95.site\\\/ar\\\/author\\\/mohammad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Reducing experiment duration with predicted control variates - Dev95","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/","og_locale":"ar_AR","og_type":"article","og_title":"Reducing experiment duration with predicted control variates - Dev95","og_description":"In 2021, we published a blog post titled \u201cIncreasing experimentation accuracy and speed by using control variates,\u201d describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolvedRead more &gt;&gt;&gt;","og_url":"https:\/\/dev95.site\/ar\/reducing-experiment-duration-with-predicted-control-variates\/","og_site_name":"Dev95","article_published_time":"2025-11-25T16:28:26+00:00","og_image":[{"url":"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0","type":"","width":"","height":""}],"author":"dev95","twitter_card":"summary_large_image","twitter_misc":{"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629":"dev95","\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631":"9 \u062f\u0642\u0627\u0626\u0642"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#article","isPartOf":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/"},"author":{"name":"dev95","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298"},"headline":"Reducing experiment duration with predicted control variates","datePublished":"2025-11-25T16:28:26+00:00","mainEntityOfPage":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/"},"wordCount":1747,"commentCount":0,"publisher":{"@id":"https:\/\/dev95.site\/#organization"},"image":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#primaryimage"},"thumbnailUrl":"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0","articleSection":["Posts"],"inLanguage":"ar","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/","url":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/","name":"Reducing experiment duration with predicted control variates - Dev95","isPartOf":{"@id":"https:\/\/dev95.site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#primaryimage"},"image":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#primaryimage"},"thumbnailUrl":"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0","datePublished":"2025-11-25T16:28:26+00:00","breadcrumb":{"@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#breadcrumb"},"inLanguage":"ar","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/"]}]},{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#primaryimage","url":"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0","contentUrl":"https:\/\/i.etsystatic.com\/inv\/508bdd\/7485804299\/inv_fullxfull.7485804299_5dhtrbg7.jpg?version=0"},{"@type":"BreadcrumbList","@id":"https:\/\/dev95.site\/reducing-experiment-duration-with-predicted-control-variates\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dev95.site\/"},{"@type":"ListItem","position":2,"name":"Reducing experiment duration with predicted control variates"}]},{"@type":"WebSite","@id":"https:\/\/dev95.site\/#website","url":"https:\/\/dev95.site\/","name":"Dev95","description":"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.","publisher":{"@id":"https:\/\/dev95.site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dev95.site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ar"},{"@type":"Organization","@id":"https:\/\/dev95.site\/#organization","name":"Dev95","url":"https:\/\/dev95.site\/","logo":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Dev95"},"image":{"@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298","name":"dev95","image":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","caption":"dev95"},"url":"https:\/\/dev95.site\/ar\/author\/mohammad\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1480","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/comments?post=1480"}],"version-history":[{"count":0,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1480\/revisions"}],"wp:attachment":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/media?parent=1480"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/categories?post=1480"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/tags?post=1480"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}