{"id":1466,"date":"2023-09-05T19:55:41","date_gmt":"2023-09-05T19:55:41","guid":{"rendered":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/"},"modified":"2023-09-05T19:55:41","modified_gmt":"2023-09-05T19:55:41","slug":"the-so-fine-real-time-ml-paradigm","status":"publish","type":"post","link":"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/","title":{"rendered":"The So-fine Real-time ML Paradigm"},"content":{"rendered":"<div id=\"dev95-3178420990\" class=\"dev95-- dev95-entity-placement\"><script async=\"async\" data-cfasync=\"false\" src=\"https:\/\/pl27862732.profitableratecpmnetwork.com\/2ad7a50e0bbc23ac6801d7b77c501463\/invoke.js\"><\/script>\r\n<div id=\"container-2ad7a50e0bbc23ac6801d7b77c501463\"><\/div><\/div><div>\n<h2>Introduction<\/h2>\n<p>Each year, Etsy hosts an event known as  \u201cCodeMosaic\u201d &#8211; an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes.  People across Etsy source ideas, organize into teams, and then have 2-3 days to build innovative proofs-of-concept that might deliver big wins for Etsy\u2019s buyers and sellers, or improve internal engineering systems and workflows.<\/p>\n<p>Besides being a ton of fun, CodeMosaic is a time for engineers to pilot novel ideas. Our team\u2019s project this year was extremely ambitious &#8211; we wanted to build a system for stateful machine learning (ML) model training and online machine learning. While our ML pipelines are no stranger to streaming data, we currently don\u2019t have any models that learn in an online context &#8211; that is, that can have their weights updated in near-real time.<\/p><div id=\"dev95-1666683524\" class=\"dev95- dev95-entity-placement\"><center>\r\n<script>\r\n  atOptions = {\r\n    'key' : '4ba6b6513c00e0ba76511f798ae56401',\r\n    'format' : 'iframe',\r\n    'height' : 50,\r\n    'width' : 320,\r\n    'params' : {}\r\n  };\r\n<\/script>\r\n<script src=\"https:\/\/www.highrevenueformat.com\/4ba6b6513c00e0ba76511f798ae56401\/invoke.js\"><\/script>\r\n\t<\/center><\/div>\n<p>Stateful training updates an already-trained ML model artifact incrementally, sparing the cost of retraining models from scratch. <em>Online learning<\/em> updates model weights in production rather than via batch processes. Combined, the two approaches can be extremely powerful. <a href=\"https:\/\/arxiv.org\/abs\/2107.07106\">A study conducted by Grubhub in 2021<\/a> reported that a shift to stateful online learning saw up to a 45x reduction in costs with a 20% increase in metrics, and I\u2019m all about saving money to make money.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"Stateful continuous retraining and online learning\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?w=1280&#038;ssl=1\" title=\"Stateful continuous retraining and online learning\"><br \/>\n<\/figure>\n<h2>Day 1 &#8211; Planning<\/h2>\n<p>Of course, building such a complex system would be no easy task. The ML pipelines we use to generate training data from user actions require a number of offline, scheduled batch jobs. As a result it takes quite a while, 40 hours at a minimum, for user actions to be reflected in a model\u2019s weights.<\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"current pipeline\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/0d445c\/5272464395\/inv_fullxfull.5272464395_mtgyerd2.jpg?w=1280&#038;ssl=1\" title=\"current pipeline\"><br \/>\n<\/figure>\n<p>To make this project a success over the course of three days, we needed to scope our work tightly across three streams:<\/p>\n<ol>\n<li><strong>Real-time training data<\/strong> &#8211; the task here was to circumvent the batch jobs responsible for our current training data and get attributions (user actions) right from the source.<\/li>\n<li><strong>A service to consume the data stream and learn incrementally<\/strong> &#8211; today, we heavily leverage TensorFlow for model training. We needed to be able to load a model&#8217;s weights into memory, read data from a stream, update that model, and incrementally push it out to be served online. <\/li>\n<li><strong>Evaluation<\/strong> &#8211; we&#8217;d have to make a case for our approach by validating its performance benefits over our current batch processes. <\/li>\n<\/ol>\n<p>No matter how much we limited the scope it wasn&#8217;t going to be easy, but we broke into three subteams reflecting each track of work and began moving towards implementation.<\/p>\n<h2>Day 2 &#8211; Implementation<\/h2>\n<p>The real-time training data team began by looking far upstream of the batch jobs that compute training data &#8211; at Etsy\u2019s Beacon Main Kafka stream, which contains bot-filtered events. By using Kafka SQL and some real-time calls to our streaming feature platform, Rivulet, we figured we could put together a realistic approach to solving this part of the problem.<\/p>\n<p>Of course, as with all hackathon ideas it was easier said than done. Much of our feature data uses the binary avro data format for serialization, and finding the proper schema for deserializing and joining this data was troublesome. The team spent most of the second day munging the data in an attempt to join all the proper sources across platforms. And though we weren&#8217;t able to write the output to a new topic, the team actually did manage to join multiple data sources in a way that generated real-time training data!<\/p>\n<p>Meanwhile the team focusing on building the consumer service to actually learn from the model faced a different kind of challenge: decision making. What type of model were we going to use? Knowing we weren\u2019t going to be able to use the actual training data stream yet &#8211; how would we mock it? Where and how often should we push new model artifacts out? <\/p>\n<p>After significant discussion, we decided to try using an Ad Ranking model as we had an Ads ML engineer in our group and the Ads models take a long time to train &#8211; meaning we could squeeze a lot of benefit out of them by implementing continuous training. The engineers in the group began to structure code that pulled an older Ads model into memory and made incremental updates to the weights to satisfy the second requirement. <\/p>\n<p>That meant that all we had left to handle was the most challenging task &#8211; evaluation. None of this architecture would mean anything if a model that was trained online performed worse than the model retrained daily in batch. Evaluating a model with more training training periods is also more difficult, as each period we\u2019d need to run the model on some held-out data in order to get an accurate reading without data leakage. <\/p>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"oof\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/171da0\/5224262572\/inv_fullxfull.5224262572_694cztlm.jpg?w=1280&#038;ssl=1\" title=\"oof\"><br \/>\n<\/figure>\n<p>Instead of performing an extremely laborious and time-intensive evaluation for continuous training like the one outlined above, we chose to have a bit more fun with it. After all, it was a hackathon! What if we made it a competition? Pick a single high-performing Etsy ad and see which surfaced it first, our continuously trained model or the boring old batch-trained one?<\/p>\n<p>We figured if we could get a continuously trained model to recommend a high-performing ad sooner, we\u2019d have done the job! So we set about searching for a high-performing Etsy ad and training data that would allow us to validate our work. <\/p>\n<p>Of course, by the time we were even deciding on an appropriate advertised listing, it was the end of day two, and it was pretty clear the idea wasn\u2019t going to play out before it was time for presentations. But still a fun thought, right?<\/p>\n<h2>Presentation takeaways and impact<\/h2>\n<p>Day 3 gives you a small window for tidying up work and slides, followed by team presentations. At this point, we loosely had these three things:<\/p>\n<ol>\n<li>Training data from much earlier in our batch processing pipelines<\/li>\n<li>A Kafka consumer that could almost update a TensorFlow model incrementally <\/li>\n<li>A few click attributions and data for a specific listing<\/li>\n<\/ol>\n<figure>\n<img data-recalc-dims=\"1\" decoding=\"async\" alt=\"final product\" src=\"https:\/\/i0.wp.com\/i.etsystatic.com\/inv\/b8e099\/5272467237\/inv_fullxfull.5272467237_68m5p29m.jpg?w=1280&#038;ssl=1\" title=\"final product\"><br \/>\n<\/figure>\n<p>In the hackathon spirit, we phoned it in and pivoted towards focusing on the theoretical of what we\u2019d been able to achieve!<\/p>\n<p>The 1st important potential area of impact was cost savings. We estimated that removing the daily \u201ccold-start\u201d training and replacing it with continuous training would save about <strong>$212K annually in Google Cloud costs<\/strong> for the 4 models in ads alone.<\/p>\n<p>This is a huge potential win &#8211; especially when coupled with the likely metrics gains coming from more reactive models. After all, if we were able to get events to models 40 hours earlier, who knows how much better our ranking could get! <\/p>\n<h2>Future directions and conclusion<\/h2>\n<p>Like many hackathon projects, there&#8217;s no shortage of hurdles getting this work into a production state. Aside from the infrastructure required to actually architect a continuous-training pipeline, we\u2019d need a significant number of high-quality checks and balances to ensure that updating models in real-time didn\u2019t lead to sudden degradations in performance. The amount of development, number of parties involved, and the breadth of expertise to get this into production would surely be extensive. However, as ML continues to mature, we should be able to enable more complex architectures with less overhead.<\/p>\n<\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1466\" class=\"pvc_stats total_only\" data-element-id=\"1466\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewbox=\"0 0 502 315\" preserveaspectratio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/i0.wp.com\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif?resize=16%2C16&#038;ssl=1\" border=\"0\" \/><\/p>\n<div class=\"pvc_clear\"><\/div>","protected":false},"excerpt":{"rendered":"<p>Introduction Each year, Etsy hosts an event known as \u201cCodeMosaic\u201d &#8211; an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes. People across Etsy source ideas, organize into<\/p>\n<div class=\"hosteria-entry-more\"><a href=\"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/\" class=\"no-underline font-light  group-hover:text-primary-800 dark:group-hover:text-primary-300 py-1\">Read more &gt;&gt;&gt;<\/a><\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1466\" class=\"pvc_stats total_only\" data-element-id=\"1466\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewbox=\"0 0 502 315\" preserveaspectratio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=\"0\" \/><\/p>\n<div class=\"pvc_clear\"><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"fp_fajr_begins":"","fp_fajr_iqamah":"","fp_dhuhr_begins":"","fp_dhuhr_iqamah":"","fp_asr_begins":"","fp_asr_iqamah":"","fp_maghrib_begins":"","fp_maghrib_iqamah":"","fp_isha_begins":"","fp_isha_iqamah":"","fp_midnight":"","fp_midnight_name":"","fp_sunrise":"","fp_single_prayer_begins_title":"","fp_single_prayer_iqamah_title":"","fp_prayer_times_for_today":"","fp_hijra_date":"","fp_fajr_name":"","fp_dhuhr_name":"","fp_asr_name":"","fp_maghrib_name":"","fp_isha_name":"","fp_sunrise_name":"","fp_currentDate":"","fp_current_time":"","fp_current_title":"","fp_current_location":"","fp_masjid_name":"","fp_prayer_title":"","fp_next_prayer_iqamah_time":"","fp_next_prayer_iqamah_title":"","fp_next_prayer_begins_time":"","fp_next_prayer_begins_title":"","fp_next_prayer_title":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[37],"tags":[],"class_list":["post-1466","post","type-post","status-publish","format-standard","hentry","category-posts"],"a3_pvc":{"activated":true,"total_views":0,"today_views":0},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>The So-fine Real-time ML Paradigm - Dev95<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/\" \/>\n<meta property=\"og:locale\" content=\"ar_AR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The So-fine Real-time ML Paradigm - Dev95\" \/>\n<meta property=\"og:description\" content=\"Introduction Each year, Etsy hosts an event known as \u201cCodeMosaic\u201d &#8211; an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes. People across Etsy source ideas, organize intoRead more &gt;&gt;&gt;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/\" \/>\n<meta property=\"og:site_name\" content=\"Dev95\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-05T19:55:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0\" \/>\n<meta name=\"author\" content=\"dev95\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629\" \/>\n\t<meta name=\"twitter:data1\" content=\"dev95\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 \u062f\u0642\u0627\u0626\u0642\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/\"},\"author\":{\"name\":\"dev95\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\"},\"headline\":\"The So-fine Real-time ML Paradigm\",\"datePublished\":\"2023-09-05T19:55:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/\"},\"wordCount\":1221,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/d8a51b\\\/5272462297\\\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0\",\"articleSection\":[\"Posts\"],\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/\",\"url\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/\",\"name\":\"The So-fine Real-time ML Paradigm - Dev95\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/d8a51b\\\/5272462297\\\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0\",\"datePublished\":\"2023-09-05T19:55:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#breadcrumb\"},\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/d8a51b\\\/5272462297\\\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0\",\"contentUrl\":\"https:\\\/\\\/i.etsystatic.com\\\/inv\\\/d8a51b\\\/5272462297\\\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dev95.site\\\/the-so-fine-real-time-ml-paradigm\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dev95.site\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The So-fine Real-time ML Paradigm\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"name\":\"Dev95\",\"description\":\"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.\",\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dev95.site\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ar\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\",\"name\":\"Dev95\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"width\":512,\"height\":512,\"caption\":\"Dev95\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\",\"name\":\"dev95\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"caption\":\"dev95\"},\"url\":\"https:\\\/\\\/dev95.site\\\/ar\\\/author\\\/mohammad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The So-fine Real-time ML Paradigm - Dev95","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/","og_locale":"ar_AR","og_type":"article","og_title":"The So-fine Real-time ML Paradigm - Dev95","og_description":"Introduction Each year, Etsy hosts an event known as \u201cCodeMosaic\u201d &#8211; an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes. People across Etsy source ideas, organize intoRead more &gt;&gt;&gt;","og_url":"https:\/\/dev95.site\/ar\/the-so-fine-real-time-ml-paradigm\/","og_site_name":"Dev95","article_published_time":"2023-09-05T19:55:41+00:00","og_image":[{"url":"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0","type":"","width":"","height":""}],"author":"dev95","twitter_card":"summary_large_image","twitter_misc":{"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629":"dev95","\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631":"6 \u062f\u0642\u0627\u0626\u0642"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#article","isPartOf":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/"},"author":{"name":"dev95","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298"},"headline":"The So-fine Real-time ML Paradigm","datePublished":"2023-09-05T19:55:41+00:00","mainEntityOfPage":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/"},"wordCount":1221,"commentCount":0,"publisher":{"@id":"https:\/\/dev95.site\/#organization"},"image":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#primaryimage"},"thumbnailUrl":"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0","articleSection":["Posts"],"inLanguage":"ar","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/","url":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/","name":"The So-fine Real-time ML Paradigm - Dev95","isPartOf":{"@id":"https:\/\/dev95.site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#primaryimage"},"image":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#primaryimage"},"thumbnailUrl":"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0","datePublished":"2023-09-05T19:55:41+00:00","breadcrumb":{"@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#breadcrumb"},"inLanguage":"ar","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/"]}]},{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#primaryimage","url":"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0","contentUrl":"https:\/\/i.etsystatic.com\/inv\/d8a51b\/5272462297\/inv_fullxfull.5272462297_1e6jr1i2.jpg?version=0"},{"@type":"BreadcrumbList","@id":"https:\/\/dev95.site\/the-so-fine-real-time-ml-paradigm\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dev95.site\/"},{"@type":"ListItem","position":2,"name":"The So-fine Real-time ML Paradigm"}]},{"@type":"WebSite","@id":"https:\/\/dev95.site\/#website","url":"https:\/\/dev95.site\/","name":"Dev95","description":"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.","publisher":{"@id":"https:\/\/dev95.site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dev95.site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ar"},{"@type":"Organization","@id":"https:\/\/dev95.site\/#organization","name":"Dev95","url":"https:\/\/dev95.site\/","logo":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Dev95"},"image":{"@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298","name":"dev95","image":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","caption":"dev95"},"url":"https:\/\/dev95.site\/ar\/author\/mohammad\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/comments?post=1466"}],"version-history":[{"count":0,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1466\/revisions"}],"wp:attachment":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/media?parent=1466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/categories?post=1466"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/tags?post=1466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}