{"id":1890,"date":"2024-01-03T22:26:50","date_gmt":"2024-01-03T22:26:50","guid":{"rendered":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/"},"modified":"2024-01-03T22:26:50","modified_gmt":"2024-01-03T22:26:50","slug":"scalable-continuous-integration-for-ios","status":"publish","type":"post","link":"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/","title":{"rendered":"Scalable Continuous Integration for iOS"},"content":{"rendered":"<div id=\"dev95-1704150300\" class=\"dev95-- dev95-entity-placement\"><script async=\"async\" data-cfasync=\"false\" src=\"https:\/\/pl27862732.profitableratecpmnetwork.com\/2ad7a50e0bbc23ac6801d7b77c501463\/invoke.js\"><\/script>\r\n<div id=\"container-2ad7a50e0bbc23ac6801d7b77c501463\"><\/div><\/div><div><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp?w=1280&#038;ssl=1\" alt=\"Scalable Continuous Integration for iOS\"><\/p>\n<p><em>Originally published on the\u00a0<\/em><a href=\"https:\/\/medium.com\/justeattakeaway-tech\/scalable-continuous-integration-for-ios-15ff33435992?ref=albertodebortoli.com\" rel=\"noreferrer\"><em>Just Eat\u00a0Takeaway\u00a0Engineering Blog<\/em><\/a><em>.<\/em><\/p><div id=\"dev95-1306151674\" class=\"dev95- dev95-entity-placement\"><center>\r\n<script>\r\n  atOptions = {\r\n    'key' : '4ba6b6513c00e0ba76511f798ae56401',\r\n    'format' : 'iframe',\r\n    'height' : 50,\r\n    'width' : 320,\r\n    'params' : {}\r\n  };\r\n<\/script>\r\n<script src=\"https:\/\/www.highrevenueformat.com\/4ba6b6513c00e0ba76511f798ae56401\/invoke.js\"><\/script>\r\n\t<\/center><\/div>\n<p>How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners.<\/p>\n<h3 id=\"problem\">Problem<\/h3>\n<p>At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscape of innovation and adaptation. Historically, JET\u2019s multiple iOS teams operated independently, each employing their distinct CI solutions.<\/p>\n<p>The original Just Eat iOS and Android teams had pioneered an in-house CI solution anchored in Jenkins. This setup, detailed in our 2021\u00a0<a href=\"https:\/\/medium.com\/justeattakeaway-tech\/the-continuous-integration-system-used-by-the-mobile-teams-28ba057ef628?ref=albertodebortoli.com\" rel=\"noopener\">article<\/a>, served as the backbone of our CI practices up until 2020. It was during this period that the iOS team initiated a pivotal migration: moving from in-house Mac Pros and Mac Minis to AWS EC2 macOS instances.<\/p>\n<p>Fast forward to 2023, a significant transition occurred within our Continuous Delivery Engineering (CDE) Platform Engineering team. The decision to adopt GitHub Actions company-wide has marked the end of our reliance on Jenkins while other teams are in the process of migrating away from solutions such as CircleCI and GitLab CI. This transition represented a fundamental shift in our CI philosophy. By moving away from Jenkins, we eliminated the need to maintain an instance for the Jenkins server and the complexities of managing how agents connected to it.\u00a0Our focus then shifted to transforming our Jenkins pipelines into GitHub Actions workflows.<\/p>\n<p>This transformation extended beyond mere tool adoption. Our primary goal was to ensure that our macOS instances were not only scalable but also configured in code. We therefore enhanced our global CI practices and set standards across the entire company.<\/p>\n<h3 id=\"desired-state-of-ci\">Desired state of CI<\/h3>\n<p>As we embarked on our journey to refine and elevate our CI process, we envisioned a state-of-the-art CI system. Our goals were ambitious yet clear, focusing on scalability, automation, and efficiency. At the time of implementing the system, no other player in the industry seemed to have implemented the complete solution we envisioned.<\/p>\n<p>Below is a summary of our desired CI state:<\/p>\n<ul>\n<li><strong>Instance setup in code<\/strong>: One primary objective was to enable the definition of the setup of the instances entirely in code. This includes specifying macOS version, Xcode version, Ruby version, and other crucial configurations. For this purpose, the HashiCorp tool Packer, emerged once again as an ideal solution, offering the flexibility and precision we required.<\/li>\n<li><strong>IaC (Infrastructure as Code) for macOS instances<\/strong>: To define the infrastructure of our fleet of macOS instances, we leaned towards Terraform, another HashiCorp tool. Terraform provided us with the capability to not only deploy but also to scale and migrate our infrastructure seamlessly, crucially maintaining its state.<\/li>\n<li><strong>Auto and Manual Scaling<\/strong>: We wanted the ability to dynamically create CI runners based on demand, ensuring that resources were optimally utilized and available when needed. To optimize resource utilization, especially during off-peak hours, we desired an autoscaling feature. Scaling down our CI runners on weekends when developer activity is minimal was critical to be cost-effective.<\/li>\n<li><strong>Automated Connection to GitHub Actions<\/strong>: We aimed for the instances to automatically connect to GitHub Actions as runners upon deployment. This automation was crucial in eliminating manual interventions via SSH or VNC.<\/li>\n<li><strong>Multi-Team Use<\/strong>: Our vision included CI runners that could be easily used by multiple teams across different time zones. This would not only maximize the utility of our infrastructure but also encourage reuse and standardization.<\/li>\n<li><strong>Centralized Management via GitHub Actions<\/strong>: To further streamline our CI processes, we intended to run all tasks through GitHub Actions workflows. This approach would allow the teams to self-serve and alleviate the need for developers to use Docker or maintain local environments.<\/li>\n<\/ul>\n<p>Getting to the desired state was a journey that presented multiple challenges and constant adjustments to make sure we could migrate smoothly to a new system.<\/p>\n<h3 id=\"instance-setup-in-code\"><strong>Instance setup in code<\/strong><\/h3>\n<p>We implemented the desired configuration\u00a0<a href=\"https:\/\/aws.amazon.com\/blogs\/compute\/building-amazon-machine-images-amis-for-ec2-mac-instances-with-packer\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">with Packer<\/a>\u00a0leveraging a number of\u00a0<a href=\"https:\/\/developer.hashicorp.com\/packer\/docs\/provisioners\/shell?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Shell Provisioners<\/a>\u00a0and variables to configure the instance. Here are some of the configuration steps:<\/p>\n<ul>\n<li>Set user password (to allow remote desktop access)<\/li>\n<li>Resize the partition to use all the space available on the EBS volume<\/li>\n<li>Start the Apple Remote Desktop agent and enable remote desktop access<\/li>\n<li>Update Brew &amp; Install Brew packages<\/li>\n<li>Install CloudWatch agent<\/li>\n<li>Install rbenv\/Ruby\/bundler<\/li>\n<li>Install Xcode versions<\/li>\n<li>Install GitHub Actions actions-runner<\/li>\n<li>Copy scripts to connect to GitHub Actions as a runner<\/li>\n<li>Copy daemon to start the GitHub Actions self-hosted runner\u00a0<a href=\"https:\/\/docs.github.com\/en\/actions\/hosting-your-own-runners\/managing-self-hosted-runners\/configuring-the-self-hosted-runner-application-as-a-service?platform=mac&amp;ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">as a service<\/a><\/li>\n<li>Set\u00a0<a href=\"https:\/\/github.com\/aws\/ec2-macos-init?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">macos-init<\/a>\u00a0modules to perform provisioning of the first launch<\/li>\n<\/ul>\n<p>While the steps above are naturally configuration steps to perform when creating the AMI, the macos-init modules include steps to perform once the instance becomes available.<\/p>\n<p>The\u00a0<code>create_ami<\/code>\u00a0workflow accepts inputs that are eventually passed to Packer to generate the AMI.<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_UTehZkRYTyeGW-xHO2uGEw.webp?resize=788%2C878&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"788\" height=\"878\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_UTehZkRYTyeGW-xHO2uGEw.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_UTehZkRYTyeGW-xHO2uGEw.webp 788w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<pre><code class=\"language-bash\">packer build \n  --var ami_name_prefix=${{ env.AMI_NAME_PREFIX }} \n  --var region=${{ env.REGION }} \n  --var subnet_id=${{ env.SUBNET_ID }} \n  --var vpc_id=${{ env.VPC_ID }} \n  --var root_volume_size_gb=${{ env.ROOT_VOLUME_SIZE_GB }} \n  --var macos_version=${{ inputs.macos-version}} \n  --var ruby_version=${{ inputs.ruby-version }} \n  --var xcode_versions='${{ steps.parse-xcode-versions.outputs.list }}' \n  --var gha_version=${{ inputs.gha-version}} \n  bare-metal-runner.pkr.hcl<\/code><\/pre>\n<p>Different teams often use different versions of software, like Xcode. To accommodate this, we permit multiple versions to be installed on the same instance. The choice of which version to use is then determined within the GitHub Actions workflows.<\/p>\n<p>The seamless generation of AMIs has proven to be a significant enabler. For example, when Xcode 15.1 was released, we executed this workflow the same evening. In just over two hours, we had an AMI ready to deploy all the runners (it usually takes 70\u2013100 minutes for a macOS AMI with 400GB of EBS volume to become ready after creation). This efficiency enabled our teams to use the new Xcode version just a few hours after its release.<\/p>\n<h3 id=\"iac-infrastructure-as-code-for-macos-instances\"><strong>IaC\u00a0<\/strong>(Infrastructure as Code)\u00a0<strong>for macOS instances<\/strong><\/h3>\n<p>Initially, we used distinct Terraform modules for each instance to facilitate the deployment and decommissioning of each one. Given the high cost of EC2 Mac instances, we managed this process with caution, carefully balancing host usage while also being mindful of the 24-hour minimum allocation time.<\/p>\n<p>We ultimately ended up using Terraform to define a single infrastructure (i.e. a single Terraform module) defining resources such as:<\/p>\n<ul>\n<li><code>aws_key_pair<\/code>, <code>aws_instance<\/code>, <code>aws_ami<\/code><\/li>\n<li><code>aws_security_group<\/code>, <code>aws_security_group_rule<\/code><\/li>\n<li><code>aws_secretsmanager_secret<\/code><\/li>\n<li><code>aws_vpc<\/code>, <code>aws_subnet<\/code><\/li>\n<li><code>aws_cloudwatch_metric_alarm<\/code><\/li>\n<li><code>aws_sns_topic<\/code>, <code>aws_sns_topic_subscription<\/code><\/li>\n<li><code>aws_iam_role<\/code>, <code>aws_iam_policy<\/code>, <code>aws_iam_role_policy_attachment<\/code>, <code>aws_iam_instance_profile<\/code><\/li>\n<\/ul>\n<p>A crucial part was to use\u00a0<code>count<\/code>\u00a0in\u00a0<code>aws_instance<\/code>, setting the value of a variable passed in from\u00a0<code>deploy_infra<\/code>\u00a0workflow. Terraform performs the necessary scaling upon changing the value.<\/p>\n<p>We have implemented a workflow to perform Terraform\u00a0<code>apply<\/code>\u00a0and\u00a0<code>destroy<\/code>\u00a0commands for the infrastructure. Only the AMI and the number of instances are required as inputs.<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_DBUpUssi6rj8DqpIL6VXkA.webp?resize=762%2C680&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"762\" height=\"680\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_DBUpUssi6rj8DqpIL6VXkA.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_DBUpUssi6rj8DqpIL6VXkA.webp 762w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<pre><code class=\"language-bash\">terraform ${{ inputs.command }} \n  --var ami_name=${{ inputs.ami-name }} \n  --var fleet_size=${{ inputs.fleet-size }} \n  --auto-approve<\/code><\/pre>\n<p>Using the name of the AMI instead of the ID allows us to use the most recent one that was generated, useful in case of name clashes.<\/p>\n<pre><code class=\"language-terraform\">variable \"ami_name\" {\n  type = string\n}\n\nvariable \"fleet_size\" {\n  type = number\n}\n\ndata \"aws_ami\" \"bare_metal_gha_runner\" {\n  most_recent = true\n\n  filter {\n    name   = \"name\"\n    values = [\"${var.ami_name}\"]\n  }\n  \n  ...\n}\n\nresource \"aws_instance\" \"bare_metal\" {\n  count         = var.fleet_size\n  ami           = data.aws_ami.bare_metal_gha_runner.id\n  instance_type = \"mac2.metal\"\n  tenancy       = \"host\"\n  key_name      = aws_key_pair.bare_metal.key_name\n  ...\n}<\/code><\/pre>\n<p>Instead of maintaining multiple CI instances with varying software configurations, we concluded that it\u2019s simpler and more efficient to have a single, standardised setup. While teams still have the option to create and deploy their unique setups, a smaller, unified system allows for easier support by a single global configuration.<\/p>\n<h3 id=\"auto-and-manual-scaling\"><strong>Auto and Manual Scaling<\/strong><\/h3>\n<p>The\u00a0<code>deploy_infra<\/code>\u00a0workflow allows us to scale on demand but it doesn\u2019t release the underlying dedicated hosts which are the resources that are ultimately billed.<\/p>\n<p>The autoscaling solution provided by AWS is great for VMs but gets sensibly more complex when actioned on dedicated hosts.\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/autoscaling\/ec2\/userguide\/auto-scaling-groups.html?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Auto Scaling groups<\/a>\u00a0on macOS instances would require a\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/license-manager\/latest\/userguide\/create-license-configuration.html?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Custom Managed License<\/a>, a\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/license-manager\/latest\/userguide\/host-resource-groups.html?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Host Resource Group<\/a>\u00a0and, of course, a\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/autoscaling\/ec2\/userguide\/launch-templates.html?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Launch Template<\/a>. Using only AWS services appears to be a lot of work to pull things together and the result wouldn\u2019t allow for automatic release of the dedicated hosts.<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_RF0qy26XVnZldt3SpFkc8w.webp?resize=1280%2C147&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"1280\" height=\"147\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_RF0qy26XVnZldt3SpFkc8w.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1000\/2024\/01\/1_RF0qy26XVnZldt3SpFkc8w.webp 1000w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1600\/2024\/01\/1_RF0qy26XVnZldt3SpFkc8w.webp 1600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_RF0qy26XVnZldt3SpFkc8w.webp 1950w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>AirBnb mention in their\u00a0<a href=\"https:\/\/medium.com\/airbnb-engineering\/flexible-continuous-integration-for-ios-4ab33ea4072f?ref=albertodebortoli.com\" rel=\"noopener\">Flexible Continuous Integration for iOS<\/a>\u00a0article that an internal scaling service was implemented:<\/p>\n<blockquote><p>An internal scaling service manages the desired capacity of each environment\u2019s Auto Scaling group.<\/p><\/blockquote>\n<p>Some articles explain how to set up Auto Scaling groups for mac instances (see\u00a0<a href=\"https:\/\/aws.amazon.com\/blogs\/compute\/implementing-autoscaling-for-ec2-mac-instances\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">1<\/a>\u00a0and\u00a0<a href=\"https:\/\/devdosvid.blog\/2021\/10\/24\/auto-scaling-group-for-your-macos-ec2-instances-fleet\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">2<\/a>) but after careful consideration, we agreed that implementing a simple scaling service via GitHub Actions (GHA) was the easiest and most maintainable solution.<\/p>\n<p>We implemented 2 GHA workflows to fully automate the weekend autoscaling:<\/p>\n<ul>\n<li>Upscaling workflow to\u00a0<code>n<\/code>, triggered at a specific time at the beginning of the working week<\/li>\n<li>Downscaling workflow to\u00a0<code>1<\/code>, triggered at a specific time at the beginning of the weekend<\/li>\n<\/ul>\n<p>We retain the capability for manual scaling, which is essential for situations where we need to scale down, such as on bank holidays, or scale up, like on release cut days, when activity typically exceeds the usual levels.<\/p>\n<p>Additionally, we have implemented a workflow that runs multiple times a day and tries to release all available hosts without an instance attached. This step lifts us from the burden of having to remember to release the hosts. Dedicated hosts take up to 110 minutes to move from the Pending to the Available state due to the\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ec2-mac-instances.html?ref=albertodebortoli.com#mac-instance-stop\" rel=\"noopener ugc nofollow\">scrubbing workflow<\/a>\u00a0performed by AWS.<\/p>\n<p>Manual scaling can be executed between the times the autoscaling workflows are triggered and they must be resilient to unexpected statuses of the infrastructure, which thankfully Terraform takes care of.<\/p>\n<p>Both down and upscaling are covered in the following flowchart:<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_uT28oUGw_PZWKCs7_IUlag.webp?resize=1280%2C1232&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"1280\" height=\"1232\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_uT28oUGw_PZWKCs7_IUlag.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1000\/2024\/01\/1_uT28oUGw_PZWKCs7_IUlag.webp 1000w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_uT28oUGw_PZWKCs7_IUlag.webp 1400w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>The autoscaling values are defined as configuration variables in the repo settings:<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_In73VTcwiWA7iMoaTEClKA.webp?resize=1280%2C501&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"1280\" height=\"501\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_In73VTcwiWA7iMoaTEClKA.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1000\/2024\/01\/1_In73VTcwiWA7iMoaTEClKA.webp 1000w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1600\/2024\/01\/1_In73VTcwiWA7iMoaTEClKA.webp 1600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_In73VTcwiWA7iMoaTEClKA.webp 1722w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>It usually takes ~8 minutes for an EC2 mac2.metal instance to become reachable after creation, meaning that we can redeploy the entire infrastructure very quickly.<\/p>\n<h3 id=\"automated-connection-to-github-actions\"><strong>Automated Connection to GitHub Actions<\/strong><\/h3>\n<p>We provide some user data when deploying the instances.<\/p>\n<pre><code class=\"language-terraform\">resource \"aws_instance\" \"bare_metal\" {\n  ami       = data.aws_ami.bare_metal_gha_runner.id\n  count     = var.fleet_size\n  ...\n  user_data = &lt;&lt;EOF\n{\n    \"github_enterprise\": \"&lt;GHE_ENTERPRISE_NAME&gt;\",\n    \"github_pat_secret_manager_arn\": ${data.aws_secretsmanager_secret_version.ghe_pat.arn},\n    \"github_url\": \"&lt;GHE_ENTERPRISE_URL&gt;\",\n    \"runner_group\": \"CI-MobileTeams\",\n    \"runner_name\": \"bare-metal-runner-${count.index + 1}\"\n}\n  EOF<\/code><\/pre>\n<p>The user data is stored in a specific folder by\u00a0<a href=\"https:\/\/github.com\/aws\/ec2-macos-init?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">macos-init<\/a>\u00a0and we implement a module to copy the content to\u00a0<code>~\/actions-runner-config.json<\/code>.<\/p>\n<pre><code>### Group 10 ###\n[[Module]]\n    Name = \"Create actions-runner-config.json from userdata\"\n    PriorityGroup = 10\n    RunPerInstance = true\n    FatalOnError = false\n    [Module.Command]\n        Cmd = [\"\/bin\/zsh\", \"-c\", 'instanceId=\"$(curl http:\/\/169.254.169.254\/latest\/meta-data\/instance-id)\"; if [[ ! -z $instanceId ]]; then cp \/usr\/local\/aws\/ec2-macos-init\/instances\/$instanceId\/userdata ~\/actions-runner-config.json; fi']\n        RunAsUser = \"ec2-user\"<\/code><\/pre>\n<p>which is in turn used by the\u00a0<code>configure_runner.sh<\/code>\u00a0script to configure the GitHub Actions runner.<\/p>\n<pre><code class=\"language-bash\">#!\/bin\/bash\n\nGITHUB_ENTERPRISE=$(cat $HOME\/actions-runner-config.json | jq -r .github_enterprise)\nGITHUB_PAT_SECRET_MANAGER_ARN=$(cat $HOME\/actions-runner-config.json | jq -r .github_pat_secret_manager_arn)\nGITHUB_PAT=$(aws secretsmanager get-secret-value --secret-id $GITHUB_PAT_SECRET_MANAGER_ARN | jq -r .SecretString)\nGITHUB_URL=$(cat $HOME\/actions-runner-config.json | jq -r .github_url)\nRUNNER_GROUP=$(cat $HOME\/actions-runner-config.json | jq -r .runner_group)\nRUNNER_NAME=$(cat $HOME\/actions-runner-config.json | jq -r .runner_name)\n\nRUNNER_JOIN_TOKEN=` curl -L \n  -X POST \n  -H \"Accept: application\/vnd.github+json\" \n  -H \"Authorization: Bearer $GITHUB_PAT\"\n  $GITHUB_URL\/api\/v3\/enterprises\/$GITHUB_ENTERPRISE\/actions\/runners\/registration-token | jq -r '.token'`\n\nMACOS_VERSION=`sw_vers -productVersion`\n\nXCODE_VERSIONS=`find \/Applications -type d -name \"Xcode-*\" -maxdepth 1 \n  -exec basename {} ; \n  | tr 'n' ',' \n  | sed 's\/,$\/n\/' \n  | sed 's\/.app\/\/g'`\n\n$HOME\/actions-runner\/config.sh \n  --unattended \n  --url $GITHUB_URL\/enterprises\/$GITHUB_ENTERPRISE \n  --token $RUNNER_JOIN_TOKEN \n  --runnergroup $RUNNER_GROUP \n  --labels ec2,bare-metal,$RUNNER_NAME,macOS-$MACOS_VERSION,$XCODE_VERSIONS \n  --name $RUNNER_NAME \n  --replace<\/code><\/pre>\n<p>The above script is run by a macos-init module.<\/p>\n<pre><code>### Group 11 ###\n[[Module]]\n    Name = \"Configure the GHA runner\"\n    PriorityGroup = 11\n    RunPerInstance = true\n    FatalOnError = false\n    [Module.Command]\n        Cmd = [\"\/bin\/zsh\", \"-c\", \"\/Users\/ec2-user\/configure_runner.sh\"]\n        RunAsUser = \"ec2-user\"<\/code><\/pre>\n<p>The\u00a0<a href=\"https:\/\/docs.github.com\/en\/actions\/hosting-your-own-runners\/managing-self-hosted-runners\/configuring-the-self-hosted-runner-application-as-a-service?platform=mac&amp;ref=albertodebortoli.com#customizing-the-self-hosted-runner-service-1\" rel=\"noopener ugc nofollow\">GitHub documentation<\/a>\u00a0states that it\u2019s possible to create a customized service starting from a provided template. It took some research and various attempts to find the right configuration that allows the connection without having to log in in the UI (over VNC) which would represent a blocker for a complete automation of the deployment. We believe that the single person who managed to get this right is\u00a0<a href=\"https:\/\/github.com\/sebsto?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">S\u00e9bastien Stormacq<\/a>\u00a0who provided the\u00a0<a href=\"https:\/\/github.com\/actions\/runner\/issues\/1056?ref=albertodebortoli.com#issuecomment-1237426462\" rel=\"noopener ugc nofollow\">correct solution<\/a>.<\/p>\n<p>The connection to GHA can be completed with 2 more modules that install the runner as a service and load the custom daemon.<\/p>\n<pre><code>### Group 12 ###\n[[Module]]\n    Name = \"Run the self-hosted runner application as a service\"\n    PriorityGroup = 12\n    RunPerInstance = true\n    FatalOnError = false\n    [Module.Command]\n        Cmd = [\"\/bin\/zsh\", \"-c\", \"cd \/Users\/ec2-user\/actions-runner &amp;&amp; .\/svc.sh install\"]\n        RunAsUser = \"ec2-user\"\n\n### Group 13 ###\n[[Module]]\n    Name = \"Launch actions runner daemon\"\n    PriorityGroup = 13\n    RunPerInstance = true\n    FatalOnError = false\n    [Module.Command]\n        Cmd = [\"sudo\", \"\/bin\/launchctl\", \"load\", \"\/Library\/LaunchDaemons\/com.justeattakeaway.actions-runner-service.plist\"]\n        RunAsUser = \"ec2-user\"<\/code><\/pre>\n<p>Using a daemon instead of an agent (see\u00a0<a href=\"https:\/\/developer.apple.com\/library\/archive\/documentation\/MacOSX\/Conceptual\/BPSystemStartup\/Chapters\/CreatingLaunchdJobs.html?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Creating Launch Daemons and Agents<\/a>), doesn\u2019t require us to set up any auto-login which on macOS is a bit of a tricky procedure and is best avoided also for security reasons. The following is the content of the daemon for completeness.<\/p>\n<pre><code class=\"language-xml\">&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;\n&lt;!DOCTYPE plist PUBLIC \"-\/\/Apple\/\/DTD PLIST 1.0\/\/EN\" \"http:\/\/www.apple.com\/DTDs\/PropertyList-1.0.dtd\"&gt;\n&lt;plist version=\"1.0\"&gt;\n  &lt;dict&gt;\n    &lt;key&gt;Label&lt;\/key&gt;\n    &lt;string&gt;com.justeattakeaway.actions-runner-service&lt;\/string&gt;\n    &lt;key&gt;ProgramArguments&lt;\/key&gt;\n    &lt;array&gt;\n      &lt;string&gt;\/Users\/ec2-user\/actions-runner\/runsvc.sh&lt;\/string&gt;\n    &lt;\/array&gt;\n    &lt;key&gt;UserName&lt;\/key&gt;\n    &lt;string&gt;ec2-user&lt;\/string&gt;\n    &lt;key&gt;GroupName&lt;\/key&gt;\n    &lt;string&gt;staff&lt;\/string&gt;\n    &lt;key&gt;WorkingDirectory&lt;\/key&gt;\n    &lt;string&gt;\/Users\/ec2-user\/actions-runner&lt;\/string&gt;\n    &lt;key&gt;RunAtLoad&lt;\/key&gt;\n    &lt;true\/&gt;    \n    &lt;key&gt;StandardOutPath&lt;\/key&gt;\n    &lt;string&gt;\/Users\/ec2-user\/Library\/Logs\/com.justeattakeaway.actions-runner-service\/stdout.log&lt;\/string&gt;\n    &lt;key&gt;StandardErrorPath&lt;\/key&gt;\n    &lt;string&gt;\/Users\/ec2-user\/Library\/Logs\/com.justeattakeaway.actions-runner-service\/stderr.log&lt;\/string&gt;\n    &lt;key&gt;EnvironmentVariables&lt;\/key&gt;\n    &lt;dict&gt; \n      &lt;key&gt;ACTIONS_RUNNER_SVC&lt;\/key&gt;\n      &lt;string&gt;1&lt;\/string&gt;\n    &lt;\/dict&gt;\n    &lt;key&gt;ProcessType&lt;\/key&gt;\n    &lt;string&gt;Interactive&lt;\/string&gt;\n    &lt;key&gt;SessionCreate&lt;\/key&gt;\n    &lt;true\/&gt;\n  &lt;\/dict&gt;\n&lt;\/plist&gt;<\/code><\/pre>\n<p>Not long after the deployment, all the steps above are executed and we can appreciate the runners appearing as connected.<\/p>\n<figure class=\"kg-card kg-image-card\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SRzNVowEas-7_g5J1nxzTw.webp?resize=1280%2C543&#038;ssl=1\" class=\"kg-image\" alt=\"Scalable Continuous Integration for iOS\" loading=\"lazy\" width=\"1280\" height=\"543\" srcset=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w600\/2024\/01\/1_SRzNVowEas-7_g5J1nxzTw.webp 600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1000\/2024\/01\/1_SRzNVowEas-7_g5J1nxzTw.webp 1000w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/size\/w1600\/2024\/01\/1_SRzNVowEas-7_g5J1nxzTw.webp 1600w, https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SRzNVowEas-7_g5J1nxzTw.webp 1712w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<h3 id=\"multi-team-use\">Multi-Team Use<\/h3>\n<p>We start the downscaling at 11:59 PM on Fridays and start the upscaling at 6:00 AM on Mondays. These times have been chosen in a way that guarantees a level of service to teams in the UK, the Netherlands (GMT+1) and Canada (Winnipeg is on GMT-6) accounting for BST (British Summer Time) and DST (Daylight Saving Time) too. Times are defined in UTC in the GHA workflow triggers and the local time of the runner is not taken into account.<\/p>\n<p>Since the instances are used to build multiple projects and tools owned by different teams, one problem we faced was that instances could get compromised if workflows included unsafe steps (e.g. modifications to global configurations).<\/p>\n<p>GitHub Actions has a documentation page about\u00a0<a href=\"https:\/\/docs.github.com\/en\/actions\/security-guides\/security-hardening-for-github-actions?ref=albertodebortoli.com#hardening-for-self-hosted-runners\" rel=\"noopener ugc nofollow\">Hardening self-hosted runners<\/a>\u00a0specifically stating:<\/p>\n<blockquote><p>Self-hosted runners for GitHub do not have guarantees around running in ephemeral clean virtual machines, and can be persistently compromised by untrusted code in a workflow.<\/p><\/blockquote>\n<p>We try to combat such potential problems by educating people on how to craft workflows and rely on the quick redeployment of the stack should the instances break.<\/p>\n<p>We also run\u00a0<a href=\"https:\/\/docs.github.com\/en\/actions\/hosting-your-own-runners\/managing-self-hosted-runners\/running-scripts-before-or-after-a-job?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">scripts before and after each job<\/a>\u00a0to ensure that instances can be reused as much as possible. This includes actions like deleting the simulators\u2019 content, derived data, caches and archives.<\/p>\n<h3 id=\"centralized-management-via-github-actions\">Centralized Management via GitHub Actions<\/h3>\n<p>The macOS runners stack is defined in a dedicated\u00a0<code>macOS-runners<\/code>\u00a0repository. We implemented GHA workflows to cover the use cases that allow teams to self-serve:<\/p>\n<ul>\n<li>create macOS AMI<\/li>\n<li>deploy CI<\/li>\n<li>downscale for the weekend*<\/li>\n<li>upscale for the working week*<\/li>\n<li>release unused hosts*<\/li>\n<\/ul>\n<p><em>* run without inputs and on a scheduled trigger<\/em><\/p>\n<p>The runners running the jobs in this repo are small t2.micro Linux instances and come with the\u00a0<a href=\"https:\/\/aws.amazon.com\/cli\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">AWSCLI<\/a>\u00a0installed. An IAM instance role with the correct policies is used to make sure that\u00a0<code>aws ec2<\/code>\u00a0commands\u00a0<code>allocate-hosts<\/code>,\u00a0<code>describe-hosts<\/code>\u00a0and\u00a0<code>release-hosts<\/code>\u00a0could execute and we used\u00a0<code>jq<\/code>\u00a0to parse the API responses.<\/p>\n<h3 id=\"a-note-on-vm-runners\">A note on VM runners<\/h3>\n<p>In this article, we discussed how we\u2019ve used bare metal instances as runners. We have spent a considerable amount of time investigating how we could leverage the\u00a0<a href=\"https:\/\/developer.apple.com\/documentation\/virtualization?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Virtualization framework<\/a>\u00a0provided by Apple to create virtual machines via\u00a0<a href=\"https:\/\/tart.run\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Tart<\/a>.<\/p>\n<p>If you\u2019ve grasped the complexity of crafting a CI system of runners on bare metal instances, you can understand that introducing VMs makes the setup sensibly more convoluted which would be best discussed in a separate article.<\/p>\n<p>While a setup with Tart VMs has been implemented, we realised that it\u2019s not performant enough to be put to use. Using VMs, the number of runners would double but we preferred to have native performance as the slowdown is over 40% compared to bare metal. Moreover, when it comes to running heavy UI test suites like ours, tests became too flaky.<\/p>\n<p>Testing the VMs, we also realised that the standard values of Throughput and IOPS on the EBS volume didn\u2019t seem to be enough and caused disk congestion resulting in an unacceptable slowdown in performance.<\/p>\n<p>Here is a quick summary of the setup and the challenges we have faced.<\/p>\n<ul>\n<li>Virtual runners require 2 images: one for the VMs (tart) and one for the host (AMI).<\/li>\n<li>We use Packer to create VM images (Vanilla, Base, IDE, Tools) with the software required based on the\u00a0<a href=\"https:\/\/github.com\/cirruslabs\/macos-image-templates?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">templates<\/a>\u00a0provided by Tart and we store the OCI-compliant images on ECR. We create these images on CI with dedicated workflows similar to the one described earlier for bare metal but, in this case, macOS runners (instead of Linux) are required as publishing to ECR is done with tart which runs on macOS. Extra policies are required on the instance role to allow the runner to push to ECR (using\u00a0<code>temporary_iam_instance_profile_policy_document<\/code>\u00a0in Packer\u2019s\u00a0<a href=\"https:\/\/developer.hashicorp.com\/packer\/integrations\/hashicorp\/amazon\/latest\/components\/builder\/ebs?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Amazon EBS<\/a>).<\/li>\n<li>Apple set a limit to the number of VMs that can be run on an instance to 2, which would allow to double the size of the fleet of runners. Creating AMIs hosting 2 VMs is done with Packer and steps include cloning the image from ECR and configuring macos-init modules to run daemons to run the VMs via Tart.<\/li>\n<li>Deploying a virtual CI infrastructure is identical to what has already been described for bare metal.<\/li>\n<li>Connecting to and interfacing with the VMs happens from within the host. Opening SSH and especially VNC sessions from within the bare metal instances can be very confusing.<\/li>\n<li>The version of macOS on the host and the one on the VMs could differ. The version used on the host must be provided with an AMI by AWS, while the version used on the VMs is provided by Apple in IPSW files (see\u00a0<a href=\"https:\/\/ipsw.me\/?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">ipsw.me<\/a>).<\/li>\n<li>The GHA runners run on the VMs meaning that the host won\u2019t require Xcode installed nor any other software used by the workflows.<\/li>\n<li>VMs don\u2019t allow for provisioning meaning we have to share configurations with the VMs via shared folders on the host with the\u00a0<code>\u2014 dir<\/code>\u00a0flag which causes extra setup complexity.<\/li>\n<li>VMs can\u2019t easily run the GHA runner as a service. The\u00a0<code>svc<\/code>\u00a0script requires the runner to be configured first, an operation that cannot be done during the provisioning of the host. We therefore need to implement an agent ourselves to configure and connect the runner in a single script.<\/li>\n<li>To have UI access (a-la VNC) to the VMs, it\u2019s first required to stop the VMs and then run them without the\u00a0<code>--no-graphics<\/code>\u00a0flag. At the time of writing, copy-pasting won\u2019t work even if using the\u00a0<code>--vnc<\/code>\u00a0or\u00a0<code>--vnc-experimental<\/code>\u00a0flags.<\/li>\n<li><a href=\"https:\/\/github.com\/shapehq\/tartelet?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Tartelet<\/a>\u00a0is a macOS app on top of Tart that allows to manage multiple GitHub Actions runners in ephemeral environments on a single host machine. We didn\u2019t consider it to avoid relying on too much third-party software and because it\u00a0<a href=\"https:\/\/github.com\/shapehq\/tartelet\/issues\/27?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">doesn\u2019t have yet<\/a>\u00a0GitHub Enterprise support.<\/li>\n<li>Worth noting that the Tart team worked on an orchestration solution named\u00a0<a href=\"https:\/\/github.com\/cirruslabs\/orchard?ref=albertodebortoli.com\" rel=\"noopener ugc nofollow\">Orchard<\/a>\u00a0that seems to be in its initial stage.<\/li>\n<\/ul>\n<h3 id=\"conclusion\">Conclusion<\/h3>\n<p>In 2023 we have revamped and globalised our approach to CI. We have migrated from Jenkins to GitHub Actions as the CI\/CD solution of choice for the whole group and have profoundly optimised and improved our pipelines introducing a greater level of job parallelisation.<\/p>\n<p>We have implemented a brand new scalable setup for bare metal macOS runners leveraging the HashiCorp tools Packer and Terraform. We have also implemented a setup based on Tart virtual machines.<\/p>\n<p>We have increased the size of our iOS team over the past few years, now including more than 40 developers, and still managed to be successful with only 5 bare metal instances on average, which is a clear statement of how performant and optimised our setup is.<\/p>\n<p>We have extended the capabilities of our Internal Developer Platform with a globalised approach to provide macOS runners; we feel that this setup will stand the test of time and serve well various teams across JET for years to come.<\/p>\n<\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1890\" class=\"pvc_stats total_only  \" data-element-id=\"1890\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewBox=\"0 0 502 315\" preserveAspectRatio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/i0.wp.com\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif?resize=16%2C16&#038;ssl=1\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Originally published on the\u00a0Just Eat\u00a0Takeaway\u00a0Engineering Blog. How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners. Problem At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscape<\/p>\n<div class=\"hosteria-entry-more\"><a href=\"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/\" class=\"no-underline font-light  group-hover:text-primary-800 dark:group-hover:text-primary-300 py-1\">Read more &gt;&gt;&gt;<\/a><\/div>\n<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_1890\" class=\"pvc_stats total_only\" data-element-id=\"1890\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewbox=\"0 0 502 315\" preserveaspectratio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/dev95.site\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=\"0\" \/><\/p>\n<div class=\"pvc_clear\"><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"fp_fajr_begins":"","fp_fajr_iqamah":"","fp_dhuhr_begins":"","fp_dhuhr_iqamah":"","fp_asr_begins":"","fp_asr_iqamah":"","fp_maghrib_begins":"","fp_maghrib_iqamah":"","fp_isha_begins":"","fp_isha_iqamah":"","fp_midnight":"","fp_midnight_name":"","fp_sunrise":"","fp_single_prayer_begins_title":"","fp_single_prayer_iqamah_title":"","fp_prayer_times_for_today":"","fp_hijra_date":"","fp_fajr_name":"","fp_dhuhr_name":"","fp_asr_name":"","fp_maghrib_name":"","fp_isha_name":"","fp_sunrise_name":"","fp_currentDate":"","fp_current_time":"","fp_current_title":"","fp_current_location":"","fp_masjid_name":"","fp_prayer_title":"","fp_next_prayer_iqamah_time":"","fp_next_prayer_iqamah_title":"","fp_next_prayer_begins_time":"","fp_next_prayer_begins_title":"","fp_next_prayer_title":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[37],"tags":[],"class_list":["post-1890","post","type-post","status-publish","format-standard","hentry","category-posts"],"a3_pvc":{"activated":true,"total_views":0,"today_views":0},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Scalable Continuous Integration for iOS - Dev95<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/\" \/>\n<meta property=\"og:locale\" content=\"ar_AR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Scalable Continuous Integration for iOS - Dev95\" \/>\n<meta property=\"og:description\" content=\"Originally published on the\u00a0Just Eat\u00a0Takeaway\u00a0Engineering Blog. How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners. Problem At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscapeRead more &gt;&gt;&gt;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/\" \/>\n<meta property=\"og:site_name\" content=\"Dev95\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-03T22:26:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp\" \/>\n<meta name=\"author\" content=\"dev95\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629\" \/>\n\t<meta name=\"twitter:data1\" content=\"dev95\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631\" \/>\n\t<meta name=\"twitter:data2\" content=\"18 \u062f\u0642\u064a\u0642\u0629\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/\"},\"author\":{\"name\":\"dev95\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\"},\"headline\":\"Scalable Continuous Integration for iOS\",\"datePublished\":\"2024-01-03T22:26:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/\"},\"wordCount\":2845,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/storage.ghost.io\\\/c\\\/ae\\\/f4\\\/aef4d625-32a2-417b-86f4-22c70a9b47a1\\\/content\\\/images\\\/2024\\\/01\\\/1_SaGE67XSSeflkTKknVOHbg.webp\",\"articleSection\":[\"Posts\"],\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/\",\"url\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/\",\"name\":\"Scalable Continuous Integration for iOS - Dev95\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/storage.ghost.io\\\/c\\\/ae\\\/f4\\\/aef4d625-32a2-417b-86f4-22c70a9b47a1\\\/content\\\/images\\\/2024\\\/01\\\/1_SaGE67XSSeflkTKknVOHbg.webp\",\"datePublished\":\"2024-01-03T22:26:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#breadcrumb\"},\"inLanguage\":\"ar\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#primaryimage\",\"url\":\"https:\\\/\\\/storage.ghost.io\\\/c\\\/ae\\\/f4\\\/aef4d625-32a2-417b-86f4-22c70a9b47a1\\\/content\\\/images\\\/2024\\\/01\\\/1_SaGE67XSSeflkTKknVOHbg.webp\",\"contentUrl\":\"https:\\\/\\\/storage.ghost.io\\\/c\\\/ae\\\/f4\\\/aef4d625-32a2-417b-86f4-22c70a9b47a1\\\/content\\\/images\\\/2024\\\/01\\\/1_SaGE67XSSeflkTKknVOHbg.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dev95.site\\\/scalable-continuous-integration-for-ios\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dev95.site\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Scalable Continuous Integration for iOS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#website\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"name\":\"Dev95\",\"description\":\"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.\",\"publisher\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dev95.site\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ar\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#organization\",\"name\":\"Dev95\",\"url\":\"https:\\\/\\\/dev95.site\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/dev95.site\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/rbrrbr-6.png?fit=512%2C512&ssl=1\",\"width\":512,\"height\":512,\"caption\":\"Dev95\"},\"image\":{\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dev95.site\\\/#\\\/schema\\\/person\\\/b807805ffe2916206b04d0938bce0298\",\"name\":\"dev95\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ar\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g\",\"caption\":\"dev95\"},\"url\":\"https:\\\/\\\/dev95.site\\\/ar\\\/author\\\/mohammad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Scalable Continuous Integration for iOS - Dev95","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/","og_locale":"ar_AR","og_type":"article","og_title":"Scalable Continuous Integration for iOS - Dev95","og_description":"Originally published on the\u00a0Just Eat\u00a0Takeaway\u00a0Engineering Blog. How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners. Problem At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscapeRead more &gt;&gt;&gt;","og_url":"https:\/\/dev95.site\/ar\/scalable-continuous-integration-for-ios\/","og_site_name":"Dev95","article_published_time":"2024-01-03T22:26:50+00:00","og_image":[{"url":"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp","type":"","width":"","height":""}],"author":"dev95","twitter_card":"summary_large_image","twitter_misc":{"\u0643\u064f\u062a\u0628 \u0628\u0648\u0627\u0633\u0637\u0629":"dev95","\u0648\u0642\u062a \u0627\u0644\u0642\u0631\u0627\u0621\u0629 \u0627\u0644\u0645\u064f\u0642\u062f\u0651\u0631":"18 \u062f\u0642\u064a\u0642\u0629"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#article","isPartOf":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/"},"author":{"name":"dev95","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298"},"headline":"Scalable Continuous Integration for iOS","datePublished":"2024-01-03T22:26:50+00:00","mainEntityOfPage":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/"},"wordCount":2845,"commentCount":0,"publisher":{"@id":"https:\/\/dev95.site\/#organization"},"image":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#primaryimage"},"thumbnailUrl":"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp","articleSection":["Posts"],"inLanguage":"ar","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/","url":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/","name":"Scalable Continuous Integration for iOS - Dev95","isPartOf":{"@id":"https:\/\/dev95.site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#primaryimage"},"image":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#primaryimage"},"thumbnailUrl":"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp","datePublished":"2024-01-03T22:26:50+00:00","breadcrumb":{"@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#breadcrumb"},"inLanguage":"ar","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/"]}]},{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#primaryimage","url":"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp","contentUrl":"https:\/\/storage.ghost.io\/c\/ae\/f4\/aef4d625-32a2-417b-86f4-22c70a9b47a1\/content\/images\/2024\/01\/1_SaGE67XSSeflkTKknVOHbg.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/dev95.site\/scalable-continuous-integration-for-ios\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dev95.site\/"},{"@type":"ListItem","position":2,"name":"Scalable Continuous Integration for iOS"}]},{"@type":"WebSite","@id":"https:\/\/dev95.site\/#website","url":"https:\/\/dev95.site\/","name":"Dev95","description":"A comprehensive platform for data and knowledge, delivering reliable content that meets the aspirations of readers and enthusiasts.","publisher":{"@id":"https:\/\/dev95.site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dev95.site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ar"},{"@type":"Organization","@id":"https:\/\/dev95.site\/#organization","name":"Dev95","url":"https:\/\/dev95.site\/","logo":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/dev95.site\/wp-content\/uploads\/2026\/07\/rbrrbr-6.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Dev95"},"image":{"@id":"https:\/\/dev95.site\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/dev95.site\/#\/schema\/person\/b807805ffe2916206b04d0938bce0298","name":"dev95","image":{"@type":"ImageObject","inLanguage":"ar","@id":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a70a73d950838b20cd80d7ebdc955737e802e8cd896044c5473b32b946c0662a?s=96&d=mm&r=g","caption":"dev95"},"url":"https:\/\/dev95.site\/ar\/author\/mohammad\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1890","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/comments?post=1890"}],"version-history":[{"count":0,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/posts\/1890\/revisions"}],"wp:attachment":[{"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/media?parent=1890"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/categories?post=1890"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev95.site\/ar\/wp-json\/wp\/v2\/tags?post=1890"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}