From Manual Mayhem to Button Clicks: Automating Infrastructure at Scale

The Setup

When I jumped into a CDN infrastructure team role in the Summer of 2023, I thought I knew what I was getting into: Ansible, Apache Traffic Control (ATC), and some infrastructure automation. What I actually got was a deep dive into a technology stack that would make even seasoned DevOps engineers sweat:

• Ansible (with its delightful YAML and variable hierarchy mysteries)
• GitHub (for version control, discussions, and existential questions)
• ATC (Apache Traffic Control—shaping CDN traffic with the finesse of a sledgehammer at first)
• InfluxDB (storing metrics while I questioned my life choices)
• Slack Automation (because alerts need personality)
• ELK Stack (Elasticsearch, Logstash, Kibana—the debugging trifecta)
• Artifactory (for storing things we build)
• GitHub Actions (CI/CD that actually works, mostly)

Oh, and security compliance. Always security compliance.

The Pain Point: CentOS 7 to Rocky Linux 9

Imagine this: Your entire infrastructure is running on CentOS 7. Support ending December 2024. You need to upgrade to Rocky Linux 9 while maintaining security compliance across the entire infrastructure AND the code.

And you're not talking about five servers. You're talking about dozens of edge servers, caching infrastructure, and configuration management systems—all interdependent, all critical.

Before Automation: The upgrade was a manual, terrifying process - 4-6 hours per upgrade cycle, manual verification, sleepless nights

After Automation: 30 minutes, automated compliance checks, Slack notifies the team when it's done

Key Takeaways

✅ Automation wins don't happen overnight — we spent months learning, failing, and iterating
✅ Variable hierarchy is your friend — document it religiously
✅ Compliance isn't a roadblock, it's a feature — bake it into your playbooks
✅ The button is the goal — infrastructure that runs itself