Leading a team of nine engineers on Industry Software DevOps team through a transition to a self-service compute platform for >200 engineers. Responsible for infrastructure, operational duties, production readiness, tooling architecture, and maintaining homegrown platform tooling and services.

• Transforming culture and processes at the local and org level from a traditional centralized DevOps team to a platform engineering model; shifting our offerings incrementally to minimize risks and maximize impact. Maintains the quarterly roadmap to align with business priorities while also triaging tech debt. Meets with tenants to understand the highest impact work for tenant teams.
• Overhauling compute infrastructure to be more flexible and to follow a repeatable lifecycle by rebuilding our pet clusters into cattle, created tooling to manage k8s resources instead of handcrafting important config. Cluster can be managed almost as ephemeral resources and now we can test dangerous cluster changes in sterile environments outside of tenant workloads to avoid impact from infrastructure changes.
• Founder and maintainer of the team's bazel monorepo, migrating service and tool code to a single repository to reduce cognitive context load, increase code reuse, reduce abandoned code, and keep dependencies up to date uniformly.
• Drove adoption of in-house CICD tooling across 19 teams and hundreds of services and jobs to standardize deployments. Tooling improved deployment feedback and ergonomics, reduced deploy failures by 20% and increased developer confidence in their deploys and their overall velocity while adding functionality that was configurable directly by service owners.

As a team of nine, we managed over 70PB of user data, handling 100 million requests per day in an on-premises, multi-datacenter, eventually consistent distributed system developed in-house. As the infrastructure lead, I oversaw approximately 8,000 physical machines, managed compliance and chargeback reporting, hardware provisioning and automation, observability, tooling, deployments, anti-entropy services, and delivered critical service features.

• Developed and automated chargeback reporting for customer data, transforming a manual and annual process into a daily automated metric covering network, API usage, storage class, and data locality charges.
  • Customers could now make informed decisions and assess how service changes impact cost.
  • Helped the team prioritize features to help teams improve performance and cut costs like allowing customers to choose replica count per dataset or which datacenter to write to.
• Developed a state-aware locking service that would approve operations on the dataset allowing operators to work safely and independently without impacting service health or causing downtime.
  • This enabled no-touch deployments and hardware failure remediation, saving weeks of developer operational toil annually.
  • Oncall pages reduced from 90/wk to 4-5/wk
  • Deployment frequency increased from several times a quarter to weekly for stateful services and twice a week for stateless.
• Compliance automation rolled 8k hosts in less than 25 days with no manual intervention and negligible service impact. As a result Blobstore was consistently in the top 5 of patched fleets across the company.
• Implemented anti-entropy services to scan large metadata dumps, billions of keys processed via MapReduce fed a key-list to repair workers to patch production datasets every week.
• Migrated stateless services from baremetal to Aurora Mesos compute to ease operational burden, improve developer velocity and service reliability as well as cut $100k from annual COGS.

Worked remote to a Switzerland-based infrastructure team that developed Microsoft Social Engagement (a social media sentiment analysis) product's backend and large Elasticsearch clusters. Later moved to a backend product team developing the account management api for Dynamics CRM.

• Owned infrastructure and agent tooling for the product.
• Owned livesite operations, infrastructure automation, observability for the product.