Building Automated AWS Permission Testing Infrastructure for CI/CD

This post was originally published on graycloudarch.com. I deployed a permission set for our data engineers five times before it worked correctly. The first deployment: S3 reads worked, Glue Data Catalog reads worked. Athena queries failed --- the query engine needs KMS decrypt through a service principal, and I'd missed the kms:ViaService condition. Second deployment: Athena worked. EMR Serverless job submission failed --- missing iam:PassRole. Third deployment: EMR submission worked. Job execution failed --- missing permissions on the EMR Serverless execution role boundary. I kept deploying, engineers kept getting blocked, I kept opening tickets. Five iterations. Two weeks. Every failure meant a data engineer opened a ticket instead of running their job. The problem wasn't that IAM is complicated --- it is, but that's expected. The problem was that I had no way to catch these issues before deploying to the account where real engineers were trying to do real work. Every bug was a prod