# Break Glass in Case of Emergency: Admin Access Without Giving Everyone Admin

*Least privilege is great, until the few people with admin access can't be reached*

We follow least privilege. Everyone gets the minimum access they need to do their job, nothing more. That means only a very small number of people have admin access to our AWS accounts.

That sounds good, and it is. But it raises a simple question: what happens if production is down and none of the admins can be reached?

Maybe they're asleep, on a plane, sick, or just not answering. The people who are around want to help, but they don't have the access to fix the problem. And you can't wait until the morning when production is down.

So we needed a backup plan. A way for someone who is not an admin to get admin access in a real emergency, but in a way that nobody can ever use quietly or casually. This is usually called **break-glass access**.

## Why not just use the root account?

The easy answer is "keep the root credentials somewhere safe and use them in an emergency". I didn't want that. Root can do everything, it's not tied to a person, and the whole point of least privilege is that nobody walks around with full power.

Giving a few more people permanent admin access wasn't the answer either. That's exactly what least privilege is trying to avoid.

## The idea

Think about the fire alarm box on the wall with "break glass in case of emergency" on it. Anyone in the building can use it, but you don't use it unless there's a real fire, and when you do, everyone knows about it.

That's what I wanted:

- A small group of people **can** get admin access if they need to
- Getting it takes a deliberate extra step, it's not part of their normal login
- The access is temporary
- The moment someone uses it, the on-call engineer gets an alert

## How it works

We use IAM Identity Center for logins, with MFA enforced at sign in. Here's the setup:

1. **A dedicated group.** Only people in this group can break the glass.
2. **A low-privilege permission set.** When they sign in, they only get read-only access (monitoring and billing). The only extra thing it allows is assuming one role, `BreakGlassAdmin`.
3. **A manual switch.** To get admin, they have to use "Switch Role" in the console and pick `BreakGlassAdmin`. Admin never shows up as an option in their normal login.
4. **Short sessions.** The admin session lasts one hour, then they have to switch again.
5. **An alert on every use.** Every time someone assumes `BreakGlassAdmin`, the on-call engineer gets notified right away.

The permission set policy is tiny. It basically says "you can assume this one role, nothing else":

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::<account-id>:role/BreakGlassAdmin"
  }]
}
```

And on the other side, lock the role's trust policy to that permission set, so nothing else can assume it:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::<account-id>:root" },
    "Action": "sts:AssumeRole",
    "Condition": {
      "ArnLike": {
        "aws:PrincipalArn": "arn:aws:iam::<account-id>:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_<PermissionSetName>_*"
      }
    }
  }]
}
```

The wildcard at the end matters. Identity Center creates the permission set role with a random suffix, and that suffix can change, so you match on the prefix.

About the one hour: switching from your Identity Center session into another role is what AWS calls role chaining, and AWS limits role chaining sessions to one hour no matter what you set on the role. For break-glass that's actually a good thing. Nobody should be sitting in an admin session all day.

## Why the extra step?

The simpler option would be to give the group an "Admin" permission set directly in the SSO portal. They sign in, click Admin, done. We didn't do that on purpose, for three reasons:

- **Admin stays out of the normal login.** Day to day, these people don't even see admin as an option.
- **Using it is one clear event.** Switching into one specific role is easy to detect and alert on.
- **It's not easy to use by accident.** You have to know what you're doing and decide to do it. That small bit of friction is the point.

## The alert

This is the part that makes the whole thing safe. Break-glass access without an alert is just admin access with extra steps.

The flow is simple:

**CloudTrail → CloudWatch Logs → metric filter → alarm → on-call**

CloudTrail records every `AssumeRole` call. We send CloudTrail to CloudWatch Logs, and a metric filter turns every assume of `BreakGlassAdmin` into a custom metric. An alarm on that metric notifies the on-call engineer.

Here's a generic version in Terraform:

```hcl
resource "aws_cloudwatch_log_metric_filter" "break_glass" {
  name           = "BreakGlassAdminAssumed"
  log_group_name = aws_cloudwatch_log_group.cloudtrail.name
  pattern        = "{ ($.eventSource = \"sts.amazonaws.com\") && ($.eventName = \"AssumeRole\") && ($.requestParameters.roleArn = \"*:role/BreakGlassAdmin\") }"

  metric_transformation {
    name      = "BreakGlassAdminAssumeCount"
    namespace = "Security"
    value     = "1"
  }
}

resource "aws_cloudwatch_metric_alarm" "break_glass" {
  alarm_name          = "BreakGlassAdminAssumed"
  namespace           = "Security"
  metric_name         = "BreakGlassAdminAssumeCount"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 1
  threshold           = 1
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.on_call.arn]
}
```

A couple of small things worth knowing here:

- `treat_missing_data = "notBreaching"` matters. Most of the time nothing matches, so there's no data at all, and you don't want the alarm complaining about that.
- Match the full role name at the end of the ARN (`*:role/BreakGlassAdmin`) instead of `*BreakGlassAdmin*`, so a role with a similar name doesn't trigger it.

Every assume also stays in CloudTrail and shows up on a CloudWatch dashboard, so after an emergency you can see exactly who used it and when.

## The rules

The tech is only half of it. The other half is the rules, and we wrote them down:

- **Only in a real emergency.** A severe outage, infrastructure that nobody can reach because of a misconfiguration, or a security incident that needs full visibility.
- **Only if nobody else can fix it.** Before breaking the glass, you confirm that no one on the team can solve it with their normal access.
- **Get out when you're done.** Leave the admin session as soon as the problem is fixed.
- **Remove temporary members.** If someone was added to the group just for this emergency, they get removed right after.

## Did it work?

We tested it once after setting it up. Someone switched into the role, and the alert fired. Since then nobody has needed it.

Honestly, that's exactly what you want from break-glass. It's like a fire extinguisher, you hope you never use it, but you check it works. If you set something like this up, test it again from time to time. Access setups drift, people leave, alert channels change, and you don't want to find out it's broken in the middle of an outage.

Least privilege and "someone can always fix production" sound like they're fighting each other. They don't have to. Keep admin access small, give a few people a locked door they can open in an emergency, and make sure everyone hears it when the door opens.\

That's all for now. If you're interested in more advanced content about WSGI, Apache, or other DevOps topics, there are many articles in the DevOps series that you might find interesting.

Mostafa-DE Fayyad

Software Engineer
