← All writing

Updated

The Lock on the Back Door

How long-lived credentials spread through software systems and what a stronger design does differently

A database password can become difficult to change long before anyone knows it has been exposed. An application needs it, a maintenance script uses it, and a scheduled report depends on it. Someone has also copied it into a runbook so another person can get things working during an outage.

Each copy had a reason to exist. When the team finally tries to rotate the password, those reasons have to be reconstructed. The application is easy to find. The script belongs to someone who has left. The report only runs at the end of the month, so a successful deployment won't tell you whether it still works.

This is part of what makes credential sprawl difficult to manage. Finding exposed values matters, and keeping secrets out of source code and chat is important. The system also needs attention if its ordinary operation keeps requiring people to distribute credentials that remain useful for months or years.

An inventory helps you understand those dependencies. Reducing how many of them need a permanent secret makes the inventory easier to maintain and gives a copied value less opportunity to become a lasting source of access.

Why the copies keep accumulating

Long-lived passwords and API keys are straightforward to use. Give the application a value, configure the receiving service to accept it, and the connection can keep working until that value expires, changes, or loses its permissions. The team doesn't have to build a process for obtaining replacement credentials during normal operation.

That simplicity becomes attractive under pressure. A developer can copy a working configuration to get a task finished. During an outage, a colleague can send a key to someone who needs access. A deployment system may receive the application's password because the team has made passing it along part of deployment.

Over time, the same credential can end up in several places with different protections and owners. A secret store may control access to one copy, while another remains in a laptop file or an old backup. Changing who can read the secret store won't invalidate those other copies.

Service accounts create a related ownership problem. An account may be created for a job with broader permissions than it needs because the team hasn't worked out the policy yet. If the account and its access aren't reviewed when the job changes, they can survive the work that originally justified them. The account's lifetime and the lifetime of any credentials issued to it both need attention.

Creating access has an immediate benefit because it lets someone finish a task. Removing it takes investigation and can interrupt a process people still depend on. That imbalance explains why old access survives even in teams that understand the risk.

Rotation depends on knowing what will break

Changing a long-lived secret can make an exposed copy unusable for future authentication. To do it reliably, the team needs to know which applications and jobs use the value and how each one will receive its replacement.

A shared database password makes that coordination harder. Several applications might pick up a change through their normal deployment, while a scheduled job reads a configuration file that nobody has updated. The job can fail days later, after the team believes the rotation succeeded.

It's understandable that operations teams become cautious. They will be responsible for the outage, and they may not have enough information to assess it. A rotation policy can be repeatedly postponed when the work required to follow it hasn't been made safe and routine.

Managed secret stores, deployment automation, and clear ownership help. They provide a controlled place to retrieve credentials and a way to test how changes reach the applications. Some secret-management systems can also generate temporary credentials. Vault's dynamic secrets, for example, use leases to manage the lifetime of credentials created for a service.

The useful distinction is how the application obtains access. If it keeps retrieving the same long-lived password, the team still has to manage that password's dependencies. If it can receive fresh, limited-duration credentials as part of normal operation, there is less need to coordinate replacements across every place an old value was stored.

Give the running application an identity

A workload identity lets an application authenticate using an identity associated with the environment and process running it. That might be a cloud role assigned to a virtual machine or serverless function, a Kubernetes service account, or a certificate issued after checks on the workload.

AWS, for example, can provide temporary role credentials to applications running on its compute services. Kubernetes supports short-lived service-account tokens that the platform refreshes. The application and its libraries still need to use those credentials correctly, including picking up replacements.

Once the workload has authenticated, policy determines what it can do. The receiving service may accept the identity credential directly, or the application may exchange it for access to another service. An application that only reads from one database should receive the permissions needed for that work.

The limited lifetime and the limited permissions are separate choices. A token that expires soon can still grant far too much access. The policy has to restrict the actions and resources available, and the receiving service has to enforce it.

With that setup, a copied credential has a shorter period in which it can be used. The application obtains replacements through the platform, so expiration becomes something the system handles routinely. For that application, people no longer need to copy an updated password into each configuration.

There are limits worth understanding before relying on that protection. An attacker who still controls the workload may be able to obtain new credentials. Expiration also doesn't necessarily end a connection that has already authenticated. With Amazon RDS IAM database authentication, for example, the token expires after 15 minutes, but that expiration doesn't terminate an established database session.

Incident response therefore needs a way to contain the workload, stop further access, and handle existing sessions where necessary. A short token lifetime helps reduce exposure, though it cannot perform all of that work.

The identity has to be trustworthy

The application still needs a reliable way to establish who it is. If the credential issuer accepts a claim that another process can easily imitate, issuing short-lived tokens just gives that process a way to request fresh access.

Cloud platforms can use their knowledge of deployed resources and assigned roles. Other systems inspect attributes of the machine and workload. SPIRE's attestation process, for example, compares information about a workload and its environment with registration rules that determine which identity it can receive.

Those rules and the systems enforcing them become security dependencies. The team needs to control who can assign an identity, who can change its permissions, and which workloads can use it. A broadly shared identity can make several applications difficult to distinguish even if every credential is temporary.

Logging needs deliberate attention too. The team should be able to trace the workload, the access it received, and the relevant policy. The exact records available depend on the platform and how auditing is configured. Adopting workload identity doesn't automatically produce a complete explanation of every access decision.

Revocation also needs to be tested in the actual system. Disabling an identity can stop new credentials from being issued while previously issued credentials or established sessions require additional action. The incident plan should account for that behavior before the team needs it during a compromise.

Make the supported approach practical

Developers are more likely to follow a secure process when it fits the way they deploy and troubleshoot applications. If every routine access request needs several tickets and manual handoffs, the team has left a practical incentive to keep reusing whatever already works.

The normal deployment process should make it possible to assign an appropriate workload identity and define its access. The application should have a supported way to refresh credentials, and operators should know how to diagnose a failed refresh. The credential issuer becomes a dependency for obtaining new access, so its availability belongs in the reliability plan. Human approval can remain part of granting unusually sensitive permissions without making someone approve every routine renewal.

Training still matters because developers choose permissions and handle data. A workable platform reduces the number of occasions on which someone has to retrieve, copy, and protect a reusable secret personally.

Employee offboarding still needs a review of the workloads that person owned. Disabling their personal account won't necessarily remove service access they helped create. Responsibility needs to transfer, and shared credentials they knew may need to change. Named workload identities and explicit policies give the team something clearer to inspect during that review.

This approach fits environments where workloads are frequently created and replaced, including cloud services, container platforms, and CI/CD systems. An existing identity layer can make adoption relatively straightforward. A small internal tool may be adequately served by a managed secret store, tightly scoped permissions, automated rotation, and an owner who tests the recovery process. Introducing another platform adds its own operational dependencies, so the risk and maintenance savings need to justify it.

I'd begin with the credentials that are copied most often, shared across the most systems, or able to reach the most sensitive resources. Those are the places where removing a long-lived dependency can reduce both exposure and the work of managing it. The team can then check that renewal, logging, and revocation behave as expected before extending the approach to more applications.