Lessons On Secrets Security From Datadog Research – Source: securityboulevard.com

Recently, Datadog released their report on attacker techniques that they saw from malicious IP addresses that affected multiple AWS environments from late 2023 through early 2024. They revealed findings from two different types of attack, each revealing a different level of sophistication.

Thankfully, no production environments were compromised, nor was any customer data accessed in these attacks. We are grateful to the Datadog team for their transparency in the report, as there are some important lessons we can learn from how attackers commonly behave from these attempts.  

A tale of two attacks

In both types of attack described in the Datadog report, and just like the majority of all attacks, attackers gained an initial foothold using ill-gotten credentials. In this case, leaked IAM user keys were to blame. We highly suggest reading the Datadog article for an in-depth look, but here is a high-level summary of what happened. 

A sophisticated attack meant to evade detection

In the first attack, after gaining initial access, the adversarial actor strengthened their foothold by creating new admin accounts, which would allow them continued access in case the initial access was terminated. Next, they focused on escalating privilege through account manipulation. After that, they looked through any service or bucket they could access to see if there was anything to leverage or steal, especially more credentials that allowed wider access. 

Ultimately, they tried to create new EC2 instances in a different region, which is when the attack ended unsuccessfully. If Datadog had not been monitoring for suspicious activities outside of their normal regions, this attack could have persisted until the bill came due. It is a good reminder that your attack surface is wider than you might initially suspect.

The Datadog team mapped this process out, summarizing one of the attacks with the following illustration. 

A less delicate approach

The second type of attack took a lot fewer steps. The attacker got in and immediately began spinning up a lot of malicious infrastructure as fast as they could. Using randomized naming, they created many new clusters to run malicious code across 17 regions. This would allow them to deploy thousands of containers to execute malicious code. 

Just like in the first attack scenario, the attacker widened their foothold as fast as they could by moving beyond the initial region. Fortunately, the Datadog team was able to work with the AWS team to find the root of the problem, in this case, allowing IAM users with static credentials to perform such costly actions. Datadog has since updated its playbooks. 

What can we do to better defend ourselves?

Even though no customers were affected or anything truly terrible resulted, the fact that these malicious actors could get so far into the Datadog infrastructure is alarming. It is a good reminder to us all to improve our security posture, especially around secrets. 

Leaked long-lived passwords continue to be a problem

Any password runs the risk of being leaked. If it is still valid 30, 90, or 365 days after it leaks, that is going to open you up to a lot of risk of a successful initial breach. If that credential is automatically rotated every 24 hours, the likelihood that it could be exploited in case of a leak drops significantly.  

For Root user passwords, it is a good idea to rotate passwords on a regular basis and regularly ensure your password reset path is secure as well.  For any other passwords, AWS has a clear path for using Secrets Manager and some simple scripting to auto-rotate secrets as often as you like. 

Always remember the principle of least privilege 

There are two things that are true about Root accounts on AWS. 1) They are powerful and, therefore, potentially dangerous. 2) They should not be used for anything beyond the highest level of administration. What you want to do instead is create users with limited permissions, scoped as tightly to the specific task as possible, following the Principle of Least Privilege.

If access is stolen or shared for one of these users, the attackers will have very limited powers and most likely would not have been able to get as far as the attackers did in Datadog’s findings.   

No password is better than short-lived passwords. 

The most secure password is one that does not exist. On AWS, and most other cloud providers, there is a clear and well-discussed path to eliminating most passwords through the use of IAM Roles. Assigning a user or application a role will grant them temporary access when it is needed. This approach also lets you get very granular with the permissions within defined roles, meaning access is only granted in tightly scoped circumstances. 

As the second type of attack showed, when an attacker creates a new user account, they don’t assign them IAM roles. They give them passwords. That should give you some pause if you are still using long-lived passwords for your trusted users. 

Remove outdated and unneeded credentials

At some point in your cloud application’s history, it is likely someone used a long-lived password to allow needed access for the app to function. In the best of scenarios, they completely removed these credentials, revoked them, and adjusted the code to use IAM rules or a secrets manager. 

There is also the scenario that those old passwords are still in Git commits from the project’s early days and are still valid.  Wouldn’t it be nice to know so you can remedy the situation if necessary?

This is where the GitGuardian Secret Detection Platform can help. Our secrets detection engine actively looks for over 400 types of secrets in your source code and in communication platforms like Slack and Jira Cloud (coming soon). 

When you first connect a code repository to GitGuardian, the platform performs a historical scan, combing back through every commit and branch in the Git history to the beginning of the project. The secrets detection engine can validate most types of common platform and service credentials, giving you a quick check if there are active credentials still lurking in your project history.

Get alerts when leaks or breaches first occur

In both of these cases researched by Datadog, the initial access was from a leaked credential. There are a number of ways an attacker can get a hold of these secrets, but based on what we have seen, they might have found them in public on GitHub. Ideally, you want to know as soon as they appear in public so you can act to protect your organization.  We can help there as well.

Honeytokens to know about leaks

GitGuardian Honeytokens are decoy credentials that don’t allow any access but instead set off alarms when someone or something tries to use them, even if it is just to validate them. If your code contains honeytokens and that code is publicly exposed on the internet, scanners, including ones from GitGuardian, will find and trigger these decoy secrets in minutes. If any real credentials are in that repo, you will be able to act immediately to revoke and replace them, keeping your application safe from unwanted access.

GitGuardian Public Monitoring to find public appearances

Just like Datadog was monitoring for activity in unused regions on AWS, It all comes down to looking in the right places. This means checking outside of your organization or even your repos. Shared credentials can be hard to find when you are only looking where you expect to find them, as was the case of Toyota, a leaked credential that was in public for 5 years before researchers found and reported it. 

GitGuardian Public Monitoring has helped many enterprises keep an eye out for leaked secrets no matter what form they take or what code they get pushed into on public GitHub. If you are not sure about the state of your secrets sprawl, we would be happy to help you with a complimentary audit of your secrets leaks on public GitHub for your enterprise. 

Learning from research 

We are very thankful for the research by our partners on the Datadog team and other security professionals who surface findings of this depth. There is a lot we can learn from these findings, even if they did not result in successful headline-grabbing breaches or involve customer data leaks. Hopefully, your organization can apply these lessons and will never fall victim to these attackers.

Security is a journey, not a constant state you can permanently achieve.  We are happy to meet you wherever you are along the path to help you keep your secrets a secret. If you need help fighting secrets sprawl and want alerts early when leaks and breaches happen, we invite you to reach out.

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog – Automated Secrets Detection authored by Dwayne McDaniel. Read the original post at: https://blog.gitguardian.com/lessons-on-secrets-security-from-datadog-research/