The Challenge
Organizations running workloads on AWS (whether fully or in a hybrid cloud) need their existing Security Operations Center (SOC), typically managed as a separate service outside of the cloud, to have operational and security visibility into the cloud environment. AWS native security tools such as AWS GuardDuty, AWS Config findings, Security Hub for compliance results, CloudTrail audit logs, VPC Flow Logs, and others provide valuable data. However security teams often operate SIEM platforms (e.g., Elastic, Splunk, Datadog, QRadar and others) that are not native to AWS and require all findings in a centralized location. This creates the following challenges:
- Fragmented and inefficient visibility: Security findings in AWS are usually scattered across the AWS Console. Although AWS provides basic observability mechanisms, they typically lack the advanced capabilities that specialized SIEMs offer, creating blind spots and making incident response difficult.
- Multi account security: Enterprise AWS landing zones span dozens or hundreds of AWS accounts and deploying security tools, securely collecting and securely storing logs across all accounts, becomes complex and difficult to manage reliably.
- Log retention and compliance requirements: Enterprises often must archive their logs in a cost-effective way, with appropriate lifecycle policies not only within the SIEM but also for intermediary S3 buckets in order to retain them for security and compliance purposes.
- Data integrity and access control: Security logs must be immutable for legal, compliance, and forensic purposes, and isolated from employees with elevated privileges and application teams who might accidentally delete or modify them.
- Integration gaps: Routing AWS findings into external SIEMs can be challenging but is straightforward with the right architecture using AWS Firehose and AWS Kinesis, which provide advanced and reliable streaming capabilities.
Solution
Implementing centralized AWS security architecture is best done with a dedicated Log Archive AWS account, that has strict security permissions and compliance locks on its storage.
- Deploying security tools: Organizations can use the AWS Terraform provider to enable AWS security tools according to their risk profiles and needs, such as AWS GuardDuty, AWS Config, CloudTrail organization trails, VPC Flow Logs, across all accounts with delegated security administration.
- Cross-account log collection: Organizational source accounts can be allowed to write S3 buckets in the AWS Log Archive account through service principals (
cloudtrail.amazonaws.com, vpc-flow-logs.amazonaws.com) with bucket policies restricted to the organization’s ID. S3 buckets must be properly configured to block public access, appropriate object lock according to the industry specific compliance mode needs, disable legacy S3 ACLs and set up restricted IAM and S3 bucket policies, use the appropriate KMS encryption, and block access organizationally wide with SCPs.
- S3 lifecycle policies: Organizations can automatically transition logs through storage tiers from Standard to Glacier and Deep Archive based on their retention requirements. Organizations often use S3 as their backup and long term storage mechanism, in order to reduce their storage costs on their main SIEM and follow the 3-2-1 backup rule.
- Send AWS logs to an external SIEM: Organizations can use Kinesis Firehose streams to efficiently route logs from the centralized S3 buckets to the external SIEM such as Elastic, Datadog, Splunk, QRadar and others. Firehose supports generic HTTP outputs, as well as direct integrations with common SIEMs, and has the capability to enrich data with lambda functions
Impact
By following an architecture similar to the one indicated in the solution, organizations can gain visibility across their AWS environments and through their existing SIEM, while maintaining cost-effective log storage:
- Centralized security visibility as AWS findings appear in the Security Operation Center alongside on-premise and possibly other public cloud logs such as Azure, GCP or private cloud logs.
- Deploying sources as IaC and in an automated manner across all AWS accounts reduces errors and ensures consistent security posture when it comes to log management
- Storage costs are reduced through efficient lifecycle policies that move logs to cheaper tiers automatically after a defined time
- Log integrity guaranteed through immutable S3 buckets in isolated log archive account with restricted access
- Compliance simplified with automated log retention policies and cross-region replication for disaster recovery
- Incident response accelerates as security analysts work from single SIEM interface instead of switching between the AWS Console and the SIEM
