AWS CloudWatch Log Group Infinite Retention Detector
Identify Amazon CloudWatch log groups defaulting to 'Never Expire' quietly accumulating gigabytes of historical log storage fees.
1. Detection Scope & Operational Boundaries
Every ArthaOps detector enforces strict epistemic honesty. We clearly separate what can be mathematically proven via agentless telemetry from what requires contextual human judgement.
What This Detector Identifies
- CloudWatch log groups where retentionInDays is null (Never Expire).
- Non-production log groups (Lambda functions, ECS task logs, API Gateway execution logs) accumulating logs older than 30 days.
- High-volume log groups with storedBytes exceeding 100 GB without an automated export or retention lifecycle.
What It Cannot Establish Alone
- Whether statutory compliance standards (e.g. RBI cyber security guidelines, SOC2, PCI-DSS) mandate retaining specific audit logs for 365+ days.
- Whether cold logs are actively referenced by Athena or OpenSearch log analysis queries.
- Whether historical ingestion charges were justified: log ingestion fees ($0.57/GB in Mumbai, $0.50/GB in us-east-1) are sunk operational costs incurred at write time; setting retention curtails future storage fees ($0.033/GB-mo) but cannot retroactively refund past ingestion charges.
2. Telemetry & Evidence Requirements
ArthaOps operates strictly via Plane 1 (Read-Only Discovery). The following CloudFormation / IAM permissions and CloudWatch metrics are evaluated:
AWS::Logs::LogGroupVerification Protocol: Tested against simulated log groups with retentionInDays == null and accumulated storedBytes.
Validation Context: Validated in production AWS environments uncovering gigabytes of debug and staging logs retained for over 2 years.
3. Reproducible AWS CLI Verification Recipe
List all CloudWatch log groups in ap-south-1 with infinite retention (retentionInDays is null) sorted by storage size.
# List log groups without retention policies in ap-south-1
aws logs describe-log-groups \
--region ap-south-1 \
--query "logGroups[?retentionInDays==null].{Name:logGroupName,StoredBytes:storedBytes,Created:creationTime}" \
--output table-----------------------------------------------------------------------------------------
| DescribeLogGroups |
+--------------------------+---------------------------------------------+--------------+
| Created | Name | StoredBytes |
+--------------------------+---------------------------------------------+--------------+
| 2025-01-15T10:00:00.000Z | /aws/lambda/staging-payment-webhook-handler | 536870912000 |
+--------------------------+---------------------------------------------+--------------+💡 StoredBytes indicates accumulated storage. A value of 536,870,912,000 bytes (500 GB) with no retention policy incurs $16.50/mo (₹1,386/mo) indefinitely.
4. Transparent Economic Model
- Calculated purely on stored historical log data ($0.033/GB-mo in Mumbai) beyond the target 30-day retention threshold.
- Ingestion charges ($0.57/GB in Mumbai, $0.50/GB in us-east-1) are sunk operational costs incurred at write time; purging stored logs stops future storage accrual but does not recover historical ingestion fees.
- Vended logs discount tiers and CloudWatch Logs Insights query scanning fees ($0.005/GB scanned) are excluded.
5. False-Positive Scenarios & Operational Risks
Never delete or terminate cloud resources blindly. These validated architectural scenarios must be reviewed before initiating remediation:
Documented False-Positive Scenarios
- Security & Audit Trails: CloudTrail digest logs and VPC flow log groups may be intentionally retained for regulatory compliance.
- Dispute Resolution Data: Payment callback logs retained for 180 days to resolve chargebacks.
Blast Radius & Remediation Risks
- Permanent Log Deletion: Applying a 30-day retention policy permanently purges all log events older than 30 days within hours. Deleted log events cannot be recovered.
6. Remediation Procedure & Rollback Runbook
Follow this ordered sequence of verified AWS CLI actions to remediate the resource with complete rollback availability:
Export Historical Logs to S3 (Optional)
If compliance requires retaining logs, export historical data to an S3 bucket with Glacier lifecycle rules.
aws logs create-export-task --log-group-name /aws/lambda/staging-payment-webhook-handler --from 0 --to $(date +%s000) --destination my-archive-bucketApply 30-Day Retention Policy
Set the retention policy to 30 days. CloudWatch will asynchronously delete events older than 30 days.
aws logs put-retention-policy --region ap-south-1 --log-group-name /aws/lambda/staging-payment-webhook-handler --retention-in-days 30Forensic Rollback Procedure
Retention policies can be removed at any time via `aws logs delete-retention-policy --log-group-name <name>`, returning the group to Never Expire. However, log events already purged cannot be recovered.
🔒 ArthaOps Assurance: ArthaOps checks log group name patterns and flags potential security/audit groups (e.g. CloudTrail, audit, auth) to prevent unintended log purging.
Log events purged after applying a retention policy cannot be undeleted by AWS Support.
How This Appears in the ArthaOps Control Plane
Staging container log group has accumulated 1,500 GB of logs over 18 months with retention set to Never Expire.
Test Your Infrastructure in 180 Seconds
Explore our zero-setup interactive sample report covering real Mumbai findings, or launch a 100% read-only agentless scan across your AWS environment.