AWS Resource Scanner & Cost Optimizer

Scan AWS resources across multiple Organizations, Accounts, and Regions to identify unused resources and optimize costs.

View Demo • Documentation • Installation • Usage

📋 Table of Contents

• Overview
• Features
  • Resource Discovery
  • Cost Analysis
  • Performance & Scalability
  • Output & Reporting
• Getting Started
  • Prerequisites
  • Multi-Account Setup
    • Manual Setup
    • Automated CloudFormation Setup
  • Installation
  • Usage and Configuration
• Documentation
  • Cost Estimation System
  • Rate Limiting
  • Worker Pool Architecture
• Example Report
• AWS Permissions
• Contributing
• License
• Contact

Overview

CloudSift is a powerful Go-based utility that helps organizations optimize their AWS infrastructure and reduce cloud spending. By scanning multiple AWS Organizations, Accounts, and Regions, it provides comprehensive insights into resource utilization and costs.

Features

Resource Discovery

CloudSift performs comprehensive scanning across various AWS services:

Compute & Storage

• EC2 Instances
  • CPU and memory utilization analysis
  • Attached EBS volume tracking
  • Instance state monitoring
• EBS Volumes & Snapshots
  • Unused volume detection
  • Orphaned snapshot identification
  • Cost optimization recommendations
• AMIs (Amazon Machine Images)
  • Unused AMI detection
  • Associated snapshot tracking
  • Age-based analysis
  • Cost impact calculation
• RDS Instances
  • Database utilization metrics
  • Idle instance detection

Networking

• Elastic IPs
  • Unattached IP detection
  • Usage patterns analysis
• Load Balancers (ELB)
  • Classic and Application LB support
  • Traffic pattern analysis
  • Idle LB detection
• VPCs
  • Resource utilization
  • Default VPC identification
• Security Groups
  • Unused group detection
  • Rule analysis

Identity & Database

• IAM Users & Roles
  • Last access tracking
  • Unused credential detection
  • Service role analysis
• DynamoDB Tables
  • Table usage metrics
  • Provisioned vs actual capacity
• OpenSearch Domains
  • Cluster utilization
  • Resource optimization

Cost Analysis

CloudSift includes a sophisticated real-time cost analysis system:

• Live Cost Estimation
  • AWS Pricing API integration
  • Smart caching system
  • Detailed cost breakdowns
  • Multiple time period projections
  • Resource lifetime calculations
  • Support for all AWS regions and pricing tiers

Performance & Scalability

• Intelligent Rate Limiting

  • Token bucket algorithm with 20 requests per second (RPS) default rate
  • Exponential backoff with smart retry strategy
  • Automatic rate adjustment based on API responses
  • Comprehensive failure handling and recovery

• High-Performance Worker Pool

  • I/O optimized worker allocation
  • Dynamic task distribution
  • Real-time performance metrics
  • Graceful shutdown handling

Output & Reporting

• HTML Reports

  • Interactive, modern UI
  • Resource filtering and sorting
  • Cost breakdown charts
  • Detailed resource metadata
  • Action recommendations

• Flexible Output Options

  • JSON for programmatic processing
  • Text-based logging with multiple verbosity levels
  • Optional S3 output storage

Getting Started

Prerequisites

1. AWS Credentials:
   Configure your AWS credentials using aws configure or set up the ~/.aws/credentials file.

Multi-Account Setup

CloudSift can operate in either single-account or multi-account mode:

• Single-Account Mode: Only requires AWS credentials with appropriate permissions
• Multi-Account Mode: Requires organization roles and an S3 bucket for storing results

Choose one of the following setup methods for multi-account scanning:

Manual Setup

If you prefer to set up the infrastructure manually or need customization, create:

• An Organization Role with permissions to list AWS Organization resources
• A Scanner Role that can be assumed in member accounts
• An S3 bucket with server-side encryption for storing scan results

The required permissions are detailed below.

AWS Permissions

Organization Role Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::<account_id>:role/<scanner_role>"
            ]
        },
        {
            "Action": [
                "organizations:ListAccounts",
                "organizations:DescribeAccount",
                "ec2:DescribeRegions"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Scanner Role Permissions

The scanner role requires the AWS-managed ReadOnlyAccess policy and the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<organization_account_id>:role/<organization_role>"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Optional S3 Permissions

If you want to enable S3-based file output, the organization role must also have permissions to read and write to an S3 bucket. The following policy grants the necessary access to <bucket_name>:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::<bucket_name>/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<bucket_name>"
        }
    ]
}

These S3 permissions are optional and only required if you intend to upload scan results to S3.

Automated CloudFormation Setup

For convenience, we provide a CloudFormation template that automatically sets up all required infrastructure, including all the IAM roles and permissions detailed above. This is entirely optional and only needed for multi-account scanning.

⚠️ Important: Deploy this template in your AWS Organization's management account.

The template creates:

• Organization Role: For querying organization-level resources
• Scanner Role: For reading resources in member accounts
• S3 Bucket: For storing scan results with server-side encryption

After deployment, note these outputs for use with CloudSift:

• OrganizationRole: Use with --organization-role
• ScannerRole: Use with --scanner-role
• BucketName: Use with --bucket
• BucketRegion: Use with --bucket-region

Installation

# Download latest release (replace VERSION with actual version)
curl -L -o cloudsift https://github.com/emptyset-io/cloudsift/releases/download/vVERSION/cloudsift_linux_amd64
chmod +x cloudsift
sudo mv cloudsift /usr/local/bin/

Usage and Configuration

CloudSift can be configured using command-line arguments, a YAML configuration file, or environment variables. The precedence order is:

1. Environment Variables (highest)
2. Configuration File (config.yaml)
3. Command-Line Arguments (lowest)

To get started quickly, use the init command to create default configuration files:

# Create a default config.yaml in the current directory
cloudsift init config

# Create a default .env file in the current directory
cloudsift init env

# Create config files in custom locations
cloudsift init config --output /path/to/config.yaml
cloudsift init env --output /path/to/.env

Listing Resources and Configurations

CloudSift provides commands to list various AWS resources and configurations:

# List available AWS credential profiles
cloudsift list profiles

# List AWS accounts in single-account mode
cloudsift list accounts

# List AWS accounts in organization mode
cloudsift list accounts --organization-role OrganizationRole

# List available resource scanners
cloudsift list scanners

Command-Line Usage

# Basic scan of current account
cloudsift scan

# Basic scan of all accounts in organization
cloudsift scan --organization-role OrganizationRole --scanner-role ScannerRole

# Scan specific resources in specific regions
cloudsift scan --scanners ebs-volumes,ec2-instances \
               --regions us-west-2,us-east-1

# Scan organization with custom roles and S3 output
cloudsift scan --organization-role OrganizationRole \
               --scanner-role ScannerRole \
               --output s3 \
               --bucket my-bucket \
               --bucket-region us-west-2

# Ignore specific resources (case-insensitive matching)
cloudsift scan --ignore-resource-ids i-1234567890abcdef0,vol-0987654321fedcba \
               --ignore-resource-names prod-server,backup-volume \
               --ignore-tags "Environment=production,KeepAlive=true"

# Use a specific config file
cloudsift scan -c /path/to/config.yaml

Global Command-Line Arguments

Flag	Description	Default
-c, --config	Path to config file	""
-p, --profile	AWS profile to use	default
--organization-role	Role for org access	""
--scanner-role	Role for scanning	""
--log-format	Log format (text/json)	text
--log-level	Log level (DEBUG/INFO/WARN/ERROR)	INFO
--max-workers	Maximum concurrent workers	32

Scan Command Arguments

Flag	Description	Default
--profile	AWS profile to use	default
--regions	Comma-separated list of regions	All regions
--scanners	Comma-separated list of scanners	All scanners
--accounts	Comma-separated list of account IDs to scan	"" (all accounts)
--output	Output type (filesystem, s3)	filesystem
--output-format, -o	Output format (json, html)	html
--bucket	S3 bucket for output	""
--bucket-region	S3 bucket region	""
--organization-role	Role for org access	""
--scanner-role	Role for scanning	""
--days-unused	Days threshold for unused resources	90
--ignore-resource-ids	Resource IDs to ignore	""
--ignore-resource-names	Resource names to ignore	""
--ignore-tags	Tags to ignore (KEY=VALUE)	""

Environment Variables

All configuration options can be set via environment variables with the CLOUDSIFT_ prefix:

Environment Variable	Description	Default
CLOUDSIFT_AWS_PROFILE	AWS profile to use	default
CLOUDSIFT_AWS_ORGANIZATION_ROLE	Role for organization access	""
CLOUDSIFT_AWS_SCANNER_ROLE	Role for scanning accounts	""
CLOUDSIFT_APP_LOG_FORMAT	Log format (text/json)	text
CLOUDSIFT_APP_LOG_LEVEL	Log level (DEBUG/INFO/WARN/ERROR)	INFO
CLOUDSIFT_SCAN_REGIONS	Comma-separated list of regions	"" (all regions)
CLOUDSIFT_SCAN_SCANNERS	Comma-separated list of scanners	"" (all scanners)
CLOUDSIFT_SCAN_ACCOUNTS	Comma-separated list of account IDs	"" (all accounts)
CLOUDSIFT_SCAN_OUTPUT	Output type (filesystem/s3)	filesystem
CLOUDSIFT_SCAN_OUTPUT_FORMAT	Output format (json/html)	html
CLOUDSIFT_SCAN_BUCKET	S3 bucket for output	""
CLOUDSIFT_SCAN_BUCKET_REGION	S3 bucket region	""
CLOUDSIFT_SCAN_DAYS_UNUSED	Days threshold for unused resources	90
CLOUDSIFT_SCAN_IGNORE_RESOURCE_IDS	Resource IDs to ignore (case-insensitive)	""
CLOUDSIFT_SCAN_IGNORE_RESOURCE_NAMES	Resource names to ignore (case-insensitive)	""
CLOUDSIFT_SCAN_IGNORE_TAGS	Tags to ignore in KEY=VALUE format (case-insensitive)	""

Configuration File

The config.yaml file can be placed in the following locations (in order of precedence):

1. Current directory (./config.yaml)
2. User's home directory ($HOME/.cloudsift/config.yaml)
3. System-wide directory (/etc/cloudsift/config.yaml)

Example configuration file:

aws:
  profile: default  # AWS profile to use (supports SSO profiles)
  organization_role: ""  # Role name to assume for organization-wide operations
  scanner_role: ""  # Role name to assume for scanning operations

app:
  log_format: text  # Log output format (text or json)
  log_level: INFO  # Set logging level (DEBUG, INFO, WARN, ERROR)
  max_workers: 8

scan:
  regions: # Leaving this list empty will scan all regions
    - us-west-2
    - us-east-1
  scanners: # Leaving this list empty will execute all scanners
    - ebs-volumes
    - ec2-instances
  accounts: # Leaving this list empty will scan all accounts
    - 123456789012
    - 098765432109
  output: filesystem
  output_format: html
  bucket: ""
  bucket_region: ""
  days_unused: 90
  
  # Ignore list configuration (all case-insensitive)
  ignore:
    resource_ids:
      - i-1234567890abcdef0
      - vol-1234567890abcdef0

    resource_names:
      - my-important-instance
      - critical-data-volume

    tags:
      Environment: production   # Will match "ENVIRONMENT: PRODUCTION"
      KeepAlive: "true"        # Will match "keepalive: TRUE"
      Project: critical        # Will match "PROJECT: CRITICAL"

Cost Estimation System

The cost estimation system provides real-time analysis using the AWS Pricing API:

Features

• Live AWS Pricing with intelligent caching
• Comprehensive coverage of all AWS regions
• Detailed cost breakdowns (hourly/daily/monthly/yearly)
• Resource-specific calculations

Cache Management

• Location: cache/costs.json
• Thread-safe concurrent operations
• Automatic cache maintenance
• Graceful handling of cache misses

Rate Limiting

CloudSift implements an intelligent rate limiting system to handle AWS API requests efficiently:

Core Features

• Token bucket algorithm with 20 requests per second (RPS) default rate
• Exponential backoff with smart retry strategy
• Automatic rate adjustment based on API responses
• Comprehensive failure handling and recovery

Configuration

The rate limiter uses the following default settings:

type RateLimitConfig struct {
    RequestsPerSecond: 20.0,              // Default API request rate
    MaxRetries:       10,                 // Maximum retry attempts
    BaseDelay:        100 * time.Millisecond, // Initial backoff delay
    MaxDelay:         120 * time.Second,      // Maximum backoff delay
}

Backoff Strategy

1. Starts with a 100ms delay on first failure
2. Exponentially increases delay with each retry (doubles each time)
3. Caps maximum delay at 2 minutes
4. Automatically resets after 5 minutes of successful operations
5. Provides detailed debug logging of backoff state

This configuration ensures efficient API usage while protecting against rate limiting and transient failures.

Worker Pool Architecture

The worker pool system is optimized for concurrent AWS API operations:

Features

• Dynamic scaling (CPU cores × 8)
• Comprehensive performance metrics
• Efficient task management
• Graceful shutdown handling

Performance Optimization

• Optimized for I/O-bound operations
• Configurable worker limits
• Built-in task prioritization

Example Report

View a sample CloudSift report here. This demonstration showcases:

• Resource utilization metrics
• Cost analysis and potential savings
• Resource details and metadata
• Usage patterns and recommendations

Note: The example uses generated sample data and does not reflect real AWS resources or costs.

Contributing

We welcome contributions! Please feel free to submit a Pull Request.

License

This project is licensed under the Mozilla Public License 2.0 (MPL-2.0). For more details, refer to the license page.

Commercial License

For full proprietary use, premium features, or additional support, contact us at support@cloudsift.io for commercial licensing options.

Contact

For questions and support, please email support@cloudsift.io