List and Visualize your AWS Services and Sub Resources in an Account

Managing and understanding AWS infrastructure across various services and resources can be challenging. This post explores how to list, aggregate, and visualize AWS services and their sub-resources within an account using the AWS resourcegroupstaggingapi, awk, and D3.js for a comprehensive, visual representation.

Utilizing AWS Resource Group Tagging API

The AWS Resource Groups Tagging API allows users to query and tag resources across AWS services. The get-resources method is particularly useful for listing all the tagged resources in your AWS account.

You can find all the supported AWS resources in the documentation.

Examples Using AWS Resource Group Tagging API and AWK

Code Example 1: Extracting Service and Resource

The first example demonstrates how to extract the service name and resource type directly from the Resource ARN using awk.

#!/usr/bin/env bash

aws resourcegroupstaggingapi get-resources | jq '.ResourceTagMappingList[].ResourceARN' | \
 awk -F '[:/]' '
            {
                if ($3 == "s3") {
                    print "Service: " $3 ", Resource: bucket"
                } else if ($3 == "sns") {
                    print "Service: " $3 ", Resource: topic"
                } else {
                    print "Service: " $3 ", Resource: " $6
                }
            }
        '

Special handling for services like S3 buckets and SNS topics is included since their resource names are more descriptive and meaningful.

Output

Service: ssm, Resource: parameter
Service: ec2, Resource: internet-gateway
Service: ec2, Resource: subnet
Service: ec2, Resource: vpc
Service: events, Resource: rule
Service: lambda, Resource: function
Service: s3, Resource: bucket
Service: wafv2, Resource: regional
Service: lambda, Resource: function
Service: lambda, Resource: function
Service: ssm, Resource: parameter
Service: s3, Resource: bucket
Service: s3, Resource: bucket

Code Example 2: Aggregating Resource Counts

Building on the first example, this script aggregates a count of resources for each service, providing a high-level overview of resource utilization.

#!/usr/bin/env bash

aws resourcegroupstaggingapi get-resources | jq -r '.ResourceTagMappingList[].ResourceARN' | \
awk -F '[:/]' '
    {
        if ($3 == "s3") {
            count["s3"]++
        } else if ($3 == "sns") {
            count["sns"]++
        } else {
            count[$3]++
        }
    }
    END {
        for (service in count) {
            print "Service: " service ", Count: " count[service]
        }
    }
'

Output

Service: events, Count: 1
Service: ssm, Count: 2
Service: ec2, Count: 3
Service: s3, Count: 3
Service: lambda, Count: 3
Service: wafv2, Count: 1

Code Example 3: Aggregating Resource Counts with Detailed Breakdown

This example further refines the output by not only counting resources per service but also including the resource type for a detailed breakdown.

#!/usr/bin/env bash

aws resourcegroupstaggingapi get-resources | jq -r '.ResourceTagMappingList[].ResourceARN' | \
awk -F '[:/]' '
    {
        resourceType = "unknown"
        if ($3 == "s3") {
            resourceType = "bucket"
        } else if ($3 == "sns") {
            resourceType = "topic"
        } else {
            resourceType = $6  
        }

        # Construct a unique key for each service-resource pair
        pair = $3 ":" resourceType
        count[pair]++
    }
    END {
        for (pair in count) {
            split(pair, s, ":")  # Split the pair back into service and resource
            service = s[1]
            resource = s[2]
            print "Service: " service ", Resource: " resource ", Count: " count[pair]
        }
    }
'

Output

Service: ec2, Resource: subnet, Count: 1
Service: ssm, Resource: parameter, Count: 2
Service: ec2, Resource: vpc, Count: 1
Service: ec2, Resource: internet-gateway, Count: 1
Service: events, Resource: rule, Count: 1
Service: s3, Resource: bucket, Count: 3
Service: lambda, Resource: function, Count: 3
Service: wafv2, Resource: regional, Count: 1

Visualizing Data with D3.js

After collecting and structuring the data, visualization can provide deeper insights and a clearer overview. The Zoomable Starburst chart from D3.js is an excellent choice for representing hierarchical data like AWS services and resources.

https://observablehq.com/@d3/zoomable-sunburst

To populate this visual, the data needs to be in a specific JSON structure compatible with D3. The Python script below reads the CSV format data and outputs the required JSON structure.

import json
import sys

# Initialize the data structure
data = {
    "name": "flare",
    "children": []
}

# A dictionary to keep track of services and their resources
services = {}

# Read from stdin
for line in sys.stdin:
    service, resource, count = line.strip().split(',')
    if service not in services:
        services[service] = {}
    services[service][resource] = int(count)

# Convert the services dictionary to the required JSON structure
for service, resources in services.items():
    children = [{"name": resource, "value": count} for resource, count in resources.items()]
    data["children"].append({"name": service, "children": children})

# Print or save the JSON output
print(json.dumps(data, indent=4))

The only modification to the AWK in example 3 above is the output, so change:

print "Service: " service ", Resource: " resource ", Count: " count[pair]

To

print service "," resource "," count[pair]

Integration with Zoomable Starburst

The final touch is to integrate the generated JSON data with the Zoomable Starburst chart, modifying labels to include counts. The working code example can be found in this gist.

Visualisation of my AWS Services and Resources

Next Iteration?

A natural next step for this and an extra level in the heirarchy would be region so group resources by Region, Service and Resource.

Use Cases

1. Cost Optimization: Understanding the distribution and count of resources can help in identifying underutilized services.
2. Security Auditing: Listing all resources can be the first step in ensuring only the necessary resources are deployed and correctly configured.
3. Infrastructure Overview: Providing a visual overview of resources for stakeholders or during team discussions.

Conclusion

By leveraging AWS’s resourcegroupstaggingapi, awk for data aggregation, and D3.js for visualization, one can effectively list and visualize all services and sub-resources in an AWS account. This approach not only aids in better understanding and managing the AWS infrastructure but also supports cost optimization and security compliance efforts. Whether you are a system administrator, DevOps engineer, or cloud architect, these tools and methods can be invaluable in maintaining a robust and efficient cloud environment.