Achieving Greater Visibility and Control: Centralized Logging on AWS Multi-Account

Alright, let's give this text a quick polish and sprinkle some sass on it!

If you're like me and you're obsessed with cloud-based architectures, then you've gotta check out the "This is My Architecture" series from AWS. Every week, they feature a new design that'll make your head spin with excitement. Today, we're gonna talk about Biogen's awesome scalable centralized logging architecture for multi-account use.

Check out this snazzy diagram:

Centralized Logging
Centralized Logging

Now, let's break down the design decisions:

  • First, we use Amazon Kinesis Firehose to capture data from CloudWatch, CloudTrail, and VPC flow logs. This helps us create logs in the correct format.

  • Then, we use Lambda (for each individual account) to classify or split the information based on what data we would need for troubleshooting and what to store for later use.

  • Next, we use S3 as a centralized backup because it's scalable, encrypted, and has life cycle policies across the logs. Plus, different accounts may want to use different visualization tools and dashboards, so having a centralized storage helps with that design.

  • Lambda then pushes the data from S3 into ElasticSearch.

  • And finally, using the application keyname in ElasticSearch, we can search and troubleshoot the application. For example, we can find out which EC2 instances belong to a particular account and then use that information to check the logs.

  • The best part? It's mostly serverless, so the architecture scales on demand and doesn't require much maintenance.

Boom, there you have it! A slick logging architecture that'll make all your developer dreams come true.

Reference: This is My Architecture, AWS