This is My Architecture Series from AWS is pretty good to see how companies are designing cloud based architectures. Every week I watch at least one from the series that catches my attention. For today’s post I’ll be focusing on a scalable centralized logging architecture for multiaccount taken from Biogen use case.

centralized-logging

Following are the key points around the design decisions:

  • Capture data from CloudWatch, CloudTrail and VPC flow logs into Amazon Kinesis Firehose. Use Kinesis Firehose to create logs in correct format.

  • Use Lambda (for each individual account) to classify or split information based on what data we would need for troubleshooting and what to store for later use.

  • S3 is used as a centralized backup here because of scalability, encryption and life cycle policies across the logs. Moreover, it is possible that different accounts may want to use different visualization tools and dashboards and, having a centralized storage helps with that design.

  • Lambda then pushes the data from S3 into ElasticSearch.

  • Using application keyname in ElasticSearch we can search and troubleshoot the application; for example, we can find which EC2 instances belong to particular account and then we can use that information to check the logs.

  • Everything here is mostly serverless. The architecture scales on demand and not much maintenance is needed.

Reference : This is My Architecture, AWS