Build reusable cloud components

UPDATE: this post references to CDKv1 some code examples might nor work 1:1 in CDKv2. The overall concep is the same.

The Challenge

Which of those sentences have you heard in your team lately:

1. We clicked it manually, because the management console does all the complicated stuff for us.
2. The template we used for the other service is too specific, we have to build it again.
3. To do it in [insert IaC framework here] requires at least x days.
4. To reuse parts of our templates we need to build some kind of templating engine around it.

If you’ve heard at least one, you maybe want to have a look at AWS CDK!

In this article I will build infrastructure code with AWS CDK and extract reusable parts with so called Constructs. I will also look at common pitfalls while doing so.

Infrastructure as Code (IaC) is - or at least should be - the norm when it comes to cloud infrastructure. Given the limitations of JSON or YAML in templating, building reusable code can still be a challenge. Especially the consequences of sentences number 4. can hurt a lot.

Building something on top of your cloud infrastructure language may seem like a good idea, and it could be a sufficient solution in the beginning. But maintaining your own templating framework has a good chance to become pretty costly in the future. An unnecessary risk if there is something well maintained out there, already.

AWS CDK

If you have already heard about AWS CDK, you know that writing IaC code gets really easy these days. Otherwise, check it out here: Getting started with the AWS CDK

With AWS CDK there is no excuse left not to IaC! In almost every case I would say: “If you can click in the management console, you can write it with the CDK as well”. For the rest it’s most likely CloudFormation itself, who can’t do it.

A practical example

Scenario: A microservice (Producer), that communicates with IoT devices, sends events to an SNS topic. A second serverless service (Consumer) will consume those events and process them. More consumers will follow.

A possible implementation of a fanout patttern could look like this:

After adding a dead letter queue and some alarming, the event subscription part of the service gets more complex. And we are not even touching the actual business logic.

Writing the ConsumerService with AWS CDK will result in the following code. The subscription part is already encapsulated in a separate function.

...
export class ServiceStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)
    const workerLambda = this.createWorkerLambda()
    const eventTopic = Topic.fromTopicArn(...)
    const alarmingTopic = Topic.fromTopicArn(...)
    const eventQueue = this.createSubscriptionResources(eventTopic, alarmingTopic)
    workerLambda.addEventSource(new SqsEventSource(eventQueue, {batchSize: 1}))
  }

  createSubscriptionResources(eventTopic: ITopic, alarmingTopic: ITopic): Queue {
    const deadLetterQueue = new Queue(this, 'DeadLetterQueue')
    const eventQueue = new Queue(this, 'EventQueue', {
      deadLetterQueue: {
        queue: deadLetterQueue,
        maxReceiveCount: 3
      }
    })

    eventTopic.addSubscription(new SqsSubscription(eventQueue))

    new Alarm(this, 'EventQueueSize', {...}).addAlarmAction(new SnsAction(alarmingTopic))
    new Alarm(this, 'DeadLetterQueueSize', {...}).addAlarmAction(new SnsAction(alarmingTopic))

    return eventQueue
  }
  createWorkerLambda(): Function {
  ...
  }
...

The whole subscription part needs less than 40 lines of code (see on Github). The generated CloudFormation template contains 190 lines of JSON, including queues, policies and event subscriptions. Even without extracting anything for reuse here, it is already less code to write and overall more readable.

The next step is to extract this code to reuse it in upcoming consumer services.

Building cloud components

The subscription code needs three input parameters:

1. event topic
2. alarming topic
3. worker lambda

and returns the event queue for the worker subscription. A perfect candidate to build an encapsulated cloud component with CDK Constructs.

In this case, everything can be set up at instantiation time. In other cases the Construct methods prepare() and validate() can be overwritten to perform final changes right before the CloudFormation code is synthesized. CDK App lifecycle. Handle with care!

The resulting Construct looks like this:

export class ConsumerSubscription extends Construct {
  eventQueue: Queue
  deadLetterQueue: Queue
  props: ConsumerSubscriptionProps

  constructor(scope: Construct, id: string, props: ConsumerSubscriptionProps) {
    super(scope, id)
    this.props = props
    this.createSubscriptionResources()
  }

  createSubscriptionResources() {
    this.deadLetterQueue = new Queue(this, 'DeadLetterQueue')
    this.eventQueue = new Queue(this, 'EventQueue', {
      deadLetterQueue: {
        queue: this.deadLetterQueue,
        maxReceiveCount: this.props.dlqRetry || 3
      }
    })
    this.props.eventTopic.addSubscription(new SqsSubscription(this.eventQueue))
    this.eventQueueSizeAlarm = new Alarm(this, 'EventQueueSize', { ... })
    this.eventQueueSizeAlarm.addAlarmAction(new SnsAction(this.props.alarmingTopic))

    this.deadLetterQueueSizeAlarm = new Alarm(this,'DeadLetterQueueSize',{ ... }))
    this.deadLetterQueueSizeAlarm.addAlarmAction(new SnsAction(this.props.alarmingTopic))
  }
}

To use this Construct, you can instantiate it with your custom props. The event queue can be accessed from the stack to enable event subscriptions. Since the given topics and the returned queue are also Constructs, CDK is still able to create all needed permissions for you.

...
export class ServiceStack extends Stack {
   ...
    const subscription = new ConsumerSubscription(this, 'Subscription',{
      eventTopic,
      alarmingTopic
    })
    lambda.addEventSource(SqsEventSource(subscription.eventQueue, {batchSize: 1}))
   ...
}

The next service can reuse this Construct and will have a more advanced subscription infrastructure, containing dead letter handling and basic alarming.

Construct vs. custom class

Or: Why should I extend Construct instead of using my own base class?”

Consistency and sharing

You can compose a higher level Construct from any number of Constructs and your Construct can be used in another cloud component. This enables you, your team or even company, to put together your Constructs like any other predefined Construct from the CDK.

Construct tree and unique idenitifiers

Each Construct knows its scope and will add new Constructs to it. This way a hierarchy of Constructs (Construct tree) is created. Every Construct gets its own id, which is used to generate the CloudFormation names of the containing resources. Those names have to be unique or will result in an error.

new Queue(this,'MyQueueName',...)
new Queue(this,'MyQueueName',...)

=> Error: There is already a Construct with name 'MyQueueName' 

Child Constructs - as part of the Construct tree - prepend their parent Construct id.

 # lib
 new Queue(this,'MyQueueName',...)

 # app
 new Queue(this,'MyQueueName',...)
 new MyConstruct(this, 'MyConstructName',...)

 # CloudFormation
 "Resources": {
   "MyQueueName...": {...   
   "MyConstructNameMyQueueName...": {...

Aspects and Tagging

AWS CDK offers Aspects as a “way to apply an operation to all Constructs in a given scope”[¹]. Aspects are classes that enable the visitor pattern by implementing the IAspect interface. You decide at which point of the Construct tree you want to start and the Aspect is applied to all child Constructs.

This can be used to write validators, that check nested Constructs for compliance like enabled encryption and many other things.

CDK offers an Aspect implementation for Tagging. Based on the scope in the Construct tree, this Aspect adds tags to the Construct and all of its children (if possible).

myConstruct.node.applyAspect(new Tag('Maintainer','MyTeamName'))
// shorthand for tagging: 
Tag.add(myConstruct, 'Maintainer', 'MyTeamName');

Pitfalls

Fixed resource names

Inside of Constructs, avoid fixed resource names as much as possible. A fixed queue name inside of your Construct will prevent you from reusing this Construct in this and any other stack in the same account. CloudFormation can’t create two queues with the same name in the same Account. A fixed S3 Bucket name is even worse, because bucket names must be globally unique.

“But i need to share the queue name for another service”: If you need to reference resource names between stacks, you can let CloudFormation generate the name and make the resource available from outside of your construct. The stack can then decide on how to share the resource name in a cross-stack scenario.

For example via AWS SSM StringParameter:

// exporting the queuename from the producer stack
new StringParameter(this, 'QueueName',{
  parameterName: '/'+ this.stackName.toLowerCase() + '/queuename',
  stringValue: myConstruct.queue.queueName
})

// importing the queuename in the consumer stack
const queuename = StringParameter.fromStringParameterName(this,'ProducerQueuename','/producerstack/queuename')

or via stack outputs:

// exporting the queuename from the producer stack
new CfnOutput(this, 'QueueName', {
  exportName: this.stackName.toLowerCase() +'-queuename',
  value: subscription.eventQueue.queueName,
})

// importing the queuename in the consumer stack
const queuename = Fn.importValue('producerstack-queuename')

Fn.importValuecreates a tight dependency between both stacks. Make sure you understand the implications of Fn:ImportValue in cross-stack references.

If it is still needed to provide a fixed name, it should be passed into the Construct and follow a consistent naming scheme.

Referencing L1 and L2 Constructs

“L1 Constructs are exactly the resources defined by AWS CloudFormation.”[¹] There is an L1 Construct for each CloudFormation resource and if you can’t find a higher level Construct fitting your needs, this is your Plan B. Higher level Constructs are more like wrappers around L1 and other higher level Constructs.

An important difference are the property types. Let us look at a standard queue to give an example. We create a standard queue and put the queue URL on a stack output.

This code will work (L2 Construct):

const queue = new Queue(this,'MyQueue')
new CfnOutput(this, 'QueueName', {
  value: queue.queueName // string '${Token[TOKEN.130]}'
})

queue.queueName returns a token. In this case, this will result in a Ref statement in CloudFormation, meaning the actual queue URL will be available once the queue has been created.

With L1 Constructs you don’t get a token back:

// fixed name
const queue = new CfnQueue(this,"MyQueue",{
  queueName: 'myqueue'
})
new CfnOutput(this, 'QueueName', {
  value: queue.queueName // string 'myqueue'
})

// dynamic name
const queue = new CfnQueue(this,"MyQueue")
new CfnOutput(this, 'QueueName', {
    value: queue.ref // string '${Token[TOKEN.130]}'
})

Instead, if you want to use values, which are created at deployment time, you have to get the value like you would in your CloudFormation template: with Ref or Fn::GetAtt.

Summary

By wrapping existing CDK Constructs with your own, it is fairly easy to create reusable cloud components. Have a look at AWS Solutions Constructs to find well crafted Constructs, see best practices or just get inspired for your own Constructs.

It will still be a challenge to identify what to extract and to choose how much configuration options should be provided. But like other software projects, principles like DRY are a good idea and avoiding tight coupling helps a lot.

Like always, you’ll improve along your way, but you need to start walking first.