In this blog post, we will create a dynamic discord bot specifically to fetch AWS DevOps blogs and bring them right into your discord server with ChatOps as the Discord Bot is a form of ChatOps working within your chat environment to keep you updated on AWS blogs.

Check out the Github code for reference : 

While there are a few examples of setting this up manually, lets dive into how this can be achieved through Infrastructure as Code with Terraform for provisioning EC2, associated AWS resources and we will leverage Ansible to configure the EC2 instance. 

In this blog, we will use the power of Terraform to provision our resources with expressive HCL syntax to define infrastructure's desired state. This way, our infrastructure becomes easily replicable and version-controlled, promoting consistency and minimizing potential errors that can occur with manual configurations. 

Here is the project structure that I have currently and lets define the resources required in each blocks.

Let us add the following in the main.tf file and please find the explanation for the code block right after the code snippet. 

 This block tells Terraform to use the AWS provider and to operate in the "us-east-1" region. The AWS provider is responsible for creating and managing resources on AWS.

 A VPC (Virtual Private Cloud) is created, which is essentially a private network in AWS where you can launch your resources. The CIDR block "10.0.0.0/16" determines the IP range for the VPC. The parameters enable_dns_hostnames and enable_dns_support are set to true to enable DNS resolution and hostname support within the VPC.

 A subnet is created within the VPC. The subnet is assigned a subset of the IP range defined in the VPC (determined by the cidrsubnet function). The subnet is set to be in the "us-east-1a" availability zone and will automatically assign public IP addresses to instances launched within it.

 A security group acting as a virtual firewall for the EC2 instance is created. This security group is configured to allow incoming traffic on ports 22 (SSH), 80 (HTTP), and 443 (HTTPS) from any IP address (0.0.0.0/0). It also allows all outbound traffic.

 An EC2 instance (a virtual server) is created with the specified AMI (Amazon Machine Image) and instance type. It's launched within the created subnet and associated with the security group. You can SSH into this instance using the specified key pair.

 An Elastic IP address is a static, public IPv4 address, which is allocated to your AWS account. In this case, an EIP is associated with the EC2 instance to provide a fixed public IP address, useful if you restart your instance or it fails for some reason.

 An internet gateway is a horizontally scaleable, redundant, and highly available VPC component that allows communication between your VPC and the internet. It's attached to your VPC.

 A route table contains rules, called routes, that determine where network traffic from your subnet or gateway is directed. In this case, a route table is created which sends all traffic (0.0.0.0/0) to the Internet Gateway.

 Finally, the route table is associated with the subnet, which means that the routing rules defined in the route table will now apply to the subnet.

At the time of writing this blog, please refer the version that was used. 

This code block is mainly about setting up your Terraform project with the necessary provider (AWS in this case) and configuring it to use Terraform Cloud for storing state and running operations.

required_providers: This block specifies which providers are necessary for your Terraform project. A provider is a plugin that Terraform uses to manage resources. In this case, it specifies that the AWS provider is required. The source attribute indicates that the AWS provider should be downloaded from the HashiCorp registry and the version attribute specifies that version 5.8.0 of the AWS provider should be used.

cloud: This block is used to configure the backend for your Terraform state. The state is a necessary component of Terraform that tracks the resources it has created. In this case, it is being configured to use Terraform Cloud, which is a service provided by HashiCorp for storing your Terraform state and running your Terraform operations. The organization attribute is used to specify the name of the Terraform Cloud organization to use.

workspaces: This block is used to specify which workspace in Terraform Cloud to use. A workspace in Terraform Cloud is a place where Terraform runs are executed. The name attribute is used to specify the name of the workspace. In this case, it specifies that the workspace "terraform-sandeep-aws" should be used.

This Terraform code defines two variables: ami_id and ami_key_pair. Here is a walkthrough of what each piece does:

: This block declares a variable named ami_id. Variables in Terraform are a way to provide customizable, user-defined values that can be used within Terraform code. Please specify the AMI ID you would like to use. 

variable "ami_key_pair": Similarly, this block declares another variable named ami_key_pair. This variable is also of type string, and its default value is set to terraform-ec2. This terraform code expects you to create EC2 key pair manually from the AWS console and save the private key information securely. 

This Terraform code block defines an output value. In Terraform, outputs are like return values for a Terraform module. Once you run terraform apply, the outputs are displayed on the command line as a convenient way for users to retrieve and use these values.

output "instance_public_ip": This line is declaring an output named instance_public_ip. The string in the quotes is the identifier for this output, which you'll use when referencing the output value.

value = aws_instance.cloudysky-ec2.public_ip: This is assigning a value to the output. In this case, the value is the public_ip attribute of the aws_instance resource named cloudysky-ec2

If you have followed so far, then you should have an fair idea on the terraform resources defined. We will use actions workflow to provision these resources, also store the terraform state in terraform cloud workspace. 

Ansible is an open-source software provisioning, configuration management, and application deployment tool. In this case, we will be using Ansible playbook to configure our EC2 instance which will be created by terraform. Primarily, we will be using playbook to install Nodejs 16 on EC2 and followed by PM2 installation process manager for production 

Here is walkthrough of the code after the code snippet. 

The following tasks of the playbook are carried out on hosts belonging to the group ec2_instance.

: This task uses the wait_for module to pause the playbook execution until SSH is available on the host.

: The setup module is used here to gather facts about the host. This was explicitly requested as a task, as gather_facts: no was specified at the beginning of the play.

: The dnf module is used to update all installed packages on the host.

: The dnf module is again used to install the specified packages (tar and xz), which will later be used for unpacking the Node.js binaries.

: The get_url module is used to download the Node.js binaries from the specified URL to the temporary location /tmp.

: The command module is used to extract the downloaded Node.js binaries to /usr/local.

: The command module is used again to verify the installed Node.js version, with the output registered into the node_version variable.

: The debug module is used to display the installed Node.js version, retrieved from node_version.stdout.

: Finally, the npm module is used to install PM2 (a process manager for Node.js applications) globally on the host.

This playbook, therefore, provides an automated way to install and configure Node.js on an EC2 instance.

Now let's look at how to create discord bot using discord.js (a popular nodejs library to build discord bots ) and you can refer for more here in the documentation here - 

We will be using the bot to fetch AWS blogs specificially devops blogs and for easier understanding, I have kept it simple to fetch top 5 devops blogs when you send the request in the ChatOps but there are several enhancements that can be made to this. 

: The necessary packages and modules are imported, including the Discord.js for interacting with the Discord API, and rss-parser to parse the RSS feed from the AWS DevOps blog. An instance of the Discord client and RSS Parser are created.

: The bot logs in using a token from the environment variables and indicates it's ready in the console.

: The bot starts listening for messages on any server it's connected to. It reacts specifically to a message containing "!devops".

: Upon receiving the "!devops" command, the bot fetches the RSS feed from the AWS DevOps blog, parses it and slices the first five items (the five most recent blog posts).

: For each of these five blog posts, it generates a rich content message, known as an 'embed', that contains the blog post's title, URL, description, and other details. These embeds are then sent to the same channel where the "!devops" command was used.

: If there's an error at any point during the "!devops" command processing, it's caught and logged to the console.

I have used Github Actions to build the discord bot application code, upload to github artifacts and then download the artifacts and deploy it to  target host (EC2)

Prior to the that, I have also used Github actions to provision the required infrastructure, configuration. 

Infrastructure Provisioning and Configuration

Lets look at the Github Actions workflow and pretty long workflow :) The idea was to share one of the approaches to do it and it can be improved with reusable workflows for better organization of workflow. 

There are 2 workflow jobs in this workflow template and one is for Terraform used in Infrastructure provisioning and 2nd job is for Ansible to infrastructure configuration. 

 is responsible to checkout the source code from github repo mentioned above and then use setup-terraform actions to plan and apply the changes to provision the required AWS resources. It uses Terraform Teams token and it was created in the terraform cloud console and I have stored it in the github secrets. Refer this terraform docs for team token - 

 It also outputs the public IP that was assigned on the EC2 instance is created. 

is responsible to create a SSH connection to the target EC2 instance, install ansible on github runner, create ansible inventory for EC2 host and then run the playbook on target to perform required configuration. 

Discord BOT application build and deployment

Let's look at how to use Github actions to build this NodeJs code and deploy to EC2 and leverage PM2 - Advanced Node Process Manager to start the BOT. 

This worklfow has couple of jobs called Build and deploy. Build job is responsible to checkout source code, build nodejs application, upload artifacts to github artifacts. Deploy job is responsible for deploying the application to EC2 using SSH using private key that I have stored in Secrets. Alternatively, you can use AWS CodeDeploy to deploy to target EC2 instance but I have gone with SSH route. 

Once everything is all set, You should see the Discord Bot online and get a response for every command you send that has !DevOps 

Watch out for more interesting blogs and happy reading! 

Build a discord bot that brings AWS devops blogs right into your Discord server. Starting from setting up our infrastructure using Terraform, we leverage Ansible to configure EC2 instances precisely as per our needs and use GitHub Actions, which facilitates continuous integration and deployment (CI/CD), ensuring that our bot is always running the latest code

Discord Bot for AWS Blogs: Leveraging Terraform, Ansible, EC2, and GitHub Actions

Hello readers! Today, I'll be guiding you step-by-step on a fascinating journey that integrates AWS-SDK with the latest version of Next.js, 13.4. The exciting features of this version, including App Router and the much-anticipated server components, make it an ideal choice for modern web applications. Also, will walk through step by step in deploying the application on AWS using terraform starting from ECR for storing container images,  route53's DNS management, ALB load balancing to ECS container orchestration. We will also leverage Github Actions to build, publish application to ECR and deploy to AWS using terraform using terraform actions. 

To make our journey tangible, we will be creating a full-fledged application using Next.js. This application will serve a comprehensive dashboard detailing the currently running ECS tasks for this app. You'll be privy to metrics such as the number of ECS tasks, the Availability Zones they operate within, their connectivity parameters, Launch type, allocated memory, and so much more.

https://github.com/SandeepKumarYaramchitti/nextjs-ecs-github-actions

Our deployment isn't just about getting our application up and running; it's about doing so securely, efficiently, and scalably. We'll set up an SSL configuration using AWS Certificate Manager, master DNS management with Route53, and ensure efficient traffic distribution with ALB. All these while orchestrating our containers using the mighty ECS.

To bring all of these components together, we'll rely on the synergy between Route53, ALB, and ECS, a trifecta that ensures our application isn't just available, but resilient, scalable, and performant and we will do it with Infrastructure as Code with Terraform. 

Continuous Deployment with GitHub Actions

But there is more! We will use Github actions for our CI/CD, we'll automate the process of building our Next.js app, publishing it to ECR, and then deploying it seamlessly to AWS using the terraform GitHub action.

1. Setting up our Next.js 13.4 Application

Create Next.js application using `create-next-app` and follow the docs 

 for the step by step process. Once you have the Next.js app up and running, install aws-sdk with npm. 

You can also refer my GitHub repo as a starting point - 

We will be building our Next.js application that will display the ECS task definition details as a card component with a header in the Next.js application. Refer the actual application for more information. 

As I mentioned, we will creating the above app using server components and App routing; lets get into the code now. 

Let us create our API using route handler that can integrate to AWS ECS service using aws-sdk. Route Handlers are only available inside the app directory. They are the equivalent of 

 need to use API Routes and Route Handlers together.

We will be using HTTP Method GET with extended helper functions from Next.js i. e 

 . Install aws-sdk if you have not done it yet. 

NextResponse from next/server helps send back responses from server-side functions in Next.js.

ECS from aws-sdk is the SDK for interacting with AWS's Elastic Container Service.

ExtractedTaskData type provides a structured way to shape the data we want to extract from the ECS task response.

maskAccountId is a helper function that masks AWS account IDs in image URLs for privacy/security reasons.

extractTaskData takes the raw data fetched from ECS and transforms it to return a structured and sanitized version.

An instance of the ECS class is created to interact with the ECS service in the us-east-1 region.

The listTasks method fetches a list of task ARNs (Amazon Resource Names) for a cluster named nextjs-cluster.

If there are no tasks in the cluster, an error is thrown.

For each task ARN in the list, detailed task descriptions are fetched using the describeTasks method.

The raw task details are sanitized using the extractTaskData function.

Finally, the sanitized task data is sent as a JSON response using Next.js's NextResponse.json().

Now, lets move on to building the frontend for the application. 

As this is a basic UI, I will be using 3 components as building blocks for my application i. e

This code represents a React component named ECSTasks that fetches and displays details of Elastic Container Service (ECS) tasks from a specified URL. Using a defined environment variable or a default URL, it makes an API request to get task data. If the API call is unsuccessful, the component displays an error message. However, upon successful data retrieval, it maps through the array of ECS tasks, presenting each task's details using a component named TaskCard. The expected shape of each ECS task's data is described by the type ExtractedTaskData. This type outlines the properties and structure of the data, such as availability zones, connectivity information, container details, and resource specifications like CPU and memory. The overall design of the component employs a flexbox layout, ensuring that the TaskCard components are centered and wrapped within the viewport.

The code defines a React component named TaskCard that visually represents the details of an Elastic Container Service (ECS) task. Leveraging a combination of text and icons from the 'react-icons' library, it displays information about the task, including availability zones, connectivity, launch type, CPU, and memory. Furthermore, for each container within the task, the component showcases container-specific details, such as the image, status, and resource allocations.

which imports the ECSTasks component to display in on the home page. 

With our app ready, it's time to containerize it. Using Docker, create an image of your application. Once done:

Set up an ECR repository on AWS. (You can refer the AWS documentation to create this. )

Now,  lets create the Github Actions workflow to build and publish image to ECR. Please setup the required secrets in your github repo. 

Create resources for ECS, ALB, and Route53.

Incorporate SSL via AWS Certificate Manager.

Apply the Terraform scripts to build out your infrastructure.

: Specifies that the AWS cloud provider is being used, with the AWS region being sourced from a variable.

: The default VPC is being used. Three default subnets are defined for three availability zones in the us-east-1 region.

: An AWS Certificate Manager (ACM) SSL certificate is created for a custom domain. A Route53 DNS record is used for domain validation.

: A new Elastic Container Service (ECS) cluster and task definition for a Next.js application is defined. The task uses a container image from an ECR repository and runs on AWS Fargate. An execution IAM role for the ECS task is also created.

: An Application Load Balancer (ALB) is set up with its security group. The ALB listens on HTTPS port 443 and forwards traffic to a target group. The target group uses HTTP and targets the ECS service.

: A DNS A record is created in Route53, pointing to the ALB, thus linking the custom domain to the load balancer.

: Defines an ECS service that runs the Next.js tasks on Fargate. The service is linked to the previously defined ALB and is configured to run three instances of the task. Another security group specific to the ECS service is also established.

In essence, the script sets up an environment where a Next.js application is containerized, then served via ECS running on Fargate, with traffic routed through an ALB, and the domain managed through Route53.

5. Automating Deployment with GitHub Actions

Create a workflow file for GitHub Actions.

Define jobs to build your Next.js app, push to ECR, and then trigger Terraform for deployment.

Commit and watch GitHub Actions work its magic!

Discover how to effectively utilize the AWS SDK within a Next.js 13.4 application with server components and deploy it on AWS using Terraform. From Route53's DNS management, ALB's load balancing capabilities to ECS's container orchestration, get an in-depth understanding of each stage. We'll also delve into the significance of secure connections and how to achieve them in your setup

Seamless Integration of Next.js 13.4 with AWS SDK: Deploying on Route53, ALB, and ECS using Terraform

This blog is a continuation of my blog series on "How I built my Cloudysky portal from ground up using some of the modern tech stacks. Please have a look at the following blogs to get the overview of the architecture. 

How I built my Cloudsky portfolio and leveraged AWS services for infrastructure and automation ?

Automating AWS Step Functions: Programmatically Initiating Workflows for Seamless Orchestration

In this blog, I will focus on provisioning OpenSearch as Infrastructure as Code (IaC) through CloudFormation Template automate the deployment and configuration of an OpenSearch cluster while ensuring secure access to the cluster's data through API Gateway and Lambda functions. 

In this example, I will be using aws cli to provision OpenSearch Cluster with the CloudFormation template.

Create a CloudFormation template in YAML or JSON format. This template will define your OpenSearch cluster's infrastructure.

Include the necessary resources, such as an AWS::OpenSearchService::Domain resource, to provision the OpenSearch cluster.

Configure the desired properties for the OpenSearch cluster, such as instance types, storage options, and network settings.

Add any additional resources or configurations required for your specific use case.

Remember to review the OpenSearch EngineVersion you would be using. Also, review the charges for OpenSearch services to avoid unnecessary charges for the service usage. Also, as a best practice keep the Opensearch domain endpoint username and password secured. In this example, I have kept them in Secrets Manager for my reference only. 

Step 1: Upload the CloudFormation Template to S3

If you haven't already, create an S3 bucket where you can upload the CloudFormation template.

Use the AWS CLI to upload the template file to your S3 bucket. For example:

Use the AWS CLI to create the CloudFormation stack, specifying the template URL and desired stack parameters. For example:

Replace <stack_name> with a unique name for your CloudFormation stack.

Update the --template-url parameter with the appropriate URL for your S3 bucket and template file.

Provide the necessary parameters and their values using the --parameters option. Adjust the parameters according to your CloudFormation template.

Wait for the CloudFormation stack creation to complete. You can use the following command to check the stack status:

Step 3: Access the Provisioned OpenSearch Cluster

Once the stack creation is complete, use the AWS Management Console or AWS CLI to retrieve the endpoint URL of your provisioned OpenSearch cluster.

In this example, I have used the sanity CMS data coming from webhook event to ingest into OpenSearch and based on your use case, you could modify it.

Here is a sample Lambda code that ingests data into OpenSearch Cluster. 

I found opensearch-py client library has good support to interact with Opensearch cluster in python. 

As a first step, you would connect to the OpenSearch domain endpoint using the client library using the secrets stored in SecretsManager and then perform insert and update operation conditionally. 

You can verify the data ingestion by logging into the OpenSearch domain endpoint URL. 

Secure access to the cluster's data through API Gateway and Lambda functions

To access the data securely, you could use various strategies  and the following uses API gateway and Lambda and this is used by my 

 to make calls to OpenSearch to get search results. 

Here is the CloudFormation template for the API gateway with custom route53 domain and also, uses API key to secure the API access. Also, it has the Lambda function resource which will used to search the data in OpenSearch cluster.

Here is the snippet for the actual Lambda function that gets invoked based on API gateway event. You could use various query DSL strategies to perform a full text search and in this case, match_phrase_prefix query worked fine for me. 

If you have followed above steps correctly, you would end up with a API URL like mine :- 

https://*****.*****/api/opensearch-results?q=DevOps

This integrated approach offers several benefits. It allows you to provision and manage your OpenSearch cluster and associated resources in a consistent and repeatable manner. API Gateway provides a secure and scalable interface for accessing the OpenSearch data, ensuring fine-grained control over access permissions and enforcing security measures. Lambda functions offer flexibility and extensibility, enabling you to implement custom business logic or data transformations as needed.

By combining IaC, API Gateway, and Lambda, you can streamline the provisioning of OpenSearch clusters, enhance security, and provide a scalable and controlled API interface for accessing and managing your data.

This blog is about Provisioning OpenSearch as Infrastructure as Code (IaC) and securing data access with API Gateway and Lambda allows you to automate the deployment and configuration of an OpenSearch cluster while ensuring secure access to the cluster's data through API Gateway and Lambda functions

Provisioning OpenSearch as Infrastructure as Code (IaC) and Securing Data Access with API Gateway and Lambda

Are you interested in building your own portfolio application from ground up using modern tech stack? Here is a blog explaining the process I followed to build my 

My inspiration for this app is to have the ability to create blog posts, cross post to multiple blogging platforms in a fully automated way,  organize and manage my blog posts at one consolidated place in admin studio, logging events, analytics, service uptime calculator and AWS resources cost calculator for my portal, showcase some of the modern features in tech, integration with external services and in the process stay current with tech.

Now, lets talk about the tech stack in details and implementation process. 

Next.js is a powerful and popular framework for building production ready applications with cutting edge features and few of them used on this app are server side rendering, Incremental Static Regeneration, optimize website performance by pre-rendering pages, enabling faster initial load times, automatic code splitting etc. I have used Tailwindcss for styling and micro link for link previews etc. to name a few. 

Check out the github repo for more information :- 

https://github.com/SandeepKumarYaramchitti/cloudysky-web-app

Sanity CMS is a flexible and composable headless content management system that allows to structure and manage content for the websites or applications. Why i went with Sanity for CMS ? 

Using the composable CMS capabilities of Sanity in conjunction with Next.js greatly enhanced the content management and development workflow for my application. With Sanity Studio, its powerful content editing interface, I was able to create customized and intuitive editing experiences tailored to my specific project requirements. The studio provides a user-friendly interface for content authors to create, edit, and organize content in a structured manner. 

I also used Next.js provided server-side rendering and static site generation (ISR) capabilities, allowing to fetch data from Sanity during the build process and pre-render pages with the latest content. This ensures that my application always displays up-to-date content without compromising performance.

Amplify Web Hosting is a feature of AWS Amplify that allows to deploy and host web applications, static websites, and single-page applications (SPAs) with ease. It provides a managed environment for building, deploying, and scaling your applications while offering features like continuous deployment, automatic SSL certificate management, and CDN (Content Delivery Network) integration for fast content delivery.

These are some of the features I really like with Amplify

Within the Amplify console, I have defined the build and deployment settings for my web application. This includes specifying the build commands, environment variables,  secrets, branch mappings, and other configuration options. Above all, this is logging that is next level for a SSR / ISR application specially as it is NextJs based app. Also, configurations are slightly different for nextjs and checkout 

This is one of my all time favorite feature on Amplify and takes effect when a pull request is created in my web application github repo, Amplify automatically detects it and triggers a preview build. The preview build creates a temporary environment that reflects the changes made in the pull request along with a preview URL. 

Amplify works well with Route 53 URLs out of the box as I could do the  DNS Configuration in Amplify console after domain registration, Amplify provides automatic provisioning and management of SSL/TLS certificates using AWS Certificate Manager (ACM). After that, Amplify will build my application and create a deployment bundle, which is then distributed and served by AWS infrastructure. In summary, Amplify leverages Route 53 for domain registration, DNS configuration, and routing of traffic to my web application - 

 It provides an integrated workflow, making it easier to manage custom domains and their associated DNS settings within the Amplify ecosystem

Sanity Webhook and AWS API Gateway Integration

Sanity provides a webhook functionality as part of its content management system. A webhook in Sanity is a mechanism that allows to receive real-time notifications or trigger actions in external systems when certain events occur within the CMS. These events can include content updates, deletions, creations, or other actions within the CMS. When the specified event occurs, Sanity will automatically send an HTTP request to the configured webhook URL, providing relevant data about the event. 

I have set up the API gateway in AWS using Cloudformation Iac that will receive the sanity webhook events on CMS content change and will forward that to a step function initiator lambda to kick of data processing.

Once the sanity webhook event is received, API gateway triggers a Lambda function to initiate the step function workflow. 

AWS State machine to transform and publish blog posts

An AWS State Machine, also known as AWS Step Functions, is a serverless workflow service provided by Amazon Web Services (AWS). It enables to build and coordinate highly scalable and resilient applications by orchestrating multiple AWS services and integrating them with custom code or business logic. In this scenario, the step function orchestrated the lambda function executions in parallel to transform data required to publish to each of the blogging platforms such as Medium, DevTo, Hashnode and also, lambda to ingest data into opensearch for data logging. 

 for more details on this. At somepoint, I will make the github source code public as well once I refactor and review for any security concerns etc. 

Upstash for serverless database for caching 

I have also used serverless data platform Upstash to caching data when data in ingested to AWS opensearch. This just helps to avoid repeated API calls to opensearch for data query. More to come in this and snippets on how this integration is done. 

I still have some work to do on my portal however I thought of sharing my experience on building the portal as a side project. Let me know if this helps to get an idea on how to design, build, deploy your own idea into production. 

Cloudsky portal built with modern techstack leveraging composable Sanity CMS, AWS services, caching with upstash, and integration with various services such as medium, devto, hashnode, spotify etc. 

How I built my Cloudsky portfolio and leveraged AWS services for infrastructure and automation

Recently, I came across a scenario that required using a programmatic approach, such as invoking Step Functions directly from a Lambda function over triggering Step Functions via API Gateway directly. 

, I came across a scenario that basically processes the Sanity CMS webhook data and initiate a state machine workflow to cross post and publish my blogs into various blogging and logging platform such as medium, Devto, Hashnode and AWS opensearch for logging. Initial intent was to just trigger and pass the data from API gateway directly to the step function workflow and that didn't really scale as I ended up doing same repeated data transformation in each state machine lambda functions. So, I thought of adding a Lambda function between the API gateway and State Machine which can do that data transformation and initiate state machine workflow by passing the transformed data. 

Here is how I was able to build this solution and I have used Cloudformation to build the required infrastructure. 

The below code snippet creates the API gateway resource

Here is the step function initiator lambda function that runs when any webhook event is received from Sanity CMS once configured. Also, note that StepFunctionInitiatorFunction will require 

StepFunctionsExecutionPolicy to allow permission to kick of the workflow. 

In addition, I have also provided SecretsManagerReadWrite which is used to fetch required credentials for data transformation. 

Also, refer the state machine and StatesExecutionRole defined in the Cloudformation template. State machine will require the reference to Amazon State Language file that defines the workflow and has StatesExecutionRole that allows 

lambda:InvokeFunction for each of the lambda functions within the workflow. 

Now, lets see the code which does the programatic call to start the workflow. This is just a snippet and can be refactored based on the need. 

The below snippet for Lambda function receives the Sanity CMS webhook event data and trims down the data to required ones. 

For easier reference, I have excluded the data transformation section and this is basically to show that any type of data manipulation can be done here. 

Also, fetches additional data from sanity connecting through sanity client and gets required credentials from secrets manager. 

Finally, it creates an instance to connect to Step Functions using AWS-SDK.

Last step of this flow is to call the stepfunctions.startExecution function with the data you would want to pass it to the step function workflow. 

There are several scenarios where using a programmatic approach, such as invoking Step Functions directly from a Lambda function, may be preferable over triggering Step Functions via API Gateway:

Fine-grained control: When you need precise control over the invocation of your Step Functions workflow and want to perform custom logic or transformations on the input data before triggering the workflow, a programmatic approach provides more flexibility. You can write code in the Lambda function to manipulate the data, apply business rules, perform validation, or make decisions dynamically before invoking the workflow.

Complex data transformations: If you have complex data transformations or data preprocessing requirements before triggering the workflow, using a Lambda function allows you to handle these tasks efficiently. You can perform data enrichment, format conversions, or data validation as needed within the Lambda function before passing the processed data to the Step Function.

Orchestration with multiple services: If your workflow involves coordination and interaction with multiple AWS services or external systems, a programmatic approach can be beneficial. The Lambda function can act as the central orchestrator, integrating with various services and systems, making API calls, retrieving data, and applying business logic to decide when and how to trigger the Step Functions workflow.

Complex routing and conditional triggering: If you need to implement complex routing and conditional triggering logic, a programmatic approach offers more control. You can use conditionals and decision-making logic within the Lambda function to determine dynamically whether to trigger the workflow based on certain conditions or business rules. This level of flexibility may not be easily achievable with API Gateway alone.

In the world of cloud-based application development, automation is key to achieving efficient and seamless orchestration of workflows. With the title "Automating AWS Step Functions: Programmatically Initiating Workflows for Seamless Orchestration," this blog dives into the powerful capabilities of AWS Step Functions and demonstrates how to leverage programmatic approaches to automate the initiation of workflows

In this blog, we will explore the process of building a GitHub actions for notifications services to publish actions workflow execution results for Slack, Discord, Teams, Chime, SNS. Also, I will share the steps to publish this to Github Actions market place. 

Explore the power of GitHub actions actions by integrating with multiple notification services including Slack, Chime, Teams, AWS SNS and more.. 

Blogs

Sandeep Yaramchitti