<![CDATA[Route to Cloud - by HPF]]>https://blogs.houessou.comRSS for NodeMon, 13 Apr 2026 12:03:16 GMT60<![CDATA[VPC Lattice in Production]]>https://blogs.houessou.com/vpc-lattice-in-productionhttps://blogs.houessou.com/vpc-lattice-in-productionFri, 12 Dec 2025 15:07:21 GMTHey guys,

I just renewed my AWS Advanced Networking Specialty, and let me tell you… this one really humbled me.

Between all the new services, the subtle networking tricks, and the fact that I hadn’t done real deep network hands-on in a little while, I definitely had a few cold sweats during the exam.

And to make it worse, I waited 10 full days for the results; easily the most stressful part.

But hey, it’s done. I passed. 🙌🏾

And now that the pressure is over, I wanted to share something interesting.

During a recent project, and also while preparing for the exam, I rediscovered a service that I think doesn’t get the love it deserves: Amazon VPC Lattice.

It ended up helping us solve a very tricky service-exposure problem in a complex hybrid, multi-hop network architecture with overlapping CIDRs.

If you’ve ever dealt with that kind of setup, you already know: this is where traditional patterns start showing their limits.

So let’s talk about it.

Context and challenge

Let me describe what we were working with.

The client operates a multi-site enterprise network. Multiple physical locations, each with its own network equipment, all interconnected through an internal enterprise backbone.

Here's the complexity: every site uses the same internal IP ranges, with overlapping CIDRs across the board. To enable cross-site communication, each site exposes services using a pool of routable IPs combined with NAT.

One site, Site A, has a hybrid connection to AWS via Direct Connect. In AWS, we have multiple VPCs (Dev, Test, Prod, and a shared endpoints VPC) all connected through a Transit Gateway.

Site A has full connectivity to all its services - both on-premises and in AWS. Other sites can consume Site A's on-prem services through the routable IP + NAT setup. Everything works as expected.

The new objective:

Enable other sites on the enterprise network to consume services running in AWS, via Site A's Direct Connect connection, while minimizing impact on the existing landing zone infrastructure.

The challenges we needed to address:

  1. AWS private IP ranges are not routable from the enterprise network

  2. Risk of IP overlap between AWS and the enterprise network

  3. Need for bidirectional connectivity with integrated DNS resolution

The traditional approaches - chaining NAT devices, allocating more routable IPs (not even an option in our case), or setting up multiple ALBs with custom DNS - all had significant drawbacks in terms of complexity or operational overhead.

We needed a solution that could abstract the service exposure, handle routing elegantly, and integrate smoothly with the existing architecture.

That's where VPC Lattice came in.

What is VPC Lattice?

Amazon VPC Lattice is a managed application networking service that fundamentally changes how we think about service-to-service communication.

At its core, VPC Lattice abstracts away the traditional networking complexity. Instead of dealing with IP addresses, routing tables, and load balancers, you work with service networks - logical application layer constructs that handle connectivity, security, and observability for you.

Here's what makes it powerful:

Protocol flexibility: It supports HTTP, HTTPS, gRPC, TLS, and TCP. Whether you're running REST APIs, microservices, or legacy TCP applications, Lattice has you covered.

Compute agnostic: Works with ECS, EKS, EC2, Lambda - basically any AWS compute service. Also works with on-prem resources via IP address. You're not locked into a specific deployment pattern.

Security built-in: Native integration with AWS IAM for authentication and authorization. You can implement zero-trust principles without building custom proxy layers.

Hybrid-ready: And this is the key part for our use case - VPC Lattice can connect services across VPCs, accounts, and even between AWS and on-premises environments.

Read more about VPC Lattice here.

VPC Lattice handles four main connectivity patterns:

  • Connecting applications within a single AWS Region

  • Enabling hybrid connectivity between AWS and on-premises systems

  • Managing internet-based application access to AWS services

  • Facilitating cross-region application communication

For our scenario, that second pattern - hybrid connectivity - was exactly what we needed.

Solution overview

Here's how we used VPC Lattice to solve the problem.

Instead of exposing individual AWS services through traditional NAT or load balancers, we created a VPC Lattice service network that acts as a controlled gateway layer.

The architecture looks like this:

The flow

  1. Services in AWS (ECS tasks, Lambda functions, EKS workloads, RDS databases) are registered as targets in VPC Lattice

  2. Each service gets a Lattice service endpoint with its own DNS name

  3. VPCs are associated with the Lattice service network - this creates network connectivity without requiring complex routing

  4. The Lattice service network is exposed to Site A through a single, stable endpoint

  5. Site A publishes this endpoint to the enterprise network using one routable IP (remember - we had limited routable IPs available)

  6. Other sites (Site B, Site C) can now consume AWS services through standard DNS resolution and HTTP/HTTPS calls

DNS resolution for on-premises consumers required a custom DNS configuration since remote sites can't directly resolve Lattice endpoints (this would resolve the non-routable IP of the Lattice endpoint):

  1. A remote site wants to call an AWS service: service1.sitea.aws.cloud

  2. The enterprise DNS returns the routable IP of Site A's NAT device - not an AWS IP

  3. Traffic hits the NAT at Site A, which rewrites the source IP (from routable to non-routable internal IP)

  4. The NAT forwards the request to the Lattice endpoint located in the internal perimeter VPC (reachable via Direct Connect and Transit Gateway)

  5. Lattice inspects the host header and path, then routes the request to the appropriate target service in the correct VPC

  6. The response flows back through the same path: Lattice → TGW → DX → NAT → Enterprise Network → Remote Site

What this gives us

Single point of exposure: Only one routable IP needed at the Site A edge. All AWS services are accessed through the Lattice service network.

No IP overlap issues: VPC Lattice operates at Layer 7 (application layer). Services are identified by DNS names, not IP addresses. This completely bypasses the overlapping CIDR problem.

Integrated service discovery: Each service registered in Lattice gets a custom DNS entry in a private hosted zone shared across VPCs (check out this post for more). For on-premises access, we used custom DNS that points to Site A's NAT, which then forwards to Lattice. This gave us consistent service names across both environments.

Fine-grained access control: We can use IAM policies and Lattice auth policies to control exactly which services are accessible from on-premises, down to specific API paths if needed.

Simplified routing: The Transit Gateway only needs to know how to reach the Lattice service network association. All the service-level routing is handled by Lattice itself.

Scalability: Adding new services is trivial - register them in Lattice, update DNS, done. No firewall rules, no NAT configuration, no routing table updates.

Why this worked

The key insight here is that VPC Lattice abstracts the network layer complexity into application-layer service connectivity. Traditional approaches would have required:

  • Multiple routable IPs (one per exposed service) - we didn't have them

  • Complex NAT chains to handle IP translation - operational nightmare

  • Custom DNS infrastructure to route requests - more moving parts

  • Manual routing table management across VPCs - doesn't scale

VPC Lattice eliminated all of that.

We went from "how do we expose these 100+ services without running out of routable IPs and creating a routing mess" to "register services in Lattice, associate services, publish one endpoint."

The on-premises teams at Site B and Site C just got a list of DNS names exposing the services. From their perspective, they're making standard HTTPS calls. They don't know (or care) about the underlying AWS networking complexity.

And from our perspective, we have centralized visibility, IAM-based security, and the ability to add or remove services without touching the underlying network infrastructure.

Alternative approaches

Before we settled on VPC Lattice, we evaluated the standard approach: combining Network Load Balancers (NLB) and Application Load Balancers (ALB).

The traditional flow would be:

DNS resolution → NAT at Site A → NLB in perimeter VPC → ALB → Target service

This requires configuring NAT per environment, creating target groups and listener rules per application at the ALB level, and managing routing across multiple layers.

Why we didn't go this route:

While the NLB+ALB approach offers very good performance (under 1ms latency) and cost, it came with deal-breakers.

Assuming the NLB and ALB live in the perimeter VPC, here's where the architecture breaks down:

  • The NLB gives you a static IP for NAT mapping (maybe one per environment if traffic segregation is required)

  • The ALB provides listener rules based on host headers or paths - great for routing

  • But ALB target groups only support EC2 instances, Lambda functions (in the same VPC/account), or IP addresses

For services running in different VPCs or accounts - which is our exact scenario - you can't just point the ALB at an ECS service or an EKS pod directly. You need static IPs.

This forces application teams in other accounts/VPCs to deploy additional NLBs in their own environments just to get a static IP that the central ALB can target. But most modern services run on containers (EKS/ECS), which typically use ALBs with dynamic, non-static IPs.

So you end up with this mess:

  • Central perimeter: NLB → ALB

  • Each application account/VPC: Another NLB just for IP stability → ALB (for the actual service) → ECS/EKS targets

That's two layers of load balancers per service, duplicated infrastructure across accounts, and significant operational overhead just to work around ALB's targeting limitations.

Other challenges:

  • Fragmented security: Policies scattered across NAT, multiple NLBs, multiple ALBs, and target security groups

  • Manual observability: No unified view. You're correlating logs across multiple load balancers, VPCs and accounts

  • High operational complexity: Every new service requires coordinating across network teams (NAT rules), central platform teams (perimeter ALB), and application teams (their own NLB)

VPC Lattice eliminated the operational overhead, centralized security and observability management, and gave us the flexibility to scale.

Key recommendations

If you're considering VPC Lattice for hybrid connectivity:

  • Evaluate your architecture complexity. If you have services across multiple VPCs and accounts, Lattice eliminates the need for application teams to deploy extra infrastructure just to get static IPs for routing.

  • Check protocol requirements. Lattice supports HTTP/HTTPS/gRPC and TCP. FTP and other legacy protocols need alternative paths.

  • Enable observability from day one. CloudWatch Logs and access logging are Lattice's biggest advantages - don't waste them.

  • Document DNS patterns clearly. The custom DNS + NAT flow for on-premises isn't intuitive. Clear docs save support tickets.

  • Weigh cost vs. operational complexity. Lattice costs more but simplifies operations dramatically. Calculate both direct costs and operational overhead.

  • Use IAM policies for service-level security. Don't rely solely on security groups. Lattice's auth policies enable fine-grained, identity-based access control.

]]><![CDATA[I've tried the AWS MCP Server]]>https://blogs.houessou.com/ive-tried-the-aws-mcp-serverhttps://blogs.houessou.com/ive-tried-the-aws-mcp-serverTue, 12 Aug 2025 12:29:30 GMTHi guys!

So years ago, GenAI was emerging and I gave a presentation about how to apply Generative AI to everyday business processes. The goal was simple: showcase what was possible with services like Amazon Bedrock and demonstrate how AI could transform how we work.

One scenario stuck with people:

Mr. Nobody wants to talk to his AWS resources.
He speaks; a bot listens; it pulls context from the environment and, when asked, takes action.

Back then, that was more vision than product. Amazon Q was still young and couldn’t reach into the console the way we wanted. If you needed an assistant for AWS infrastructure, you had to build your own.

Fast-forward to today: Amazon Q can now provide details about your resources right inside the console. The thing that felt “impossible” turned into a menu option. So why bother building your own at all?

Because the path from proof-of-concept to production reality teaches you when to build, when to buy, and how the underlying architecture is changing.

Goals & Constraints

For this scenario, my goals were simple:

  • Make an application that will work simpler with AI

  • Boost day-to-day productivity

  • Plug in enterprise context (AWS resources across multiple accounts)

  • Keep the overall infrastructure fast and cheap enough to matter

At that time, building a custom solution was the only way to explore this possibility. I knew I needed:

  • An LLM to understand natural language requests (Claude with Amazon Bedrock)

  • Custom tools to interact with AWS services/resources

  • An agent to orchestrate everything together

The plan on paper looked clean:

User → Agent → Custom Tools → AWS APIs → Answer/Action

I followed the standard pattern for AI agents at the time (at least to my knowledge then). I built custom Python functions to serve as tools for the agent:

def search_ec2_instances_by_tag(tag_key):
    """Custom tool to search EC2 instances by tag"""
    ec2 = boto3.client('ec2')
    response = ec2.describe_instances(
        Filters=[{'Name': f'tag:{tag_key}', 'Values': ['*']}]
    )
    # Process and return results...

def start_ec2_instance(instance_id):
    """Custom tool to start an EC2 instance"""
    ec2 = boto3.client('ec2')
    response = ec2.start_instances(InstanceIds=[instance_id])
    # Handle response...

Each tool required careful implementation, error handling, and testing. The LangChain integration looked like this:

tools = [
    Tool(
        name="search_ec2_instances_by_tag",
        func=search_ec2_instances_by_tag,
        description="Lists EC2 instances by tag key"
    ),
    Tool(
        name="start_ec2_instance", 
        func=start_ec2_instance,
        description="Starts an EC2 instance"
    )
    # ... more custom tools
]

agent = initialize_agent(
    tools, 
    llm, 
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    memory=memory,
    verbose=True
)

The tradeoffs

This approach worked but it came with significant challenges:

  • Every new AWS action was another tool to build, test, and document.

  • The wrappers had to keep up with AWS API changes.

  • Capability matched my free time, not business demand.

  • I was basically shipping a tiny, bespoke AWS CLI.

Great for a talk and demos. Less great for production and scale.

From Custom Tools to MCP

While Q matured inside the console, the Model Context Protocol (MCP) landed with a different promise: standardize how AI apps talk to tools and data. No more bespoke plumbing for every integration. Just a protocol and servers that expose capabilities in a consistent way.

AWS Labs shipped two official MCP servers I cared about:

  • aws-api: access to AWS operations

  • aws-docs: fast, contextual access to AWS documentation

The architecture simplified overnight:

My code shrank while the surface area grew.

class SimpleAgent:
    """Simplified agent that directly uses Bedrock and MCP servers."""

    def __init__(self, app_config, mcp_client, aws_session=None):
        self.app_config = app_config
        self.mcp_client = mcp_client
        self.aws_session = aws_session or boto3.Session()
        self.bedrock_client = None
        self.conversation_history = []
        self._initialized = False
        self._last_tool_calls = []

    async def _analyze_intent(self, user_message: str) -> Dict[str, Any]:
        """Analyze user intent and determine what tools to use."""
        ...
        # Get available tools
        available_tools = self.mcp_client.get_available_tools()
        tools_description = self._format_tools_for_prompt(available_tools)

        # Create intent analysis prompt with improved context awareness
        ...

What changed for me

  • Fewer moving parts. I dropped a lot of LangChain-specific orchestration and custom wrappers.

  • First-class AWS. Instead of my custom EC2 functions, I now have access to the full AWS CLI through the aws-api server

  • Built-in documentation. The aws-docs server provides something I never implemented in my custom solution: intelligent documentation access.

When Custom Still Wins

Q is excellent when you live inside the console. But custom assistants still win when you need:

  • Cross-tool workflows. AWS + Jira + GitHub + PagerDuty in one thread.

  • Multi-cloud or on-prem bridges. Q won’t run your GCP/Azure/VMware playbooks.

  • Custom guardrails. Your change windows, naming rules, org policy, and hand-offs.

  • Deep domain context. Proprietary docs, acronyms, runbooks, and tribal knowledge.

In other words: when your workflow isn’t just “ask AWS something,” custom still pays off.

Key Takeaways

Building this custom AWS assistant taught me several lessons that I think will remain relevant for a long time (I know, AI is moving fast…):

  1. Standards > Snowflakes. MCP changed the shape of the problem. I write less glue and deliver more capability.

  2. Build → Buy → Blend. Building teaches you enough to know what to buy later and eventually what to keep custom.

  3. Reduce learning debt. The value of the proof-of-concept wasn’t the code; it was the intuition about where agents help and where they get in the way.

Conclusion

More than the result, the lesson was understanding how the GenAI technology stack keeps evolving. Whether you're using Amazon Q (or similar), building custom solutions, or exploring MCP servers, understanding these architectural patterns helps you make better decisions about when to build, when to buy, and when to migrate.

The Model Context Protocol represents the current state of this evolution: standardized ways to integrate AI with external systems that remain valuable even as managed services continue expanding their capabilities.

The complete implementation, including both the custom tool approach and the MCP-based solution, is available in my GitHub repository.

]]><![CDATA[Automate Blog Writing with AWS Bedrock & Step Functions]]>https://blogs.houessou.com/automate-blog-writing-with-aws-bedrock-and-step-functionshttps://blogs.houessou.com/automate-blog-writing-with-aws-bedrock-and-step-functionsTue, 03 Jun 2025 00:05:38 GMTHi guys!

Ever came back from a conference full of ideas... and zero time to write them down? That was me at AWS re:Invent 2024. I had notes, voice memos, and way too many ideas — but no time (or energy) to turn them into blog posts.

So I built a tool to do it for me.

Let me show you how I used Step Functions, Lambda, and Bedrock to automate the whole thing — from raw document to publish-ready blog post.

Overview

The Problem

Writing takes time. Especially when your ideas are trapped in voice recordings, transcripts, or long-form notes.

I needed a way to go from this:

“Hey future me, remember to write about the GenAI session tomorrow…”

To this:

A clean, structured blog post with a nice intro, summary, and sections — ready to copy-paste.

What I Built

I created a document processing pipeline that can:

  • Take in different types of files (audio, text, images)

  • Summarize the content with Amazon Bedrock

  • Generate a blog post draft with structure and flow

  • Save it automatically — ready to review or publish

It’s built entirely with serverless AWS services. No infra to manage.

Architecture

Here’s the pipeline stack:

  • Amazon S3: File input (like .txt, .jpeg, .mp3, .docx) and output storage

  • S3 Event Notifications: Automatically trigger the workflow when files are uploaded

  • AWS Step Functions: Orchestrates the entire workflow with smart routing and parallel processing

  • AWS Transcribe: Converts audio/video files to text

  • AWS Rekognition: Extracts text and labels from images

  • AWS Lambda: Multiple functions for preprocessing, content formatting, and output generation

  • Amazon Bedrock (Claude): AI-powered content analysis and blog post generation

The Flow

Here's how it works in real life:

  1. Upload the files to S3 - text documents, audio recordings, images, or any content you want to turn into a blog post.

  2. The upload triggers an AWS Step Functions workflow via S3 Event Notifications.

  3. Smart routing - The system automatically detects your file type and processes accordingly:

    • Audio/video files get transcribed first

    • Images get processed for visual analysis

    • Text go straight to content processing

  4. All content (whether from text, transcribed audio, or image analysis) gets cleaned up and formatted consistently. This ensures everything is ready for the AI to work with

  5. The formatted content goes to Amazon Bedrock which invokes Claude 3.5 Haiku to analyze the material and generate a polished blog post.

  6. The system saves both a summary and the final blog post in markdown format, ready to publish.

All done in a few minutes.

Prompt Engineering

Prompt engineering was the key to unlocking the full potential of this workflow. I followed best practices from Anthropic to design a clear, structured prompt that delivers accurate output from Claude.

  • Defined Role: “You are an expert content analyst…” sets Claude’s persona and aligns its responses with the task.

  • Clear Delimiters: <transcript>…</transcript> separates input from instructions to avoid confusion.

  • Precise Formatting Rules: Word limits, Markdown headers, and a strict JSON schema ensure reliable and consistent formatting.

  • Example I/O: A sample transcript and expected JSON response provide Claude with a concrete model to follow.

  • Strict Output Schema: By requiring exactly two escaped JSON fields (summary and blog_post), the output is always machine-readable and ready to use without post-processing.

Here’s the prompt I used with Claude for this project:

You are an expert content analyst and technical writer. Your role is to transform transcripts into structured, accessible content for digital platforms.
   <transcript>
   [Insert transcript content here]
   </transcript>

   Task Instructions
   Your task is to analyze the provided transcript and create two complementary outputs that serve different audience needs:
   1. Executive Summary: A concise overview for busy readers who need key takeaways quickly
   2. Detailed Blog Post: An in-depth article for readers seeking comprehensive understanding

   <detailed_requirements>
   Summary Requirements
   - Maximum 150 words
   - Capture the most important points and main themes
   - Write in clear, accessible language
   - Focus on actionable insights or key conclusions

   Blog Post Requirements  
   - Write in engaging, professional Markdown format
   - Use proper header hierarchy (hash for main title, double hash for sections, triple hash for subsections)
   - Include logical flow: compelling introduction → structured body content → strong conclusion  
   - Add at least one relevant media placeholder: ![Descriptive alt text](image-url-placeholder)
   - Incorporate technical details and examples from the transcript
   - Make content scannable with good formatting and clear transitions
   </detailed_requirements>

   <examples>
   Example Input Transcript:
   "Today we discussed the implementation of microservices architecture at our company. We found that breaking down our monolithic application into smaller services improved our deployment speed by 40%. However, we also encountered challenges with service discovery and inter-service communication. The key lessons were: start small, invest in monitoring, and have a clear API strategy."

   Example Output:
   {
   "summary": "The team successfully implemented microservices architecture, achieving 40% faster deployments by breaking down their monolithic application. Key challenges included service discovery and inter-service communication. Critical success factors identified were: starting with small services, investing in comprehensive monitoring, and establishing a clear API strategy from the beginning.",
   "blog_post": "Microservices Migration: Lessons from the Trenches\n\nIntroduction\n\nTransitioning from monolithic to microservices architecture represents one of the most significant technical decisions modern development teams face...\n\nKey Results\n\n![Microservices Architecture Diagram](image-url-placeholder)\n\nOur migration delivered impressive results:\n- 40% improvement in deployment speed\n- Enhanced team autonomy and development velocity\n\nChallenges Encountered\n\nService Discovery\nOne of our biggest hurdles was implementing effective service discovery...\n\nConclusion\n\nOur microservices journey taught us that success requires careful planning, gradual implementation, and robust monitoring infrastructure."
   }
   </examples>

   Processing Instructions
   Before generating your final response, take time to analyze the transcript systematically:

   <thinking>
   Think through these steps:
   1. What are the 3-5 most important points in this transcript?
   2. Who is the likely audience for this content?
   3. What technical concepts need explanation or context?
   4. What would make an engaging blog post structure for this topic?
   5. Where would visual elements (diagrams, charts, screenshots) add value?
   </thinking>

   Output Format
   Return your response as a properly formatted JSON object with exactly these two fields. Ensure all JSON syntax is correct, with proper escaping of quotes and newlines:

   {
   "summary": "Your 150-word summary here",
   "blog_post": "Your complete Markdown blog post here"
   }

   If the transcript is unclear, incomplete, or lacks sufficient information for a meaningful analysis, include this in your summary and create a shorter blog post that acknowledges the limitations while extracting what value is possible.

Why Step Functions + Bedrock?

Step Functions made it easy to manage the flow:

  • Visual execution

  • Built-in retries

  • Clean separation between tasks

Bedrock let me call Claude without hosting anything.

It’s fast, scalable, and reusable.

How I Used It at re:Invent

At re:Invent, I recorded voice notes every day.

At night, I dropped them into S3.

The next morning, I had a draft ready to post — with:

  • An intro,

  • Key takeaways,

  • Structured sections,

  • And a nice little closing paragraph.

I even used the drafts to help prepare LinkedIn posts faster. Huge win.

Other Use Cases

This pipeline is more than just blogging. You could use the same framework to:

  • Turn meeting transcripts into reports.

  • Summarize research papers for internal newsletters.

  • Convert interviews into publish-ready articles.

  • Feed summaries into a chatbot or FAQ generator.

Basically: turn unstructured input info into structured output.

Wrap-Up

This little project saved me hours of post-event work. It’s also a great example of how to embed GenAI into your workflow, not just for fun, but for real productivity.

Whether you’re a cloud architect, a content creator, or somewhere in between, integrating Bedrock with automation tools like Step Functions opens up real possibilities.

Want to build your own version? The full source code is coming soon.

]]><![CDATA[Exploring RAG Systems, Resilience, and AI at AWS re:Invent Days 3 & 4]]>https://blogs.houessou.com/exploring-rag-systems-resilience-and-ai-at-aws-reinvent-days-3-4https://blogs.houessou.com/exploring-rag-systems-resilience-and-ai-at-aws-reinvent-days-3-4Sat, 07 Dec 2024 13:28:15 GMTAWS re:Invent 2024 continues to be an incredible journey of learning, innovation, and a few surprises. Here's my recap of days 3 and 4, filled with technical insights, hands-on labs, and even some live entertainment.

Day 3: Exploring AI Innovations and Practical Architectures

Operational Excellence

The day started with an insightful workshop that showcased best practices and strategies - most of them relying on the use of GenAI or ML-powered services - to ensure workload stability, performance, and security while meeting organizational goals. Key points included:

  • To ensure high performance and scalability, it's important to test workloads thoroughly before they go live. Distributed Load Testing (DLT) makes this easier by automating large-scale tests to find and fix potential bottlenecks.

  • To ensure resilience, eliminate single points of failure and test system strength. AWS tools like Resilience Hub and Fault Injection Simulator help check recovery plans. For proactive monitoring, we can use services like Amazon DevOps Guru to detect and solve issues before they affect operations.

Building Scalable RAG Applications Using Amazon Bedrock and Knowledge Bases

This session focused on designing and scaling Retrieval-Augmented Generation (RAG) systems with Amazon Bedrock. It covered practical strategies, including:

• Data Optimization: Effective preprocessing techniques to streamline data pipelines for RAG systems.

• Knowledge Base Integration: Using external knowledge sources to enrich and improve LLM-generated outputs.

• Scalable: Infrastructure design patterns for building resilient systems capable of handling the unique demands of RAG-based applications.

The addition of structured data retrieval in Bedrock Knowledge Bases aligns perfectly with RAG principles, simplifying the integration of complex datasets and enabling new possibilities for customer support, analytics, and content generation use cases.

Amazon Nova Foundation Models

Amazon announced a new family of foundation models called Amazon Nova, featuring four distinct models with different capabilities: Nova Micro - A text-only model with 128K context, Nova Light - A multimodal model for fast processing, Nova Pro - An advanced multimodal model, and Nova Premiere - The most capable model for complex reasoning.

This session highlighted Amazon’s Nova series of AI foundation models, showcasing their advanced intelligence, fast processing speeds, excellent cost-effectiveness, and, most importantly, easy integration with Bedrock.

Day 4: Hands-On Labs and Inspirational Keynotes

Keynote by Dr. Werner Vogels: Managing Complexity

Dr. Vogels delivered a standout keynote on designing for resilience and simplicity in modern architectures. Key insights:

  • Breaking Down Monoliths: Modular designs reduce complexity and allow for better fault isolation.

  • Building for Failure: Systems should not only withstand failures but also adapt and recover from them.

  • Simplicity in Innovation: Balancing innovation with operational simplicity ensures long-term scalability.

His talk was inspiring and packed with actionable advice, providing a clear roadmap for handling complexity in cloud-native systems.

Prompt Engineering with Amazon Bedrock

This hands-on session was all about crafting better prompts to improve LLM responses. The steps involved:

  1. Deploying a Single-Page Application: Using Amazon S3, AWS Lambda, and API Gateway for user interaction with Bedrock models.

  2. Experimenting with Prompts: Testing and fine-tuning prompts in an Amazon SageMaker notebook.

  3. Observing the Impact: Witnessing how small changes to prompts can significantly enhance the output quality.

This lab showed how important prompt design is for getting the most out of generative AI. It was also my first experience with AWS SimuLearn, where you converse with an AI chatbot to identify technical and business requirements, and then transition to a self-paced lab to deploy and test the solution.

Build and Deploy LLM Tools Using LLM Agents

This session focused on creating intelligent tools using Amazon Bedrock. Key activities included:

  • Agent Configuration: Actions Groups, linking APIs, and associating Lambda functions to create actionable agents.

  • Testing and Deployment: Running scenarios to test agent behavior and deploying them for real-world use.

  • Practical Integration: Techniques for embedding agents into existing workflows to improve automation and efficiency.

It was a highly interactive session that showed how intelligent agents could drive innovation in business processes.

Replay Party: A Night of Fun and Music

The day ended with the re:Play Party, a chance to unwind and have some fun. The highlight? Seeing Weezer perform live!

I also tried my hand at an RC car game and nearly won. These lighter moments added a personal touch to what has been an intellectually intense event.

Reflections and Key Takeaways

Days 3 and 4 at AWS re:Invent have been packed with learning and hands-on practice. From exploring the latest AI designs to understanding the nuances of operational resilience, each session has left me with valuable insights. And, of course, the Replay Party was a perfect way to balance work and play.

As my time at re:Invent 2024 comes to the end, I’m grateful for the opportunity to learn, connect, and grow in this dynamic environment.

]]><![CDATA[AWS re:Invent 2024: Insights from the First Two Days]]>https://blogs.houessou.com/aws-reinvent-2024-insights-from-the-first-two-dayshttps://blogs.houessou.com/aws-reinvent-2024-insights-from-the-first-two-daysWed, 04 Dec 2024 07:37:26 GMTAWS re:Invent 2024 has proven to be an exhilarating journey filled with deep learning, inspiring conversations, and plenty of walking! Here’s a recap of my first two days at this extraordinary event.

Day 1: Workshops and Networking

VPC Security Approaches

The day kicked off with an intensive workshop on layered VPC security. This level 300 session provided hands-on experience with:

  • Advanced routing configurations

  • Security layer implementation

  • Best practices for secure Amazon VPC deployment

AWS Control Tower Customizations

My second session of the morning explored Control Tower customizations. It focused on scaling account provisioning while maintaining security compliance, using frameworks like AFT and CfCT. These tools help organizations scale account provisioning and enforce security compliance with ease.

The customization capabilities allow for dynamic adjustments, ensuring compliance without sacrificing agility.

Afternoon Deep Dive: Hybrid and Edge networking architectures

The sessions highlighted combining hybrid architectures, including Local Zones and Outposts, to meet latency and connectivity requirements. Discussions also covered physical connectivity considerations for Outposts deployments and enterprise-scale solutions.

A key takeaway was understanding hybrid and edge networking solutions like Local Zones and AWS Outposts, which bring computational capabilities closer to users for guaranteed low-latency performance. While both options can serve the same purpose, it is important to note that AWS Outposts requires active connectivity to the control plane - which resides in AWS VPC.

Day 2: Keynotes, Certified Lounge,

Keynote by Matt Garman

The second day started with an insightful keynote by AWS CEO Matt Garman. He announced groundbreaking advancements in AI and cloud infrastructure. Highlights included the development of “Ultracluster,” a massive AI supercomputer built with AWS Trainium chips, and the launch of Trainium2 and Trainium3 chips, offering significant performance improvements. AWS also unveiled the “Nova” series of AI foundation models and introduced Aurora DSQL for faster database performance, reinforcing its leadership in innovation and scalability.

Certified Lounge: A Quiet Break and Cool Swag

Before heading to the Expo, I made a stop at the Certified Lounge. It was a quiet space to relax, network with fellow AWS-certified professionals, and collect some stickers to personalize (yet again) my laptop. This small yet meaningful moment added a personal touch to my re:Invent experience.

Technical Insights: Sessions Attended

Multi-Tenant Serverless Architectures for Agility

This session focused on strategies for optimizing multi-tenant serverless architectures to enhance agility.

Key discussion points included:

  • Network Isolation Strategies: Ensuring tenant data remains isolated in multi-tier SaaS applications.

  • Routing Decisions: Techniques for efficient traffic routing between application tiers.

  • Tenant Deployment Models: Exploring options for deploying tenant-specific resources.

  • Connectivity Patterns: Leveraging tools like PrivateLink and AWS Lattice for secure and scalable service discovery and communication.

This session highlighted how multi-tenant architectures can balance agility and security while scaling to meet customer demands.

Robust Easy-to-Scale Architecture with Cells

One standout session explored cell-based architecture, a design approach aimed at improving scalability and resilience by isolating workloads into independent cells.

Key principles included:

  • Fault Isolation Boundaries: Limit the blast radius of failures to individual cells.

  • Dimensional Monitoring: Drill down into site, AZ, or cell-specific metrics for better observability while reducing noise with aggregated alarms.

  • Staggered Deployments: Test changes in individual cells before scaling to the entire environment.

By adopting this methodology, organizations can build fault-tolerant systems and safely scale their infrastructure with minimal downtime.

Reflections and Key Takeaways

The first two days at AWS re:Invent 2024 have been exciting and full of learning. Meeting new people, exploring ideas, and seeing the energy around innovation have made it a great experience. I’m excited to see what the rest of the event has in store!

]]><![CDATA[Countdown to AWS re:Invent 2024]]>https://blogs.houessou.com/countdown-to-aws-reinvent-2024https://blogs.houessou.com/countdown-to-aws-reinvent-2024Fri, 08 Nov 2024 04:11:58 GMTAs a Cloud Architect, AWS re:Invent is more than just another conference—it's a chance to explore AWS's latest innovations, learn from experts, and connect with industry peers. This year, I'm particularly excited about sessions on layered security, generative AI, hybrid cloud architectures, and resilience strategies.

Each day is filled with advanced content and hands-on labs that I can't wait to share with our team at Cofomo.

Day 1: Mastering Network Security and Enhancing AI-Powered Apps

The conference kicks off with NET303-R, focusing on multi-layered security strategies for Amazon VPC. I’m looking forward to hands-on guidance on multi-account, multi-region VPC network security, Route 53 DNS Firewall, AWS Network Firewall, and traffic mirroring—key elements for improving network segmentation and secure data flow.

In session FWM301-R, I’m excited to learn about the new features of AWS Amplify, especially its integration with generative AI. As a newcomer to this field, I'm excited to see how Amplify, along with AWS CDK and Amazon Bedrock, can make it easier to build AI-powered applications and improve full-stack development for more engaging user experiences.

Day 2: Architecting for Resilience Across Regions and Scaling Multi-Tenant Apps

One of the sessions I must attend, ARC309-R1, will cover the key considerations between single-region and multi-region architectures. With real-world failover scenarios, this session focuses on ensuring that our applications stay resilient in any situation.

Another highlight, SVS322, will dive into serverless multi-tenant architectures using AWS Lambda, Amazon SQS, and EventBridge, with a focus on agility and data isolation. I’m particularly interested in learning best practices for optimizing performance and scalability in multi-tenant environments.

Day 3: Generative AI and Operational Resilience

In SUP401, AWS's managed generative AI services take center stage. The hands-on workshop will cover architecture readiness, anomaly detection, and debugging strategies, which are crucial for building operational resilience. This advanced workshop supports our goal of improving proactive risk management for client workloads.

Day 4: Building AI-Driven Tools with Amazon Bedrock

Day 4 will focus on using Amazon Bedrock to develop smart, personalized tools. In sessions TNC227 and TNC231, I’ll explore how to leverage Bedrock and AWS Lambda to create LLM-driven applications. Learning about prompt engineering techniques will be a valuable addition to my toolkit for building more tailored and effective solutions.

And much more…

Each session offers practical insights, from boosting security and resilience to using AI-driven innovations. I'm excited to connect with AWS enthusiasts, gain new perspectives, and share valuable takeaways. Stay tuned for updates and session highlights!

]]><![CDATA[How to Set Up a Secure Hybrid Network on AWS]]>https://blogs.houessou.com/how-to-set-up-a-secure-hybrid-network-on-awshttps://blogs.houessou.com/how-to-set-up-a-secure-hybrid-network-on-awsMon, 22 Jul 2024 10:00:52 GMTEstablishing a secure and scalable hybrid network is essential for organizations looking to extend their on-premises infrastructure to the cloud. A robust infrastructure forms the backbone of any successful cloud workload.

This article provides a guide to deploying a secure hybrid network on AWS using Terraform. It leverages services such as Transit Gateway, VPC, Direct Connect and/or S2S VPN, Route 53, and AWS Network Firewall to ensure robust security and centralized management.

Architecture Overview

Target Architecture

The architecture uses AWS Transit Gateway as a central hub, connecting multiple VPCs and on-premises networks in a hub-and-spoke model. This setup allows for centralized and dynamic routing, supporting up to 20 Gbps per connection and enabling efficient peering regionally and inter-regionally.

Key Components

  1. AWS Transit Gateway

    The AWS Transit Gateway acts as a central hub, simplifying the network by connecting multiple VPCs and on-premises networks. This setup uses a hub-and-spoke model, where the Transit Gateway is the hub, and the connected networks are the spokes. Here are some key considerations for the Transit Gateway:

    • Routing: Transit Gateway uses route tables to manage traffic flow between connected VPCs and on-premises networks. Each attachment (VPC, VPN, Direct Connect) can be linked to a specific route table, allowing precise control over traffic flow.

    • Static and Dynamic Routing: Transit Gateway supports both static and dynamic routing. Static routes are manually set, while dynamic routing uses Border Gateway Protocol (BGP) for automatic route updates, ensuring efficient traffic flow and less manual configuration.

    • Appliance Mode: This feature ensures that both incoming and outgoing traffic pass through the same network interface and Availability Zone, maintaining consistent routing throughout the flow.

    • Centralized Management: Efficient peering and centralized management are achieved via AWS Resource Access Manager (RAM).

  2. VPC Sharing

    We already know what a VPC is. VPC sharing allows you to share VPCs with multiple AWS accounts, facilitating centralized management of network resources while reducing the number of VPCs needed. Some key features include:

    • Resource Sharing and Centralized Management: Subnets are shared across accounts, enabling resource management from a central account while maintaining security boundaries with Security Groups and Network ACLs (micro-segmentation).

    • Cost Efficiency: By reducing the number of VPCs, VPC sharing helps lower the costs associated with inter-VPC data transfer fees.

This design choice implies that different workloads ENIs stay in the same subnet or VPC. You can choose to deploy VPC per environment as displayed on the diagram or follow your organization's compliance rules.

  1. Centralized Ingress/Egress and Inspection

    To enhance security, the architecture centralizes ingress and egress traffic.

    Egress Traffic

    • NAT Gateway: Centralized NAT Gateway handles outbound traffic from private subnets. This setup reduces the need for multiple NAT Gateways, simplifying management and reducing costs. It is also easy to apply restrictive policies for workloads that do not allow access to the Internet.

    • AWS Network Firewall: Outbound traffic is inspected by the AWS Network Firewall before reaching the NAT Gateway.

Ingress Traffic

  • Application Load Balancer (ALB) and AWS WAF: The ALB, combined with AWS WAF, protects against web exploits and bots, ensuring that only legitimate traffic reaches your applications. ALB advanced request routing can be used to route traffic to multiple workloads

  • AWS Network Firewall: Inbound traffic is inspected by the Network Firewall in the Perimeter before reaching the application servers.

Inspection

VPC-to-VPC or OnPrem-to-VPC traffic can be inspected by implementing a centralized Inspection VPC. You can choose which traffic is inspected by setting the appropriate routes on the corresponding Transit Gateway routing table.

  1. Hybrid DNS Resolution

    Effective DNS resolution is critical for hybrid networks. This architecture leverages AWS Route 53 for both internal and hybrid DNS resolution.

    Private Hosted Zones

    • Private Hosted Zones: These zones allow DNS records to be visible only within specific VPCs. By associating private hosted zones with VPCs, internal DNS resolution is enabled, ensuring that internal services can communicate efficiently.

    • VPC Attachments: Private hosted zones can be attached to multiple VPCs, allowing for seamless DNS resolution across different network segments within the AWS environment.

Route 53 Resolver

  • Inbound Resolver: The inbound resolver endpoints enable DNS queries from on-premises networks to be resolved within AWS. This setup ensures that on-premises systems can resolve AWS internal DNS names.

    What happens is that DNS servers OnPrem forward DNS queries for your AWS private hosted zones to the Inbound resolver endpoint which has a network interface in the Endpoint VPC. All private hosted zones should be associated with the VPC Endpoint allowing DNS resolution.

  • Outbound Resolver: The outbound resolver endpoints allow DNS queries from within AWS VPCs to be forwarded to external DNS servers. You can add Resolver rules for domains hosted on OnPrem.

Services Endpoint

Centralizing access to VPC private endpoints enables secure communication between VPCs and services such as Amazon S3, EKS, SSM, and other AWS services. This approach reduces the need for internet gateways, enhances security, and simplifies network architecture management.

Deployment

In this section, we’ll provide an overview of deploying this architecture using Terraform. You can find the complete code in the GitHub repository.
As illustrated in the diagram, the deployment uses two AWS accounts: one for Network resources and another for Perimeter resources. Additionally, the on-premises resources are deployed in a separate region, with an EC2 instance acting as the customer gateway.

The Terraform code is structured into modules that handle different components of the architecture:

  1. Transit Gateway: Sets up the Transit Gateway as well as its routing tables. The TGW is shared with the principals provided in the ram_principals variable.

  2. Perimeter VPC: Creates the VPC responsible for egress and ingress traffic handling, including Network Firewalls, NAT Gateways, and Application Load Balancers. The corresponding TGW route table is associated with this VPC.

  3. Endpoint VPC: Endpoint services are listed in the variable services. TGW route table propagation and association are also added in the configuration.

  4. Development and Production VPCs: Establishes isolated environments for development and production workloads, utilizing centralized VPC endpoints for connectivity to AWS Services. These VPCs are shared with the organization principals listed in the subnet_sharing_principals variable. TGW route table propagation and association are also added in the configuration.

  5. On-Premises VPC: Sets up a simulated on-premises network environment in a different region, including VPN connections for secure communication with cloud VPCs. We use strongSwan and bird with an EC2 instance to handle IPSec VPN and BGP propagation.

This modular approach ensures that each part of the network is configured correctly and can be easily managed and updated. The architecture provides a robust and secure hybrid network foundation, facilitating seamless integration and efficient management of both cloud and on-premises resources.

Conclusion

This architecture provides a scalable and secure multi-VPC AWS network infrastructure using AWS Transit Gateway, VPC sharing, centralized ingress/egress inspection, and hybrid DNS resolution. By following this guide, organizations can extend their on-premises infrastructure to the AWS cloud, ensuring strong security and centralized management.

For further questions or detailed walkthroughs, feel free to reach out or leave a comment below. Happy networking!

Good to read:

Building a Scalable and Secure Multi-VPC AWS Network Infrastructure

Landing Zone Accelerator on AWS

AWS Security Reference Architecture

]]><![CDATA[Deploying a sample to-do app on AWS - Container Version]]>https://blogs.houessou.com/todo-app-on-ecs-awshttps://blogs.houessou.com/todo-app-on-ecs-awsSun, 13 Feb 2022 23:35:05 GMTHi guys!

Recently I have been studying for the new AWS SysOps certification (which I passed already) and as you may already know, this exam is more on the practical side of things. Since I am not currently employed as a Cloud Engineer/Cloud Administrator, I decided to build and deploy a container version of my sample serverless to-do app in other to prepare for the most technical aspects of the exam.

The application's goal remains the same - allow logged-in users to manage their to-do list. The underlining architecture however will change a lot, especially the backend.

Services like ECS and ECR, Elastic Load Balancers and Autoscaling Group, API Gateway, and Cognito will be leveraged. We will make sure that our resources are securely deployed behind a VPC with the proper subnet configuration (NACLs, Route Tables, and Security groups), and will deploy a highly available application across availability zones. We also want to build a resilient application, properly monitored and maintained with appropriate CloudWatch alarms and actions (auto repairs, scaling policies...).

Overview & Basic Functionality

Refer to my previous blog post to get an overview of the application and learn more about its functionalities. All the code can be found here. There is no running demo for this app but you can still check out the serverless version's UI here.

Application Components

The biggest change with this version of the application is that we decided to deploy services backed by docker containers. And since we are deploying on AWS, Elastic Container Service is our choice to host our backend containers. Along with ECS, we need to have a well-defined VPC with networks spanning multiple availability zones for HA. The below image should provide a good overview of each layer of the app and especially the technical components involved in the backend.

app-components.png

Let's go through our backend layer components:

Services container

The application services (Main and File services) are coded as Python functions served via a Flask App. The Flask server is responsible for routing the requests to the corresponding functions i.e.

# get todos for the provided path parameter userID
@app.route('/<userID>/todos', methods=['GET'])
def getTodos(userID):
    foo bar...
    ...
    return todolist     # as JSON

Even though both services use the same design, they are built-in 2 separate container images stored in the AWS Elastic Container Registry. The reason for that choice is for the purpose of decoupling - to isolate the services so that an issue with one container does not affect the whole application.

When building the container image, make sure to provide a specific tag and version as it is going to be used and eventually modified (version) during deployments. You should also avoid using latest for your production images; it refers to the latest image built without a version specified. Your deployment/update pipeline should add a version to modified images.

ECS Service and Cluster

ECS allows you to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster (logical group of services). This is called a service. It defines the deployment configuration for your tasks. I like to compare it to an autoscaling group for tasks/containers. It specifies where our tasks will run (instances that are part of the cluster) and also how they register themselves to the load balancer to receive traffic (ports, target groups).

ECS Task

This is the instantiation of the task definition where we specify how the service's containers should be built. The task definition contains information such as container image, environment variables (i.e. the different DynamoDB tables name), container port, and log configuration for each container. I decided to define both service containers in one task definition even though they are not linked and do not share information. This will ensure that we always have the same number of containers per service per task. In the Kubernetes world, that will mean having both containers running inside a single pod so both can scale at the same time. I would compare a Task to a Pod.

Depending on your application, you can choose to define one service container per task definition, where services will benefit from being independently scalable.

Application Load Balancer

With a task definition registered, we are ready to provide the infrastructure needed for our backend. Rather than directly exposing our services to the Internet, we will provide an Application Load Balancer (ALB) to sit in front of our services tiers. This would enable our frontend website code to communicate with a single DNS name while our backend service would be free to elastically scale in and out based on demand or if failures occur and new containers need to be provisioned.

For this application, we want to create one target group per service - so two target groups in total. By doing this, we make sure that the health of our services is evaluated individually and that some unhealthy containers for the main service do not affect the files service. Having two target groups will also help the load balancer identify what request goes to what containers (service) based on the path. The ALB Listener rules will help with that: main service requests go to main service containers, files service requests go to files service containers.

Authentication

Authentication is handled by AWS Cognito + API Gateway. We use a Cognito user pool to store users' data. When a user logs in and a session is established with the app, the session token and related data are stored at the Frontend and sent over to the API endpoints. API Gateway then validates the session token against Cognito and allows users to perform application operations. We use the API endpoints as HTTP proxies which forward traffic to the ALB. Requests are routed based on their path to the appropriate service containers.

Data Layer

DynamoDB and S3 are used to store all todos and related data. Our Flask app will be performing all Database and S3 operations connecting to the table and bucket, and getting requests from the frontend. DynamoDB and S3 are serverless services that provide auto-scaling along with high availability and durability.

Application Architecture

architecture.png

The main difference with the serverless version of this app is on the backend side. Instead of API + Lambda, we now have API + ALB + ECS + plus some magic to make them work together. We also have to manage how the ECS stack scales to accommodate traffic. Let's describe the backend.

VPC and components

Before we get to define your ECS service and task, we need to build a secure and highly available infrastructure to host our containers.

  • For the sake of cost-saving, we went with the default VPC and preexisting subnets.
  • A total of 6 subnets is required to achieve HA - 3 private and 3 public subnets. Make sure to set up your routing tables accordingly.
  • We configured 2 security groups: one for the ALB allowing ingress traffic from the internet, and one for the containers instances (EC2s hosting the containers) allowing ingress only from the ALB security group. Here, make sure to allow traffic from port 31000 to 61000 (those ports are probably used by the Load balancer to map target groups).

IAM

We created 4 IAM roles as follows:

  • ECS Service Role: Amazon ECS uses the service-linked role named AWSServiceRoleForECS to enable Amazon ECS to call AWS APIs on your behalf. The AWSServiceRoleForECS service-linked role trusts the ecs.amazonaws.com service principal to assume the role. The role permissions policy allows Amazon ECS to complete actions on resources such as Rules which allow ECS to attach network interfaces to instances, Rules which allow ECS to update load balancers, Rules that let ECS interact with container images, Rules that let ECS create and push logs to CloudWatch.

  • ECS Task Role: With IAM roles for Amazon ECS tasks, you can specify an IAM role that can be used by the containers in a task. This is where we defined access permissions to the DynamoDb table and the S3 bucket. You also need to add rules to Allow the ECS Tasks to download images from ECR and
    Allow the ECS tasks to upload logs to CloudWatch.

  • EC2 Instance Role: This role is required for the container instance to be able to serve a cluster and also push logs to Cloudwatch.

  • The last role is for the application auto-scaling group for ECS which allows it to automatically scale resources (containers) based on a predefined scaling policy.

Application Load Balancer

The load balancer is deployed on the public subnets and will have the below components:

2 Target Groups

Since our application has 2 services, we need to create one target group for each, in other to forward requests to proper containers and also monitor the health of services independently.

ALB Listner and Listerner Rules

The listener rules will handle requests routing to the appropriate service based on a path pattern:

  • requests with paths like ' **/todos** ' will be forwarded to the main service containers
  • requests with paths like ' **/files** ' will be forwarded to the files service containers

Auto-scaling

We need to implement auto-scaling in two places:

Application auto-scaling

How do we scale our service containers when there is a traffic spike? Application auto-scaling. We created a CloudWatch Alarm to check for the HTTPCode_ELB_5XX_Count metric on the load balancer and trigger a scaling policy when the count is 10 for a period of 60 seconds. When triggered, the application scaling policy doubles the number of tasks, which in our case scales both services at the same time.

EC2 auto-scaling

So they have been a lot of HTTPCode_ELB_5XX errors and our containers have increased in number to accommodate the traffic. Containers run on EC2 and more containers mean fewer resources available for our container instances. We need to set up an auto-scaling group to spin up new instances when necessary. The container Instances will be deployed in the private subnets.

API Gateway

You may ask yourself why we need an API gateway while we already have the ALB which can be accessed on the frontend to serve requests. Well, the reason for having an API gateway here is to handle authentication with Cognito. It is also quite easier to set CORS in the API gateway. After authorizing the request against Cognito, the API will forward it to ALB which acts as an HTTP proxy.

Now that we have a description of how the components work together, we can discuss the deployment methods for the application.

Security

Security groups are defined so that only the load balancer can talk to the containers. No traffics coming from the internet is directly routed to the containers. The backend is isolated in private networks. Also, requests must be authorized by API Gateway before they get to the load balancer.

With the current configs, the backend sends data over the internet to retrieve/store data from DynamoDB and S3. You can do it in a more secure way by implementing VPC endpoints for the DynamoDB table and the S3 bucket. All backend traffic will then reside in the private networks.

IaC and Deployment Pipeline

The will be 4 different pipelines to deploy our application: one per backend service, one for core resources of the backend, and one for the frontend. We decided to use a Github repository and Github actions for workflows.

Frontend

The frontend pipeline is pretty straightforward. The code is pushed to the S3 website bucket whenever the frontend folder is updated in the repository.

frontend-pipeline.png

Backend core components

By core components, I mean VPC and related resources, ALB, ECS Cluster service and task, Launch configuration for EC2, API gateway, IAM roles... All resources are defined inside a SAM template. There is also a different template to spin up the website (CloudFront, S3, OAI). Every time the templates changed, a Cloudformation changeset is generated and the stack gets updated.

backend-core-pipeline.png

Backend services

The backend code is dockerized and images are pushed to our private ECR repository. The ECS task definition is then updated with the new image tag and redeployed to ECS. This happens every time the backend code is updated. Each service got a separate pipeline.

backend-services-pipline.png

Takeaways

Go for Serverless!

While I had fun spinning up containers with ECS, I found the operations burden (setting up the VPC, ensuring HA with scaling group and policies...) a lot heavier than when deploying serverless resources. Not to talk about the cost of having EC2s running at all times - especially when you cannot predict traffic. It is true that some applications use cases require the exclusive use of containers. But now with more AWS services being serverless friendly, it is easier to redesign existing apps to take advantage of that. And if you absolutely need to run containers, maybe because the app requires a rich ecosystem, well AWS App Runner can do it for you while abstracting a lot of infrastructure provisioning. And if you need deeper control over your infrastructure, go for Kubernetes.

PS: I am preparing for the CKA exam at the time of publishing this blog post.

]]><![CDATA[#CloudGuruChallenge: Multi-Cloud Madness]]>https://blogs.houessou.com/image-recognition-multicloudhttps://blogs.houessou.com/image-recognition-multicloudSun, 17 Oct 2021 23:02:56 GMTHi guys, back with another #CloudGuruChallenge. This time, it is all about a multi-cloud madness inspired by @ScottPletcher from ACG. The goal of this challenge is to architect and build an image upload and recognition process using no less than three different cloud providers. Feel free to read about the challenge instructions here.

Whilst it seems to be simple, there is still a big uncertainty because the architecture is spanning across three different cloud providers.

Here is my approach to this challenge.

Overview

I will go through the overall choice of architecture and services for the app and how I deployed it. All pertaining code can be find here.

Application web UI

About the App

The app functionality is simple on its own. It just provides analysis data from an uploaded image. The data is then stored into a NoSQL database along with the path to the image. The uploaded image + data are also displayed back to the user.

appflow.png

Application Components

Now that we have a basic understanding of the app, let's see how all of these functionalities translate to different technical components. Below image should provide a good overview of each layer of the app and the technical components involved.

app-components.png

Let's go through each component.

Frontend

The Frontend for the app is built of simple HTML and Javascript. There are two points of communication with the backend:

  • Upload an Image: I have used Javascript SDK to store the uploaded image on S3. Before being uploaded, the name of the file is made unique using UUID. This name is saved and later used to retrieve the image's data.

  • Get analysis data of the uploaded image: the frontend calls an API which triggers a Lambda function. The above saved file name is sent as path parameter.

Backend

I decided to host the brain of the app on AWS. This part of the infrastructure is basically made of Lambda functions and API Gateway. For image recognition, I've used Azure Computer Vision API, and to save the results, I've opted for Google Cloud Firestore. This setup is really efficient as it mostly uses managed services from the different cloud providers. We will not have to manage servers or extra configuration - other than those required to make the services work together.

Now that we have information about the various components and services involved in the app, let's focus on how to integrate them in other to obtain a technical architecture.

Architecture

architecture.png

Frontend

Website: The static html, javascript and css files generated for the website will be stored in an S3 bucket. The S3 bucket is configured to host a website and will provide an endpoint through which the app can be accessed. To have a better performance on the frontend, the S3 bucket is selected as an Origin to a CloudFront distribution. The CloudFront will act as a CDN for the app frontend and provide faster access through the app pages.

Upload picture: A Javascript function handles the file upload to a separate S3 bucket (files bucket). Before being uploaded, the name of the file is made unique using UUID. The function saves the file name in a local storage variable. This name is later used as the path parameter of the request sent to the API to retrieve image's data.

Backend

Our application's backend relies on services from all 3 providers. Let's break it down.

Azure

The purpose of our app being to extract rich information from images, I have decided to give Azure Cognitive services a try and deployed a Computer Vision API to serve this purpose. The service is fairly easy to deploy and offers a publicly accessible API which can be integrated with the other parts of the app. Since the API is accessible through the internet, we don't have to worry much about extra config to allow the communication. An access key is provided to add to your request when calling the API (although not the most secure option).

AWS

On the AWS side we have deployed two Lambda functions to handle the application logic.

  • analyzeImage: triggered when an image is added to the files bucket. Retrieves the file url from the events and send a request to the publicly accessible Azure Computer Vision API. The response of this request contains the analysis data and is store in a dictionary. The function now stores the data into a Cloud Firestore database collection. A new document is created in the collection for each analysis.

  • getAnalysis: retrieves the data store in the Cloud Firestore based on the file name. This function is backing a REST API to communicate with the frontend.

We can see that both lambda functions interact with the Cloud Firestore database hosted in GCP. Additional configuration is required for communication between AWS and GCP resources. More on that later.

GCP

Here we will deploy the NoSQL database to store the analysis data. We will create a Cloud Firestore collection and use the SDK for python to perform read and write operations.

Access GCP resources from AWS

GCP Workload Identity federation will allow us to access GCP resources from AWS without the need for service account keys.

How it works

Workload identity federation contains two components: workload identity pools and workload identity providers. Workload identity pools is a logical container of external identities (in our case AWS roles), whereas Workload identity providers are the entities that contain the relative metadata about the relationship between the external identity provider (AWS, Azure. etc.) and GCP. For example, providers can contain information like AWS account IDs, IAM role ARNs, etc.

In addition to these components, we need to configure attribute mappings. Attributes are metadata attached to the external identity token that supply information to GCP via attribute mappings. Attributes can be combined with conditions to secure your tokens so that they can only be used by approved external identities during the workload identity federation process. Examples of attributes include name, email, or user ID.

Accessing GCP from AWS

  • AWS: Create an IAM role for our Lambda functions

  • GCP: Create a workload identity pool - allows us to organize and manage providers

gcloud iam workload-identity-pools create REPLACE_ME_POOL_ID \
    --location="global" \
    --description="REPLACE_ME_POOL_DESCRIPTION" \
    --display-name="REPLACE_ME_POOL_NAME"
  • GCP: Create an Identity pool provider
gcloud iam workload-identity-pools providers create-aws REPLACE_ME_PROVIDER_NAME \
    --workload-identity-pool="REPLACE_ME_POOL_ID" \
    --account-id="REPLACE_ME_AWS_ACCOUNT_ID" \
    --location="global"

  • GCP: Create a Service Account - needed to give access to the GCP services that our lambda access, in our case Cloud Firestore: roles/iam.workloadidentityuser (to impersonate the SA) roles/datastore.user (read and write on firestore database)

  • GCP: Allow External Identities to impersonate a service account - required to allow AWS services (our lambda functions) access the GCP resources with the same roles and permissions as the service account created above

gcloud iam service-accounts add-iam-policy-binding awsrole@datapath.iam.gserviceaccount.com \
    --role=roles/iam.workloadIdentityUser \
    --member="principalSet://iam.googleapis.com/projects/REPLACE_ME_GCP_PROJECT_ID/locations/global/workloadIdentityPools/REPLACE_ME_POOL_ID/attribute.aws_role/REPLACE_ME_ROLE_ARN"
  • GCP: Generate Google credentials - will be used in the lambda functions by exposing the GOOGLE_APPLICATION_CREDENTIALS environment variable.
gcloud iam workload-identity-pools create-cred-config \
projects/REPLACE_ME_GCP_PROJECT_ID/locations/global/workloadIdentityPools/REPLACE_ME_POOL_ID/providers/REPLACE_ME_PROVIDER_NAME \
    --service-account=awsrole@datapath.iam.gserviceaccount.com \
    --output-file=configoutput.json \
    --aws

Note: All CLI commands are done in the GCP console.

The diagram bellow describes the exchange process before our resources in AWS can access the Firestore database in GCP.

workload-identity-federation.png

To better understand it, please refer to this blog which outlines the different steps involved.

IaC and Deployment Method

The application frontend and backend services hosted on AWS are defined as SAM templates. I have created two different stacks which contain resources from each side. Although I could have used a solution like Terraform (which I am yet to learn and master) to codify the entire infrastructure, I decided to deploy the Azure Computer Vision API and the GCP Firestore database directly from the console and CLI.

As for automated deployments, I am comfortable using GitHub Actions as I find it easy to understand. Each service - frontend and backend - is deployed using a separate deployment pipeline as follow:

Frontend

frontend-pipeline.png

AWS Backend

backend-pipeline.png

Takeaways

Although presenting some benefits for reliability, a multi-cloud architecture presents a lot of concerns when it comes to security, and it think this is what this cloud guru challenge is all about.

Communication from AWS resources to GCP services used to be implemented by creating a Service Account - a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in GCP. This approach represent a big security risk because it relies on service account keys (file containing authentication info) to access GCP APIs.

By using workload identity federation, we adopt a more secure method (keyless application authentication mechanism) which allows applications running in AWS (or Azure or on-premises) to federate with an external Identity Provider and call Google Cloud resources without using a service account key. Hopefully I was able to show you how this works in a real environment.

As a next step, I am planning to make access between cloud providers totally private (abstracted from the internet). Don't know how to do it yet, but we'll figure it out!

As always feel free to let me know in the comments section, how you'd have complete this challenge.

]]><![CDATA[Deploying a sample serverless to-do app on AWS]]>https://blogs.houessou.com/sample-todo-app-awshttps://blogs.houessou.com/sample-todo-app-awsWed, 04 Aug 2021 12:58:16 GMTHi guys! In this post, we'll be building a sample todo app on AWS with Python. We will build a web application which enables logged in visitors to manage their todo list. We will use the AWS Serverless Application Model SAM Framework to deploy the backend services (API, Lambda, DynamoDB and Cognito) and will host the frontend on S3 behind a CloudFront distribution. The frontend is pretty basic with no fancy visuals (I am no frontend dev :p). We will try to focus on how the resources are created and deployed on AWS.

Overview

I will go through the overall setup of the app and how I deployed it. Mostly this will be a theoretical post but all the code can be found in the GitHub repo.

Application web UI

About the App

Before I go into the architecture, let me describe what the app is about and what it does. The app is a todo list manager which helps a user manage and track his/her todo list along with their files or attachments. The user can also find specific todos through the search.

Basic Functionality

appflow.png The image above should describe the app's basic functionalities.

User/Login Management

Users are able to log in to the app using provided credentials. There is a self-register functionality and once a user is registered, the app provides a capability for the user to log in using those credentials. It also provides a logout option for the user.

Search Todo

Users are able to perform a keyword search and the app shows a list of todos that contain that keyword in the name. The search only searches on todos that the logged-in user has created. So it has the access boundary and doesn’t show Recipes across users.

Add New Todo

Users can add new Todos to be stored in the app. There are various details that can be provided for each Todo. Users can also add notes for each Todo.

Support for files

Users can upload todo files for each Todo. The app provides a capability where users can select and upload a local file or download existing files while adding notes to a Todo. The file can be anything, from a text file to an image file. The app stores it in an S3 bucket and serve it back to the user via CloudFront.

Application Components

Now that we have a basic functional understanding of the app, let's see how all of these functionalities translate to different technical components. The below image should provide a good overview of each layer of the app and the technical components involved in each layer.

app-components.png

Let's go through each component:

Frontend

The Front end of the app is built of simple HTML and Javascript. All operations and communications with the backend are performed via various REST API endpoints.

Backend

Backend for the app is built with Lambda Functions triggered by REST APIs. It provides various API endpoints to perform application functionalities such as adding or deleting todos, adding or deleting todo files, etc. The REST APIs are built using API Gateway. The API endpoints perform all operations of connecting with the functions, authenticating, etc. CORS is enabled for the API so it only accepts requests from the frontend.

Data Layer

DynamoDB Table is used to store all todos and related data. The lambda functions will be performing all Database operations connecting to the Table and getting requests from the frontend. DynamoDB is a serverless service and it provides auto-scaling along with high availability.

Authentication

The authentication is handled by AWS Cognito. We use a Cognito user pool to store users' data. When a user logs in and a session is established with the app, the session token and related data are stored at the FrontEnd and sent over the API endpoints. API Gateway then validates the session token against Cognito and allow users to perform application operations.

File Service

There is a separate service to handle file management for the application. The File service is composed of Javascript function using AWS SDK (for upload files operations), Lambda functions + API Gateway for API calls for various file operations like retrieving file info, deleting file, etc, S3 and DynamoDB to store files and files information. The files are served back to the user through the app using a CDN (Content Delivery Network). The CDN makes serving the static files faster and users can access/download them faster and easier.

Application Architecture

Now that we have some information about the various components and services involved in the app, let's move on to how to place and connect these different components to get the final working application.

architecture.png

Frontend

The static HTML, JavaScript, and CSS files generated for the website will be stored in an S3 bucket. The S3 bucket is configured to host a website and will provide an endpoint through which the app can be accessed. To perform better on the frontend, the S3 bucket is selected as an Origin to a CloudFront distribution. The CloudFront will act as a CDN for the app frontend and provide faster access through the app pages.

Lambda Functions for backend services logic

All the backend logic is deployed as AWS Lambda functions. Lambda functions are totally serverless and our task is to upload our code files to create the Lambda functions along with setting other parameters. Below are the functions which are deployed as part of the backend service:

Todos Service

  • getTodos : retrieve all todos for a userID
  • getTodo : return detailed information about one todo based on the todoID attribute
  • addTodo : create a todo for a specific user based on the userID
  • completeTodo : update todo record and set completed attribute to TRUE based on todoID
  • addTodoNotes : update todo record and set the notes to attribute to the specified value based on todoID
  • deleteTodo : delete a todo for a specific user based on the userID and todoID

Files Service

  • getTodoFiles : retrieve all files which belong to a specified todo
  • addTodoFiles : add files as attachments to a specified todo
  • deleteTodoFiles: delete selected file for specified todo

API Gateway to expose Lambda Functions

To expose the Lambda functions and make them accessible by the Frontend, AWS API Gateway is deployed. API Gateway defines all the APIs' endpoints and routes the requests to the proper Lambda function in the backend. These API gateway endpoints are called by the frontend. Each application service has its own API (keeping services as separate as possible for decoupling purpose) with deployed routes as follow:

Todos Service

  • getTodos : /{userID}/todos
  • getTodo : /{userID}/todos/{todoID}
  • deleteTodo : /{userID}/todos/{todoID}/delete
  • addTodo : /{userID}/todos/add
  • completeTodo : /{userID}/todos/{todoID}/complete
  • addTodoNotes : /{userID}/todos/{todoID}/addnotes

Files Service

  • getTodoFiles : /{todoID}/files
  • addTodoFiles : /{todoID}/files/upload
  • deleteTodoFiles : /{todoID}/files/{fileID}/delete

The addTodoFiles API route triggers the addTodoFiles function which only records the file information like fine name and file path/key to a DynamoDB table. The same table is queried by the getTodoFiles function to display returned files information. The actual operation to upload the files to S3 is performed by a Javascript function in the Frontend code. I found it better to do it that way to prevent a large amount of data going through the lambda functions and thus increasing response time and cost.

Database

DynamoDB tables are used to serve as databases. We have two tables for respectively the Todos Service and the Files Service. The search functionality of the app is handled by simple DynamoDB query requests. We can deploy a DynamoDB Accelerator in front of the tables to increase performance if needed. Below is the configuration of the tables:

Todos Service To keep things simple, each document in DynamoDB will represent one todo with attributes as follow:

  • todoID : unique number identifying todo, will serve as primary key
  • userID : ID of the user who created the todo, will serve as a sort key
  • dateCreated : date todo has been created, today's date
  • dateDue : date the todo is due, user-provided
  • title : todo title, user-provided
  • description : todo description, user-provided
  • notes : additional notes for todo, can be added anytime after todo is created, blank by default
  • completed : true or false if todo is marked as completed

Files Service

  • fileID : unique number identifying file, will serve as primary key
  • todoID : ID of belonging todo item, will serve as sort key
  • fileName : name of the uploaded file
  • filePath : URL of the uploaded file for downloads

File Storage

To support the file management capability of the application, file storage needs to be deployed. I am using an S3 bucket as the storage for the files which are uploaded from the app. The file service API calls the AWS S3 API to store the files in the bucket. To serve the files back to the user, a CloudFront distribution is created with the S3 bucket as the origin. This will serve as the CDN to distribute the static files faster to the end users.

IaC and Deployment Method

The application backend services are defined as SAM templates. Each service has its own template and resources are configured to be as independent as possible. I am using automated deployments for the whole application environment - frontend and 2 backend services. Each service is deployed using a separate deployment pipeline to maintain optimal decoupling. The components below are used as part of the deployment pipeline:

  • One GitHub Repository for code commits
  • A separate branch for Prod changes (master branch as Dev)
  • Various paths, one per service - Frontend, Backend Todos Service, and Backend Files Service
  • Any commit to a service path in a specified branch (Prod or Dev) automatically tests and deploys changes to the service in the appropriate environment.
  • GitHub Actions backed by docker containers to build and deploy services

FrontEnd

frontend-pipeline.png

Backend

backend-pipeline.png

Takeaways

Hopefully, I was able to describe in detail the system architecture which I would use for a basic todo-list management app. This application is designed solely for training purposes and there is a lot of room for improvement. I will continue working on making the deployment more secure, highly available, and fault tolerant. This post should give you a good idea about how to design a basic full stack and fully serverless architecture for an app using the microservices pattern.

]]><![CDATA[Building A Modern Application using the AWS Serverless Application Model Framework]]>https://blogs.houessou.com/aws-modern-applicationhttps://blogs.houessou.com/aws-modern-applicationMon, 05 Jul 2021 16:06:32 GMTHi guys! In this blog, we'll be building a modern application on AWS with Python following this tutorial. We will build a sample website called Mythical Mysfits that enables visitors to adopt a fantasy creature (mysfit) as pet. The main difference with the tutorial is that we will use the AWS Serverless Application Model SAM Framework to deploy the backend services - API, Lambda (instead of Fargate), DynamoDB and Cognito).

We will use all the Frontend provided code from the tutorial - just working on the backend services here with Python as our programming language.

Application Architecture arch-diagram.png

The steps to complete this tutorial are:

  • Build a static website to serve static content - S3 + CloudFront
  • Enable users to retrieve, filter, like and adopt mysfits - API Gateway + AWS Lambda + DynamoDB - microservice #1
  • Enable users authentication - Cognito
  • Enable users to contact the Mythical Mysfits staff via a Contact Us button - API Gateway + AWS Lambda + DynamoDB + SNS - microservice #2
  • Capture user behavior with a clickstream analysis - API Gateway + Lambda + Kinesis - microservice #3
  • Use Machine Learning to recommend a Mysfit - API Gateway + Lambda + SageMaker - microservice #4

GitHub repository: https://github.com/hpfpv/mythicalmysfits-aws

Created web app: https://mythicalmysfits.houessou.com

Alright, let's break this down.

Static website with S3 and CloudFront

website.png This one is straight forward. Use the aws cli to create a bucket and copy the xx/web/index.html file. Modify the bucket policy to allow public read and set the bucket to serve static website content: s3bucket1.png For this project, we have also implemented a CloudFront distro with a custom domain name. Accessing the bucket display an HTML page with a list of mysfits stored in a dict variable in the code. We need to load the mysfits from a DynamoDB Table for dynamic operations like get and update.

Backend microservice #1: Operations on Mysfits + user authentication

micrservice 1.png This microservice is deployed using the SAM Framework and requires 4 REST APIs coupled with 4 Lambda functions (getmyfits, getmyfit, likemysfit, adoptmysfit) which perform QUERY and UPDATE actions on the Mysfits DynamoDB Table also created in the same stack. Since adopt and like mysfit operations are allowed for signed in users only, we need to create a Cognito User Pool and Client to be set as authorizer for those functions.

In the SAM template file, we will provisionne below resources:

DynamoDB Table to store mysfits dynamoDB.png Make sure to set the GlobalSecondaryIndex to allow filtering on GoodEvil and LawChaos attributes.

Cognito User Pool and Client

We need to add an authorizer to our API Gateway in other to authenticate and authorize users before they could like or adopt a mysfit. For that, we first need to create a Cognito user pool and client in our stack.

  • Cognito User Pool UserPool.png

  • Cognito User Pool Client userpoolclient.png

  • Cognito User Pool Domain

userpooldomain.png We can now reference this cognito user pool and client as authorizer for our main HTTP API.

Main HTTP API httpapi.png The authorizer settings has been added to the HTTP API properties. It uses a JWT configuration with our cognito user pool as issuer and user pool client as audience. I have also added the CORS settings to allow GET and POST requests only from the website.

Lambda functions associated to the main HTTPApi

Lambda code is written in Python and performs CRUD operations on the DynamoDB table containing mysfits items based on the event received - in our case the HTTPApi path and request parameters.

I have set the 4 functions below:

  • getmysfits: Retrieve all mysfits for the main page and performs filtering based on GoodEvil or LawChaos value

SAM ressource getmysfits.png Function code

# Returns a list of filtered mysfits based on queryParameters
def queryMysfitItems(filter, value):
    # Use the DynamoDB API Query to retrieve mysfits from the table that are
    # equal to the selected filter values.
    response = client.query(
        TableName='MysfitsTable',
        IndexName=filter+'Index',
        KeyConditions={
            filter: {
                'AttributeValueList': [
                    {
                        'S': value
                    }
                ],
                'ComparisonOperator': "EQ"
            }
        }
    )
    mysfitList = getMysfitsJson(response["Items"])  # getMysfitsJson adds mysfits attributes to a dict that matches the JSON response structure
    return json.dumps(mysfitList)

# Returns all mysfits list   
def getmysfits():
    response = client.scan(TableName='MysfitsTable')
    logging.info(response["Items"])
    mysfitList = getMysfitsJson(response["Items"]) 
    return json.dumps(mysfitList)

def lambda_handler(event, context):
    if (event["rawQueryString"] == ""): # check the presence of queryParameters in the request
        print("Getting all values")
        items = getmysfits()
    else:
        print("Getting filtered values")
        data = event["queryStringParameters"].items()
        for key, value in data:
            items = queryMysfitItems(key, value)
    return {
        'statusCode': 200,
        'headers': {
            'Access-Control-Allow-Origin': '*',
            'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token',
            'Access-Control-Allow-Methods': '*',
            'Content-Type': 'application/json'
        },
        'body': items
    }
  • getmysfit: Return one item based on path parameters {MysfitsId}

SAM resource getmysfit.png

  • likemysfit: Increment the like value for a specified mysfit

SAM resource likemysfit.png

  • adoptmysfit: Update the adopt value to TRUE for a specified mysfit

SAM resource adoptmysfit.png

When writing your code, remember to break down things as much as possible to keep your functions simple. Build your SAM template and test the functions and APIs locally then deploy to AWS (which will create a CloudFormation stack with the resources specified in the template file). Validate your config by testing the APIs with POSTMAN. You can use the AWS hosted UI which provides an OAuth 2.0 authorization server with built-in webpages that can be used to sign up and sign in users (required to like and adopt a mysfit).

At this stage, we have successfully created the microservice needed to serve the frontend (retrieve all mysfits, filter mysfits, like and adopt mysfits). We only need to update the frontend html files by adding the HTTP API url, Cognito user pool and Cognito user pool client.

Backend microservice #2: Enable users to contact the Mythical Mysfits

questions microservice 2.png

This microservice is deployed as a seperate CloudFormation stack and also using the SAM Framework. It requires 1 REST API coupled with 1 Lambda function which writes in a DynamoDB table to allow users to send questions through a form on our website. Once a question is posted in the table, a stream triggers another function which uses SNS SDK to publish the question (received as an event) as a topic message. All resources required are defined in the SAM template file:

DynamoDB Table to store questions questions dynamotable.png DynamoDB table to store users questions. Stream enabled with NEW_IMAGE view type.

Questions SNS Topic questions sns topic.png SNS Topic which to retrieve questions from the dynamodb table and send notification to topic subscribers (Mythical Mysfits staff).

Questions API questions http api.png HTTP Api which will trigger our lambda function to post a question to the questions table. CORS settings has been set to allow GET and POST requests only from our website.

*Lambda functions

  • postquestion(): retrieve posted question from the request body and save it to the questions table SAM resource post question.png

Function code

client = boto3.client('dynamodb')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info(event)
    eventBody = json.loads(event["body"])
    question = {}
    question["QuestionId"] = {
        "S": str(uuid.uuid4())
        }
    question["QuestionText"] = {
        "S": eventBody["questionText"]
        }
    question["UserEmailAddress"] = {
        "S": eventBody["email"]
        }

    response = client.put_item(
        TableName=os.environ['MYSFITS_QUESTIONS_TABLE'],
        Item=question
        ) 
    logger.info(response)   
    responseBody = {}
    responseBody["status"] = "success"
    return {
        'statusCode': 200,
        'headers': {
            'Access-Control-Allow-Origin': 'https://mythicalmysfits.houessou.com',
            'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token',
            'Access-Control-Allow-Methods': 'GET',
            'Content-Type': 'application/json'
        },
        'body': json.dumps(responseBody)  
    }
  • publishquestion(): publish the newly posted question from the questions table to SNS topic SAM resource publish question.png Function code
sns = boto3.resource('sns')
topic = sns.Topic(os.environ['TOPIC_ARN'])
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info(event)
    try:
        for record in event['Records']:
            if record['eventName'] == 'INSERT':
                question = record.get('dynamodb').get('NewImage')
                logger.info(question)
                QuestionText = question["QuestionText"]
                UserEmailAddress = question["UserEmailAddress"]
            response = topic.publish(
                Message = 'FROM EMAIL: ' + UserEmailAddress['S'] + '  QUESTION: ' + QuestionText['S'] ,
                Subject = 'Question from :' + UserEmailAddress['S'] ,
                MessageStructure = 'string'
            )
            print(str(response) + ' has been published!')
        return response
    except Exception as er:
        print(er)
        print('Couldn't publish message to SNS')

Build your SAM template and test the functions and APIs locally then deploy to AWS (which will create a CloudFormation stack with the resources specified in the template file). At this stage, we have successfully created the microservice needed to allow users to post questions. We only need to update the frontend html files by adding the questions HTTP API url.

Microservices 3 and 4 coming soon...

]]><![CDATA[#CloudGuruChallenge - Improve application performance using Amazon ElastiCache]]>https://blogs.houessou.com/cloudguruchallenge-elasticachehttps://blogs.houessou.com/cloudguruchallenge-elasticacheSat, 12 Jun 2021 14:36:50 GMTThe purpose of this challenge is to implement a Redis cluster using Amazon ElastiCache to cache database queries in a simple Python application. More details on the challenge here.

I decided to deploy all the resources as part of a CloudFormation stack with an EC2 Instance preloaded with user data to make sure that my application could be replicated easily. When it comes to CF templates, you need to visualize the whole resources engaged for the application as well as the required prerequisites i.e. AZ, network, security groups, ACLs... Also think about all the parameters like database account and password, environment variables for EC2...

I wrote a simple CF template which creates below resources in a stack:

  • Amazon S3 bucket to retrieve and store application code from GitHub repository. Using only a GitHub repo may be an easier and better approach
  S3Bucket: #S3 Bucket
    DeletionPolicy: Retain
    Type: AWS::S3::Bucket
    Properties:
      BucketName: "S3bucketname"
      Tags:
        - Key: key1
          Value: value1
  • EC2 Instance profile associated with a role with read and write permissions on the S3 bucket - to be able to pull the code on the EC2 instance from S3
  EC2Role: #IAM Role for EC2 instance profile (S3 bucket access)
    Type: AWS::IAM::Role
    DependsOn: S3Bucket
    Properties:
      Tags:
        - Key: key1
          Value: value1
      Description: "description"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: "policyname"
          PolicyDocument:
            Version: "2012-10-17"
            Statement: 
              - Effect: Allow
                Action: 's3:*'
                Resource: 
                  - arn:aws:s3:::s3bucketname
                  - arn:aws:s3:::s3bucketname/*

  EC2InstanceProfile: #EC2 Instance profile
    Type: AWS::IAM::InstanceProfile
    DependsOn: EC2Role
    Properties:
      Path: /
      Roles:
        - !Ref EC2Role
  • Security Group for EC2 instance
  EC2SG: #SecurityGroup for EC2 Instance
    Type: AWS::EC2::SecurityGroup
    Properties: 
      GroupDescription: "SecurityGroup for EC2 Instance - Allow all"
      GroupName: "ec2securitygroupname"
      SecurityGroupIngress:
        - 
          IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          CidrIp: 0.0.0.0/0
      Tags: 
        - Key: key1
          Value: value1
  • Security Group for RDS database and ElastiCache cluster - referencing the EC2 instance SG: Only allow traffic from specified EC2 instance
  DatabaseSG: #SecurityGroup for database
    Type: AWS::RDS::DBSecurityGroup
    DependsOn: EC2SG
    Properties:
      GroupDescription: "description"
      DBSecurityGroupIngress: 
        - EC2SecurityGroupName: !Ref EC2SG #referencing EC2 instance SG as allowed inbound traffic
      Tags: 
        - Key: key1
          Value: value1

  ElastiCacheSG: #SecurityGroup for ElastiCache
    Type: AWS::EC2::SecurityGroup
    DependsOn: EC2SG
    Properties: 
      GroupDescription: "description"
      GroupName: "securitygroupname"
      SecurityGroupIngress: 
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupName: !Ref EC2SG #referencing EC2 instance SG as allowed inbound traffic
      Tags: 
        - Key: key1
          Value: value1
  • RDS instance with PostgreSQL as engine
DBInstance: #RDS database Instance
    Type: AWS::RDS::DBInstance
    DependsOn: 
      - DatabaseSG
    Properties:
      Engine: "postgres"
      DBInstanceIdentifier: "identifiername"
      DBInstanceClass: db.t2.micro
      AllocatedStorage: 20
      DBName: "dbname"
      MasterUsername: !Ref DatabaseAccount #referencing the master user account parameter
      MasterUserPassword: !Ref DatabasePassword #referencing the master password parameter
      StorageType: gp2
      MaxAllocatedStorage: 20
      DBSecurityGroups: 
        - !Ref DatabaseSG
      Tags:
        - Key: key1
          Value: value1
  • ElastiCache for Redis cluster for caching
  ElastiCache: # ElastiCache for Redis Cluster
    Type: AWS::ElastiCache::CacheCluster
    DependsOn: ElastiCacheSG
    Properties:
      ClusterName: "redisclustername"
      CacheNodeType: cache.t2.micro
      Engine: Redis
      VpcSecurityGroupIds:
        - !GetAtt ElastiCacheSG.GroupId #referencing the ElastiCache SG created above
      NumCacheNodes: 1
      Tags:
        - Key: key1
          Value: value1
  • EC2 Instance with bootstrap code to install prerequisites and run the python application.
  EC2: #EC2 Instance for app
    Type: AWS::EC2::Instance
    DependsOn: 
      - EC2SG
      - EC2InstanceProfile
      - ElastiCache
      - DBInstance 
    Properties: 
      ImageId: ami-0aeeebd8d2ab47354
      InstanceType: t2.micro
      KeyName: keyname #specify an existing key name
      SecurityGroups:
        - !Ref EC2SG
      IamInstanceProfile: !Ref EC2InstanceProfile
      UserData: 
        !Base64 |
        #!/bin/bash
        sudo mkdir /home/app
        sudo aws s3 cp s3://hpf-acg-elasticache/app/ /home/app --recursive
        sudo yum -y update
        sudo yum -y install python3
        sudo yum -y install postgresql
        sudo cp /home/app/config/.pgpass /home/ec2-user/.pgpass
        sudo chmod 0600 /home/ec2-user/.pgpass
        sudo chown ec2-user:ec2-user /home/ec2-user/.pgpass
        export PGPASSFILE='/home/ec2-user/.pgpass'
        export REDIS_URL=redis://redisendpoint_URL:6379
        psql -h rdsendpointURL -U postgres -f /home/app/install.sql databasename
        cd /home/app
        python3 -m venv /home/app
        source /home/app/bin/activate
        python3 -m pip install --upgrade pip
        pip install -r requirements.txt 
        python3 /home/app/app.py
      Tags:
        - Key: key1
          Value: value1
        - Key: "Name"
          Value: "ec2instancename"
  • As parameters, I provided the RDS database account username and password plus Redis endpoint URL as environment variable
Parameters:
  DatabaseAccount: #DB master account
    Description : "The database admin account. Default is postgres"
    Type : String
    Default: postgres
    MinLength : 1
    MaxLength : 41
    AllowedPattern : ^[a-zA-Z0-9]*$
  DatabasePassword: #DB master account password
    NoEcho: True
    Description : "The database admin account password"
    Type : String
    MinLength : 1
    MaxLength : 41
    AllowedPattern : ^[a-zA-Z0-9]*$

CloudFormation template diagram CF.jpg

To be able to access the app via the public IP of the EC2 instance, add the code below to the user data and copy the /app/config/nginx-app.conf file to the /etc/nginx/conf.d/ folder of your EC2.

        sudo amazon-linux-extras install nginx1
        sudo chmod -R 755 /home/app
        sudo chown -R ec2-user:nginx /home/app
        sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf-orig
        sudo cp /home/app/config/nginx-app.conf /etc/nginx/conf.d/nginx-app.conf
        sudo systemctl start nginx
        sudo systemctl enable nginx

This CloudFormation template is not perfect and not complete either. A lot more parameters can be added as well as mappings.

Now that the app is running, you can see the Elapsed time is always above 5 secondes: 5.09140s, 5.08107s, 5.06325s...

Lets add the caching layer with the following settings:

  • Check the Redis cache before querying the database.
  • If a cache miss occurs, query the database and update the cache with the results.

I updated the app code by first isolating the RDS query process in a separate function with sql code as parameter:

def query(sql):
    # connect to database listed in database.ini
    conn = connect()
    if(conn != None):
        cur = conn.cursor()
        cur.execute(sql)
        # fetch one row
        retval = cur.fetchone()
        # close db connection
        cur.close() 
        conn.close()
        print("PostgreSQL connection is now closed")
        return retval
    else:
        return None

Next step is the Redis cache initialization:

# Read the Redis credentials from the REDIS_URL environment variable.
REDIS_URL = os.environ.get('REDIS_URL')

# Initialize the cache
cache = redis.Redis.from_url(REDIS_URL)

# Time to live for cached data - 10 seconds for this example
TTL = 10

Finally, I added a fetch function to check Redis cache first and update Redis in case of a miss:

def fetch(sql):
    result = cache.get(sql)
    if result:
        return json.loads(result)
    result = query(sql)
    cache.setex(sql,TTL, json.dumps(result)) # if result not found in redis, update redis cache
    return result

Let's run the app:

  • The first request also take above 5 seconds to complete.
  • From the fourth request we notice an elapsed time of 0.00150 second and lower
  • After waiting 10 seconds (which is the value of the cache TTL), the time elapsed is back to 5+ seconds

Before Redis before redis.png

After Redis after redis.png after redis2.jpg

You can find more information on improving application performance with ElastiCache on AWS docs.

This challenge was very fun to work on. I had a good time practicing on CF templates to deploy the app. I will further my learnings by trying to implement ElastiCache for session-store.

]]><![CDATA[#CloudGuruChallenge – Event-Driven Python on AWS]]>https://blogs.houessou.com/cloudguruchallenge-event-driven-python-on-awshttps://blogs.houessou.com/cloudguruchallenge-event-driven-python-on-awsTue, 01 Jun 2021 21:56:06 GMTI started this challenge as part of my training after completing last month's Azure Resume challenge. This one was particularly challenging for me because I had no such experience in Python when starting the project. I learnt on the go with a lot of resources from stack overflow and AWS docs. The goal of this challenge is to automate an ETL processing pipeline for COVID-19 data using Python and cloud services (AWS here). You can find the steps for this #CloudGuruChallenge here.

Let's begin!

Extract and Transform

The first part of this challenge is all about importing and manipulating data from 2 csv files and importing selected data to a DynamoDB Table. After a lot of googling, I decided to use pandas dataframes to store the csv data and perfom the required transformation (conversion of the date field in a date object, joining data from 2 dataframes, removing non-US data...). The csv import and data transformation were handled separately by 2 different modules.

  • import_csv
import ssl
import pandas
ssl._create_default_https_context = ssl._create_unverified_context

def usdata():
    uscovid19 = []
    csvurl = 'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us.csv'
    uscovid19 = pandas.read_csv(csvurl, delimiter=',')
    #print (uscovid19)
    return uscovid19

def recovered():
    recovered = []
    recurl = 'https://raw.githubusercontent.com/datasets/covid-19/master/data/time-series-19-covid-combined.csv'
    recovered = pandas.read_csv(recurl, delimiter=',', usecols=[0,1,4])
    recovered = recovered[recovered['Country/Region'] == 'US']
    recovered = recovered.drop('Country/Region', axis = 'columns')
    #print (recovered)
    return recovered
  • data transformation
import import_csv
import pandas

def main():
    try:
        uscovid19 = import_csv.usdata()
        recovered = import_csv.recovered()

        uscovid19['date'] = pandas.to_datetime(uscovid19['date'])
        recovered['Date'] = pandas.to_datetime(recovered['Date'])

        alldata = pandas.merge(uscovid19, recovered, left_on='date', right_on='Date')
        alldata = alldata.drop('Date', axis='columns')

        return alldata
    except Exception as er:
        print(er)

The data transformation module calls import_csv and returns the combined csv data as a pandas.dataframe

Load data into DynamoDB

First I created a DynamoDB table with date (String) as the partition key and enabled DynamoDB Streams.

Screen Shot 2021-06-01 at 8.58.29 PM.png

Since this code will be triggered by a CloudWatch Rule, the data load process has to be conditional depending on the count of items in the table compared to the count of items in the dataframe.

  • Initial data load (no items in DynamoDB table)

I got the count of items by scanning the table dynamotable.scan(Select = 'COUNT')['Count']. When the count returns 0, the code bellow is triggered:

def initial_load(dataframe, dynamotable):
    try:
        datacount = int(len(dataframe.index)) 
        for row in range(0, datacount):
            date = str(dataframe.loc[row, 'date'])
            cases = str(dataframe.loc[row, 'cases'])
            deaths = str(dataframe.loc[row, 'deaths'])
            Recovered = str(dataframe.loc[row, 'Recovered'])
            dynamotable.put_item(
                Item={
                    'date': date,
                    'cases': cases,
                    'deaths': deaths,
                    'Recovered': Recovered
                }
            )
        print ('First load of Items completed successfully')
        finalcount = dynamotable.scan(Select = 'COUNT')['Count'] 
        response = finalcount 
        return response
    except Exception as er:
        print(er)
  • Appending the table with new items

When the count of items in the dataframe is greater than the count of items in the table, the code bellow is triggered to add only new items (new days). This code is not complete for me, as it only add new items after the last item in the table. No control has been implemented to check and add new items at any other row than the last one.

def append_data(dataframe, dynamotable):
    try:
        dynamocount = int(dynamotable.scan(Select = 'COUNT')['Count'])
        datacount = int(len(dataframe.index)) 
        for row in range(dynamocount, datacount):
            date = str(dataframe.loc[row, 'date'])
            cases = str(dataframe.loc[row, 'cases'])
            deaths = str(dataframe.loc[row, 'deaths'])
            Recovered = str(dataframe.loc[row, 'Recovered'])
            dynamotable.put_item(
                    Item={
                        'date': date,
                        'cases': cases,
                        'deaths': deaths,
                        'Recovered': Recovered
                    }
                )       
        dynamotable.scan()
        print ('DynamoDB table updated successfully')
        finalcount = dynamotable.scan(Select = 'COUNT')['Count'] 
        response = finalcount - dynamocount
        return response
    except Exception as er:
        print(er)

Both codes return the number of added items to the table (just to be able to log it). Below is the main code which calls the initial load or the append data modules based on the items count. I also added a part for no new items in dataframe vs table.

def load_data(dataframe, dynamotable):
    print('---------------------')
    print('Begining data Laod')
    print('---------------------')
    dynamocount = int(dynamotable.scan(Select = 'COUNT')['Count'])
    datacount = int(len(dataframe.index))
    count = datacount - dynamocount
    print('Items in DynamoDB table: ' + str(dynamocount))
    print('Items in dataframe: ' + str(datacount))
    if dynamocount == 0:
        print('Table first load of items')
        response = initial_load(dataframe, dynamotable)
        print( str(response) + ' ITEMS CREATED')
    elif dynamocount == datacount:
        print('NO NEW ITEMS')
    elif dynamocount != datacount:
        print('Appending table with new items')
        response = append_data(dataframe, dynamotable)
        print( str(response) + ' ITEMS ADDED')
    print('---------------------')
    print ('Data load completed!!')
    print('---------------------')

Those modules are part of the same package which has been deployed to AWS Lambda as a function. The handler calls the load_data module with the DynamoDB table and the dataframe as arguments. Running the function in Lambda resulted in several errors due to missing numpy dependencies even-though pandas and numpy lib were added to the function package . Numpy is required when using pandas in Python. I eventually came this post with the required python deployment package to run pandas and numpy in AWS Lambda with python 3.6 runtime. I just had to add my modules and it worked just fine. Since this function update the DynamoDB table, don't forget to set appropriate permission to the execution role.

CloudWatch or EventBridge rule to trigger ETL function

Here I simply created a CloudWatch rule to trigger my function on a schedule. You will need to add a resource based policy to the function to allow lambda:InvokeFunction permission. Screen Shot 2021-06-01 at 9.25.43 PM.png

Notification

When the database has been updated, the code should trigger an SNS message to notify any interested consumers that the ETL job has completed. The message should include the number of rows updated in the database.

  • Lambda function to publish message to SNS topic

I created another function triggered by DynamoDB streams with below configuration: Screen Shot 2021-06-01 at 9.36.16 PM.png

The code run through the events (DynamoDB stream) and get the total number of items added or deleted based on event name and then publish a message to SNS topic.

def get_stream_insert(event, context):
    try:
        new_rows_count = 0
        for record in event ['Records']:
            if record['eventName'] == 'INSERT':
                new_rows_count += 1
        return new_rows_count
    except Exception as er:
        print(er)

def get_stream_remove(event, context):
    try:
        del_rows_count = 0
        for record in event ['Records']:
            if record['eventName'] == 'REMOVE':
                del_rows_count += 1
        return del_rows_count
    except Exception as er:
        print(er)

def main(event, context):
    try:
        sns = boto3.resource('sns')
        topic = sns.Topic('arn:aws:sns:us-east-1:XXXXXXX:xxxxxxx')
        newItems = str(get_stream_insert(event, context))
        delItems = str(get_stream_remove(event, context))
        response = topic.publish(
            Message = 'The ETL job is completed. The table has ' + newItems + ' new items and ' + delItems + ' deleted items.' ,
            Subject = 'ETLCovid19 job status',
            MessageStructure = 'string'
        )
        print(str(response) + ' has been published!')
        return response
    except Exception as er:
        print(er)
        print('Couldnt publish message to SNS')

IaC

This part is about defining the created resources in code using CloudFront. I created a yaml template to import all the resources in a CloudFormation stack. Very useful guide on CF here.

Resources:
#Import S3 bucket
  ETLCovid19Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: "BucketName"
      Tags:
        - Key: "xxxxxxxx"
          Value: "xxxxxxxx"

#Import DynamoDB Table to store data
  ETLCovid19Table:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain
    Properties:
      TableName: "TableName"
      BillingMode: PROVISIONED
      AttributeDefinitions: 
        -
          AttributeName: "date"
          AttributeType: "S"
        -
          AttributeName: "cases"
          AttributeType: "S"
        - 
          AttributeName: "deaths"
          AttributeType: "S"
        - 
          AttributeName: "recovered"
          AttributeType: "S"
      KeySchema: 
        - AttributeName: "date"
          KeyType: "HASH"
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
      Tags:
        - Key: "xxxxxxxx"
          Value: "xxxxxxxxxxx"

#Import ETLCovid19 Lambda Function
  ETLCovid19Function:
    DependsOn:
      - ETLCovid19Bucket
      - ETLCovid19Table
    Type: AWS::Lambda::Function
    DeletionPolicy: Retain
    Properties:
      FunctionName: "FunctionName"
      Handler: "ETLCovid19.main"
      Role: "arn:aws:iam::XXXXXXXXXXXXXX:role/service-role/xxxxxxxxx"
      Code:
        S3Bucket: "xxxxxxxxxxxxxx"
        ZipFile: "xxxxxx.zip"
      Runtime: "python3.6"
      Tags:
        - Key: "xxxxxxxxxx"
          Value: "xxxxxxxxxxx"

#Import ETLTriggerSNS Lambda Function
  ETLCovid19SNSFunction:
    DependsOn:
      - ETLCovid19Bucket
      - ETLCovid19Table
    Type: AWS::Lambda::Function
    DeletionPolicy: Retain
    Properties:
      FunctionName: "FunctionName"
      Handler: "trigger_sns.main"
      Role: "arn:aws:iam::XXXXXXXXXXXXXX:role/service-role/xxxxxxxxxxxxxxxx"
      Code:
        S3Bucket: "xxxxxxxxxx"
        ZipFile: "xxxxxxx.zip"
      Runtime: "python3.8"
      Tags:
        - Key: "xxxxxxxx"
          Value: "xxxxxxxxxxx"

#Import SNS Topic for notification
  ETLCovid19Topic:
    DependsOn:
     - ETLCovid19SNSFunction
    Type: AWS::SNS::Topic
    DeletionPolicy: Retain
    Properties:
      TopicName: "TopicName"
      Subscription:
        - Endpoint: "xxx@xxxxx.xx"
          Protocol: "email"
      Tags:
        - Key: "xxxxxxxxx"
          Value: "xxxxxxxxx"

#Import SQS queue for retries
  ETLCovid19Queue:
    DependsOn:
      - ETLCovid19Topic
    Type: AWS::SQS::Queue
    DeletionPolicy: Retain
    Properties:
      QueueName: "QueueName"
      VisibilityTimeout: 30
      Tags:
        - Key: "xxxxxxxx"
          Value: "xxxxxxxxxx"

#Import CloudWatch Rule
  ETLCovid19CloudWatchRule:
    DependsOn: ETLCovid19Function 
    Type: AWS::Events::Rule
    DeletionPolicy: Retain
    Properties: 
      Description: "Rule to trigger Function ACG_ETLCovid19"
      Name: "Name"
      ScheduleExpression: "rate(5 days)"
      State: ENABLED
      Targets: 
        - Arn: "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:xxxxxxxxxx"
          Id: "xxxxxxxxxxxx"

Here's a template view in the Designer Screen Shot 2021-06-01 at 5.53.29 PM.png

Imported resources Screen Shot 2021-06-01 at 5.32.09 PM.png

CI/CD

Deployed my functions to GitHub repository for change control and automatic deployment to AWS Lambda on test pass with GitHub actions. The repo structure is as below:

- Repository folder
  - backend
    - Function-1-folder
    - Function-2-folder
    - CloudFormation

Test files for each function are located in the function folder. Running python test using Nose2. Below are the code for the workflows.

  • Update on function 1

    name: Test and Deploy to Function 1
on: 
push:
    branches: [ main ]
    paths:
      - 'backend/Function-1-folder/**'          
jobs:
deploy_source:
  name: Test and Deploy to Lambda
  runs-on: ubuntu-latest
  steps:
    - name: checkout source code
      uses: actions/checkout@v1

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: 'us-east-1'

    - name: Setup Python 3.6 Environment
      uses: actions/setup-python@v1
      with:
        python-version: 3.6

    - name: 'Test functions with Nose2'
      run: |
        pushd './backend/Function-1-folder/'
        pip install awscli
        python -m pip install --upgrade pip
        pip3 install pandas
        pip3 install numpy
        pip3 install nose2
        pip3 install boto3    
        python -m nose2 test_import_csv.test_import_csv test_transformation.test_transformation

    - name: Install zip
      uses: montudor/action-zip@v1

    - name: Zip Package
      run: zip -qq -r Function1.zip .
      working-directory: ./backend/Function-1-folder   

    - name: Deploy to Lambda
      uses: appleboy/lambda-action@master
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: 'us-east-1'
        function_name: Function1
        zip_file: ./backend/Function-1-folder/Function1.zip

    - name: Upload package to S3 bucket
      uses: qoqa/action-s3-cp@v1.1
      env:
        AWS_REGION: 'us-east-1'
        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_S3_PATH: '/backend/Fucntion1.zip'
        FILE: './backend/Function-1-folder/Function1.zip'

  • Update on function 2

Paste the same code as above and modify as appropriate.

Results

Below is a dashboard created in QuickSight using the data in the DynamoDB table. This article explains how to visualize DynamoDB data in QuickSight using Athena data connectors.

Screen Shot 2021-06-01 at 10.35.04 PM.png

]]><![CDATA[#CloudGuruChallenge – Cloud Resume]]>https://blogs.houessou.com/cloudguruchallenge-cloud-resumehttps://blogs.houessou.com/cloudguruchallenge-cloud-resumeMon, 24 May 2021 00:20:06 GMTThe goal of this challenge is to create your resume hosted on Azure or AWS. I recently achieved the AWS Solutions Architect certification and I was so excited about it that I decided to go for the SysOps exam. Learning for the exam, I came across this video on youtube talking about How many certs were needed to get a cloud job. You may have guessed it: hands-on experience and aptitude is your most important asset. That's the reason why I decided to take this challenge I first saw on A Cloud Guru.

I completed the challenge both on AWS and Azure.

Azure

1- First created a CosmosDB account (SQL Core) with a container to store the site visits count

CosmosDB.png

2- The second part was to create a Function to retrieve and update the value of the visits count. Decided to go with Python but I can say that this language is note really appropriate to use with Azure Functions. You have to create you package outside of Azure + not so many sample for CRUD operations on CosmosDB. Definitely had some difficulties here but eventually sort them out with this article. Bellow is a part of the code of the function I deployed to Azure using vscode.

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')
    getcount = container.read_item(item='visits', partition_key='visits')
    getcount['count'] = getcount['count'] + 1
    siteVisited = container.upsert_item(body=getcount)

    #print('count: {0}'.format(response.get('count')))
    counts = '{0}'.format(siteVisited.get('count'))

    return func.HttpResponse ( 
        status_code=200,
        headers = {
            'Access-Control-Allow-Origin': '*',
            'Access-Control-Allow-Headers': 'Content-Type,Date,x-ms-session-token',
            'Access-Control-Allow-Credentials': 'false',
            'Content-Type': 'application/json'
        },
        body = counts
    )

This function returns only the count value. Don't forget to add the response headers in the return statement. One thing I like on Azure is the fact that your HTTP trigger functions come with a publicly accessible URL directly (as opposed to AWS where you have to separately create the API as a trigger for the function). Now you have to properly configure CORS for the function to be triggered by the API.

3- When I was sure that my function was working and returning the visits count value through GET method, now was the part for the frontend website. Downloaded a template from themezy and updated it with my resume information. At this point, the challenge was to write the JavaScript code to retrieve the count value. I did a lot of googling here. I knew I had to use the fetch API but I wasn't getting it to work. Finally came out with this code:

const url = 'your function URL';
fetch(url, {
    method: "GET",
    headers: {
        "Content-Type": "application/json",
        "Accept": "application/json"
    },
})
.then(response => {
    return response.json()
})
.then(response => {
    console.log("Calling API to get visits counts");
    nbvisited = response;
    console.log(nbvisited);
    document.getElementById('nbvisited').innerText = nbvisited;
})
.catch(function(error) {
  console.log(error);
});

4- The next step was to create an Azure Blob storage setup as a static website. Then create a CDN with the blob storage as the origin and configure HTTPS with your custom domain. You need to look at the CORS settings of the blob storage as well.

5- CI/CD with GitHub

I have deployed my project to a GitHub repository and created workflows to update the frontend (update blob storage and purge Azure CDN) on push. From Azure functions App, you can configure CI/CD with GitHub as deployment center. You function will be updated on Azure when modified on GitHub. However, the created workflow doesn't include code testing (because I use Python). So I added a testing step in the workflow file: testing using nose2. If you use Python as well, make sure that your functions are importable as module otherwise nose2 will not pick them.

Workflow for backend

name: Test and Deploy function to Azure Function App

on:
  push:
     branches: [ main ]
     paths:
        - 'backend/**'
  workflow_dispatch:

env:
  AZURE_FUNCTIONAPP_PACKAGE_PATH: 'backend' # set this to the path to your web app project, defaults to the repository root
  PYTHON_VERSION: '3.8' # set this to the python version to use (supports 3.6, 3.7, 3.8)

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: 'Checkout GitHub Action'
      uses: actions/checkout@v2

    - name: Setup Python ${{ env.PYTHON_VERSION }} Environment
      uses: actions/setup-python@v1
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: 'Resolve Project Dependencies Using Pip'
      shell: bash
      run: |
        pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
        python -m pip install --upgrade pip
        pip install -r requirements.txt --target=".python_packages/lib/site-packages"
        popd

    - name: 'Test functions with Nose2'
      shell: bash
      run: |
        pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
        python -m pip install --upgrade pip
        pip install nose2
        pip install azure-functions
        pip install azure-cosmos
        python -m nose2

    - name: 'Run Azure Functions Action'
      uses: Azure/functions-action@v1
      id: fa
      with:
        app-name: 'hpfgetresumecounter'
        slot-name: 'production'
        package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
        publish-profile: ${{ secrets.AzureAppService_PublishProfile_a7603b82524748699547273836b49631 }}

6- Results

Visit my cloud resume on Azure here.

You can checkout my GitHub repository for this project.

AWS

I actually completed this challenge on AWS before doing it is Azure. I tend to prefer AWS as for me it more easily understandable. The steps are pretty much identical except for the better Python integration on AWS

1- Created a DynamoBD table to store my website visits count as item. Straight forward here, just give your table a name and specify the primary key. No need to set API and data model.

2- Created a Lambda function to update the visits count in DynamoDB table and return the value of the count. Also make sure to mention the appropriate headers in the return statement. You need to set an execution role for the function which has read and write access to your DynamoDB table.

import json
import boto3

dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    table = dynamodb.Table('NameofTable')
    siteVisited = table.update_item(
        Key={
            'id': 'visits'
        },
        UpdateExpression='SET counts = counts + :val',
        ExpressionAttributeValues={
            ':val': 1
        },
        ReturnValues="UPDATED_NEW"
    )
    getCount = table.scan()
    counts = str(siteVisited['Attributes']['counts'])
    return {
        'statusCode': 200,
        'headers': {
            'Access-Control-Allow-Origin': '*',
            'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token',
            'Access-Control-Allow-Credentials': 'false',
            'Content-Type': 'application/json'
        },
        'body': json.dumps(counts)
    }

This function also returns only the count value.

3- Created an API Gateway trigger for the function. Make sure to give lambda:InvokeFunction permission for the function to the API Gateway. Also check CORS settings in case of issue.

4- From there, it is almost the same with Azure:

  • JavaScript code to fetch the API is identical. Just put the URL of the AWS API Gateway

  • Created an S3 bucket to store the website files. No need to set it as static website or publicly available

  • Created a CloudFront distribution with the S3 bucket as origin. Configure OAI to allow access to the bucket only from the CloudFront distribution (you have to create an identity)

  • Configured a custom domain with custom domain AWS managed SSL certificate for HTTPS

5- CI/CD with GitHub

I have uploaded my project to a GitHub repository and created workflows to update the frontend (update S3 bucket and invalidate CloudFront distribution) on push. Uploaded the Python function package to the backend folder of the repository and created a workflow file to test and deploy it to Lambda. Testing using nose2. If you use Python as well, make sure that your functions are importable as module otherwise nose2 will not pick them.

Workflow for backend

name: Test and Deploy to Lambda
on: 
  push:
      branches: [ main ]
      paths:
        - 'backend/**'

jobs:
  deploy_source:
    name: Test and Deploy to Lambda
    runs-on: ubuntu-latest
    steps:
      - name: checkout source code
        uses: actions/checkout@v1

      - name: 'Configure AWS credentials'
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: 'us-east-1'

      - name: 'Test functions with Nose2'
        run: |
          pushd './backend'
          pip install awscli
          python -m pip install --upgrade pip
          pip install nose2
          pip install boto3
          python -m nose2

      - name: Deploy to Lambda
        uses: appleboy/lambda-action@master
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: 'us-east-1'
          function_name: HPFResumeVisitCounter
          source: backend/HPFResumeVisitCounter/GetResumeCounter.py

6- Results

Visit my cloud resume on AWS here.

You can checkout my GitHub repository for this project.

]]>