Enhancing Performance with Amazon DynamoDB: A Personal Journey
Written on
Chapter 1: Introduction to DynamoDB
Recently, I embarked on a personal project to delve deeper into Amazon DynamoDB. While I had dabbled with DynamoDB for minor tasks previously, my main expertise lies in relational databases, making the transition to NoSQL quite a learning curve.
What Exactly is DynamoDB?
DynamoDB is Amazon Web Services' fully managed NoSQL database solution. Unlike traditional relational databases that organize data in structured rows and columns, DynamoDB stores information in key-value pairs. This allows for a more flexible data model, where the only required schema consists of the table's keys: the partition key and the sort key. After that, your data can remain schema-less, though your application must be equipped to handle it appropriately.
My Early Misstep
In my project, I utilized Python and accessed DynamoDB through the boto3 library, all running on AWS Lambda. I developed a data model called collection to facilitate basic CRUD (Create, Read, Update, Delete) operations.
Here’s an excerpt of my initial code:
# Example of initial code
Can you spot the issue?
Each time I performed a database operation, I was re-initializing the boto3 resource and the DynamoDB table. This oversight was easy to make, especially with GitHub Copilot assisting in writing these functions. It wasn't until my fiancé, Myles Loffler, reviewed my code that he highlighted this as an anti-pattern, particularly in the context of using boto3 clients within Lambda functions.
Note: This anti-pattern can arise whenever you're repeatedly re-initializing the same boto3 resource, not just with DynamoDB.
How I Resolved It
Fortunately, the fix was both straightforward and enhanced the overall quality of my code! I created a new utility file to manage my database connections and incorporated the following functions:
# Example of new utility functions
Next, I refactored my CRUD functions to utilize get_dynamo_table instead of directly calling boto3.
# Example of refactored CRUD functions
You might have noticed the use of global variables in my database utility. Although this is often frowned upon, I employed them based on Lambda best practices that recommend leveraging execution environment reuse.
The Impact of Changes
These modifications resulted in four significant improvements to my code:
Performance
This was the most notable benefit! To validate the impact, I deployed two versions of the application: one with the original code and another with the revised code, testing workflows that involved CRUD operations. Below are traces of my application before implementing the changes, which included deleting an item and running a query:
The delete action took 84 ms, while the query took 76 ms. Here are the traces after applying the changes:
After the changes, the delete and query actions took only 7 ms and 12 ms, respectively. Although these are small time differences, they accumulate and contribute to a significantly faster and more responsive application.
Code Clarity
This new approach enhances readability and will ultimately reduce the amount of code in my project. I no longer need to import boto3 or access environment variables each time I interact with the database. With a single function for DynamoDB access, any future changes will only need to be made in one location.
Testability
Testing became cleaner as well. Previously, I had to mock boto3 at multiple locations, which was complex and cumbersome. Now, I can simply mock get_dynamo_table, making the tests much more straightforward.
Reusability
The new utility functions are agnostic to the table name, which is retrieved through the get_dynamo_table_name function in settings. As my project expands, this code will be reusable across other Lambda functions that may utilize different DynamoDB tables.
As I mentioned earlier, I am still relatively new to Amazon DynamoDB and continue to learn!
Chapter 2: Learning Resources
To deepen your understanding of DynamoDB and best practices, consider watching the following videos:
This session from AWS re:Invent 2021 explores advanced design patterns in DynamoDB, providing valuable insights for developers.
In this follow-up from AWS re:Invent 2020, additional advanced design patterns for Amazon DynamoDB are discussed, which can enhance your application development strategies.