Vivek Shukla - Programming

Setup Redis TLS With Self-signed Certificate

2025-09-08T00:00:00+00:00

This method is only recommended when connecting to redis server via local network.

Generate Server Certificates

Generate Cert and Key file for self signed TLS, then move both these files to redis directory and set the proper permissions. Delete the Key file and copy the cert file to the client machine which will connect to the redis-server.

cd ~

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout redis_tls.key -out redis_tls.crt \
  -subj "/CN=YourOrg"

sudo cp redis_tls.key redis_tls.crt /etc/redis/ && rm redis_tls.key
sudo chmod 600 /etc/redis/redis_tls.key
sudo chmod 644 /etc/redis/redis_tls.crt
sudo chown redis:redis /etc/redis/redis_tls.key
sudo chown redis:redis /etc/redis/redis_tls.crt

Update Redis Configuration File

This command will append the value to redis.conf (redis configuration file). Make sure to update the password.

It will disable the non-tls support and enable the tls connection at port 6379 with previously generated key and cert files.

sudo tee -a /etc/redis/redis.conf << EOF
bind * -::*
port 0
tls-port 6379
tls-cert-file /etc/redis/redis_tls.crt
tls-key-file /etc/redis/redis_tls.key
tls-auth-clients no
tls-prefer-server-ciphers yes
requirepass YOUR_PASSWORD_HERE
EOF

Restart and enable redis

sudo systemctl restart redis-server
sudo systemctl enable redis-server

Check status with

sudo systemctl status redis-server

If any error then check journal (log)

sudo journalctl -u redis-server -f

Add Redis Certificate to Trusted CA Store in Client Machine

# Copy your certificate to CA certificates directory
sudo mv redis_tls.crt /usr/local/share/ca-certificates/redis_tls.crt

# Update CA certificates
sudo update-ca-certificates

Redis Connection URL

Since we want to connect to redis using TLS connection, so the protocol in the url will be rediss (with double “ss”).

rediss://:<your_password_here>@<redis_server_ip>:6379/0#insecure

How I Use Cargo Workspace to structure my Rust projects

2025-09-03T00:00:00+00:00

I’ve learn to use Cargo Workspace from Olivia’s post in bluesky, which you should checkout. This article is a way of self-documenting since I’ve come back to it more often.

What is a Cargo Workspace

A cargo workspace is collection of one or more library or binary crates. Cargo workspace let’s you break your rust project into multiple different crates (or packages), which may aid you in better project structure and management.

Why use Cargo Workspace

Your rust project has grown big and is hard to manage
Multiple people work on a project, so breaking your project into multiple crates can help with better ownership and less conflict among developers
You want to reduce your project’s compile time

Read this (external link): Cutting Down Rust Compile Times From 30 to 2 Minutes With One Thousand Crates

Creating a Workspace

Create a new workspace directory

mkdir runner
cd runner

Now create Cargo.toml with following content

[workspace]
resolver = "3"

Here we are using resolver version 3, which is the latest version and comes in Rust 2024 edition (Rust 1.84+). Resolver tells cargo how to resolve the dependencies.

Create a binary crate

We are going to use cargo new command to create a binary crate. When no argument is passed, cargo new command will create binary crate by default, you can use --lib as argument to create library crate or --bin to explicit that you want binary crate.

cargo new run

Now your workspace will look like this:

.
├── Cargo.toml
└── run
    ├── Cargo.toml
    └── src
        └── main.rs

If you look at your workspace’s Cargo.toml you will notice that cargo has already added the newly added crate run against the members key.

[workspace]
resolver = "3"
members = ["run"]

Adding lib crates

Let’s add two lib crates

cargo new participant --lib
cargo new track --lib

Updated structure:

.
├── Cargo.lock
├── Cargo.toml
├── participant
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── run
│   ├── Cargo.toml
│   └── src
│       └── main.rs
└── track
    ├── Cargo.toml
    └── src
        └── lib.rs

Adding dependencies

If you try to simply add a dependency like this

cargo add uuid --features v4

‼️ it will fail and you will see this error

error: `cargo add` could not determine which package to modify. Use the `--package` option to specify a package. 
available packages: participant, run, track

Since inside our Cargo workspace we have 3 packages, therefore cargo is unable to determine to which package dependency belongs to. ✅ We can solve this by using --package arg.

cargo add uuid --features v4 --package participant

Dependency Resolution

Since in cargo workspace we can have multiple crates, which means we have to add dependency to each crate separately and in that sometimes there may be a version mismatch.

Like with uuid in participant crate, it is using version 1.18.1 and now you if you try to add uuid of different version in run crate, for example if we want to stick to only 1.17 version of the uuid crate:

cargo add uuid@=1.17 --features v4 --package run

But wait, we got error

error: failed to select a version for `uuid`.
    ... required by package `run v0.1.0 (/home/vivek/Documents/code/hands-on/runner/run)`
versions that meet the requirements `=1.17` are: 1.17.0

all possible versions conflict with previously selected packages.

  previously selected package `uuid v1.18.1`
    ... which satisfies dependency `uuid = "^1.18.1"` of package `participant v0.1.0 (/home/vivek/Documents/code/hands-on/runner/participant)`

failed to select a version for `uuid` which could resolve this conflict

It’s because a Cargo workspace can only have one Cargo.lock file, which is shared between all the members of the workspace, so it can only have one version of the dependency across the workspace. So instead of pinning to =1.17 version of uuid, we had used just 1.17, cargo would have choosen uuid@1.18.1 since that was already being used by participant crate.

In order to resolve the dependency across workspace, cargo will try to find the most suitable version of a crate, mostly by downgrading to the appropriate version.

Better way to declare dependency

It’s very normal to have common dependencies across crates, that is why Cargo Workspace let us declare dependencies in workspace’s Cargo.toml and then you can re-use the same dependency inside of your crate.

# Cargo.toml

[workspace.dependencies]
uuid = { version = "=1.17", features = ["v4"] }

And then in your crates you can use the dependency with:

# participant/Cargo.toml
[dependencies]
uuid = { workspace = true }

# run/Cargo.toml
[dependencies]
uuid = { workspace = true }

Importing members as dependencies

A common way to import a crate (or package) inside another crate is like this:

# run/Cargo.toml
[dependencies]
participant = { path = "../participant" }
track = { path = "../track" }

# track/Cargo.toml
[dependencies]
participant = { path = "../participant" }

But now if you had to do this for each of the crate in your project and sometime there can be a lot of crates. Also the path is relative to the current Cargo.toml file, so if you dare re-structure your workspace, it can backfire and you will have another mess to deal with.

However we can use the same way of delcaring the dependency in the workspace’s Cargo.toml and then just use it inside the members by telling cargo to use workspace’s dependency.

# Cargo.toml
[workspace]
resolver = "3"
members = ["participant", "run", "track"]

[workspace.dependencies]
participant = { path = "participant" }
track = { path = "track" }

# run/Cargo.toml
[dependencies]
participant = { workspace = true }
track = { workspace = true }

# track/Cargo.toml
[dependencies]
participant = { workspace = true }

Multiple Binaries

When you have multiple binaries in your workspace, cargo run won’t work since it doesn’t know which binary crate to run. So instead explicitly pass the crate name:

cargo run --package my_bin_crate

Cheat sheet

Create workspace:

# Cargo.toml
[workspace]
resolver = "3"

Adding bin and lib to workspace:

# creates binary crate
cargo new my_bin

# crates lib crate
cargo new my_lib --lib

Importing dependencies:

# Add dependencies in workspace's Cargo.toml
[workspace.dependencies]
uuid = { version = "=1.17", features = ["v4"] }

# import in members like this: my_lib/Cargo.toml
[dependencies]
uuid = { workspace = true }

Import members like this:

# Cargo.toml
[workspace]
resolver = "3"
members = ["my_bin", "my_lib"]

[workspace.dependencies]
my_lib = { path = "my_lib" }

# my_bin/Cargo.toml
[dependencies]
my_lib = { workspace = true }

Other commands:

# run if multiple binaries exist
cargo run --package my_bin

# build specific binary
cargo build --package my_bin

# build all
cargo build --all

# test all
cargo test --all

How I use Postgres DB as Key Value Store in Rust with SQLx

2025-09-02T00:00:00+00:00

For my smaller application (or personal) or when I’m starting out a new app, I would like to keep the dependencies of my program to minimum. Since I’m going to need a DB anyway, so instead of using Redis and adding one more component to my tech stack, I would like to use Postgres as my key value store.

Prematurely thinking about scaling your system when you don’t even know if it would be successful enough to actually need all those bells and whistle is sure way to extend the deadline and procrastinate.

Overview

We are going to create a trait called PgStore, which will implement all the functions to set, get, pop and delete the value.

Then you can implement this trait for your own types. Your types must derive serde’s Serialize and Deserialize for this to work.

Dependencies

You need the following dependencies in your Cargo.toml

chrono = { version = "0.4", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0" }
sqlx = { version = "0.8", features = [
    "runtime-tokio",
    "tls-rustls-ring-native-roots",
    "migrate",
    "postgres",
    "macros",
    "chrono",
] }

Creating static PgPool connection

I don’t like to pass around DB connection across functions or threads or tasks. Instead I like to use tokio’s OnceCell which is a thread safe Cell which can be written to only once, it’s thread safe which means we can use async code in it.

use sqlx::PgPool;
use tokio::sync::OnceCell;

static PG_POOL: OnceCell<PgPool> = OnceCell::const_new();

pub async fn pg_pool() -> &'static PgPool {
    PG_POOL
        .get_or_init(async || {
            PgPool::connect(std::env("DATABASE_URL").expect("DATABASE_URL is not set"))
                .await
                .expect("❌ PgPool connection failed")
        })
        .await
}

Now we can just use pg_pool().await to fetch the db pool connection wherever we need.

DB Migration

We are creating a new table kv_store in our Postgres DB. If you notice we are using UNLOGGED keyword to create the table, this tells postgres to not write WAL (write-ahead log) for kv_store table, which improves the performance at the expanse of data durability.

CREATE UNLOGGED TABLE "kv_store" (
    "key" VARCHAR(2048) PRIMARY KEY,
    "value" TEXT NOT NULL,
    "expires" TIMESTAMPTZ NOT NULL
);

PgStore Trait

We are creating PgStore trait which will have methods to store, retrieve and delete the value in the kv_store table.

use chrono::{DateTime, Utc};
use serde::Serialize;
use serde::de::DeserializeOwned;
use std::time::Duration;

#[allow(async_fn_in_trait)]
pub trait PgStore: Serialize + DeserializeOwned {
    const EXPIRE_IN: usize; // seconds

    fn key_format(key: String) -> String;

    fn get_expire_time() -> DateTime<Utc> {
        Utc::now() + Duration::from_secs(Self::EXPIRE_IN as u64)
    }

    async fn set_ex(&self, key: String) -> Result<(), String> {
        let value =
            serde_json::to_string(&self).map_err(|_| "Failed to serialize value.".to_string())?;

        sqlx::query!(
            r#"INSERT INTO kv_store ("key", "value", expires) VALUES ($1, $2, $3)
            ON CONFLICT ("key") DO UPDATE SET "value" = $2, expires = $3"#,
            Self::key_format(key),
            value,
            Self::get_expire_time(),
        )
        .execute(db::pg_pool().await)
        .await
        .map_err(|_| "Failed to set value in kv_store".to_string())?;

        Ok(())
    }

    // it will just fetch the value if it exists
    async fn get(key: String) -> Result<Self, String> {
        let record = sqlx::query!(
            r#"SELECT "value" FROM kv_store WHERE "key" = $1"#,
            Self::key_format(key)
        )
        .fetch_one(db::pg_pool().await)
        .await
        .map_err(|_| "Failed to get value from kv_store".to_string())?;

        serde_json::from_str(&record.value).map_err(|_| "Failed to deserialize value".to_string())
    }

    // this will fetch the value while extending the expire time
    async fn get_ex(key: String) -> Result<Self, String> {
        let record = sqlx::query!(
            r#"UPDATE kv_store SET expires=$2 WHERE "key"=$1 RETURNING "value""#,
            Self::key_format(key),
            Self::get_expire_time(),
        )
        .fetch_one(db::pg_pool().await)
        .await
        .map_err(|_| "Failed to get value from kv_store".to_string())?;

        serde_json::from_str(&record.value).map_err(|_| "Failed to deserialize value".to_string())
    }

    // it will fetch the value while deleting the key
    async fn get_del(key: String) -> Result<Self, String> {
        let record = sqlx::query!(
            r#"DELETE FROM kv_store WHERE key=$1 RETURNING "value""#,
            Self::key_format(key)
        )
        .fetch_one(db::pg_pool().await)
        .await
        .map_err(|_| "Failed to get value from kv_store".to_string())?;

        serde_json::from_str(&record.value).map_err(|_| "Failed to deserialize value".to_string())
    }

    async fn del(key: String) -> Result<(), String> {
        sqlx::query!(
            r#"DELETE FROM kv_store WHERE key=$1"#,
            Self::key_format(key)
        )
        .execute(db::pg_pool().await)
        .await
        .map_err(|_| "Failed to delete value".to_string())?;

        Ok(())
    }
}

By defining our trait like PgStore: Serialize + DeserializeOwned, we are telling that whichever type is going to implement PgStore trait it must also have Serialize and DeserializeOwned implemented.

Following are the brief explaination of PgStore’s attributes:

EXPIRE_IN: This will be used to set the to be expired in seconds. It must be implemented by the Type you are using it for.

key_format: This function needs to be implemented. It will be used as a way to generate key.

set_ex: This method will store the type against the passed key in DB, overwriting value if it already exists.

get: It will fetch the value for a given key, error will be returned if it does not exist.

get_ex: Fetches the value while resetting the expire time.

get_del: Fetch the value while deleting the key. It’s more like pop.

del: Just deletes the value. Doesn’t return error even if key doesn’t exist.

Cleaning up expired keys

Then we have a helper function pgstore_cleanup which will cleanup expired keys every 60 seconds. You must spawn to run this function in the background for all the time your program is running.

pub async fn pgstore_cleanup() {
    loop {
        // tracing::info!("🟡 PgStore cleanup is running...");
        sqlx::query!(r#"DELETE FROM kv_store WHERE expires < NOW()"#)
            .execute(db::pg_pool().await)
            .await
            .ok();
        tokio::time::sleep(Duration::from_secs(60)).await;
    }
}

Usage

#[derive(serde::Serialize, serde::Deserialize)]
pub struct Game {
    pub id: usize,
    pub name: String,
    pub genre: String,
}

impl PgStore for Game {
    const EXPIRE_IN: usize = 3600;

    fn key_format(key: String) -> String {
        format!("game-{}", key)
    }
}

#[tokio::main]
async fn main() {
    // make sure to run this to delete expired keys regularly
    tokio::spawn(pgstore_cleanup());

    let game = Game {
        id: 1,
        name: "Skyrim".to_string(),
        genre: "rpg".to_string(),
    };

    // storing value in kv_store
    // it will be store with key: "game-1"
    let _ = game.set_ex(game.id.to_string()).await;

    // fetching value
    if let Ok(v) = Game::get(1.to_string()).await {
        println!("fetched game with name: {}", v.name);
    }

    // deleting key
    let _ = Game::del(1.to_string()).await;
}

Similarly you can do this with your own types, as long as they derive serde’s Serialize and Deserialize.

Minify HTML in Axum Middleware

2025-09-01T00:00:00+00:00

Minifying HTML is one of the simplest way to save bandwidth and decrease the time in transit. We can easily lower page size by 30-50% by minifying HTML.

We will be using minify-html crate for this. With the use of Axum middleware, minification will be applied to all the eligible routes.

Using axum middleware for HTML minification is template agnostic which means it will work with any template engine such as Askama, Tera, Maud, etc., as long as response type is HTML, it should work fine.

Cargo.toml

Add minify-html to your Cargo.toml.

...
minify-html = "0.16.4"
...

Basic Usage of minify-html

minify-html provides minify function which accepts array of bytes and reference to configuration struct which tells it how to minify.

let cfg = minify_html::Cfg {
        keep_closing_tags: true,
        keep_html_and_head_opening_tags: true,
        keep_comments: false,
        minify_doctype: false,
        minify_css: true,
        minify_js: true,
        ..Default::default()
    };

let mut code: &[u8] = b"<p>  Hello, world!  </p>";
let minified = minify_html::minify(&code, &cfg);
assert_eq!(minified, b"<p>Hello, world!</p>".to_vec());

Creating Axum Middleware

We are using Axum’s axum::middleware::map_response middleware to modify the response on the fly by passing an async function.

Steps to be performed by middleware:

Only move forward if the Content-Type header of the reponse is text/html
Use into_parts method of response to consume the response body give out parts (headers, status code, etc.) and body (actual html content)
Using to_bytes function we collect and convert axum::body::Body type into Bytes
Then using minify_html::minify function we minify the html
Finally build the new response using Response::from_parts with the parts that we extracted earlier and new body (minified html)

use std::sync::LazyLock;
use axum::{
    body::{Body, to_bytes},
    http::header,
    response::Response,
};

static MINIFY_CFG: LazyLock<minify_html::Cfg> = LazyLock::new(|| minify_html::Cfg {
    keep_closing_tags: true,
    keep_html_and_head_opening_tags: true,
    minify_doctype: false,
    minify_css: true,
    minify_js: true,
    ..Default::default()
});

pub async fn minify_html_response(response: Response<Body>) -> Response<Body> {
    let content_type = response
        .headers()
        .get(header::CONTENT_TYPE)
        .map(|h| h.to_str().unwrap_or_default())
        .unwrap_or_default();

    if content_type.contains("text/html") {
        let (parts, body) = response.into_parts();
        let bytes = to_bytes(body, usize::MAX).await.unwrap_or_default();
        let minified = minify_html::minify(&bytes, &MINIFY_CFG);
        let new_response = Response::from_parts(parts, Body::from(minified));
        return new_response;
    }

    response
}

Note: We are using LazyLock to initialize minify_html::Cfg only once, since we are going to need the reference to it for each minify call.

Plugging into the Router:

let router = axum::Router::new()
    .route("/", get(index_page))
    .layer(axum::middleware::map_response(minify_html_response));

Benchmark

I’ve done very basic benchmarking on this

Overhead: The overhead of extracting response body and minifying adds around 600 to 800 µs.

Size Reduction: This will be varied for everyone and depends on the content they serve. However, I’ve consistently seen at least 30% reduction in HTML document size.

Using SQLite Database in Rust with Sqlx

2025-04-13T00:00:00+00:00

SQLite is self-contained, file based relational database. If you have a simple use case and want to avoid complex databases then SQLite is one of the best choice.

We will use sqlx crate to interact with SQLite database.

Why Sqlx?

Async support
Supports migrations
Compile time type safety
Supports SQLite, Postgres and MySQL

Setup

cargo init sqlite-demo

Cargo.toml

Add the following dependencies to your Cargo.toml file. We are using tokio runtime and sqlite features of sqlx.

[package]
name = "sqlite-demo"
version = "0.1.0"
edition = "2024"

[dependencies]
sqlx = { version = "0.8", features = [
    "runtime-tokio",
    "sqlite",
    "migrate",
] }
tokio = { version = "1.44.2", features = ["full"] }

Connect to SQLite Database

Here we are using SqliteConnectOptions to connect to SQLite database. We are also creating the database file if it doesn’t exist. We are using SqliteJournalMode::Delete to delete the journal file after commit so that we will only have one file in the database directory.

Since we want file based database, we are using sqlite://main.db as connection string. For memory based database, we can use sqlite::memory: as connection string.

use sqlx::{ConnectOptions, Connection, sqlite::SqliteConnectOptions};
use std::str::FromStr;

#[tokio::main]
async fn main() {
    let mut conn = SqliteConnectOptions::from_str("sqlite://main.db")
        .unwrap()
        .create_if_missing(true)
        .journal_mode(sqlx::sqlite::SqliteJournalMode::Delete)
        .connect()
        .await
        .expect("Failed to connect to database");

    // Create the users table if it doesn't exist
    sqlx::query(
        "CREATE TABLE IF NOT EXISTS users (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            name TEXT NOT NULL,
            age INTEGER
        )",
    )
    .execute(&mut conn)
    .await
    .unwrap();

    let _ = conn.close().await;
}

Creating Connection Pool

We will use SqlitePoolOptions to create a connection pool.

We will create a helper function db_conn to create a connection pool. This function will return a connection pool that can be used to interact with the database.

use sqlx::{
    Pool, Sqlite,
    sqlite::{SqliteConnectOptions, SqlitePoolOptions},
};
use std::{str::FromStr, time::Duration};

#[tokio::main]
async fn main() {
    let conn = db_conn().await;

    // Create the users table if it doesn't exist
    sqlx::query(
        "CREATE TABLE IF NOT EXISTS users (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            name TEXT NOT NULL,
            age INTEGER
        )",
    )
    .execute(&conn)
    .await
    .unwrap();

    let _ = conn.close().await;
}

async fn db_conn() -> Pool<Sqlite> {
    let db_url = "sqlite://main.db";

    SqlitePoolOptions::new()
        .max_connections(20)
        .idle_timeout(Duration::from_secs(60))
        .acquire_timeout(Duration::from_secs(5))
        .connect_with(
            SqliteConnectOptions::from_str(db_url)
                .unwrap()
                .create_if_missing(true)
                .journal_mode(sqlx::sqlite::SqliteJournalMode::Delete),
        )
        .await
        .unwrap()
}

Using OnceCell to create a connection pool

We can use OnceCell to create a connection pool. This will ensure that the connection pool is created only once and can be shared across multiple threads. So that we won’t to pass connection pool to every function, we can just use this helper function to get the connection pool and it will re-use the same one.

use sqlx::{
    Pool, Sqlite,
    sqlite::{SqliteConnectOptions, SqlitePoolOptions},
};
use std::{str::FromStr, time::Duration};
use tokio::sync::OnceCell;

static DB_CONN: OnceCell<Pool<Sqlite>> = OnceCell::const_new();

async fn db_conn() -> Pool<Sqlite> {
    let db_url = "sqlite://main.db";

    DB_CONN
        .get_or_init(|| async {
            SqlitePoolOptions::new()
                .max_connections(20)
                .idle_timeout(Duration::from_secs(60))
                .acquire_timeout(Duration::from_secs(5))
                .connect_with(
                    SqliteConnectOptions::from_str(db_url)
                        .unwrap()
                        .create_if_missing(true)
                        .journal_mode(sqlx::sqlite::SqliteJournalMode::Delete),
                )
                .await
                .unwrap()
        })
        .await
        .clone()
}

Handling Database Migrations

We will use sqlx-cli to handle migrations.

cargo install sqlx-cli

Create migration file for users table.

sqlx migrate add users

This will create a migration file in migrations directory with empty migration file for users table. Add the following content to the newly created migration file.

-- Add migration script here
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    age INTEGER
);

Embed the migration file in the executable.

use sqlx::{
    Pool, Sqlite,
    sqlite::{SqliteConnectOptions, SqlitePoolOptions},
};
use std::{str::FromStr, time::Duration};
use tokio::sync::OnceCell;

#[tokio::main]
async fn main() {
    // this will make sure migrations are added to the executable
    sqlx::migrate!()
        .run(&db_conn().await)
        .await
        .expect("Migrations went wrong :(");
}

static DB_CONN: OnceCell<Pool<Sqlite>> = OnceCell::const_new();

async fn db_conn() -> Pool<Sqlite> {
    let db_url = "sqlite://main.db";

    DB_CONN
        .get_or_init(|| async {
            SqlitePoolOptions::new()
                .max_connections(20)
                .idle_timeout(Duration::from_secs(60))
                .acquire_timeout(Duration::from_secs(5))
                .connect_with(
                    SqliteConnectOptions::from_str(db_url)
                        .unwrap()
                        .create_if_missing(true)
                        .journal_mode(sqlx::sqlite::SqliteJournalMode::Delete),
                )
                .await
                .unwrap()
        })
        .await
        .clone()
}

Full Code with Example Queries

main.rs

use sqlx::{
    Pool, Row, Sqlite,
    prelude::FromRow,
    sqlite::{SqliteConnectOptions, SqlitePoolOptions},
};
use std::{str::FromStr, time::Duration};
use tokio::sync::OnceCell;

#[tokio::main]
async fn main() {
    sqlx::migrate!()
        .run(&db_conn().await)
        .await
        .expect("❌ Migrations went wrong :(");

    sqlx::query("INSERT INTO users (name, age) VALUES (?, ?)") // Use ? for placeholders
        .bind("Alice")
        .bind(30)
        .execute(&db_conn().await)
        .await
        .unwrap();

    let query = sqlx::query("SELECT count(*) AS count FROM users")
        .fetch_one(&db_conn().await)
        .await
        .unwrap();

    println!("Count: {:?}", query.get::<i64, &str>("count"));

    let users = sqlx::query_as::<_, User>("SELECT id, name, age FROM users")
        .fetch_all(&db_conn().await)
        .await
        .unwrap();

    for user in users {
        println!("Found user: {:?}", user);
    }
}

static DB_CONN: OnceCell<Pool<Sqlite>> = OnceCell::const_new();

async fn db_conn() -> Pool<Sqlite> {
    let db_url = "sqlite://main.db";

    DB_CONN
        .get_or_init(|| async {
            SqlitePoolOptions::new()
                .max_connections(20)
                .idle_timeout(Duration::from_secs(60))
                .acquire_timeout(Duration::from_secs(5))
                .connect_with(
                    SqliteConnectOptions::from_str(db_url)
                        .unwrap()
                        .create_if_missing(true)
                        .journal_mode(sqlx::sqlite::SqliteJournalMode::Delete),
                )
                .await
                .unwrap()
        })
        .await
        .clone()
}

#[derive(Debug, FromRow)]
struct User {
    id: i64,
    name: String,
    age: Option<i32>,
}

Self Host PostgreSQL With TLS

2025-02-03T00:00:00+00:00

In this tutorial we will go through how to self-host PostgreSQL Database and setup TLS to securely connect.

Installing PostgreSQL

Postgres APT repository configuration:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh

Install PostgreSQL 16

sudo apt install postgresql-16

After installing PostgreSQL, it creates a new user postgres, which is the superuser of the database. We can switch to postgres user in ubuntu to access postgresql command prompt:

sudo -i -u postgres

Now to create new user with passowrd and with privilege to create new db. It will ask you to enter your password, which will not be logged into history hence this is the recommended way to set password.

createuser your_db_user --pwprompt --createdb

Now we can exit from postgres account

exit

We will add new user in our Ubuntu with the same name as the new user we created in postgresql. (Optional) Add -m to create home directory for the user.

sudo useradd -s /bin/bash your_db_user

Accessing psql as the new user we just created. Since we do not have any database associated with our user therefore we are using ‘-d postgres’ to use postgres db for now.

sudo -u your_db_user psql -U your_db_user -d postgres

Create your database.

CREATE DATABASE your_db_name;

This is how you can connect with your db if not already connected.

\c your_db_name;

Setting Up SSL

It’s important that we secure our communication with the database while accessing it remotely. We are going to setup SSL certificate using certbot from LetsEncrypt.

Install certbot

sudo snap install core; sudo snap refresh core
sudo apt-get remove certbot
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Obtain certificate

sudo certbot certonly --standalone -d psql.example.com

Find PostgreSQL data directory

sudo -u postgres psql -U postgres -c 'SHOW data_directory'

Create a renewal hook file

sudo nano /etc/letsencrypt/renewal-hooks/deploy/postgresql.deploy

Paste the following

umask 0177
DOMAIN=psql.example.com
DATA_DIR=/var/lib/postgresql/16/main
cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem $DATA_DIR/server.crt
cp /etc/letsencrypt/live/$DOMAIN/privkey.pem $DATA_DIR/server.key
chown postgres:postgres $DATA_DIR/server.crt $DATA_DIR/server.key

Save and exit the file.

Give file executable permission

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/postgresql.deploy

Get the path of PostgreSQL configuration file

sudo -u postgres psql -U postgres -c 'SHOW config_file'

Open the file in edit mode and update the SSL section of the file as below:

sudo nano /etc/postgresql/16/main/postgresql.conf

ssl = on
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_prefer_server_ciphers = on

Locate the Connection Settings section and verify the listen_address is to * for all addresses. Make sure the line is not commented out.

listen_addresses = '*'

Edit pg_hba.conf file which is in the same dir as config file:

sudo nano /etc/postgresql/16/main/pg_hba.conf

Add the following line to enable connection over ssl for any user and database:

# TYPE DATABASE USER ADDRESS METHOD

hostssl all all 0.0.0.0/0 scram-sha-256
hostssl all all ::/0 scram-sha-256

Although in above config it’s shown to allow connection from all IPv4 and IPv6 addresses, however from the security point of view only allow the ip address of your connecting machine (client).

Renew the SSL certificate

sudo certbot renew --force-renewal

Restart PostgreSQL

sudo service postgresql restart

Connect to your db remotely

psql -d "dbname=your_db_name sslmode=require" -h psql.example.com -U your_db_user

AES Encryption in Rust

2024-03-22T00:00:00+00:00

What we will cover

Brief explanation of AES encryption
AES-256 encryption and decryption
Full Rust code for AES-256 encryption

Introduction

AES stands for Advanced Encryption Standard. It is a type of Symmetric key encryption algorithm, which means the same key is used for encryption and decryption.

In this tutorial we will use aes-gcm crate from Rust Crypto.

There are 3 type of AES encryption: AES-128, AES-192 and AES-256, which represents encryption Key of length 128 bit (16 byte), 192 bit (24 byte) and 256 bit (32 byte) respectively.

Both are pretty secure and hard to crack but for obvious reasons AES-256 is the most secure one.

AES encryption requires key and nonce (for gcm variant) to encrypt/decrypt the data.

Key

Key means encryption key, which is used to encrypt and decrypt the data. Length of the key matters, AES-128 requires key of 16 byte and AES-256 requires key of 32 byte.

For the simplicity from now on we will focus on AES-256, but the core concept is similar even for AES-128.

Key is byte array of length 32, where each element is of 1 byte. We can either generate the key using aes-gcm crate itself or we can use our own key. It’s prefferable to use our own key, so let’s see it in action:

Create new crypto project and add aes-gcm crate.

cargo new crypto
cargo add aes-gcm

main.rs

use aes_gcm::{
    aead::{Aead, AeadCore, KeyInit, OsRng},
    Aes256Gcm, Key, Nonce
};

fn main() {
    let key_str = "thiskeystrmustbe32charlongtowork";
    let key = Key::<Aes256Gcm>::from_slice(key_str.as_bytes());
}

It’s important that the string/str used for key must be 32 char long, since AES-256 requires key length of 32 byte.

Note: If you want to use key string of any length (less or more than 32), you can use Key Derivation Function to generate secure 32 byte long key. Argon2, PBKDF2 are some of the fitting choices for this.

Nonce

Nonce means “number used only once”. It is an array of random numbers (type: u8) that is unique to each operation. For AES encryption the required size of Nonce is 12 byte.

For each encryption operation the same key can be used but the Nonce must be unique, so we generate Nonce each time we encrypt data.

Since nonce is required to decrypt the data, it is stored along with the encrypted data itself.

let nonce = Aes256Gcm::generate_nonce(&mut OsRng);

Encryption

We will create encrypt function which will take key_str and plaintext string. This will make our code more readable and easy to use.

use aes_gcm::{
    aead::{Aead, AeadCore, KeyInit, OsRng},
    Aes256Gcm, Key, Nonce
};

fn main() {
    let plaintext = "vivekshuk.la".to_string();

    let key_str = "thiskeystrmustbe32charlongtowork".to_string();

    let encrypted_data = encrypt(key_str, plaintext);
}

fn encrypt(key_str: String, plaintext: String) -> Vec<u8> {
    let key = Key::<Aes256Gcm>::from_slice(key_str.as_bytes());
    let nonce = Aes256Gcm::generate_nonce(&mut OsRng);

    let cipher = Aes256Gcm::new(key);

    let ciphered_data = cipher.encrypt(&nonce, plaintext.as_bytes())
        .expect("failed to encrypt");

    // combining nonce and encrypted data together
    // for storage purpose
    let mut encrypted_data: Vec<u8> = nonce.to_vec();
    encrypted_data.extend_from_slice(&ciphered_data);

    encrypted_data
}

Here we are:

creating instance of Aes256Gcm with our encryption key, which will be used for encryption.
we need to convert our plaintext from string to byte array for the encrypt method.
combine nonce and ciphered_data together

Decryption

Here first we will separate out nonce and ciphered data by splitting encrypted data vector at 12th position (because we used nonce of length 12).

Then we create Nonce instance from our nonce array that we got.

After decrypting we will get byte array which we can pass to String::from_utf8 function to create our original string value back.

fn decrypt(key_str: String, encrypted_data: Vec<u8>) -> String {
    let key = Key::<Aes256Gcm>::from_slice(key_str.as_bytes());

    let (nonce_arr, ciphered_data) = encrypted_data.split_at(12);
    let nonce = Nonce::from_slice(nonce_arr);

    let cipher = Aes256Gcm::new(key);

    let plaintext = cipher.decrypt(nonce, ciphered_data)
        .expect("failed to decrypt data");

    String::from_utf8(plaintext)
        .expect("failed to convert vector of bytes to string")
}

Storage

There is one issue you might encounter that the encrypt function will give out Vector of numbers (u8), which is not easy to store. Of course you might convert it to string by joining them with comma but it’s not good solution.

To solve this we can convert our vector of numbers to hex string, this will make storage and operation much easier for us.

Let’s install hex crate for this.

cargo add hex

hex crate provides encode (vec to string) and decode (string to vec) function for easy conversion.

let encoded_string = hex::encode(encrypted_data);

let decoded_vec = hex::decode(encoded_string)
    .expect("failed to decode hex string into vec");

Full Code

In this final version we have made few changes to incorporate hex conversion within the encrypt and decrypt function itself.

use aes_gcm::{
    aead::{Aead, AeadCore, KeyInit, OsRng},
    Aes256Gcm, Key, Nonce
};

fn main() {
    let plaintext = "vivekshuk.la".to_string();

    let key_str = "thiskeystrmustbe32charlongtowork".to_string();

    let encrypted_data = encrypt(key_str.clone(), plaintext);

    println!("encrypted_data: {:?}", encrypted_data.clone());

    let original = decrypt(key_str, encrypted_data);

    println!("original: {:?}", original);
}

fn encrypt(key_str: String, plaintext: String) -> String {
    let key = Key::<Aes256Gcm>::from_slice(key_str.as_bytes());
    let nonce = Aes256Gcm::generate_nonce(&mut OsRng);

    let cipher = Aes256Gcm::new(key);

    let ciphered_data = cipher.encrypt(&nonce, plaintext.as_bytes())
        .expect("failed to encrypt");

    // combining nonce and encrypted data together
    // for storage purpose
    let mut encrypted_data: Vec<u8> = nonce.to_vec();
    encrypted_data.extend_from_slice(&ciphered_data);

    hex::encode(encrypted_data)
}

fn decrypt(key_str: String, encrypted_data: String) -> String {
    let encrypted_data = hex::decode(encrypted_data)
        .expect("failed to decode hex string into vec");

    let key = Key::<Aes256Gcm>::from_slice(key_str.as_bytes());

    let (nonce_arr, ciphered_data) = encrypted_data.split_at(12);
    let nonce = Nonce::from_slice(nonce_arr);

    let cipher = Aes256Gcm::new(key);

    let plaintext = cipher.decrypt(nonce, ciphered_data)
        .expect("failed to decrypt data");

    String::from_utf8(plaintext)
        .expect("failed to convert vector of bytes to string")
}

Python Poetry Cheat Sheet

2023-10-30T00:00:00+00:00

Poetry is an amazing tool for dependancy management in Python, it solves a lot of problems of pip. Here in this Poetry Cheat Sheet we have useful commands that we may need frequently.

Initializing

New Project

poetry new poetry-demo

In Pre-existing Directory

cd pre-existing-project
poetry init

Environments

Displaying the environment information

poetry env info

Using your own virtual environment

Keep virtualenv in project root

Switching between environments

poetry env use /full/path/to/python

poetry env use 3.7

poetry env use system

Listing the environments associated with the project

poetry env list

Deleting the environments

poetry env remove /full/path/to/python
poetry env remove python3.7
poetry env remove 3.7
poetry env remove test-O3eWbxRl-py3.7

Use the --all option to delete all virtual environments at once.

poetry env remove --all

Dependencies

Add new dependency

poetry add <package name>

Removing dependency

poetry remove <package name>

Installing dependencies

poetry install

Updating dependencies to their latest versions

This will fetch the latest matching versions (according to your pyproject.toml file) and update the lock file with the new versions. (This is equivalent to deleting the poetry.lock file and running install again.)

poetry update

Adding a dependency to a group

poetry add pytest --group test

poetry add isort --group dev

Installing group dependencies

without

poetry install --without test, docs

with

poetry install --with docs

only

poetry install --only main

Removing dependencies from a group

poetry remove mkdocs --group docs

Synchronizing dependencies

Poetry supports what’s called dependency synchronization. Dependency synchronization ensures that the locked dependencies in the poetry.lock file are the only ones present in the environment, removing anything that’s not necessary.

poetry install --sync

poetry install --without dev --sync
poetry install --with docs --sync
poetry install --only dev

Export

poetry export -f requirements.txt --output requirements.txt

--format (-f): The format to export to (default: requirements.txt). Currently, only constraints.txt and requirements.txt are supported.
--output (-o): The name of the output file. If omitted, print to standard output.
--extras (-E): Extra sets of dependencies to include.
--without: The dependency groups to ignore.
--with: The optional dependency groups to include.
--only: The only dependency groups to include.
--without-hashes: Exclude hashes from the exported file.
--without-urls: Exclude source repository urls from the exported file.
--with-credentials: Include credentials for extra indices.

Packaging

Build

poetry build

Publish

poetry publish

Configuration

You can find all the configuration options here.

pyenv Cheat Sheet

2023-10-22T00:00:00+00:00

Here is the pyenv cheat sheet of all the basic commands list.

What is pyenv

pyenv is Python version management tool, it lets you easily switch between multiple versions of Python.

What it does

Lets you change the global Python version on a per-user basis.
Provides support for per-project Python versions.
Allows you to override the Python version with an environment variable.

List available python versions to install

This will list all the available version of python

pyenv install --list

You can filter it using grep. Here we are filtering list by python version:

pyenv install --list | grep " 3\.11"

Installing Python Version

pyenv install <version>

add -v flag to see install progress output

List Installed Python Versions

pyenv versions

Astrick (*) indicate current active version.

See List Of All pyenv Commands

pyenv commands

Activating Python Version

global

Sets the given version as global python version.
It overrides system’s default python version.

pyenv global <version>

local

Use it to set directory / project specific python version.
It can override python version set by global

pyenv local <version>

shell

Sets shell specific python version.
The python version stays active until shell has been closed or by using --unset flag.
It can override python version set by local and global

pyenv shell <version>

Virtual Environments

Create Virtual Environments

pyenv virtualenv <version> <env_name>

Set virtual environment as default for dir/project

pyenv local <env_name>

Activate Virtual Environment

pyenv activate <env_name>

Deactivate Virtual Environment

pyenv deactivate

Automate Postgres Database Backup and Upload to S3

2023-10-20T00:00:00+00:00

Database is a very critical system for an application. To prepare for any disaster we all must take regular backups of our database.

In this tutorial we will create a bash script to automate postgres database backup from a docker container and native postgres installation then upload to S3 based storage system. We will use crontab to create a cron job that will run everyday at 12 PM.

What we will cover

Write db backup automation script for docker and non-docker based system
Upload backup file to AWS S3
Setup the cron job for daily backup

Steps

Move to your project folder, where docker-compose.yml is.
Take the database backup in container
Copy the backup file from container filesystem to our desired location
Delete the backup file inside the container
Upload backup file to S3
Delete the local backup file

Code snippet below covers step 1 to 4. But it is not optimal, later we will make modification so that we can easily automate it and make it re-usable.

cd project_dir

docker compose exec db_service_name sh -c "pg_dump -U db_user db_name > filename"

docker compose cp db_service_name:filename destination/filename

docker compose exec db_service_name rm filename

Upload to S3

To upload file to S3, we need to have aws cli installed, you can follow this guide from AWS docs to install awscli.

After installing awscli, you need to run aws configure and provide your key id and key secret. You can use any S3 based storage system here.

aws s3 cp filename s3://your-bucket

If you are using some other S3 based storage system then you’ll have to provide custom --endpoint-url argument here, make sure you read the respective docs for this.

Full Script

We have laid out all the steps and code to accomplish that, now let’s modify in such a way that we can easily automate it and make it reusable.

The filename is created based on the current datetime of the system, so that we can avoid ambiguity and exactly know how old is the backup file.

For docker compose

#!/bin/bash

############## ✅ Update These Values ##################
directory="your-project-dir"
home="/home/<user>"

db_service_name="db"
db_user="db_user"
db_name="db_name"

#####################################################
############ 🔴 CHANGE WITH CAUTION 🔴 #############
date=$(date +"%Y-%m-%d-%H:%M:%S")
filename=$db_name"_"$date".sql"

cd $home
cd $directory
docker compose exec $db_service_name sh -c "pg_dump -U $db_user $db_name > $filename"

docker compose cp $db_service_name:$filename $home"/"$filename
docker compose exec $db_service_name rm $filename

cd $home
#####################################################
#####################################################

############## ✅ CAN CHANGE BELOW ##################
aws s3 cp $filename s3://your-bucket

rm $filename

At the top, you’ve to update variable values based on your project and directory structure, and the main code will use that to backup the database.

At the end of script, we have aws command to copy the backup file to S3, here you need to update bucket name and if you are using other S3 storage provider (other than AWS), then --endpoint-url argument with appropriate value.

For Native Postgres Installation

#!/bin/bash

############## ✅ Update These Values ##################
home="/home/<user>"

db_user="db_user"
db_name="db_name"

#####################################################
############ 🔴 CHANGE WITH CAUTION 🔴 ##############
date=$(date +"%Y-%m-%d-%H:%M:%S")
filename=$db_name"_"$date".sql"

ch $home
pg_dump -U $db_user $db_name > $filename

#####################################################

############## ✅ CAN CHANGE BELOW ##################
aws s3 cp $filename s3://urlify-db-backup

rm $filename

When running pg_dump in the native installation, there are two ways postgres will try to authenticate the user.

Peer based authentication: It will check if the current user (system logged in user) is same as user passed to pg_dump.
If peer auth fails then postgres looks for .pgpass file in home directory for the password.
If both fails then it might ask for password.

So, make sure you use 1st method and run the script on the same user as the one passed to pg_dump.

Setup Cron Job

To automate backup and upload of our database, we will use crontab. It comes built-in with almost all the Linux distros and are very easy to setup.

Give your script executable permission:

chmod +x db_backup.sh

Open crontab:

crontab -e

This will open up crontab in your prefered choice of editor. Use the below statement to run the backup script everyday at 12 PM.

0 12 * * * /home/<user>/db_backup.sh >> cron.log 2>&1

Create cron.log file

touch cron.log

Use full path if cron.log is not in current user’s home folder.