Vivek Shukla - Backup

Vivek Shukla - Backup I am a Software Engineer from India. I am currently building Daestro. I work in Rust, Javascript and Python. Zola 2025-03-18T00:00:00+00:00 https://vshukla.com/tags/backup/atom.xml Plausible Docker Volume Backup 2025-03-18T00:00:00+00:00 2025-03-18T00:00:00+00:00 Vivek Shukla https://vshukla.com/snippets/plausible-docker-volume-backup/

Backup Plausible data from docker volumes and upload using rclone

<pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-comment">#!/bin/bash</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable">home</span><span class="z-keyword z-operator">=</span><span class="z-string">"/home/<update>/"</span></span> <span class="giallo-l"><span class="z-variable">dest</span><span class="z-keyword z-operator">=</span><span class="z-string">"<update>"</span></span> <span class="giallo-l"><span class="z-variable">bucket</span><span class="z-keyword z-operator">=</span><span class="z-string">"<update>"</span></span> <span class="giallo-l"><span class="z-variable">pg_container_id</span><span class="z-keyword z-operator">=</span><span class="z-string">"<update>"</span></span> <span class="giallo-l"><span class="z-variable">ch_container_id</span><span class="z-keyword z-operator">=</span><span class="z-string">"<update>"</span></span> <span class="giallo-l"><span class="z-variable">passphrase</span><span class="z-keyword z-operator">=</span><span class="z-string">"<update>"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable">date</span><span class="z-keyword z-operator">=</span><span>$(</span><span class="z-entity z-name">date</span><span class="z-string"> +"%Y%m%d%H%M%S"</span><span>)</span></span> <span class="giallo-l"><span class="z-variable">pl_repo</span><span class="z-keyword z-operator">=</span><span class="z-variable">$home</span><span class="z-string">"/"</span><span class="z-variable">$date</span><span class="z-string">"_plausible.tar.gz"</span></span> <span class="giallo-l"><span class="z-variable">pl_repo_gpg</span><span class="z-keyword z-operator">=</span><span class="z-variable">$pl_repo</span><span class="z-string">".gpg"</span></span> <span class="giallo-l"><span class="z-variable">pg_data</span><span class="z-keyword z-operator">=</span><span class="z-variable">$home</span><span class="z-string">"/"</span><span class="z-variable">$date</span><span class="z-string">"_db-data.tar.gz"</span></span> <span class="giallo-l"><span class="z-variable">pg_data_gpg</span><span class="z-keyword z-operator">=</span><span class="z-variable">$pg_data</span><span class="z-string">".gpg"</span></span> <span class="giallo-l"><span class="z-variable">ch_data</span><span class="z-keyword z-operator">=</span><span class="z-variable">$home</span><span class="z-string">"/"</span><span class="z-variable">$date</span><span class="z-string">"_event-data.tar.gz"</span></span> <span class="giallo-l"><span class="z-variable">ch_data_gpg</span><span class="z-keyword z-operator">=</span><span class="z-variable">$ch_data</span><span class="z-string">".gpg"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># stopping postgres and clickhouse containers before backup</span></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> container stop</span><span class="z-variable"> $pg_container_id $ch_container_id</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># creating gzip file</span></span> <span class="giallo-l"><span class="z-entity z-name">tar</span><span class="z-constant z-other"> --exclude=images --exclude-vcs -czf</span><span class="z-variable"> $pl_repo</span><span class="z-constant z-other"> -C</span><span class="z-variable"> $home</span><span class="z-string"> plausible</span></span> <span class="giallo-l"><span class="z-entity z-name">sudo</span><span class="z-string"> tar</span><span class="z-constant z-other"> -czf</span><span class="z-variable"> $pg_data</span><span class="z-constant z-other"> -C</span><span class="z-string"> /var/lib/docker/volumes/ plausible_db-data</span></span> <span class="giallo-l"><span class="z-entity z-name">sudo</span><span class="z-string"> tar</span><span class="z-constant z-other"> -czf</span><span class="z-variable"> $ch_data</span><span class="z-constant z-other"> -C</span><span class="z-string"> /var/lib/docker/volumes/ plausible_event-data</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># starting containers</span></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> container start</span><span class="z-variable"> $pg_container_id $ch_container_id</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># encrypting</span></span> <span class="giallo-l"><span class="z-support z-function">echo</span><span class="z-variable"> $passphrase</span><span class="z-keyword z-operator"> |</span><span class="z-entity z-name"> gpg</span><span class="z-constant z-other"> --batch --yes --passphrase-fd</span><span class="z-constant z-numeric"> 0</span><span class="z-constant z-other"> -o</span><span class="z-variable"> $pl_repo_gpg</span><span class="z-constant z-other"> --symmetric</span><span class="z-variable"> $pl_repo</span></span> <span class="giallo-l"><span class="z-support z-function">echo</span><span class="z-variable"> $passphrase</span><span class="z-keyword z-operator"> |</span><span class="z-entity z-name"> gpg</span><span class="z-constant z-other"> --batch --yes --passphrase-fd</span><span class="z-constant z-numeric"> 0</span><span class="z-constant z-other"> -o</span><span class="z-variable"> $pg_data_gpg</span><span class="z-constant z-other"> --symmetric</span><span class="z-variable"> $pg_data</span></span> <span class="giallo-l"><span class="z-support z-function">echo</span><span class="z-variable"> $passphrase</span><span class="z-keyword z-operator"> |</span><span class="z-entity z-name"> gpg</span><span class="z-constant z-other"> --batch --yes --passphrase-fd</span><span class="z-constant z-numeric"> 0</span><span class="z-constant z-other"> -o</span><span class="z-variable"> $ch_data_gpg</span><span class="z-constant z-other"> --symmetric</span><span class="z-variable"> $ch_data</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># saving to s3</span></span> <span class="giallo-l"><span class="z-entity z-name">rclone</span><span class="z-string"> copy</span><span class="z-variable"> $pl_repo_gpg $dest</span><span class="z-string">:</span><span class="z-variable">$bucket</span></span> <span class="giallo-l"><span class="z-entity z-name">rclone</span><span class="z-string"> copy</span><span class="z-variable"> $pg_data_gpg $dest</span><span class="z-string">:</span><span class="z-variable">$bucket</span></span> <span class="giallo-l"><span class="z-entity z-name">rclone</span><span class="z-string"> copy</span><span class="z-variable"> $ch_data_gpg $dest</span><span class="z-string">:</span><span class="z-variable">$bucket</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment"># deleting local copies</span></span> <span class="giallo-l"><span class="z-entity z-name">rm</span><span class="z-constant z-other"> -f</span><span class="z-variable"> $pl_repo $pg_data $ch_data</span></span> <span class="giallo-l"><span class="z-entity z-name">rm</span><span class="z-constant z-other"> -f</span><span class="z-variable"> $pl_repo_gpg $pg_data_gpg $ch_data_gpg</span></span></code></pre> Automate Postgres Database Backup and Upload to S3 2023-10-20T00:00:00+00:00 2023-10-20T00:00:00+00:00 Vivek Shukla https://vshukla.com/tech/automate-postgres-database-backup-and-upload/

In this tutorial we will create a bash script to automate postgres database backup from a docker container and native postgres installation then upload to S3 based storage system.

<p>Database is a very critical system for an application. To prepare for any disaster we all must take regular backups of our database.</p> <p>In this tutorial we will create a bash script to automate <a href="/tech/postgres-database-backup-from-docker-container">postgres database backup from a docker container</a> and native postgres installation then upload to S3 based storage system.</p> <h2 id="what-we-will-cover"><a class="zola-anchor" href="#what-we-will-cover" aria-label="Anchor link for: what-we-will-cover">🔗</a>What we will cover</h2> <ol> <li> <p>Write db backup automation script for docker and non-docker based system</p> </li> <li> <p>Upload backup file to AWS S3</p> </li> <li> <p>Setup the cron job for daily backup</p> </li> </ol> <h2 id="steps"><a class="zola-anchor" href="#steps" aria-label="Anchor link for: steps">🔗</a>Steps</h2> <ol> <li> <p>Move to your project folder, where docker-compose.yml is.</p> </li> <li> <p>Take the database backup in container</p> </li> <li> <p>Copy the backup file from container filesystem to our desired location</p> </li> <li> <p>Delete the backup file inside the container</p> </li> <li> <p>Upload backup file to S3</p> </li> <li> <p>Delete the local backup file</p> </li> </ol> <p>Code snippet below covers step 1 to 4. But it is not optimal, later we will make modification so that we can easily automate it and make it re-usable.</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-support z-function">cd</span><span class="z-string"> project_dir</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose exec db_service_name sh</span><span class="z-constant z-other"> -c</span><span class="z-string"> "pg_dump -U db_user db_name > filename"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose cp db_service_name:filename destination/filename</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose exec db_service_name rm filename</span></span></code></pre><h2 id="upload-to-s3"><a class="zola-anchor" href="#upload-to-s3" aria-label="Anchor link for: upload-to-s3">🔗</a>Upload to S3</h2> <p>To upload file to S3, we need to have aws cli installed, you can follow <a rel="noopener external" target="_blank" href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">this guide from AWS docs to install awscli</a>.</p> <p>After installing awscli, you need to run <code>aws configure</code> and provide your key id and key secret. You can use any S3 based storage system here.</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span>aws s3 cp filename s3</span><span class="z-punctuation z-separator">:</span><span class="z-keyword z-operator">//</span><span>your</span><span class="z-keyword z-operator">-</span><span>bucket</span></span></code></pre> <p>If you are using some other S3 based storage system then you’ll have to provide custom <code>--endpoint-url</code> argument here, make sure you read the respective docs for this.</p> <h2 id="full-script"><a class="zola-anchor" href="#full-script" aria-label="Anchor link for: full-script">🔗</a>Full Script</h2> <p>We have laid out all the steps and code to accomplish that, now let’s modify in such a way that we can easily automate it and make it reusable.</p> <p>The filename is created based on the current datetime of the system, so that we can avoid ambiguity and exactly know how old is the backup file.</p> <h3 id="for-docker-compose"><a class="zola-anchor" href="#for-docker-compose" aria-label="Anchor link for: for-docker-compose">🔗</a>For docker compose</h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-comment">#!/bin/bash</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">############## ✅ Update These Values ##################</span></span> <span class="giallo-l"><span class="z-variable">directory</span><span class="z-keyword z-operator">=</span><span class="z-string">"your-project-dir"</span></span> <span class="giallo-l"><span class="z-variable">home</span><span class="z-keyword z-operator">=</span><span class="z-string">"/home/<user>"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable">db_service_name</span><span class="z-keyword z-operator">=</span><span class="z-string">"db"</span></span> <span class="giallo-l"><span class="z-variable">db_user</span><span class="z-keyword z-operator">=</span><span class="z-string">"db_user"</span></span> <span class="giallo-l"><span class="z-variable">db_name</span><span class="z-keyword z-operator">=</span><span class="z-string">"db_name"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">#####################################################</span></span> <span class="giallo-l"><span class="z-comment">############ 🔴 CHANGE WITH CAUTION 🔴 #############</span></span> <span class="giallo-l"><span class="z-variable">date</span><span class="z-keyword z-operator">=</span><span>$(</span><span class="z-entity z-name">date</span><span class="z-string"> +"%Y-%m-%d-%H:%M:%S"</span><span>)</span></span> <span class="giallo-l"><span class="z-variable">filename</span><span class="z-keyword z-operator">=</span><span class="z-variable">$db_name</span><span class="z-string">"_"</span><span class="z-variable">$date</span><span class="z-string">".sql"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-support z-function">cd</span><span class="z-variable"> $home</span></span> <span class="giallo-l"><span class="z-support z-function">cd</span><span class="z-variable"> $directory</span></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose exec</span><span class="z-variable"> $db_service_name</span><span class="z-string"> sh</span><span class="z-constant z-other"> -c</span><span class="z-string"> "pg_dump -U </span><span class="z-variable">$db_user $db_name</span><span class="z-string"> > </span><span class="z-variable">$filename</span><span class="z-string">"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose cp</span><span class="z-variable"> $db_service_name</span><span class="z-string">:</span><span class="z-variable">$filename $home</span><span class="z-string">"/"</span><span class="z-variable">$filename</span></span> <span class="giallo-l"><span class="z-entity z-name">docker</span><span class="z-string"> compose exec</span><span class="z-variable"> $db_service_name</span><span class="z-string"> rm</span><span class="z-variable"> $filename</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-support z-function">cd</span><span class="z-variable"> $home</span></span> <span class="giallo-l"><span class="z-comment">#####################################################</span></span> <span class="giallo-l"><span class="z-comment">#####################################################</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">############## ✅ CAN CHANGE BELOW ##################</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3 cp</span><span class="z-variable"> $filename</span><span class="z-string"> s3://your-bucket</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">rm</span><span class="z-variable"> $filename</span></span></code></pre> <p>At the top, you’ve to update variable values based on your project and directory structure, and the main code will use that to backup the database.</p> <p>At the end of script, we have aws command to copy the backup file to S3, here you need to update bucket name and if you are using other S3 storage provider (other than AWS), then <code>--endpoint-url</code> argument with appropriate value.</p> <h3 id="for-native-postgres-installation"><a class="zola-anchor" href="#for-native-postgres-installation" aria-label="Anchor link for: for-native-postgres-installation">🔗</a>For Native Postgres Installation</h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-comment">#!/bin/bash</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">############## ✅ Update These Values ##################</span></span> <span class="giallo-l"><span class="z-variable">home</span><span class="z-keyword z-operator">=</span><span class="z-string">"/home/<user>"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable">db_user</span><span class="z-keyword z-operator">=</span><span class="z-string">"db_user"</span></span> <span class="giallo-l"><span class="z-variable">db_name</span><span class="z-keyword z-operator">=</span><span class="z-string">"db_name"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">#####################################################</span></span> <span class="giallo-l"><span class="z-comment">############ 🔴 CHANGE WITH CAUTION 🔴 ##############</span></span> <span class="giallo-l"><span class="z-variable">date</span><span class="z-keyword z-operator">=</span><span>$(</span><span class="z-entity z-name">date</span><span class="z-string"> +"%Y-%m-%d-%H:%M:%S"</span><span>)</span></span> <span class="giallo-l"><span class="z-variable">filename</span><span class="z-keyword z-operator">=</span><span class="z-variable">$db_name</span><span class="z-string">"_"</span><span class="z-variable">$date</span><span class="z-string">".sql"</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">ch</span><span class="z-variable"> $home</span></span> <span class="giallo-l"><span class="z-entity z-name">pg_dump</span><span class="z-constant z-other"> -U</span><span class="z-variable"> $db_user $db_name</span><span class="z-keyword z-operator"> ></span><span class="z-variable"> $filename</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">#####################################################</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-comment">############## ✅ CAN CHANGE BELOW ##################</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3 cp</span><span class="z-variable"> $filename</span><span class="z-string"> s3://urlify-db-backup</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">rm</span><span class="z-variable"> $filename</span></span></code></pre> <p>When running pg_dump in the native installation, there are two ways postgres will try to authenticate the user.</p> <ol> <li> <p><a rel="noopener external" target="_blank" href="https://www.postgresql.org/docs/current/auth-peer.html">Peer based authentication</a>: It will check if the current user (system logged in user) is same as user passed to pg_dump.</p> </li> <li> <p>If peer auth fails then postgres looks for <code>.pgpass</code> file in home directory for the password.</p> </li> <li> <p>If both fails then it might ask for password.</p> </li> </ol> <p>So, make sure you use 1st method and run the script on the same user as the one passed to pg_dump.</p> <h2 id="setup-cron-job"><a class="zola-anchor" href="#setup-cron-job" aria-label="Anchor link for: setup-cron-job">🔗</a>Setup Cron Job</h2> <p>To automate backup and upload of our database, we will use crontab. It comes built-in with almost all the Linux distros and are very easy to setup.</p> <p>Give your script executable permission:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">chmod</span><span class="z-string"> +x db_backup.sh</span></span></code></pre> <p>Open crontab:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">crontab</span><span class="z-constant z-other"> -e</span></span></code></pre> <p>This will open up crontab in your prefered choice of editor. Use the below statement to run the backup script everyday at 12 PM.</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>0 12 * * * /home/<user>/db_backup.sh >> cron.log 2>&1</span></span></code></pre> <p>Create cron.log file</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">touch</span><span class="z-string"> cron.log</span></span></code></pre> <p>Use full path if cron.log is not in current user’s home folder.</p>