Vivek Shukla - File Upload

How To Store and Serve Private Media Files in Django

2023-09-14T00:00:00+00:00

In this blog post we will see how to use AWS S3</a> to store and serve private media files in django. S3 provides a layer of security by making object private and accessible to those with the valid presigned url</a> and is only valid for however long owner decides.

We will continue from our last blog post on How To Store Django Static and Media Files in AWS S3</a>. In this post we will only discuss parts which involves setting up private media.

🔗</a>Storage Backend</h2>

Like we had MediaStorage</code> for user uploaded public media files, we will need to create a new storage backend class for Private media files.

In your_project/custom_storage.py</code>

class</span> PrivateMediaStorage</span>(</span>S3Boto3Storage</span>):</span></span>
    location</span> =</span> "private"</span></span>
    default_acl</span> =</span> "private"</span></span>
    file_overwrite</span> =</span> False</span></span>
    custom_domain</span> =</span> False</span></span></code></pre>
location</strong>: This is the directory in the s3 bucket where all the private files will go. You can change the name as per your need.</p>
default_acl</strong>: ACL</a> stands for Access Control List, this enables us to manage access to bucket and objects. Here we are passing private</code> as we want the objects to be private.</p>
file_overwrite</strong>: We do not want files to be over-written if they have the same name.</p>
custom_domain</strong>: Earlier we’ve set AWS_S3_CUSTOM_DOMAIN</code> in our settings for static and media url generation, however here we want to generate a signed url so we need to set custom domain to False.</p>
🔗</a>Model</h2>
It is better to explicitly define the private media storage class in model fields so that we do not unnecessarily complicate other file fields which uses public MediaStorage.</p>
Therefore we need to explicitly pass the PrivateMediaStorage</code> to the model field to store it as private media and generate signed url when we serve.</p>
In your_project/your_app/models.py</code></p>
class</span> PrivateFile</span>(</span>models</span>.</span>Model</span>):</span></span>
    file</span> =</span> models</span>.</span>FileField</span>(storage</span>=</span>PrivateMediaStorage</span>()</span>,</span> upload_to</span>=</span>"files"</span>)</span></span>
    uploaded_on</span> =</span> models</span>.</span>DateTimeField</span>(auto_now_add</span>=</span>True</span>)</span></span>
</span>
    def</span> __str__</span>(</span>self</span>):</span></span>
        return</span> str</span>(</span>self</span>.</span>uploaded_on</span>.</span>date</span>())</span></span></code></pre>
file</strong>: This is the field which we are using to point to our private media. You can see we are passing our custom storage class to models.FileField which will now store all our uploads with the settings that we defined for PrivateMediaStorage</code> class.</p>
🔗</a>Conclusion</h2>
You can use the file field as you usually do, now all the uploaded files will be private. While calling field you will get a presigned url which will allow users to access the file.</p>
🔗</a>Things to keep in mind</h2>

FileField and ImageField in django only stores the location of the file and is created as varchar in database which has max_length of 100. So, if your file location is long then you may want to increase it’s allowed length by passing value to max_length</code>.</p>
FileField</a></p>
</blockquote>

The presigned url is only valid for 1 hour by default, you can change this by setting a different value for querystring_expire</code> (in seconds) in PrivateMediaStorage class.</p>
</blockquote>

How To Store Django Static and Media Files in AWS S3 2023-09-06T00:00:00+00:00 In this guide we will use django-storages package to manage our static and media files. Let’s walkthrough the whole process on how to store django static and media files in aws s3. S3 is a cloud based object storage service provided by AWS (Amazon Web Services), it stands for Simple Storage Service (S3). Using AWS S3 to serve Static and Media files in Django makes sense since Django is not good at serving those and S3 provides a lot more feature around these. 🔗</a>S3 Bucket</h2> 🔗</a>Create a bucket</h3> Let’s create a S3 Bucket where our files will be stored. Few things to keep in mind: ACL must be enabled, so that we can control object permission from our Django </li> Uncheck all “Block Public Access settings for this bucket” settings and check the acknowledge box </li> </ol> Create S3 bucket </figcaption></figure> 🔗</a>CORS</h3> After bucket is created successfully, now we need to update the CORS settings of the bucket so that if content is being served to different host then your browser might block it. Open the bucket </li> Go to the Permissions tab </li> At the end of the page there will be CORS settings </li> Put the JSON given below, make sure to put the host which will use the S3 files, you can put multiples. </li> </ul> [ { "AllowedHeaders": ["*"], "AllowedMethods": ["HEAD", "GET"], "AllowedOrigins": ["yourdomain.com"], "ExposeHeaders": [] } ]</code></pre>🔗</a>IAM User</h2> An IAM user allow us to access AWS resources programmatically. We need IAM credentials to upload the files to S3. 🔗</a>Create IAM User</h3> Go to IAM page in your AWS Console. </li> Go to Users then Add users. </li> Fill out the name that you want to give this user. </li> Go ahead and create the user, you can skip the permission. </li> </ol> Create IAM user </figcaption></figure> 🔗</a>Setting up Permission</h3> Open IAM user which you intend on using. </li> Make sure you are on Permissions (default) tab. </li> From “Add permissions” dropdown select “Create inline policy”. </li> Select JSON policy editor instead of Visual and paste below JSON. Make sure to update the bucket name. </li> Give the policy appropriate name and save the changes. </li> </ol> { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": ["arn:aws:s3:::your-bucket-name", "arn:aws:s3:::your-bucket-name/*"] } ] }</code></pre>🔗</a>Get the access key</h3> To use the IAM user in our django project we will need to generate access key. Open user page, go to Security credentials tab </li> Scroll down to Access keys section and click on Create access key button </li> Select “Local code” from the use case menu </li> </ol> IAM user access key </figcaption></figure> 🔗</a>Setting Up Django</h2> 🔗</a>Installing necessary packages</h3> We will be using django-storages</a> package to manage our connection with AWS S3 and to upload our files. We will also need boto3</a> package to connect to AWS, which django-storages uses. pip install boto3 django-storages 🔗</a>Storage Backend</h3> django-storages provides S3Boto3Storage</code> class for media storage and S3StaticStorage</code> for static files. We will subclass S3Boto3Storage</code> to create our custom storage class so that we can customize certain things. In your_project/custom_storage.py</code> from django.conf import settings from storages.backends.s3boto3 import S3Boto3Storage class StaticStorage(S3Boto3Storage): location = settings.STATICFILES_LOCATION default_acl = "public-read" querystring_auth = False class MediaStorage(S3Boto3Storage): location = settings.MEDIAFILES_LOCATION default_acl = "public-read" file_overwrite = False</code></pre>🔗</a>Update settings.py</h3> We also need to update our settings.py file with media and static files settings: STATICFILES_DIRS = [ BASE_DIR / "static_assets", ] AWS_S3_ACCESS_KEY_ID = "your_aws_key" AWS_S3_SECRET_ACCESS_KEY = "your_aws_secret" AWS_S3_REGION_NAME = "ap-south-1" # change to your region AWS_STORAGE_BUCKET_NAME = "your_bucket_name" AWS_DEFAULT_ACL = None AWS_S3_CUSTOM_DOMAIN = f"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com" # static files STATICFILES_LOCATION = "static" STATIC_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/{STATICFILES_LOCATION}/" # media files MEDIAFILES_LOCATION = "media" MEDIA_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/{MEDIAFILES_LOCATION}/" # storage backends STORAGES = { "default": {"BACKEND": "your_project.custom_storage.MediaStorage"}, # for media "staticfiles": {"BACKEND": "your_project.custom_storage.StaticStorage"}, } AWS_S3_OBJECT_PARAMETERS = { "CacheControl": "max-age=2592000", }</code></pre> We have done all the setup for our static and media files to be uploaded to our S3 bucket. You can test it out in your django project. You can also checkout the tutorial on how to create file upload api using django rest framework</a>. Django Rest Framework File Upload API 2023-06-21T00:00:00+00:00 Uploading file via API is a very common requirement. In this guide we will learn to build Django Rest Framework</a> File Upload API. 🔗</a>Setup</h2> Django: 4.2 </li> Django Rest Framework: 3.14.0 </li> Project Name: fileupload</code> </li> App Name: api</code> </li> </ul> 🔗</a>Add your app in INSTALLED_APPS</h3> In fileupload/settings.py</code> INSTALLED_APPS = [ ..... 'rest_framework', 'api', # your django app ]</code></pre>🔗</a>Update MEDIA path in settings.py</h3> We need to set the path where we want the uploaded files to be stored. In settings.py</code> MEDIA_URL = '/media/' MEDIA_ROOT = BASE_DIR / 'media'</code></pre>🔗</a>Update project’s urls.py file</h3> For Django to be able to serve media files we need to enable this using static</code> helper function that Django provides. This only works in DEBUG mode and only if folder location is local. source</a> In fileupload/urls.py</code> from django.conf import settings from django.conf.urls.static import static from django.contrib import admin from django.urls import include, path urlpatterns = [ path('admin/', admin.site.urls), path('api/', include('api.urls', namespace='api')), ] urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)</code></pre>🔗</a>Model</h2> Let’s create a Model where we store the file path (as file</code>) and file upload time (as uploaded_on</code>). So the uploaded file will be stored in media</code> folder and file</code> field will contain it’s path. In your api/models.py</code> class UploadedFile(models.Model): file = models.FileField() uploaded_on = models.DateTimeField(auto_now_add=True) def __str__(self): return self.uploaded_on.date()</code></pre> Run the migration python manage.py makemigrations python manage.py migrate</code></pre>🔗</a>Serializer</h2> Creating a ModelSerializer for our Model UploadedFile</code>. ModelSerializer creates serializer fields based on Model passed and comes with buil-in save</code> method that saves passed data into database. In your api/serializers.py</code> from rest_framework import serializers from .models import UploadedFile class FileUploadSerializer(serializers.ModelSerializer): class Meta: model = UploadedFile fields = ('file', 'uploaded_on',)</code></pre>🔗</a>View</h2> Now we create the view class FileUploadAPIView</code> (sub-class of APIView) to save the uploaded file. Few things to note: Generally the JSONParser</code> is set as the default parser in DRF so that we can send the request in JSON format. However it is not advised/convenient to upload file in JSON. </li> form-data</code> is the best way to upload the file, to enable this we need to pass MultiPartParser</code> and FormParser</code> as our parser_classes</code> in our view. </li> Only MultiPartParser</code> can be passed to enable file uploading but as per DRF API Guide on MultiPartParser</a>, we should use both parser classes to fully support HTML form data. </li> </ul> In api/views.py</code> from rest_framework import status from rest_framework.parsers import FormParser, MultiPartParser from rest_framework.response import Response from rest_framework.views import APIView from .serializers import FileUploadSerializer class FileUploadAPIView(APIView): parser_classes = (MultiPartParser, FormParser) serializer_class = FileUploadSerializer def post(self, request, *args, **kwargs): serializer = self.serializer_class(data=request.data) if serializer.is_valid(): # you can access the file like this from serializer # uploaded_file = serializer.validated_data["file"] serializer.save() return Response( serializer.data, status=status.HTTP_201_CREATED ) return Response( serializer.errors, status=status.HTTP_400_BAD_REQUEST )</code></pre>🔗</a>Urls</h2> Let’s connect our view in urls to be able to receive requests on it. In api/urls.py</code> from django.urls import path from .views import FileUploadAPIView, ImageUploadAPIView app_name = 'api' urlpatterns = [ path('upload-file/', FileUploadAPIView.as_view(), name='upload-file'), ]</code></pre>🔗</a>Testing API Endpoint</h2> Start the server python manage.py runserver</code></pre>🔗</a>Postman</h3> File upload using postman </figcaption></figure> Method: POST (as per our view) </li> URL: http://localhost:8000/api/upload-file/ </li> Body Type: form-data </li> Values to be passed file as KEY (change the type to File in dropdown), Browse and select a file for VALUE</li> </ul> </li> </ul> 🔗</a>cURL</h3> curl --location --request POST 'http://localhost:8000/api/upload-file/' --form 'file=@"/path/to/yourfile.pdf"'</code></pre>🔗</a>NodeJS - Axios</h3> var axios = require('axios'); var FormData = require('form-data'); var fs = require('fs'); var data = new FormData(); data.append('file', fs.createReadStream('/path/to/yourfile.pdf')); var config = { method: 'post', url: 'http://localhost:8000/api/upload-file/', headers: { ...data.getHeaders(), }, data: data, }; axios(config) .then(function (response) { console.log(JSON.stringify(response.data)); }) .catch(function (error) { console.log(error); });</code></pre> Checkout this guide to learn how to store static and media files in AWS S3</a>

Vivek Shukla - File Upload

How To Store and Serve Private Media Files in Django

How To Store Django Static and Media Files in AWS S3

🔗</a>S3 Bucket</h2> 🔗</a>Create a bucket</h3> Let’s create a S3 Bucket where our files will be stored. Few things to keep in mind:</p> ACL must be enabled, so that we can control object permission from our Django</p> </li>

🔗</a>Create a bucket</h3> Let’s create a S3 Bucket where our files will be stored. Few things to keep in mind:</p> ACL must be enabled, so that we can control object permission from our Django</p> </li>

🔗</a>Create IAM User</h3> Go to IAM page in your AWS Console.</p> </li> Go to Users</strong> then Add users</strong>.</p> </li> Fill out the name that you want to give this user.</p> </li>

🔗</a>Get the access key</h3> To use the IAM user in our django project we will need to generate access key.</p> Open user page, go to Security credentials</strong> tab</p> </li> Scroll down to Access keys</strong> section and click on Create access key</strong> button</p> </li>

Django Rest Framework File Upload API