Migrating your Django Project to Heroku

Migrating your Django Project to Heroku

by Real Python advanced databases devops django web-dev

In this tutorial, we’ll be taking a simple local Django project, backed by a MySQL database, and converting it to run on Heroku. Amazon S3 will be used to host our static files, while Fabric will automate the deployment process.

The Project is a simple message system. It could be a todo app or a blog or even a Twitter clone. To simulate a real-live scenario, the Project will first be created with a MySQL backend, then converted to Postgres for deployment on Heroku. I’ve personally had five or six projects where I’ve had to do this exact thing: convert a local Project, backed with MySQL, to a live app on Heroku.

Setup

Pre-requisites

  1. Read the official Django Quick Start guide over at Heroku. Just read it. This will help you get a feel for what we’ll be accomplishing in this tutorial. We’ll be using the official tutorial as a guide for our own, more advanced deployment process.
  2. Create an AWS account and set up an active S3 bucket.
  3. Install MySQL.

Let’s begin

Start by downloading the test Project here, unzip, then activate a virtualenv:

Shell
$ cd django_heroku_deploy
$ virtualenv --no-site-packages myenv
$ source myenv/bin/activate

Create a new repository on Github:

Shell
$ curl -u 'USER' https://api.github.com/user/repos -d '{"name":"REPO"}'

Make sure to replace the all caps KEYWORDS with your own settings. For example: curl -u 'mjhea0' https://api.github.com/user/repos -d '{"name":"django-deploy-heroku-s3"}'

Add a readme file, initialize the local Git repo, then PUSH the local copy to Github:

Shell
$ touch README.md
$ git init
$ git add .
$ git commit -am "initial"
$ git remote add origin https://github.com/username/Hello-World.git
$ git push origin master

Be sure to change the URL to your repo’s URL that you created in the previous step.

Set up a new MySQL database called django_deploy:

SQL
$ mysql.server start
$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 1
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
mysql> CREATE DATABASE django_deploy;
Query OK, 1 row affected (0.01 sec)
mysql>
mysql> quit
Bye

Update settings.py:

Python
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'django_deploy',
        'USER': 'root',
        'PASSWORD': 'your_password',
    }
}

Install the dependencies:

Shell
$ pip install -r requirements.txt
$ python manage.py syncdb
$ python manage.py runserver

Run the server at http://localhost:8000/admin/, and make sure you can log in to the admin. Add a few items to the Whatever object. Kill the server.

Convert from MySQL to Postgres

Note: In this hypothetical situation, let’s pretend that you have been working on this Project for a while using MySQL and now you want to convert it to Postgres.

Install dependencies:

Shell
$ pip install psycopg2
$ pip install py-mysql2pgsql

Set up a Postgres database:

SQL
$ psql -h localhost
psql (9.2.4)
Type "help" for help.
michaelherman=# CREATE DATABASE django_deploy;
CREATE DATABASE
michaelherman=# \q

Migrate data:

Shell
$ py-mysql2pgsql

This command creates a file called mysql2pgsql.yml, containing the following info:

YAML
mysql:
  hostname: localhost
  port: 3306
  socket: /tmp/mysql.sock
  username: foo
  password: bar
  database: your_database_name
  compress: false
destination:
  postgres:
    hostname: localhost
    port: 5432
    username: foo
    password: bar
    database: your_database_name

Update this for your configuration. This example just covers the basic conversion. You can also include or exclude certain tables. See the full example here.

Transfer the data:

Shell
$ py-mysql2pgsql -v -f mysql2pgsql.yml

Once the data is transferred, be sure to update your settings.py file:

Python
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": "your_database_name",
        "USER": "foo",
        "PASSWORD": "bar",
        "HOST": "localhost",
        "PORT": "5432",
    }
}

Finally, resync the database, run the test server, and add another item to the database to ensure the conversion succeeded.

Add a local_settings.py file

By adding a local_settings.py file, you can extend the settings.py file with settings relevant to your local environment, while the main settings.py file is used solely for your staging and production environments.

Make sure you add local_settings.py to your .gitignore file in order to keep the file out of your repositories. Those who want to use or contribute to your project can then clone the repo and create their own local_settings.py file specific to their own local environment.

Although this method of using two settings files has been convention for a number of years, many Python developers now use another pattern called The One True Way. We may look at this pattern in a future tutorial.

Update settings.py

We need to make three changes to our current settings.py file:

Change DEBUG mode to false:

Python
DEBUG = False

Add the following code to the bottom of the file:

Python
# Allow all host hosts/domain names for this site
ALLOWED_HOSTS = ['*']

# Parse database configuration from $DATABASE_URL
import dj_database_url

DATABASES = { 'default' : dj_database_url.config()}

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

# try to load local_settings.py if it exists
try:
  from local_settings import *
except Exception as e:
  pass

Update the database settings:

Python
# we only need the engine name, as heroku takes care of the rest
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
    }
}

Create your local_settings.py file:

Shell
$ touch local_settings.py
$ pip install dj_database_url

Then add the following code:

Python
from settings import PROJECT_ROOT, SITE_ROOT
import os

DEBUG = True
TEMPLATE_DEBUG = True

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": "django_deploy",
        "USER": "foo",
        "PASSWORD": "bar",
        "HOST": "localhost",
        "PORT": "5432",
    }
}

Fire up the test server to make sure everything still works. Add a few more records to the database.

Heroku setup

Add a Procfile to the main directory:

Shell
$ touch Procfile

and add the following code to the file:

Text
web: python manage.py runserver 0.0.0.0:$PORT --noreload

Install the Heroku toolbelt:

Shell
$ pip install django-toolbelt

Freeze the dependencies:

Shell
$ pip freeze > requirements.txt

Update the wsgi.py file:

Python
from django.core.wsgi import get_wsgi_application
from dj_static import Cling

application = Cling(get_wsgi_application())

Test your Heroku settings locally:

Shell
$ foreman start

Navigate to http://localhost:5000/.

Looking good? Let’s get Amazon S3 running.

Amazon S3

Although it is hypothetically possible to host static files in your Heroku repo, it’s best to use a third party host, especially if you have a customer-facing application. S3 is easy to use, and it requires only a few changes to your settings.py file.

Install dependencies:

Shell
$ pip install django-storages
$ pip install boto

Add storages and boto to your INSTALLED_APPS in “settings.py”

Add the following code to the bottom of “settings.py”:

Python
# Storage on S3 settings are stored as os.environs to keep settings.py clean
if not DEBUG:
   AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
   AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
   AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
   STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
   S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
   STATIC_URL = S3_URL

The AWS environment-dependent settings are stored as environmental variables. So we don’t have to set these from the terminal each time we run the development server, we can set these in our virtualenv activate script. Grab the AWS Bucket name, Access Key ID, and Secret Access Key from S3. Open myenv/bin/activate and append the following code (make sure to add in your specific info you just pulled from S3):

Python
# S3 deployment info
export AWS_STORAGE_BUCKET_NAME=[YOUR AWS S3 BUCKET NAME]
export AWS_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX

Deactivate and reactivate your virtualenv, then launch the local server to make sure the changes took affect:

Shell
$ foreman start

Kill the server, then update the requirements.txt file:

Shell
$ pip freeze > requirements.txt

Push to Github and Heroku

Let’s backup our files to Github before PUSHing to Heroku:

Shell
$ git add .
$ git commit -m "update project for heroku and S3"
$ git push -u origin master

Create a Heroku Project/Repo:

Shell
$ heroku create <name>

Name it whatever you’d like.

PUSH to Heroku:

Shell
$ git push heroku master

Send the AWS environmental variables to Heroku

Shell
$ heroku config:set AWS_STORAGE_BUCKET_NAME=[YOUR AWS S3 BUCKET NAME]
$ heroku config:set AWS_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
$ heroku config:set AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX

Collect the static files and send to Amazon:

Shell
$ heroku run python manage.py collectstatic

Add development database:

Shell
$ heroku addons:add heroku-postgresql:dev
Adding heroku-postgresql on deploy_django... done, v13 (free)
Attached as HEROKU_POSTGRESQL_COPPER_URL
Database has been created and is available
! This database is empty. If upgrading, you can transfer
! data from another database with pgbackups:restore.
Use `heroku addons:docs heroku-postgresql` to view documentation.
$ heroku pg:promote HEROKU_POSTGRESQL_COPPER_URL
Promoting HEROKU_POSTGRESQL_COPPER_URL to DATABASE_URL... done

Now sync the DB:

Shell
$ heroku run python manage.py syncdb

Data transfer

We need to transfer the data from the local database to the production database.

Install the Heroku PGBackups add-on:

Shell
$ heroku addons:add pgbackups

Dump your local database:

Shell
$ pg_dump -h localhost  -Fc library  > db.dump

In order for Heroku to access db dump, you need to upload it to the Internet somewhere. You could use a personal website, dropbox, or S3. I simply uploaded it to the S3 bucket.

Import the dump to Heroku:

Shell
$ heroku pgbackups:restore DATABASE http://www.example.com/db.dump

Test

Let’s test to make sure everything works.

First, update your allowed hosts to your specific domain in settings.py:

Shell
ALLOWED_HOSTS = ['[your-project-name].herokuapp.com']

Check out your app:

Shell
$ heroku open

Fabric

Fabric is used for automating deployment of your application.

Install:

Shell
$ pip install fabric

Create the fabfile:

Shell
$ touch fabfile.py

Then add the following code:

Python
from fabric.api import local

def deploy():
   local('pip freeze > requirements.txt')
   local('git add .')
   print("enter your git commit comment: ")
   comment = raw_input()
   local('git commit -m "%s"' % comment)
   local('git push -u origin master')
   local('heroku maintenance:on')
   local('git push heroku master')
   local('heroku maintenance:off')

Test:

Shell
$ fab deploy

Have questions or comments? Join the discussion below.

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

Master Real-World Python Skills With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal.


Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!

Keep Learning