Using Docker Compose based Python Interpreter in PyCharm

20-Nov-2020

Intro

I was involved with different kinds of Django projects in the past. Back then the standard approach of attaching your debug interpreter is by creating a virtual environment in your python project. We debug using PyCharm at that time. JetBrains has generously gave us free open source license to use the whole suite of JetBrains tools.

However, back then the notion of developing a Django project from inside a container is not so common. We already manage some of our projects as microservices described by a (or several) docker-compose file. To debug with this kind of setup, we are using SSH daemon inside our container. Then we setup PyCharm so that it treats the docker container as a remote interpreter. This is working for quite a long time (almost 5 years). Then finally JetBrains released some supports to allow interpreters inside a docker-compose configuration.

Things didn’t transition smoothly back then. First issue that I remember is that the interpreter forgets all of the environment variables declared in the docker-compose file. Since Django used environment variable to override it’s settings file, this setup were unusable. So we keep doing the old ways, using remote interpreter.

Recent PyCharm version is making it more difficult to set up a remote interpreter. As of now, in version 2020.2, when we set up a remote interpreter, PyCharm implicitly defined deployment configuration. This is not needed if you are using a container, because the files were already mounted there, you don’t need to copy it again using sftp, etc. However the new interface is quite confusing because you can’t disable the setting at first. You are only allowed to delete the deployment configuration after you’ve made the configuration (funny, eh?). I’ve also tried different approach, such as creating the SSH configuration first, then set it as remote interpreter, but still it generate the deployment configuration.

So, fed up with this, I decided to try the docker-compose-based interpreter again. I noticed several improvements:

  1. You can now include multiple docker-compose recipes (useful for overriding local config on top of production config)
  2. You are allowed to include an environment file (they fixed the main deal breaker from previous issue)
  3. You can map local directory with the directory inside container (so that PyCharm knew it was the same thing)
  4. When you create a Run Configuration, the environment the interpreter use is the one coming from docker-compose. So, you won’t need to redeclare your environment variables again. Hurray!

Requirements

This articles are going to be some sort of hands on labs/workshop. You can skim thru, but it is best if you do it at your own pace.

We are going to assume this basic understanding of technical skills:

  1. Git and how to use it to clone project from Github
  2. Docker, Docker-compose, and how to use their CLI
  3. Linux based environment, or MacOS, or WSL.
  4. Python and Django basic understanding
  5. Debugging methods/terminologies

In addition to that, since we are using PyCharm, you need PyCharm Pro Edition to use it’s debugging features.

Usual setup without IDE

In order to demonstrate how PyCharm enhance our development workflow — and not ruining it or conflicting with the current workflow —. I create a small sample repo here so you can see our basic Django setup. My explanations will refer to that repo. Clone the repo to start experimenting on it.

We are going to setup a small microservice based django server. The repo contains two folders

django_project
and
deployment
. The
deployment
folder contains orchestration script to run the project. The
django_project
is where the actual Django app is located.

To run the project, we (by we, I mean my colleague and myself) usually go into the deployment folder and run

make
scripts to spin up the server. We don’t have those now since we want to be a bit more technical and dive in. We are going to run docker-compose directly.

docker-compose up --build -d

This is just to warm up docker. We want to build our initial images and see it running. One good rule of thumb to help you later setup production build: Prepare one initial docker-compose that can be run immediately without having to do further customization so developers can easily check the code out.

If you open your browser and navigate to http://localhost/ . You will see the default welcome screen. It will only says

Welcome in localhost
.

Since using an IDE is a choice, I want to believe that running the project immediately should not depend heavily on the choice of IDE. The developer should be able to run this out of the box.

However, this is a minimal setup. When doing some custom development works, or deployment, you will have to change some deployment configuration. To illustrate this, I’ve prepared a simple template

.template.env
and
docker-compose.override.template.yml
.

But before we proceed, shutdown the current stack first, since we are going to change the compose config.

docker-compose down

Copy the file

.template.env
as just
.env
. This is docker compose environment config file. When interpolating variables in the docker-compose files, generally the precedence will be:

default value —> env file —> shell environment variables

This means if a variable is defined in the env file but not in the shell, then docker-compose will use whatever value defined in the env file. This is very flexible and support declarative deployment. For example, in the previous docker-compose command, if you look at the basic recipe

docker-compose.yml
You will see that we have defined environment variable inside django container to take value from the shell with some default value:

services:
  django:
    environment:
      DJANGO_SETTINGS_MODULE: mysite.settings
      DEBUG: ${DEBUG:-False}
      SITENAME: ${SITENAME:-localhost}

With this kind of variable passing, we can expect that the environment variables are accessible from within Django settings so we can make a fully config based deployment.

Now look at the

.env
file you just copied.

COMPOSE_PROJECT_NAME=mysite
COMPOSE_FILE=docker-compose.yml

DEBUG=True
SITENAME=myothersite.test

DJANGO_HTTP_PORT=8080
HTTP_PORT=80

We override some settings. Notably

DEBUG
variable to become
True
because we want to enable Django debug mode. You can experiment by changing other variables too. For the top two variables:
COMPOSE_PROJECT_NAME
and
COMPOSE_FILE
were docker-compose internal variables. Changing
COMPOSE_PROJECT_NAME
will change your docker-compose stack namespace (useful to quickly associate which stack with which project). Meanwhile,
COMPOSE_FILE
can control which docker-compose file are going to be included by
docker-compose
command.

We are going to create an extra compose file to override some deployment config. Change value into

COMPOSE_FILE=docker-compose.yml:docker-compose.override.yml
. A useful note worth knowing, if you omit
COMPOSE_FILE
variable entirely (make it empty value), then
docker-compose
default behaviour is to look for
docker-compose.yml
and
docker-compose.override.yml
if it exists. By the way, the precedence matter. Rightmost file mentioned in the variable will override files on the left.

Now, create

docker-compose.override.yml
from the template
docker-compose.override.template.yml
.

You can see the content like this:

version: '3'
services:
  django:
    command: python manage.py runserver 0.0.0.0:8080
    volumes:
    - ../django_project:/home/web/django_project

  nginx:
    volumes:
    - ./sites-enabled:/etc/nginx/conf.d

Since we are going to be in ‘development mode’, we use django manage.py server. We also mount our local directory so the container will always have latest changes in our files. We also mount nginx config in case we are dealing with production mode settings. You can even add more overrides like

environments
or
ports
depending on the need.

Now that we’re ready, spin up the stack (without rebuilding)

docker-compose up -d

You can then check again http://localhost:8080/ . Now it says

Welcome in myothersite.test
, which is the value of
SITENAME
that we declared in the
.env
file. It will also spawn a Django debug server in port 8080 by default. We setup two server like this because Django debug server supports hot reload. So if you change codes or templates, you may want to see the changes immediately. In that case, you can navigate to http://localhost:8080/ to see the changes.

This is as far as our native setup goes without IDE. Basically you do the development in the files in your host machine, but the python and Django server were deployed using container.

One other thing worth mention is how we run unittest. Since Django and Python is in a container, we run Django tests like this:

docker-compose exec django python manage.py test

As you can see, the command becomes very long. That’s why we store

Makefile
to provide a shortcut command in the deployment directory.

Before continuing the next section, don’t forget to shut down the stack

docker-compose down

IDE Setup Using PyCharm

To see how much PyCharm improve our workflow, using the same repo, open the folder in PyCharm.

Docker-compose setup

PyCharm bundled the docker-compose plugin integration by default. If for some reason you can’t use it, refer to official doc.

Open the

docker-compose.yml
file in the
deployment
directory. You will see that the lines are annotated by arrows like this:

docker-compose.yml file

You can click the arrow. Click the double arrow in the

services:
line. PyCharm will deploy that recipe for you. The service tab will appear to let you know that the compose file are deployed.

As you can see, it also pick up the

COMPOSE_PROJECT_NAME
that you specify in
.env
file.
mysite
is the name of the stack and you can drill it down to see the services. This tab offer some controls too. You can redeploy, stop, or down the deployment with just a click. Right click the deployment name to see the context menu.

After some sightseeing you will notice that what you deploy (even though it picks up

.env
file) is only the recipe from
docker-compose.yml
file. It is evident if you see the service log of django, it shows uWSGI running and not Django debug server.

So, edit the PyCharm configuration. You can do it from the deployment context menu then Edit Configuration, or deployment configuration bar on the top menu. Basically add the file

docker-compose.override.yml
in the Compose files field. So you will have something like this:

Now you can spin up/down your deployment stack with just a button click from your Services tab. By clicking deploy, you can update your stack deployment and recreate a fresh service. It is now running using Django Debug Server.

It’s quite convenient now to edit your docker-compose file and apply the changes to the deployment.

In a typical development session, you mostly run the deployment once, do some coding, and then validate it and maybe do some debugging. So we are going to set up that workflow too.

Python interpreter setup

We first setup the Python Interpreter.

Open PyCharm Project settings, then navigate to Project Interpreter. Click the gear icon and click Add. You will be given several options to select the source of your interpreter. Choose docker-compose. Fill in the settings, which consists of Configuration file(s) (Put

docker-compose.yml
and
docker-compose.override.yml
in that order), and the service (Pick
django
). It should looks like in the image below. Click OK.

It might take some times for PyCharm to build it’s custom images (with pycharm helpers inside the containers).

Next, you want to set project path mappings. The location of codes in your repo in your host computer are different with the locations inside the container. That’s why PyCharm needs to know it. Specify this project path mappings. In our case we map

django_project
into
/home/web/django_project
inside the container. This is as reflected in the volume declarations of our
docker-compose.override.yml
. In your own project, you need to decide by yourself what are the paths that needed to be mapped out, because it can be more than one.

Screenshot below can be used as visual guide:

Once you are done, click OK. Wait a bit for PyCharm to build the helper skeletons. Once it’s finish rebuilding, you can click Python Console tab to load the python console in PyCharm. This is the same Python interpreter that Django will use. You can also check that the variables from

.env
file carried over nicely as seen in the screenshots:

Note that this is a Python interpreter, not Django shell. To use Django, you need to enable Django integration.

Open Project Settings again, type Django in the search bar**.** In the Language and Frameworks menu, select Django. Enable Django Support. Fill in all relevant information for your project. In our example, the Django project root is in

django_project
directory, and the settings module is in
mysite.settings
. See screenshot below for reference:

After setting this up, whenever you navigate to the Python Console, you will get Django Console instead:

It’s the same thing you get from running

docker-compose exec django python manage.py shell
. Isn’t that sweet? As you can see, the settings file and environment variables from
.env
are properly loaded and evaluated as Django settings.

Django debug server setup

Now, let’s step up further and create a Django server run configuration.

Mark

django_project
directory as a Source Directory by right clicking the directory and select Mark Directory as > Sources Root . If it’s not detected already, mark the
django_project/mysite/templates
directory as Templates Folder. You will now be able to activate code completions in that folder.

To create a new Run/Debug Config (for Django now), click the Configuration selector, or navigate from menu Run > Edit Configuration. As you can see, you already have docker-compose run configuration. We now want to add Django run config. Click the + button and choose Django Server. Most of our config resides in

.env
file. So we don’t actually need to modify anything else here besides the target server address to match the port that we expose to in the container (currently 8080). See the screenshot for reference:

Click OK. Then you can click the Run Button with Django config selected (or whatever the name you gave for the config in previous step). Run button will run the Django Debug Server as usual, meanwhile Debug button will attach PyCharm debugger to Django. Let’s check what Debug button do.

Debug mode panel will show up after you click Debug button. It basically recreates service to attach debugger. You can also see the log in that panel, which is nice because you don’t need to go inside the container to see that now.

As with any debugging session, you can attach breakpoint to any of your python code in the sources folder. With PyCharm you can even attach breakpoint in the template! You can debug in realtime when the template is currently rendered. This is such an awesome features. Here’s what it looks like when you are currently debugging Django Template:

You can do many things in this mode, such as inspecting variables and so on. If for some reason this template debugger doesn’t work for you, check out the documentation and make sure that you set the current template language to be Django instead of Jinja.

Test runner setup

Then, the next tips that I would like to add is about running Django test.

One of the way is to click your test file or test folders, or even the whole Django app module then right click, then click Run test from the context menu. It will create a Run test configuration and run it. Since all of our config are declarative in the

.env
file, we don’t need to edit the config, unless you need to modify specific settings. Here’s what it looks like when we do that.

Other way of running the test, if you need just to run a single specific test, you can click the arrow button (also can be seen in the screenshot) in the line where the specific test method are declared in the test file. Then PyCharm will only run that.

Some (optional) beta setup

Finally, my last tips here is something that I really waiting for in PyCharm. From my conversation with @gamesbrainiac in twitter: https://twitter.com/gamesbrainiac/status/1320793098658762753 . It seems you can store Run configuration as a file now. So, if your project is complex and you have crafted specific run configuration. You can share it to your other dev in the team as well. Especially if the deployment is the exact same (because we are using declarative docker-based deploy). You can use it to allow developer to deploy latest changes to staging server with just a click. However this is more of a devops related topic. So, in general you can share run config to allow your colleague to run unittest or local deployment the exact same way. Just make sure that you don’t store sensitive data in it.

To demonstrate how feasible is this. First, let us imagine that we haven’t made any run configuration. We want to use what’s already been given in the repository. Open the repo inside PyCharm. The Run Configuration list will be empty. However, if there are run configuration in the repo, it can be detected. First open the Edit Configuration menu. As you can see there are 3 broken Configuration:

For the docker-compose configuration there is no other editing needed so you can just click OK and run the config.

For Django-based config. You need to enable Django support in the project settings first. The dialog conveniently give you Fix button so you can click on it.

Proceed to fix the problem by following the wizard. Typically what you need to do is define the interpreter again and enable Django support. After that you can run the config again.

I have somewhat conflicting opinion on this share run configuration thing. This kind of workflow seems pretty promising, but at the moment it feels like a beta feature and not solid enough. What it lacks at the moment is to store the interpreter information in a modular way, so it can be imported easily. This feature also not easily discoverable. If you store your run config as a file and persisted in a git repo, then your colleague can only discover it by opening the file and noticing that PyCharm gave warning that it is a run configuration file, which then you can proceed on clicking the warning and making it available in your local PyCharm installations. So take this last tips as optional beta.

When this feature are going to be solid enough, I think it is going to make PyCharm to be a very ideal Python IDE to collaborate. I imagine we can just put the config inside the repo and then your colleague can just do first setup with relative ease to run the project. Totally incredible.

Conclusions

I have demonstrated some amazing PyCharm capabilities for microservice based python debugging setup. It also has a potential to be one stop shop for creating a unified IDE setup for a team of developers, which can be productive for projects where the developer come and go. This will enable new developer to get in the workflow much easily, as everything has already been prepared from the start.

It’s also important to notice that this seamless integration were only possible if your project setup is declarative from the start. It’s also ok for developer who are not using PyCharm to start the project, but the developer who uses PyCharm is also using the same exact configuration. So that means, even though they have different method when they do their own development, they share the same, repeatable project config. It will allow them to use the same config and tailor it for their own specific IDE.


Rizky Maulana Nugraha

Written by Rizky Maulana Nugraha
Software Developer. Currently remotely working from Indonesia.
Twitter FollowGitHub followers