Everything You Need to Know About Python Environment Variables

Environment variables are powerful mechanisms for storing configuration details of your development project flexibly and securely. They store key-value pairs of data accessible by programming languages such as Python and keep that information hidden from prying eyes. In this tutorial, you will learn must-know concepts and techniques to create and manage environment variables in Python.

What Are Environment Variables?

Environment variables are name-value pairs stored somewhere safe in your operation system. Most often, they look like this:

API_KEY=1akn4nka-aqwej3

In Python, you can use such variables to store sensitive information related to your development project. Sensitive information can be:

• API keys to access others’ applications
• Username and passwords to log in to applications through scripts
• Database credentials
• anything that might cause a security issue if exposed accidentally.

For example, if you write a Python script that openly uses your AWS credentials and accidentally commit that script to GitHub, there is a chance malicious parties discover it and significantly increase your AWS bill (this is known to happen).

Another benefit of environment variables is configurability. You can easily adjust settings (database URLs, file paths) by modifying environment variables without changing your code. This feature is especially helpful if you use the same setting in multiple parts of your project.

So, let’s learn how to work with them in Python.

How To Retrieve Environment Variables in Python’s os Module

To retrieve existing environment variables in your system, you can use the os module. It has the .getenv function to retrieve variables:

import os
os.getenv("USER")

'bexgboost'

Above, I am retrieving the system user name, which is built into all systems. There are many others like the home path:

os.getenv("HOME")

'/home/bexgboost'

Or the path to your Conda executable:

os.getenv("CONDA_EXE")

'/home/bexgboost/anaconda3/bin/conda'

So, when you run conda commands, that's how your terminal knows which application to run.

But what happens if you try to retrieve a variable that doesn't exist:

os.getenv("MYDATABASE")

The .getenv() function doesn't return anything, but we can change that behavior. You can use its default parameter to return a custom value if a variable doesn't exist:

os.getenv("COMPUTER_PASSWORD", default="Nope!")

'Nope!'

The .getenv() function is the best method to retrieve existing variables.

However, in other sources, you might see different methods like the .environ attribute, which returns a dictionary containing all environment variables:

current_directory = os.environ.get("PWD", None)
current_directory

'/home/bexgboost/articles/2024/4_april'

Since it is a dictionary, you can use brackets notation (not recommended) or the .get() function to retrieve a value.

os also has access to perhaps the most important system variable called PATH. PATH holds the absolute paths to all the executables installed on your system, so it is pretty long:

os.getenv("PATH")[:46]

'/home/bexgboost/.local/bin:/home/bexgboost/bin'

Each path in PATH is separated by a colon. Let's count the length of PATH on my system:

def count_path_items():
   items = os.getenv("PATH").split(":")

   return len(items)

count_path_items()

79

79! Not bad.

Using the python-dotenv Library to Manage Environment Variables in Python Effectively

Setting custom environment variables

Now that we know how to extract existing variables, let’s see how to set custom ones. The first method is using the os.environ dictionary:

# Create a new environment variable
os.environ["NEW_VARIABLE"] = "42"  # Always expects a string value

os.getenv("NEW_VARIABLE")

'42'

However, this method isn’t very useful as all new variables will be lost once the current session ends.

Fortunately, there is a more lasting method of storing environment variables by using .env files (pronounced dot-env). These files have the same syntax, which means they will work on any operating system. Here is what a sample .env file looks like:

CUSTOM_API_LINK=https://myapi.com/v1/api
SNOWFLAKE_USERNAME=bexgboost
MYSQL_DATABASE_PASSWORD=as3ndf03

For now, it defines only three variables. To read this file and extract its contents, we will use a library called python-dotenv:

$ pip install python-dotenv

Then, in a Python script or a Jupyter Notebook we will import its load_dotenv function and call it:

from dotenv import load_dotenv

load_dotenv()

True

If it can’t find a .env file within the current directory, it will search all parent directories and return True if found.

I stored the above .env file in my home directory ~/.env; that's why load_dotenv could find it.

To create your own files, you can use the touch command in the terminal or use a text editor like VSCode:

$ touch ~/.env

The syntax of .env files is pretty flexible. For example, you can set variables with multi-line values (must have quotes):

LONG_ENV="This is an environment
variable with a long message"

Use escape characters:

LONG_ENV="This value uses an \"escape character"

You can also add comments to the file for future reference:

# You can also add comments anywhere with a hashtag
CUSTOM_API_LINK=https://myapi.com/v1/api
SNOWFLAKE_USERNAME=bexgboost
MYSQL_DATABASE_PASSWORD=as3ndf03

It is also possible to use the value of one variable inside another using the ${VAR_NAME} syntax:

FIRST_NAME=John
LAST_NAME=Doe
FULL_NAME="I am ${FIRST_NAME} ${LAST_NAME}"

Retrieving custom environment variables with python-dotenv

Retrieving custom variables is the same as before - just use the os module:

import os

os.getenv("CUSTOM_API_LINK")

'https://myapi.com/v1/api'

os.getenv("SNOWFLAKE_USERNAME")

'bexgboost'

Make sure that you call load_dotenv first.

If your .env file is located somewhere unreachable to load_dotenv, you can pass a direct path to it like below:

load_dotenv("/home/bexgboost/somewhere/unreachable/.env")

True

Working with .env files in Jupyter

Once you install Python-dotenv, you don't have to import it into Python if you are using Jupyter. The library comes with a Jupyter magic method that automatically loads it:

# Run this anywhere in the notebook to load EVs from .env
%load_ext dotenv
%dotenv

Then, you can use the os module again:

import os

os.getenv("SNOWFLAKE_USERNAME")

'bexgboost'

Best Practices For Using Environment Variables in Python

In this section, we will list some best practices to ensure environment variables stay safe and fully usable.

1. Always add .env files to .gitignore so that you don't accidentally commit them to GitHub.

# Create .gitignore if it doesn't exist
$ touch .gitignore
$ echo ".env" >> .gitignore

2. While defining variables use descriptive, all UPPERCASE names.

# Good
DEBUG=True
LOGGING_VERBOSITY=3

# Bad
debug=True
LV=3

3. Create a single .env file to store system-wide variables you need in every project.

$ touch ~/.env

4. Control the scope of your .env files. Create separate files for different parts of your development workflow like below:

$ touch development/.env.development
$ touch testing/.env.testing
$ touch production/.env.production

The load_dotenv function will recognize them.

If you create multiple .env files with different names, change your .gitignore file so that all of them are excluded from git indexing:

# Pay attention to the asterisk *
$ echo ".env*" >> .gitignore

Conclusion

In this article, we have learned all essential concepts and techniques to effectively manage environment variables in Python. We discovered how to retrieve existing variables with the Python OS module and how to create custom values with python-dotenv library. We also learned how to write .env files that conform to best practices.

If you want to learn more about Python software engineering, check out our course, Software Engineering Principles in Python.

You can also check out our Python Programming Skill Track, which will take you from a beginner to an advanced Pythonista with hands-on exercises and projects.