Why Docker
Dependency Conflicts
A lot of applications depend on other software, sometimes called dependencies or libraries. It could happen that two different apps require different versions of some dependency. This is called a dependency conflict.
Ubuntu, for one, resolves this by fixing a version for commonly used dependencies for every Ubuntu version, and expecting applications to ensure compatibility with the specified library versions. These cannot be upgraded until you upgrade the Ubuntu version. Applications that require some uncommon dependencies can either take the risk with an external dependency as they do with APT) or package the dependencies along with the application itself (as snaps, flatpaks, and AppImages do).
The most common solution to this problem in app development and hosting is a virtual environment. All dependencies are stored in a file - python calls it requirements.txt - and those dependencies are installed by anyone trying to install the application on their computers.
Security
There is always a risk that hackers can execute arbitrary commands on your computer. The best one can do is to reduce "attack surfaces" - reduce the possibly vulnerable components - and to have barriers in place so that arbitrary commands cannot do much damage. This is also the motivation behind having separate users for each server.
Barriers can come in many forms.
The most basic version is a chroot jail - a linux command to make every subsequent command believe that the rest of the filesystem does not exist. This is hard to set up, though. I've tried and didn't feel confident. You need to know which services you need to copy and which ones you need to link.
At the other extreme end is virtual machines - which is a whole computer running inside your computer, sharing some of its resources. Generally easier to set up, but it takes up a lot of extra resources.
A popular middle ground is "containerization", like Docker. Containers are like an extended version of chroot jails, functioning as a fully-fledged server computer running within another.
Docker installation instructions can be found here.
To save space on your root drive and use external storage instead, the Docker data folder can be specified in /etc/docker/daemon.json as {"data-root": "/path/to/newlocation"}
An Example Docker Command
docker run --name postgres-container -p 127.0.0.1:5432:5432 -v postgres-volume:/var/lib/postgresql -e POSTGRES_PASSWORD=mysecretpassword -d postgres
I can explain.
In Docker, an image is a snapshot of working software. A container is when that image has been run, has its own processes and data associated with it. A container running Postgres can be created from a image using docker run <options> postgres.
Containers can be allotted names automatically, or we can manually name it "postgres-container" with the --name postgres-container or --name <your-chosen-name> option.
When the container runs, it takes API calls on port 5432. The host machine can be part of multiple networks and have different IP addresses on each. We can have any requests to the IP address 127.0.0.1 (also known as localhost, it is accessible only to other services on the same computer) and port 5432 be forwarded to port 5432 of the container with the -p 127.0.0.1:5432:5432 or -p <host-ip-address>:<host-port>:<container-port> option.
The data in a container cannot be easily accessed by other applications, and is lost when the container is destroyed. We can allot a volume (storage space on the host computer) called "postgres-volume" that stores everything in the container's /var/lib/postgresql location (where postgres stores its data) with the -v postgres-volume:/var/lib/postgresql or -v <volume-name>:<mount-location> option.
Postgres also requires a password to be passed in as an environment variable, which is passed with -e POSTGRES_PASSWORD=mysecretpassword or -e <ENV_VAR_NAME>=<ENV_VAR_VALUE> option.
Finally, we have the container run in the background with the -d option.
Container Management
Containers can be stopped with docker container stop <container-name> and restarted with docker container start <container-name> after a restart.