How to upgrade my postgres in docker container while maintaining my data? 10.3 to latest 10.x or to 12.x

I have a 10.3 postgres docker container in my production and localhost.

In a previous question, I had to restore a dump that was archived in 10.5. Thanks to the answer I use plainformat to do so. But this is a temporary solution.

I like to know if there’s an easy way to upgrade my postgres version for my docker container in localhost and production.

In localhost, i have many databases and schema for development and exploration purposes.

In production, there are far fewer but of course the data is far more important.

I like to upgrade to a new version of postgres without jeopardizing the data and schema.

In localhost, my host OS is macOS 10.15 catalina. In production, the host OS is ubuntu headless server edition 18.04

Both production and localhost use teh same Dockerfile config

FROM postgres:10.3

COPY ./maintenance /usr/local/bin/maintenance
RUN chmod +x /usr/local/bin/maintenance/*
RUN mv /usr/local/bin/maintenance/* /usr/local/bin 
    && rmdir /usr/local/bin/maintenance

I did find this https://github.com/docker-library/postgres/issues/37#issuecomment-431317584 but I don’t have the conceptual understanding of what this comment is suggesting.

Also I found this library https://github.com/bwbroersma/docker-postgres-upgrade

Not sure how these two approaches are the same or different

Hope to get someone experienced with both Docker and Postgres for advice here.

What I have tried

This is my original local.yml for docker. Local because for local development environment.

version: "3.7"

volumes:
  postgres_data_local: {}
  postgres_backup_local: {}

services:
  django:
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    image: eno-a3-django_local_django
    depends_on:
      - postgres
      - mailhog
      - redis
    volumes:
      - .:/app
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres
    ports:
      - "8000:8000"
    command: /start

  postgres:
    build: ./compose/production/postgres/
    image: eno-a3-django_production_postgres
    volumes:
      - postgres_data_local:/var/lib/postgresql/data
      - postgres_backup_local:/backups
    env_file:
      - ./.envs/.local/.postgres
    ports:
      - "5432:5432"

  mailhog:
    image: mailhog/mailhog:v1.0.0
    ports:
      - "8025:8025"

  redis:
    build: ./compose/production/redis/
    container_name: redis
    restart: always

And then I thought I would create a new docker container.

So I changed to this

version: "3.7"

volumes:
  postgres_data_local: {}
  postgres_backup_local: {}

services:
  django:
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    image: eno-a3-django_local_django
    depends_on:
      - postgres
      - mailhog
      - redis
      - postgres_new
    volumes:
      - .:/app
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres
    ports:
      - "8000:8000"
    command: /start

  postgres:
    build: ./compose/production/postgres/
    image: eno-a3-django_production_postgres
    volumes:
      - postgres_data_local:/var/lib/postgresql/data
      - postgres_backup_local:/backups
    env_file:
      - ./.envs/.local/.postgres
    ports:
      - "5432:5432"

  postgres_new:
    build: ./compose/production/postgres_new/
    image: eno-a3-django_production_postgres_new
    volumes:
      - postgres_data_local:/var/lib/postgresql/data
      - postgres_backup_local:/backups
    env_file:
      - ./.envs/.local/.postgres_new
    ports:
      - "15432:5432"

  mailhog:
    image: mailhog/mailhog:v1.0.0
    ports:
      - "8025:8025"

  redis:
    build: ./compose/production/redis/
    container_name: redis
    restart: always

Notice how I use the same volumes for the postgres_new container.

The Dockerfile for postgres_new is

FROM postgres:10.13

COPY ./maintenance /usr/local/bin/maintenance
RUN chmod +x /usr/local/bin/maintenance/*
RUN mv /usr/local/bin/maintenance/* /usr/local/bin 
    && rmdir /usr/local/bin/maintenance

When I run my docker build, and logged in using port 15432, I can see my old database schema etc.

It appears that both containers can share the same data via the same volume.

And then I restore into this Docker container using a 10.5 archive file and it succeeded.

My commands I use to restore are like this in my host OS

docker cp ~/path/to/10.5.dump eno-a3-django_postgres_new_1:/backups
docker exec eno-a3-django_postgres_new_1 pg_restore -U debug -d 1013-replicatelive /backups/10.5.dump

Because I thought both containers are talking to the same volume, but when I connect to the old postgres container via 5432, I noticed that the new database that I create via the new postgres container 10.13, it was not there.

And it appears to work. Can I simply destroy the older container without accidentally destroying my existing data?

However..

When I changed some database value in a database (that I thought is common in the new postgres container) via port 5432 hence via the old postgres container, the change was not seen in the corresponding database in the new postgres container.

After I commented out the old postgres container in local.yml

I then only docker up the new postgres container and making it use the host port 5432. I can now see both the new schema (restored via host port 15432) and also the changes in the common database schema. So I guess this method works.

But why does it work? Is it because the volume is reused?