Dokcer-compose部署MinIO服务及Nginx反代

发布于 2021-08-10  849 次阅读


我又是为什么不用AWS呢?

为什么要自建S3?

主要还是出于成本考虑,BuyVM的硬盘挂载只要5美元1T,算是很低的价格了,跟其他的服务商比差了几倍。
我的BuyVM邀请连接

另一方面御三家的S3服务收费非常不透明,成本根本无法估计,实在是麻烦。
最后如果不信任大厂的数据隐私保护,自建S3也是个不错的选择。

BuyVM购买及挂载硬盘

BuyVM的后台相当反人类,有BuyVM主页https://buyvm.net/
下单用的https://my.frantech.ca/
以及后台https://manage.buyvm.net/
买完服务器之后所有的操作都是在后台完成。

注册的时候要保持IP手机号地址一致,不然似乎会被识别为欺诈。

购买完成后会有个邮件,点进去设置控制台密码,或者直接重置密码也可以。

进入后台之后直接重装系统,重装时可以设置root密码。然后把硬盘在后台挂载到VPS上。

重装完之后ssh进入系统,运行fdisk -l检查磁盘是否识别。第一个外接硬盘一般是显示sda,运行fdisk /dev/sda分区,按提示操作。
如果大于2T需要用parted,运行apt-get install parted安装。
运行parted /dev/sda分区,print显示信息,mklabel gpt设置分区表,运行mkpart primary 0GB 3299GB进行分区,这里的空间大小按照print的信息来即可,完成后退出parted。mkfs /dev/sda格式化,mkdir /data然后mount /dev/sda /data挂载,nano /etc/fstab编辑fstab,加上/dev/sda /data ext4 defaults 0 0让磁盘自动加载。最后运行df -h检查是否成功。

MinIO部署

Github项目主页
开源S3项目里算是一家独大了,所以也没什么好挑的,就拿来部署呗。

部署以官方文档为基础,附上我的docker-compose.yaml和nginx.conf。

docker-compose.yaml:

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: minio/minio:RELEASE.2021-08-05T22-01-19Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    MINIO_ROOT_USER: stsecurity
    MINIO_ROOT_PASSWORD: xxxxxxxxxxxx
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - /disk1/data1-1:/data1
      - /disk1/data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - /disk1/data2-1:/data1
      - /disk1/data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - /disk1/data3-1:/data1
      - /disk1/data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - /disk1/data4-1:/data1
      - /disk1/data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/:/var/log/nginx/
      - /var/www/html/:/www/data/
    ports:
      - "80:80"
      - "443:443"
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

server {
    listen      80;
    server_name s3.stsecurity.moe;
    location /.well-known/acme-challenge/ {
        root /www/data;
    }
    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen      80;
    server_name console.s3.stsecurity.moe;
    location /.well-known/acme-challenge/ {
        root /www/data;
    }
    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen      9001;
    location / {
        return 301 https://console.s3.stsecurity.moe$request_uri;
    }
}



    server {
        listen       443 ssl;
        listen  [::]:443 ssl;
        server_name  s3.stsecurity.moe;
    ssl_certificate     /var/log/nginx/ssl/miniofullchain.cer;
    ssl_certificate_key /var/log/nginx/ssl/miniokey.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen       443 ssl;
        listen  [::]:443 ssl;
        server_name  console.s3.stsecurity.moe;
    ssl_certificate     /var/log/nginx/ssl/consolefullchain.cer;
    ssl_certificate_key /var/log/nginx/ssl/consolekey.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
        location /ws/ {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }

    }
}

相比docs提供的来说改了不少,主要是为了能让nginx实现反代+ssl,并且让服务运行在443端口。

MINIO_ROOT_USER的长度至少为3,不然没法运行,我改配置的时候被坑了一把。
单独为控制台设置了域名,不然只能运行在其他端口,实在是难受。控制台里头也得加上websocket的反代,不然Diagnostic功能会报错。

使用MinIO

用户的Accesskey和Secretkey不能太短,不然没法授权,但是在创建的时候不会报错。

用起来就跟S3差不多,设置好host就行了。

重启Nginx可以使用docker exec -it minio_nginx_1 nginx -s reload命令。


Sup