Obsidian-Main/03. 資料收集/HTTP Server/Nginx.md

Reverse Proxy(Layer4)

03. 資料收集/HTTP Server/Nginx#Reverse Proxy所用的方法雖然可以反向代理多個網站，但是對於像是Trojan這種TLS不行被中斷的服務來說，會導致handshake失敗，所以需要用Nginx的stream來做Layer 4的轉發。

docker-compose.yaml

需要先把nginx.conf與mime.types給copy到data目錄下。 依序執行下面2個命令：

sudo docker run --rm -it nginx cat /etc/nginx/nginx.conf > nginx.conf
sudo docker run --rm -it nginx cat /etc/nginx/mime.types > mime.types

然後：

mkdir data ;\
mv nginx.conf mime.types data

建立docker-compose.yaml

version: '3'
services:
  nginx_reverseproxy_l4:
    container_name: nginx
    restart: always
    image: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./data:/etc/nginx

修改data/nginx.conf：

events {
    worker_connections  1024; <-- event 不用動
}

stream {
    map $ssl_preread_server_name $backend_name {
        tjn.awin.one             trojan;
        storj.awin.one           swag;
    }

    # trojan
    upstream trojan {
        server 192.168.1.31:443;
    }

    # swag
    upstream swag {
        server 192.168.1.20:44320;
    }

    # 监听 443 并开启 ssl_preread
    server {
        listen      80 reuseport;
        listen      443 reuseport;
        listen      [::]:443 reuseport;
        proxy_pass  $backend_name;
        ssl_preread on;
    }
}

http {
    ...                     <-- http 不用動
}

2022/05/17更新：

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    map $ssl_preread_server_name $backend_name {
        tjn.awin.one             trojan;
        storj.awin.one           swag;
        blog.awin.one            swag;
        gitea.awin.one           swag;
    }

    # trojan
    upstream trojan {
        server 192.168.1.31:443;
    }

    # swag
    upstream swag {
        server 192.168.1.20:44320;
    }

    # 监听 443 并开启 ssl_preread
    server {
        listen      443 reuseport;
        listen      [::]:443 reuseport;
        proxy_pass  $backend_name;
        ssl_preread on;
    }
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    ## 新增這一段，沒有這一段會造成http無法連線，所以拿憑證會失敗
    server {
        listen 80;
        server_name tjn.awin.one;

        location / {
            proxy_pass http://192.168.1.31:80;

            # 把 IP、Protocol 等 header 都一起送給反向代理的 server
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        }
    }
}

重點來源：

1. Trojan 共用 443 端口方案 - 程小白
2. NaiveProxy + Trojan + 多HTTPS站点共存，复用443端口 | 心底的河流
3. $ssl_preread_server_name的官方說明：Module ngx_stream_ssl_preread_module

Reverse Proxy

重要：確定docker與docker-compose已經安裝好。 參考RaspberryPi#Docker與RaspberryPi#docker-compose

Use SWAG docker

swag(之前叫做letsencrypt)是linuxserver.io包裝的Nginx webserver和reverse proxy的container。

Setup HTTPS

1. 建立folder

   mkdir -p ~/dockers/linuxserverswag ; cd ~/dockers/linuxserverswag
   

2. 建立docker-compose.yaml：

   vim docker-compose.yaml
   

   填入內容如下：

   version: "2.1"
   services:
     swag:
       image: ghcr.io/linuxserver/swag
       container_name: swag
       cap_add:
         - NET_ADMIN
       environment:
         - PUID=1000
         - PGID=1000
         - TZ=Asia/Taipei
         - URL=awin.one
         - SUBDOMAINS=wildcard
         - VALIDATION=dns
         - CERTPROVIDER= #optional
         - DNSPLUGIN=cloudflare #optional
         - PROPAGATION= #optional
         - DUCKDNSTOKEN= #optional
         - EMAIL=awinhuang@gmail.com
         - ONLY_SUBDOMAINS=false #optional
         - EXTRA_DOMAINS= #optional
         - STAGING=false #optional
         - MAXMINDDB_LICENSE_KEY= #optional
       volumes:
         - ./config:/config
       ports:
         - 44320:443
         - 8020:80 #optional
       restart: unless-stopped
   
   

3. 先跑一次：

   sudo docker-compose up
   

   會發現有錯誤，這是正常的，錯誤訊息像這樣：

   swag    | Unsafe permissions on credentials configuration file: /config/dns-conf/cloudflare.ini
   swag    | Cleaning up challenges
   swag    | Error determining zone_id: 9103 Unknown X-Auth-Key or X-Auth-Email. Please confirm that you have supplied valid Cloudflare API credentials. (Did you enter the correct email address and Global key?)
   swag    | ERROR: Cert does not exist! Please see the validation error above. Make sure you entered correct credentials into the /config/dns-conf/cloudflare.ini file.
   

   按ctrl + c退出。這時候config目錄也會有swag所mapping出來的相關檔案。修改config/dns-conf/cloudflare.ini：

   vim config/dns-conf/cloudflare.ini
   

   把config/dns-conf/cloudflare.ini改為：

   # Instructions: https://github.com/certbot/certbot/blob/master/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py#L20
   # Replace with your values
   
   With global api key:
   dns_cloudflare_email = awinhuang@gmail.com
   dns_cloudflare_api_key = <YOUR_API_KEY_FROM_CLOUDFLARE>
   
   # With token (comment out both lines above and uncomment below):
   #dns_cloudflare_api_token = 0123456789abcdef0123456789abcdef01234567
   

4. 再跑一次：

   sudo docker-compose up
   

   這一次就可以順利取得認證了，訊息像這樣：

   swag    | IMPORTANT NOTES:
   swag    |  - Congratulations! Your certificate and chain have been saved at:
   swag    |    /etc/letsencrypt/live/awin.one/fullchain.pem
   swag    |    Your key file has been saved at:
   swag    |    /etc/letsencrypt/live/awin.one/privkey.pem
   swag    |    Your certificate will expire on 2021-04-26. To obtain a new or
   swag    |    tweaked version of this certificate in the future, simply run
   swag    |    certbot again. To non-interactively renew *all* of your
   swag    |    certificates, run "certbot renew"
   swag    |  - If you like Certbot, please consider supporting our work by:
   swag    |
   swag    |    Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   swag    |    Donating to EFF:                    https://eff.org/donate-le
   swag    |
   swag    | New certificate generated; starting nginx
   swag    | Starting 2019/12/30, GeoIP2 databases require personal license key to download. Please retrieve a free license key from MaxMind,
   swag    | and add a new env variable "MAXMINDDB_LICENSE_KEY", set to your license key.
   swag    | [cont-init.d] 50-config: exited 0.
   swag    | [cont-init.d] 60-renew: executing...
   swag    | The cert does not expire within the next day. Letting the cron script handle the renewal attempts overnight (2:08am).
   swag    | [cont-init.d] 60-renew: exited 0.
   swag    | [cont-init.d] 70-templates: executing...
   swag    | [cont-init.d] 70-templates: exited 0.
   swag    | [cont-init.d] 99-custom-files: executing...
   swag    | [custom-init] no custom files found exiting...
   swag    | [cont-init.d] 99-custom-files: exited 0.
   swag    | [cont-init.d] done.
   swag    | [services.d] starting services
   swag    | [services.d] done.
   swag    | nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)
   swag    | Server ready
   

   最後一行的swag | Server ready表示server已經跑起來了。先按下ctrl + c退出，再來設定reverse proxy。

5. 修正config/dns-conf/cloudflare.ini的安全性問題

   cd ~/dockers/linuxserverswag ; chmod 600 config/dns-conf/cloudflare.ini
   

Setup reverse proxy

1. 建立folder：

   cd ~/dockers/linuxserverswag ; mkdir -p config/nginx/sites-available config/nginx/sites-enabled 
   

2. 建立以下檔案：

   • config/nginx/sites-available/common.conf，內容：

     add_header Strict-Transport-Security    "max-age=31536000; includeSubDomains" always;
     add_header X-Frame-Options              SAMEORIGIN;
     add_header X-Content-Type-Options       nosniff;
     add_header X-XSS-Protection             "1; mode=block";
     

   • config/nginx/sites-available/common_location.conf，內容：

     proxy_set_header    X-Real-IP           $remote_addr;
     proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
     proxy_set_header    X-Forwarded-Proto   $scheme;
     proxy_set_header    Host                $host;
     proxy_set_header    X-Forwarded-Host    $host;
     proxy_set_header    X-Forwarded-Port    $server_port;
     

   • config/nginx/sites-available/storj.conf，內容：

     upstream storj {
         server         192.168.1.11:14002;
     }
     
     server {
         listen         443 ssl;
         server_name    storj.awin.one;
     
         include        /config/nginx/sites-available/common.conf;
         include        /config/nginx/ssl.conf;
     
         location / {
             proxy_pass http://192.168.1.11:14002/;
             include    /config/nginx/sites-available/common_location.conf;
         }
     }
     
     

3. 在config/nginx/sites-enabled裡面建立要enable的config：

   cd config/nginx/sites-enabled ; ln -s ../sites-available/storj.conf . ; cd -
   

4. 修改config/nginx/nginx.conf：

   vim config/nginx/nginx.conf
   

   找到include /config/nginx/site-confs/*;這一行，把它comment掉，在下面新增一行：

   include /config/nginx/sites-enabled/*.conf;
   

5. 啟動swag：

   cd ~/dockers/linuxserverswag ; sudo docker-compose up -d
   

Restart

cd ~/dockers/linuxserverswag ; sudo docker-compose restart

Update certification

1. 進到dockr的swag bash裡面：sudo docker exec -it swag /bin/bash
2. 輸入 certbot renew
3. !

Trouble shooting

• 如果遇到類似下面的錯誤：

  ERROR: for swag  Cannot start service swag: driver failed programming external connectivity on endpoint swag (7c527d046631e0957de0b831ca25bed296de76e2eb96378964cb0110d7fb017d): Bind for 0.0.0.0:443 failed: port is already allocated.
  

  表示有其他程式佔住了80 port，可能是其他docker container或是service，必須先關閉它們。¹

參考來源

1. linuxserver/docker-swag: Nginx webserver and reverse proxy with php support and a built-in Certbot (Let's Encrypt) client. It also contains fail2ban for intrusion prevention.
2. How to set up an easy and secure reverse proxy with Docker, Nginx & Letsencrypt
3. SWAG setup - LinuxServer.io

---

• 參考
  • NGINX Docs | NGINX Reverse Proxy
  • Nginx 設定反向代理 Reverse Proxy
  • 用 Nginx 伺服器建立反向代理
  • 俄羅斯不愧是戰鬥民族：nginx - iT 邦幫忙::一起幫忙解決難題，拯救 IT 人的一天

新增網站

新網站自帶SSL

進到Rasperberry Pi（192.168.1.20）之後，切換到~/dockers/nginx_reverseproxy_l4，並編輯data/nginx.conf，在裡面加入新網站的網域與要轉址的ip。 例如，要加入一個trojan VPN，trojan VPN本身就有SSL加密，所以不用再經過swag，data/nginx.conf的改變如下： !

新http網站

如果新的網站只是一般的http網站，那便把它掛到swag後面，這樣就可以經由https來訪問，假如要加入一個blog網站，但因為我們有2層的reverse proxy，第一層是layer 4，第2層是swag，所以如果是自帶https的要掛到layer 4之後，沒有https要先由layer 4轉到swag再轉到實際的server上。 步驟如下：

1. 進到Rasperberry Pi（192.168.1.20）
2. 設定nginx_reverseproxy_l4，這邊我們需要把「沒有https」的網站由nginx_reverseproxy_l4導引到linuxserverswag

   cd ~/dockers/nginx_reverseproxy_l4
   vim data/nginx.conf
   

   加入下圖紅框的設定： !
3. 切換到~/dockers/linuxserverswag/config/nginx/sites-available
4. 新增一個confing檔，例如叫做blog.conf，內容如下：

   upstream blog {
       server         192.168.1.30:80;  ## 網址
   }
   
   server {
       listen         443 ssl;
       server_name    blog.awin.one;    ## 網域
   
       include        /config/nginx/sites-available/common.conf;
       include        /config/nginx/ssl.conf;
   
       location / {
           proxy_pass http://192.168.1.30:80/;   ## 網址
           include    /config/nginx/sites-available/common_location.conf;
       }
   }
   

   改變3個有註解的地方就可以了。
5. 剛剛的config檔，新增到sites-enabled裡面
   1. cd ~/dockers/linuxserverswag/config/nginx/sites-enabled
   2. ln -s ../sites-available/blog.conf .
6. 重新啟動swag：cd ~/dockers/linuxserverswag ; sudo docker-compose restart

---

1. 啟動docker時出現「Cannot start service :driver failed programming external connectivity on endpoint xxx, Bind for 0.0.0.0:80 failed: port is already allocated 」 ↩︎