云原生K8s的主页 - PintereStory

I recently noticed a post on Reddit which said "Why NGINX doesn’t support hot reloading?". Weirdly, NGINX, the world’s largest web server, doesn’t support hot reloading? Does it mean we all use nginx -s reload incorrectly? With this question, let’s check how NGINX reload works.

What is NGINX

NGINX is a cross-platform open-source web server developed in C language. According to statistics, among the top 1000 highest traffic websites, over 40% of them are using NGINX to handle the enormous requests.

Why is NGINX so popular

What advantages does NGINX have to beat the other web servers and always keep a high utilization rate?

The main reason is that NGINX is designed for handling high concurrency issues. Therefore, NGINX could provide a stable and efficient service while handling enormous concurrent requests. Moreover, compared to competitors of the same age, such as Apache and Tomcat, NGINX has many extraordinary designs like advanced event-driven architecture, a fully asynchronous mechanism to handle network IO, and an extremely efficient memory management system. These remarkable designs help NGINX entirely use servers’ hardware resources and make NGINX the web server's synonyms.

Apart from the above, there are other reasons like:

Highly modular design makes NGINX own enormous feature-rich official and extended third-party modules.
The freest BSD License makes developers willingly contribute to NGINX.
The support of hot reloading enables NGINX to provide 24/7 services.

Among the above reasons, hot reloading is our main topic today.

What does Hot Reload do

What is our expected hot reloading? First, I think client-side users shouldn’t be aware of the server reloading. Second, servers or upstream services should achieve dynamic loading and handle all user requests successfully without downtime.

Under what circumstances do we need hot reloading? In the Cloud Native era, microservices have become so popular that more and more application scenarios require frequent server-side modifications. These modifications, such as reverse proxy of domain online/offline, upstream address changes, and IP allowlist/blocklist change, are related to hot reloading.

So how does NGINX achieve hot reloading?

The Principle of NGINX Hot Reloading

When we execute the hot reload command nginx -s reload, it sends a HUP signal to the NGINX’s master process. When the master process receives the HUP signal, it will open listening ports sequentially and start a new worker process. Therefore, two worker processes (old & new) will exist simultaneously. After the new worker process enters, the master process will send a QUIT signal to the old worker process to gracefully shut it down. When the old worker receives the QUIT signal, it will first close the listening handler. Now, all new connections will only enter the new worker process, and the server will shut down the old process once it processes all remaining connections.

In theory, could NGINX’s hot reloading perfectly meet our requirements? Unfortunately, the answer is no. So, what kind of downsides does NGINX’s hot reloading have?

NGINX reload causes downtime

Too frequent hot reloading would make connections unstable and lose business data.

When NGINX executes the reload command, the old worker process will keep processing the existing connections and automatically disconnect once it processes all remaining requests. However, if the client hasn’t processed all requests, they will lose business data of the remaining requests forever. Of course, this would raise client-side users’ attention.
In some circumstances, the recycling time of the old worker process takes so long that it affects regular business.

For example, when we proxy WebSocket protocol, we can’t know whether a request has been processed because NGINX doesn’t parse the header frame. So even though the worker process receives the quit command from the master process, it can’t exit until these connections raise exceptions, time out, or disconnect.

Here is another example, when NGINX performs as the reverse proxy for TCP and UDP, it can’t know how often a request is being requested before it finally gets shut down.

Therefore, the old worker process usually takes a long time, especially in industries like live streaming, media, and speech recognition. Sometimes, the recycling time of the old worker process could reach half an hour or even longer. Meanwhile, if users frequently reload the server, it will create many shutting down processes and finally lead to NGINX OOM, which could seriously affect the business.

# always exist in old worker process:
nobody 6246 6241 0 10:51 ? 00:00:00 nginx: worker process
nobody 6247 6241 0 10:51 ? 00:00:00 nginx: worker process
nobody 6247 6241 0 10:51 ? 00:00:00 nginx: worker process
nobody 6248 6241 0 10:51 ? 00:00:00 nginx: worker process
nobody 6249 6241 0 10:51 ? 00:00:00 nginx: worker process
nobody 7995 10419 0 10:30 ? 00:20:37 nginx: worker process is shutting down  <= here
nobody 7995 10419 0 10:30 ? 00:20:37 nginx: worker process is shutting down
nobody 7996 10419 0 10:30 ? 00:20:37 nginx: worker process is shutting down

In summary, we could achieve hot reloading by executing nginx -s reload, which was enough in the past. However, microservices and Cloud Native’s rapid development make this solution no longer meet users’ requirements.

If your business’s update frequency is weekly or daily, then this NGINX reloading could still satisfy your needs. However, what if the update frequency turns hourly or per minute? For example, suppose you have 100 NGINX servers, and it reloads once an hour, then it would need to reload 2400 times per day; If the server reloads per minute, then it will need to reload 8,640,000 times per day, which is unacceptable.

We need a solution without process switching, and it could also achieve immediate content updates.

Hot Reloading That Takes Effects Immediately In Memory

When Apache APISIX was born, it was designed to resolve the NGINX hot reloading issue. APISIX is developed based on NGINX and Lua’s tech stacks, and it is a cloud-native, high-performance, fully dynamic microservice API gateway that uses etcd as its core configuration center. There is no need to reboot the server to reload the new server config, meaning any upstream services, routes, or plugin changes will not require a server reboot. But how could APISIX eliminate the limitations of NGINX to achieve perfect hot reloading based on the fact that APISIX is developed upon the NGINX tech stack?

First, let’s see the software architecture of Apache APISIX:

APISIX could achieve the perfect hot reloading since it puts all configurations into the APISIX Core and Plugin Runtime so that they could use the dynamic assignments. For example, when NGINX needs to configure parameters inside configuration files, each modification would only take effect after reload. To dynamically configure routes, Apache APISIX only configures one specific single server with only one location. We have to use this location as the main entry so that all requests would first go through it, and then APISIX Core would dynamically assign specific upstreams for them. Apache APISIX’s route module could support adding/removing, modifying, and deleting routes while the server runs. In other words, it could achieve dynamic reloading. None of these changes would bring users’ attention and affect regular business.

Now, let’s introduce a classic scenario; for example, if we want to add a reverse proxy for a new domain, we only need to create an upstream in APISIX and add a new route. NGINX doesn’t need to reboot during this process. Here is another example for the plugin’s system: APISIX could use an IP-restriction plugin to achieve the IP allowlist/blocklist feature. All these feature updates are dynamic and don’t need the server reboot. Thanks to etcd, the configuration strategy could achieve instant updates using add-ons, and all configs could take effect immediately and provide the best user experience.

Conclusion

NGINX hot reloading would have one old and one new worker process under some circumstances, which causes additional resource waste. Also, too frequent hot reloading could cause a slight chance of total business data loss. Under the background of Cloud Native and microservices, the service updates become more and more frequent, and the strategy to manage API also varies, which leads to new requirements for hot reloading.

NGINX’s hot reloading no longer satisfies the business requirements. It is time to switch to Apache APISIX, an API gateway with a more advanced hot reloading strategy in the Cloud Native era. In addition, after switching to APISIX, users could have dynamic and uniform management of API services, and it could significantly improve management efficiency.

ingress-nginx 基于 OpenResty 实现了一套负载均衡机制，既然基于 OpenResty，当然也就少不了 OpenResty 的功能支持，在官方 Github 上可以看到在 ingress-nginx 中使用 Lua 插件的文档说明。

当你看完文档，需要注意以下几点：

支持的执行阶段只包括 init_worker，rewrite，header_filter，body_filter，以及 log，但不包含常用的 access 及 content 等。
插件安装的位置位于/etc/nginx/lua/plugins目录下，插件的名称与目录名保持一致，文件名都为 main.lua，比如官方默认提供的hello_world插件，完整的路径就是 /etc/nginx/lua/plugins/hello_world/main.lua。

插件的格式需要返回一个table表，示例如下：

local _M = {};

--[[
@ rewrite 方法名、即上面第一点提到的执行阶段名称。
@ 以下代码等效于：
rewrite_by_lua_block {
  ngx.header["hello"] = "1"
}
--]]
function _M.rewrite()
  ngx.header["hello"] = "1"
end

return _M

通过上面的介绍，我们已经知道如何编写一个在 ingress-nginx 中运行的插件了。但这还只是解决了问题的第一步，接下来我们需要进一步的了解如何在 ingress-nginx 中使用插件。首先，我们需要将插件的代码拷贝进 nginx-ingress-controller 的容器中，那么通常我们可以使用以下方式：

将代码打包到nginx-ingress-controller的镜像内，大概的代码示例如下，然后启动 ingress-nginx 服务时使用该镜像作为代替：

# 基于nginx-ingress-controller基础镜像打包新镜像
FROM xxx/google_containers/nginx-ingress-controller:v1.1.0
# 将插件目录及文件拷贝到/etc/nginx/lua/plugins内
COPY ./hello_world/ /etc/nginx/lua/plugins

使用 PVC，将 Lua 的代码保存在 PVC 中，这样的好处是方便测试，修改代码不用重新打包镜像等。下面仅为示例，具体请根据自身环境自行配置。

# 找到对应的pod，这里以常见的namespace为ingress-nginx为例
kubectl get pods -n ingress-nginx
# 编辑对应POD
kubectl edit -n ingress-nginx pod/YOUR-POD-NAME

apiVersion: v1
kind: Pod
spec:
  ---xxxxxx---
  containers:
    volumeMounts:
      - mountPath: /etc/nginx/lua/plugins/hello_world
        name: plugin_hello_world
volumes:
- hostPath:
    path: /var/lib/hello_world # 插件在node中保存的路径
    type: Directory
  name: plugin_hello_world

现在代码文件已经加入到 nginx-ingress-controller 容器内了，现在我们还需要最后一步，将插件加入到插件列表中使其生效。

这一步比较容易踩坑，主要跟当前你环境中的 ingress-nginx 版本有关系，在 v1.0.0-alpha.1 以后，增加了在ConfigMap中直接配置插件列表的配置，具体可参见更新记录。

如果你的版本比上述版本新，则只需要在 ConfigMap 配置中增加配置即可。于是可以继续查看上述使用 PVC 中提到的 POD 信息，查看容器启动参数内的 --configmap 配置。

# 这里假设--configmap配置为 `ingress-nginx/nginx-configuration`，实际使用命令行时以你自身环境配置为准
kubectl edit -n ingress-nginx cm/nginx-configuration

在对应的 yml 配置文件内，将插件名 hello_world 加入到插件列表中，保存完后，nginx 会自动 reload，随后通过curl命令等请求ingress-nginx内配置的server，即可测试检查是否包含 lua 插件添加的对应响应头、以确保插件配置生效。

data:
  plugins: hello_world, xxxx

如果测试不成功，很大可能是上面提到的原因，你的ingress-nginx版本较老，不支持在ConfigMap中设置plugins字段来使插件生效。我们知道，ingress-nginx 配置 –configmap ConfigMap 的作用，本质上是将指定 ConfigMap 中的配置当成数据，然后使用指定的 go text/template 模板来生成最终的nginx配置文件，所以查看配置文件是否生效的最直接方式，就是直接进入容器内，查看对应的 nignx.conf 是否包含我们更新的配置部分。

# 进入到上述的pod容器内
kubectl exec -it -n ingress-nginx YOUR-POD-NAME -- /bin/bash
# 查看nginx.conf配置，目录/etc/nginx
vim nginx.conf

在打开的nginx.conf文件中，我们搜索plugins.init，如果能看到类似以下的代码，说明我们的配置是成功了的，这时如果测试看不到效果，则需要检查 nginx 是否自动重启以使配置生效了。

plugins.init({ "hello_world", "xxx" })

如果你看到 plugin.init() 仍然是空的状态，则需确认下是不是上面提到的版本太低的问题。对于这种较低版本，我们只能曲线救国，在模板配置中写死需要调用的插件列表。继续查看上述的 POD 信息，

# 查看对应的template配置
kubectl describe -n ingress-nginx pod/YOUR-POD-NAME | grep 'template'
# 查看对应的ConfigMap配置，确认template名称
kubectl get -n ingress-nginx cm
# 编辑对应的模板
kubectl edit -n ingress-nginx cm/TEMPLATE-NAME

在上述编辑中，找到模板内的 plugins.init 部分，将对应的插件名称加入到初始化列表 table 中，保存并确保使用新模板重新生成了 nginx.conf 后，我们应该就能看到插件生效后的效果了。

以上就是在ingress-nginx中使用 Lua 插件中的大体流程，另外需要说明的是，假如你的插件基于ffi，需要使用.so动态链接库，则需要将对应的.so文件拷贝到/usr/lib等系统默认动态库存放的路径中，否则需要将存放的路径加入到LD_LIBRARY_PATH环境变量中，或者在ffi.load库时使用绝对路径。

还有的在实际使用中想针对指定的server生效，就像这个Issue中提到的那样，则可以参照里面提到的方式，在annotation里加入set plugin_NAME "on" 这样的变量，然后在插件中先判断 ngx.var.plugin_NAME == "on" 是否成立再执行插件的逻辑。

1. 背景

2. 原因分析

3. 说明一下DefaultStorageClass

4. 解决办法

卷模式

What is NGINX

Why is NGINX so popular

What does Hot Reload do

The Principle of NGINX Hot Reloading

NGINX reload causes downtime

Hot Reloading That Takes Effects Immediately In Memory

Conclusion

1、基本配置

2、网络模型

端口转发 forwarded_port

私有网络 private_network

公有网络 public_network

3、自动创建的 NAT 网卡

4、遇到的问题

Host-only Network 和 Bridged Network 网段冲突