[GH-ISSUE #18] worker process exited on signal 11 #145

Closed
opened 2026-03-13 16:32:56 +03:00 by kerem · 10 comments
Owner

Originally created by @SionInory on GitHub (Jan 15, 2021).
Original GitHub issue: https://github.com/ADD-SP/ngx_waf/issues/18

I use a docker web tools called Portainer,when I try to access it,the nginx occured error.

2021/01/15 13:27:46 [alert] 1#1: worker process 30 exited on signal 11
2021/01/15 13:27:47 [alert] 1#1: worker process 31 exited on signal 11
2021/01/15 13:27:47 [alert] 1#1: worker process 32 exited on signal 11
2021/01/15 13:27:48 [alert] 1#1: worker process 33 exited on signal 11
2021/01/15 13:27:48 [alert] 1#1: worker process 34 exited on signal 11 

I also find that when waf_mode STD;,the error occurred,when waf_mode CC;,the nginx runs well.

Originally created by @SionInory on GitHub (Jan 15, 2021). Original GitHub issue: https://github.com/ADD-SP/ngx_waf/issues/18 I use a docker web tools called ` Portainer `,when I try to access it,the nginx occured error. ``` 2021/01/15 13:27:46 [alert] 1#1: worker process 30 exited on signal 11 2021/01/15 13:27:47 [alert] 1#1: worker process 31 exited on signal 11 2021/01/15 13:27:47 [alert] 1#1: worker process 32 exited on signal 11 2021/01/15 13:27:48 [alert] 1#1: worker process 33 exited on signal 11 2021/01/15 13:27:48 [alert] 1#1: worker process 34 exited on signal 11 ``` I also find that when `waf_mode STD;`,the error occurred,when `waf_mode CC;`,the nginx runs well.
kerem 2026-03-13 16:32:56 +03:00
  • closed this issue
  • added the
    bug
    label
Author
Owner

@ADD-SP commented on GitHub (Jan 15, 2021):

先说个题外话,从您的 Github 活动记录来看猜测是中国人,如果是的话以后用中文交流吧。
Looking at your activity record in Github, I think you are Chinese, if so, let's communicate in Chinese, if not, let's continue in English.


Hi, Could you please tell me the version of ngx_waf, the version of NGINX and the output of nginx -V.

<!-- gh-comment-id:760678757 --> @ADD-SP commented on GitHub (Jan 15, 2021): 先说个题外话,从您的 Github 活动记录来看猜测是中国人,如果是的话以后用中文交流吧。 Looking at your activity record in Github, I think you are Chinese, if so, let's communicate in Chinese, if not, let's continue in English. *** Hi, Could you please tell me the version of `ngx_waf`, the version of `NGINX` and the output of `nginx -V`.
Author
Owner

@SionInory commented on GitHub (Jan 15, 2021):

nginx version: nginx/1.19.6
built by gcc 8.3.0 (Debian 8.3.0-6) 
built with OpenSSL 1.1.1d  10 Sep 2019
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.19.6/debian/debuild-base/nginx-1.19.6=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

ngx_waf 的版本是最新master分支的版本
使用的是动态模块加载

<!-- gh-comment-id:760687934 --> @SionInory commented on GitHub (Jan 15, 2021): ``` nginx version: nginx/1.19.6 built by gcc 8.3.0 (Debian 8.3.0-6) built with OpenSSL 1.1.1d 10 Sep 2019 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.19.6/debian/debuild-base/nginx-1.19.6=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' ``` `ngx_waf `的版本是最新` master `分支的版本 使用的是动态模块加载
Author
Owner

@ADD-SP commented on GitHub (Jan 15, 2021):

我是用 nginx-1.19.6 、Docker 镜像 portainer/portainer 和 master 分支的 ngx_waf,并未复现此问题。
nginx 配置

    server {
        listen 80;
        server_name xxxxxx;

        waf on;
        waf_rule_path /path/to/rules/;
        waf_mode STD;
        waf_mult_mount off;
        waf_cc_deny_limit 500 60;

        access_log /path/to/log main;
        location ~ "^/cloud/portainer(/?.*)" {
            proxy_pass http://127.0.0.1:9000$1$is_args$args;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

可不可以用 gdb 调试一下 core 文件,看看代码挂在哪一行?记得修改编译参数关掉优化。

我同时测试了 nginx-1.18.0,也没有问题。鉴于您的 nginx 版本属于开发版,可能不稳定,是否可以使用 nginx-1.18.0 重新测试一下呢?

<!-- gh-comment-id:760739147 --> @ADD-SP commented on GitHub (Jan 15, 2021): 我是用 `nginx-1.19.6` 、Docker 镜像 `portainer/portainer` 和 master 分支的 `ngx_waf`,并未复现此问题。 nginx 配置 ```text server { listen 80; server_name xxxxxx; waf on; waf_rule_path /path/to/rules/; waf_mode STD; waf_mult_mount off; waf_cc_deny_limit 500 60; access_log /path/to/log main; location ~ "^/cloud/portainer(/?.*)" { proxy_pass http://127.0.0.1:9000$1$is_args$args; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ``` 可不可以用 `gdb` 调试一下 `core` 文件,看看代码挂在哪一行?记得修改编译参数关掉优化。 我同时测试了 `nginx-1.18.0`,也没有问题。鉴于您的 nginx 版本属于开发版,可能不稳定,是否可以使用 `nginx-1.18.0` 重新测试一下呢?
Author
Owner

@SionInory commented on GitHub (Jan 15, 2021):

抱歉,对于nginx模块的编写不大熟悉,可能无法为你提供有效的帮助。
但经过我的测试,只有当服务配置了ssl访问的时候,才会出现此问题,并且当waf_mode STD;, waf_rule_path被注释时,不会出现该问题。
附上我的nginx 配置。

 server {
    listen     9090 ssl http2;
    server_name   xxxxxxx;

    ssl_certificate      /cert/server.pem;
    ssl_certificate_key  /cert/server.key;

    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    waf on;
    waf_rule_path /cert/rules/;
    waf_mode STD;
    waf_cc_deny_limit 1000 60;

    location / {
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_pass http://Portainer:9000;
    }
    error_page 497 https://$server_name:9090$request_uri;
    }
<!-- gh-comment-id:760763869 --> @SionInory commented on GitHub (Jan 15, 2021): 抱歉,对于` nginx `模块的编写不大熟悉,可能无法为你提供有效的帮助。 但经过我的测试,只有当服务配置了`ssl`访问的时候,才会出现此问题,并且当` waf_mode STD; `, `waf_rule_path`被注释时,不会出现该问题。 附上我的`nginx `配置。 ``` server { listen 9090 ssl http2; server_name xxxxxxx; ssl_certificate /cert/server.pem; ssl_certificate_key /cert/server.key; ssl_session_cache shared:SSL:1m; ssl_session_timeout 5m; ssl_ciphers HIGH:!aNULL:!MD5; ssl_prefer_server_ciphers on; waf on; waf_rule_path /cert/rules/; waf_mode STD; waf_cc_deny_limit 1000 60; location / { proxy_set_header Host $http_host; proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Connection $http_connection; proxy_set_header Upgrade $http_upgrade; proxy_pass http://Portainer:9000; } error_page 497 https://$server_name:9090$request_uri; } ```
Author
Owner

@ADD-SP commented on GitHub (Jan 15, 2021):

复制了配置还是无法复现。您的 nginx 是安装宿主机上还是 docker 里呢?如果在 docker 里是哪个 docker 镜像呢?


调试 core 文件并不需要您了解 nginx 模块开发,您可以抽空按照下列步骤操作。

  1. configure 脚本的参数 --with-cc-opt='xxxx -g -O3 xxx' 修改为 --with-cc-opt='xxxx -g -O0 xxx'
  2. 关闭 nginx,重新编译 nginx 及模块,并进行替换。
  3. nginx.conf 顶部添加一行 worker_rlimit_core 10000m;
  4. 在当前 shell 执行 ulimit -c unlimited
  5. 启动 nginx,并记住当前 shell 的目录。
  6. 访问 Portainer 页面,触发错误。
  7. 此时应该会在 nginx 的日志目录下生成 core 文件,或者在启动 nginx 时 shell 所在的目录。
  8. 关闭 nginx。
  9. 在同一个 shell 执行 gdb nginx /path/to/core
  10. 复制输出的内容发出来我来看一下。
<!-- gh-comment-id:760785975 --> @ADD-SP commented on GitHub (Jan 15, 2021): 复制了配置还是无法复现。您的 nginx 是安装宿主机上还是 docker 里呢?如果在 docker 里是哪个 docker 镜像呢? *** 调试 `core` 文件并不需要您了解 `nginx` 模块开发,您可以抽空按照下列步骤操作。 1. 将 `configure` 脚本的参数 `--with-cc-opt='xxxx -g -O3 xxx'` 修改为 `--with-cc-opt='xxxx -g -O0 xxx'`。 2. 关闭 `nginx`,重新编译 `nginx` 及模块,并进行替换。 3. 在 `nginx.conf` 顶部添加一行 `worker_rlimit_core 10000m;`。 4. 在当前 shell 执行 `ulimit -c unlimited`。 5. 启动 `nginx`,并记住当前 shell 的目录。 5. 访问 `Portainer` 页面,触发错误。 6. 此时应该会在 `nginx` 的日志目录下生成 `core` 文件,或者在启动 `nginx` 时 shell 所在的目录。 7. 关闭 nginx。 8. 在同一个 shell 执行 `gdb nginx /path/to/core`。 9. 复制输出的内容发出来我来看一下。
Author
Owner

@ADD-SP commented on GitHub (Jan 15, 2021):

另一个问题,访问 Portainer 页面会出错,访问其它页面呢?比如一个本地的静态网页文件。规则文件是否为默认的规则文件?如果不是那么做了哪些改动?

<!-- gh-comment-id:760790462 --> @ADD-SP commented on GitHub (Jan 15, 2021): 另一个问题,访问 `Portainer` 页面会出错,访问其它页面呢?比如一个本地的静态网页文件。规则文件是否为默认的规则文件?如果不是那么做了哪些改动?
Author
Owner

@SionInory commented on GitHub (Jan 15, 2021):

我的nginx是装在Docker中的,使用的是latest镜像,规则文件为默认规则文件,访问其他页面的日志如下,
待我调试过core文件后,将内容发给您。

117.xx.xx.xx - - [15/Jan/2021:18:19:14 +0800] 8888 "GET / HTTP/2.0" xxxx:8888 200 384 1089 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"  0.007 0.006,
2021/01/15 18:19:14 [alert] 1#1: worker process 52 exited on signal 11,
2021/01/15 18:19:15 [alert] 1#1: worker process 53 exited on signal 11,
2021/01/15 18:19:15 [alert] 1#1: worker process 54 exited on signal 11,
2021/01/15 18:51:51 [alert] 58#58: *2060 ngx_waf: [BLACK-URL][(?i)(?:\.(?:bak|inc|old|mdb|sql|backup|java|class))$], client: 117.xx.xx.xx, server: xxxx, request: "GET /www.bak HTTP/2.0", host: "xxxx:8888",
117.xx.xx.xx - - [15/Jan/2021:18:51:51 +0800] 8888 "GET /www.bak HTTP/2.0" xxxx:8888 403 400 114 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15"  0.000 -
<!-- gh-comment-id:760858292 --> @SionInory commented on GitHub (Jan 15, 2021): 我的`nginx`是装在`Docker`中的,使用的是`latest`镜像,规则文件为默认规则文件,访问其他页面的日志如下, 待我调试过`core`文件后,将内容发给您。 ``` 117.xx.xx.xx - - [15/Jan/2021:18:19:14 +0800] 8888 "GET / HTTP/2.0" xxxx:8888 200 384 1089 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" 0.007 0.006, 2021/01/15 18:19:14 [alert] 1#1: worker process 52 exited on signal 11, 2021/01/15 18:19:15 [alert] 1#1: worker process 53 exited on signal 11, 2021/01/15 18:19:15 [alert] 1#1: worker process 54 exited on signal 11, 2021/01/15 18:51:51 [alert] 58#58: *2060 ngx_waf: [BLACK-URL][(?i)(?:\.(?:bak|inc|old|mdb|sql|backup|java|class))$], client: 117.xx.xx.xx, server: xxxx, request: "GET /www.bak HTTP/2.0", host: "xxxx:8888", 117.xx.xx.xx - - [15/Jan/2021:18:51:51 +0800] 8888 "GET /www.bak HTTP/2.0" xxxx:8888 403 400 114 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" 0.000 - ```
Author
Owner

@SionInory commented on GitHub (Jan 15, 2021):

我得到了三个core文件,分别为core.9373 core.9554 core.9555
core.9373

[New LWP 9373]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker pr'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000564d4775daf4 in ngx_http_waf_handler_check_black_args (r=0x564d48833430, 
    out_http_status=0x7ffea083b2a0) at ../ngx_waf/inc/ngx_http_waf_module_check.h:388
388             ngx_int_t rc = ngx_regex_exec(p->regex, pargs, NULL, 0);

其余两个core出错位置均与第一个相同

<!-- gh-comment-id:761002550 --> @SionInory commented on GitHub (Jan 15, 2021): 我得到了三个`core`文件,分别为`core.9373` `core.9554` `core.9555` `core.9373` ``` [New LWP 9373] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `nginx: worker pr'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000564d4775daf4 in ngx_http_waf_handler_check_black_args (r=0x564d48833430, out_http_status=0x7ffea083b2a0) at ../ngx_waf/inc/ngx_http_waf_module_check.h:388 388 ngx_int_t rc = ngx_regex_exec(p->regex, pargs, NULL, 0); ``` 其余两个`core`出错位置均与第一个相同
Author
Owner

@ADD-SP commented on GitHub (Jan 15, 2021):

我大概猜了一下出错的地方,不知道有没有用。您可以按照下面的步骤重新测试。

  1. 进入 ngx_waf 的目录,执行 git pull --force && git checkout -b bugfix-SIGSEGV origin/bugfix-SIGSEGV
  2. 保持 configure 脚本的参数不变,但是要重新执行一遍 configure 脚本。
  3. 关闭 nginx
  4. 重新编译 nginx 及其模块,编译完毕后进行替换。
  5. 替换规则,因为规则有改动。
  6. 启动 nginx
  7. 访问 Portainer 页面,检查是否出现 worker process exited on signal 11 错误。Portainer 页面可能出现错误,这是由默认规则导致的,可以用过查看 error.log 并删除指定规则来解决。
  8. 如果出现错误麻烦提供一下调试 core 文件时的输出。如果没有出错请告知并关闭本 issue。

注意,本次修复是在 ngx_waf 的测试版上进行的修复,测试版的其它改动见 v3.0.3-beta-3

<!-- gh-comment-id:761128117 --> @ADD-SP commented on GitHub (Jan 15, 2021): 我大概猜了一下出错的地方,不知道有没有用。您可以按照下面的步骤重新测试。 1. 进入 `ngx_waf` 的目录,执行 `git pull --force && git checkout -b bugfix-SIGSEGV origin/bugfix-SIGSEGV`。 2. 保持 `configure` 脚本的参数不变,但是要重新执行一遍 `configure` 脚本。 3. 关闭 `nginx`。 4. 重新编译 `nginx` 及其模块,编译完毕后进行替换。 6. 替换规则,因为规则有改动。 7. 启动 `nginx`。 8. 访问 `Portainer` 页面,检查是否出现 `worker process exited on signal 11 `错误。`Portainer` 页面可能出现错误,这是由默认规则导致的,可以用过查看 `error.log` 并删除指定规则来解决。 9. 如果出现错误麻烦提供一下调试 `core` 文件时的输出。如果没有出错请告知并关闭本 issue。 *** 注意,本次修复是在 `ngx_waf` 的测试版上进行的修复,测试版的其它改动见 [v3.0.3-beta-3](https://github.com/ADD-SP/ngx_waf/releases/tag/v3.0.3-beta-3)。
Author
Owner

@SionInory commented on GitHub (Jan 16, 2021):

十分感谢,问题已经解决。

<!-- gh-comment-id:761294450 --> @SionInory commented on GitHub (Jan 16, 2021): 十分感谢,问题已经解决。
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/ngx_waf#145
No description provided.