Почему наблюдается большое отклонение во времени ожидания сервера nginx

У меня есть сервер Nginx, который действует как обратный прокси-сервер и статический файловый сервер.

Мои пользователи жалуются, что иногда реакция медленная.

Я попытался изучить файл .her, чтобы лучше понять проблему, и обнаружил кое-что необычное.

      time wait on https://cdnjs.cloudflare.com****
[10.00600000781566, 10.09000002155453, 12.657999974526462, 30.81500001040101, 50.140000049091874]

time wait on  https://fonts.gstatic.com/****
[0.8139999584183073, 0.8409999709799933, 1.4160000515654616, 2.3279999482259086, 79.60999999868869]

time wait on my server:
[59.73200001836568, 59.820000056281685, 60.4199999724552, 60.530999979116025, 60.79299996264279, 61.397000021353364, 61.590999948196114, 61.89499994969368, 62.058999961778525, 68.06700001706928, 68.25800005129724, 68.29399997562915, 68.88300005169958, 69.07899998541922, 69.38299999047071, 69.6550000514686, 69.6710000243038, 69.69899996588379, 69.76500002556294, 70.05899996621906, 70.57600003184378, 76.85900000137836, 256.38100003309546, 267.7780000296906, 280.696999967508, 461.3320000257194, 465.38599997801333, 476.49299997053294, 484.6409999888614, 748.1390000376775]

Как видите, очень большое отклонение времени ожидания в запросах одного и того же ресурса. Это происходит вcdnjs.cloudflare.comиfonts.gstatic.comЭто уважаемый сервер, но на моем сервере разница больше.

Сначала я подумал, что это проблема с кешированием. Но за указанное выше время ожидания все запросы кэшируются, за исключением 1, я проверилMISSзапрос$request_timeв Nginx и это0.4ясно, что это не источник проблемы.

Сервер потребляет 10 % пропускной способности, а процессор не перегружен (загрузка менее 40 % в пиковое время).

Может кто-нибудь, пожалуйста, помогите мне понять, почему такая большая разница во времени ожидания? Проблема с моим сервером или клиентом? Как я могу точно определить проблему?

Это мой файл конфигурации сервера (Nginx — openresty):

      events {
  worker_connections 1024;
}

env SERVER_BACKEND_NAME;
env SERVER_CDN_NAME;
env SERVER_CDN_SPESIFIC_NAME;

http {

    error_log /var/errors/externalNginx.http.error_1 error;
    
    
  # The "auto_ssl" shared dict should be defined with enough storage space to
  # hold your certificate data. 1MB of storage holds certificates for
  # approximately 100 separate domains.
  lua_shared_dict auto_ssl 1m;
  # The "auto_ssl_settings" shared dict is used to temporarily store various settings
  # like the secret used by the hook server on port 8999. Do not change or
  # omit it.
  lua_shared_dict auto_ssl_settings 64k;

  # A DNS resolver must be defined for OCSP stapling to function.
  #
  # This example uses Google's DNS server. You may want to use your system's
  # default DNS servers, which can be found in /etc/resolv.conf. If your network
  # is not IPv6 compatible, you may wish to disable IPv6 results by using the
  # "ipv6=off" flag (like "resolver 8.8.8.8 ipv6=off").
  resolver 127.0.0.11;

  # Initial setup tasks.
  init_by_lua_block {
    auto_ssl = (require "resty.auto-ssl").new()

    -- Define a function to determine which SNI domains to automatically handle
    -- and register new certificates for. Defaults to not allowing any domains,
    -- so this must be configured.
    auto_ssl:set("allow_domain", function(domain)
      return true
    end)

    auto_ssl:init()
  }

  init_worker_by_lua_block {
    auto_ssl:init_worker()
  }
    
    # Internal Cdn nginx backend
    upstream cdnnginx_backend { 
        server cdnnginx;
    }

    
    # Limits - so it will be harder to DOS me
    limit_req_log_level    warn;
    limit_req_zone         $binary_remote_addr zone=login:10m rate=10r/m;


    # HTTPS cdn server - will later on become our image cdn as well
     server {
        listen 443 ssl http2;
        server_name  ${SERVER_IMAGE_AND_FILE_CDN_NAME}; 
        error_log /var/errors/externalNginx.${SERVER_IMAGE_AND_FILE_CDN_NAME}.error_1 error;

    # Dynamic handler for issuing or returning certs for SNI domains.
        ssl_certificate_by_lua_block {
            auto_ssl:ssl_certificate()
        }

    # This is for the Config service so we can pass big enogh information on the headers
    large_client_header_buffers 4 32k;

    ssl_certificate /etc/resty-default-ssl/resty-auto-ssl-fallback.crt;
    ssl_certificate_key /etc/resty-default-ssl/resty-auto-ssl-fallback.key;

        # Enable gzip compression
        gzip on;
        gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
        sendfile on;

        # Serve the compressed HTML file
        location /saved.html {
                root /var/compenion;

        # we need to clean it in the most external service
        more_clear_headers 'Server';

        # this is against CORS policy - we want everyone to be able to access this
        more_set_headers 'Access-Control-Allow-Origin: *';
        }

    location / {
        # we need to clean it in the most external service
        more_clear_headers 'Server';

        return 404 'Nanana';
        }
  }


  # HTTPS cdn - backword competability to mother existing links video.malkali.com
  server {
        listen 443 ssl http2;
        server_name my.server.com;
        error_log /var/errors/externalNginx.my.server.com.error_1 error;

        # Dynamic handler for issuing or returning certs for SNI domains.
        ssl_certificate_by_lua_block {
        auto_ssl:ssl_certificate()
        }

        ssl_certificate /etc/resty-default-ssl/resty-auto-ssl-fallback.crt;
        ssl_certificate_key /etc/resty-default-ssl/resty-auto-ssl-fallback.key;

    
        location /Health {
        return 200 "ComasTas";
        }

    
        # This check HealthCheck of caching nginx
        location /Health2 {
            proxy_pass http://cdnnginx_backend/Health2;

            # this is against CORS policy - we want everyone to be able to access this
            more_set_headers 'Access-Control-Allow-Origin: *';
        }

    # this will check the proxy health check
    location /1/Health {
        proxy_pass http://cdnnginx_backend/Health3;

        # this is against CORS policy - we want everyone to be able to access this
        more_set_headers 'Access-Control-Allow-Origin: *';
    }

    # This will pass normal links forward (signed and unsigned?) - should enforce similar links only to pass forward and reduce attacks
    location ~  "^\/(?:[0-9A-Fa-f]{2}){16}\/(?:[0-9A-Fa-f]{2}){16}\/(?:.?[^\/]+)$" {        
        # solve the byte-range requestr support
        proxy_force_ranges on;
        more_set_headers 'Accept-Ranges: bytes';

        # solve the issue of gateway timeout
        proxy_read_timeout 300s;

        set $continue_url $uri;
        if ($arg_sig)
        {
            set $temp_cache 1;
        }

        if ($arg_refere)
        {
            set $temp_cache 2$temp_cache;
        }
        
        if ($temp_cache = 1)
        {
            set $continue_url $uri?sig=$arg_sig;
        }

        if ($temp_cache = 21)
        {
            set $continue_url $uri?sig=$arg_sig&refere=$arg_refere;
        }

        if ($temp_cache = 2)
        {
            set $continue_url $uri?refere=$arg_refere;
        }

        # solve pre-flight issue
        if ($request_method = OPTIONS ) {
            # this is against CORS policy - we want everyone to be able to access this
            more_set_headers 'Access-Control-Allow-Origin: *';
            
            more_set_headers 'Access-Control-Allow-Headers: refere, Origin';

                add_header Content-Length 0;
                add_header Content-Type text/plain;
                return 200;
            }

        # add the header of the request host so we will be able to route insode
        proxy_set_header X-Forwarded-Host $scheme://$http_host;

        proxy_pass      http://cdnnginx_backend$continue_url;

        # we need to clean it in the most external service
        more_clear_headers 'Server';

        # clear our internals evidence
        more_clear_headers 'V1Latency';
        more_clear_headers 'V1RequestTime';
        more_clear_headers 'ProxyCache';
        more_clear_headers 'S3priority';
        more_clear_headers 'V1internalcache';

        # this is against CORS policy - we want everyone to be able to access this
        more_set_headers 'Access-Control-Allow-Origin: *';
        more_set_headers 'Access-Control-Allow-Headers: refere, Origin';

        # @@@ For dbg only
        more_set_headers 'start_time: $msec';
        more_set_headers 'total_time: $request_time';
    }


    error_page 404 403 500 502 503 /error-page.html;
    location = /error-page.html {
            internal;
        return 404 "Nanana";
    }

    location = / {
        return 404 "Nanana";
    }
   
  }

  # HTTP server
  server {
    listen 80;
    
    location /HealthCheck {
    return 200;
    }

    # Endpoint used for performing domain verification with Let's Encrypt.
    location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }
  }

  # Internal server running on port 8999 for handling certificate tasks.
  server {
    listen 127.0.0.1:8999;

    # Increase the body buffer size, to ensure the internal POSTs can always
    # parse the full POST contents into memory.
    client_body_buffer_size 128k;
    client_max_body_size 128k;

    location / {
      content_by_lua_block {
        auto_ssl:hook_server()
      }
    }
  }
}

Я пытаюсь исследовать сам мой сервер:

iftop:

      TX:             cum:    816MB   peak:   36.8Mb                                                                                                                               rates:   17.3Mb  24.2Mb  26.6Mb RX:                     272MB           13.2Mb                                                                                                                                        7.76Mb  7.92Mb  8.90Mb TOTAL:                 1.06GB           47.3Mb                                                                                                                                        25.1Mb  32.2Mb  35.5Mb

top:

      top - 08:26:56 up 5 days, 22:33,  2 users,  load average: 0.40, 0.27, 0.44
Tasks: 271 total,   1 running, 270 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us,  0.4 sy,  0.0 ni, 98.8 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :  31986.2 total,  12924.5 free,   3720.9 used,  15340.8 buff/cache
MiB Swap:   1024.0 total,   1004.5 free,     19.5 used.  27610.2 avail Mem

Сервер имеет пропускную способность 1 Гбит/с и мощный процессор.

Он использует менее 15% доступного процессора и пропускную способность менее 1 Гбит/с.

Как я могу дополнительно изучить этот вопрос?