Performance Tuning NGINX

Transcription

1

2 Performance Tuning NGINX

3 Name: Amir Rawdat Currently: Technical Marketing Engineer at NGINX inc. Previously: - Customer Applications Engineer at Nokia inc.

4 Multi-Process Architecture with QPI Bus

5 Web Server Topology wrk nginx

6 Reverse Proxy Topology wrk nginx nginx 6

7 Technical Specifications # Sockets # Cores per Socket # Threads per Core Model Name RAM OS NIC Client Intel(R) Xeon(R) CPU E GHz 128 GB Ubuntu Xenial 40GbE QSFP+ Web Server & Reverse Proxy Intel(R) Xeon(R) Platinum GHz 192 GB Ubuntu Xenial 40GbE QSFP+

8 Multi-Processor Architecture

9 9 #1 Duplicate NGINX Configurations

10 Multi-Processor Architecture

11 NGINX Configuration (Instance 1) user root; worker_processes 48 ; worker_cpu_affinity auto ; worker_rlimit_nofile ; error_log /home/ubuntu/access.error error;... 11

12 NGINX Configuration (Instance 2) user root; worker_processes 48 ; worker_cpu_affinity auto ; worker_rlimit_nofile ; error_log /home/ubuntu/access.error error;.. 12

13 Deploying NGINX Instances $ nginx c /path/to/configuration/instance-1 $ nginx c /path/to/configuration/instance-2 $ ps aux grep nginx nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_0.conf nginx: worker process nginx: worker process nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_1.conf nginx: worker process nginx: worker process $ pkill nginx 13

14 14 #2 Additional NGINX Configuration Directives

15 Web Server (Instance 1) events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; server { listen :443 backlog= reuseport; root /usr/share/nginx/bin;

16 Web Server (Instance 2) events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; server { listen :443 backlog= reuseport; root /usr/share/nginx/bin; 16

17 Reverse Proxy (Instance 1) server { listen :443 ssl backlog= reuseport; ssl_certificate /etc/ssl/certs/nginx.pem; ssl_certificate_key /etc/ssl/private/nginx.key; ssl_session_cache off; ssl_session_tickets off; location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass upstream webserver_0 { server :80; keepalive 200; 17

18 Reverse Proxy (Instance 2) server { listen :443 ssl backlog= reuseport; ssl_certificate /etc/ssl/certs/nginx.pem; ssl_certificate_key /etc/ssl/private/nginx.key; ssl_session_cache off; ssl_session_tickets off; location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass upstream webserver_1 { server :80; keepalive 200; 18

19 Performance Test Results 19

20 Web Server (Instance 1) events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; server { listen :443 backlog= reuseport; root /usr/share/nginx/bin;

21 Web Server (Instance 2) events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; server { listen :443 backlog= reuseport; root /usr/share/nginx/bin2; 21

22 Performance Test Results 22

23 Performance Test Results 23

24 24 #3 Performance Tip: Linux Sysctl Settings

25 Linux Sysctl Settings Increase memory thresholds to prevent packet dropping sysctl -w net.ipv4.tcp_rmem= sysctl -w net.ipv4.tcp_wmem= Increase the size of the processor queues sysctl -w net.core.net_dev_max_backlog= Setting the maximum TCP buffer sizes sysctl -w net.core.rmem_max= sysctl -w net.core.wmem_max=

26 Linux Sysctl Settings Disable TCP timestamps sysctl -w net.ipv4.tcp_timestamps=0 Defines the local port range that is used by TCP and UDP to choose the local port sysctl -w net.ipv4.ip_local_port_range = Enable reuse of TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. sysctl -w net.ipv4.tcp_tw_reuse = 1 26

27 Linux Sysctl Settings 27

28 28 4 Performance Tip: Enabling RSS and TPS

29 Enabling RSS and TPS /etc/init.d/irq_balance stop git clone cd /mlnx-en/ofed-scripts./set_irq_affinity_bynode.sh <numa node id> <interface-name> set_irq_affinity -x local <interface-name> 29

30 Enabling RSS and TPS 30

31 Enabling RSS and TPS 31

32 Performance Test Results 32

33 33 Performance Numbers with NGINX and Intel QuickAssist

34 Performance with QuickAssist

35 Performance with QuickAssist 35

36 Summary Deploy two nginx instances Using additional nginx configuration directives Linux sysctl parameter tuning Setting IRQ affinity using RSS and TPS Consult appendix for additional information and performance tips 36

37 Thank you Contact information here

38 7 Appendix

39 Client Traffic Script taskset -c 0-21,44-65 wrk -t 44 -c d 180s -H 'Connection: Close' :443/$1 >> output.txt & taskset -c 22-43,66-87 wrk -t 44 -c d 180s -H 'Connection: Close' :443/$1 >> output1.txt & ## $1 is the requested static file size

40 Architectural Specifications

41 Architectural Specifications 41

42 Technical Specifications # Sockets # Cores per Socket # Threads per Core Model Name RAM OS NIC nbdw Intel(R) Xeon(R) CPU E GHz 128 GB Ubuntu Xenial 40GbE QSFP+ Web Server Intel(R) Xeon(R) Platinum GHz 192 GB Ubuntu Xenial 40GbE QSFP+

43 Technical Specifications # Sockets # Cores per Socket # Threads per Core Model Name RAM OS NIC Client Intel(R) Xeon(R) CPU E GHz 128 GB Ubuntu Xenial 40GbE QSFP+ Reverse Proxy Intel(R) Xeon(R) Platinum GHz 192 GB Ubuntu Xenial 40GbE QSFP+ Web Server Intel(R) Xeon(R) CPU E GHz 128 GB Ubuntu Xenial 40GbE QSFP+

44 Reverse proxy (Instance 1) user root; worker_processes 48 ; worker_cpu_affinity auto ; worker_rlimit_nofile ; error_log /home/ubuntu/access.error crit; events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; 44

45 Reverse proxy (Instance 1) server { listen :443 ssl backlog= reuseport; ssl_certificate /etc/ssl/certs/nginx.pem; ssl_certificate_key /etc/ssl/private/nginx.key; ssl_session_cache off; ssl_session_tickets off; location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass upstream webserver_0 { server :80; keepalive 200; 45

46 Reverse proxy (Instance 2) user root; worker_processes 48 ; worker_cpu_affinity auto ; worker_rlimit_nofile ; error_log /home/ubuntu/access.error crit; events { worker_connections ; http { access_log off; keepalive_timeout 315; keepalive_requests ; sendfile on; tcp_nopush on; tcp_nodelay on; 46

47 Reverse proxy (Instance 2) server { listen :443 ssl backlog= reuseport; ssl_certificate /etc/ssl/certs/nginx.pem; ssl_certificate_key /etc/ssl/private/nginx.key; ssl_session_cache off; ssl_session_tickets off; location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass upstream webserver_1 { server :80; keepalive 200; 47