squid負載過高,必然DOWN機,墾請高手來分析

fugangyun

操作系統(tǒng):solaris 10
服務(wù)器:SUN E2900   16G內(nèi)存    4CPU   2個10000轉(zhuǎn)SCSI硬盤
df -h

/dev/dsk/c1t0d0s3       20G   4.4G    15G    23%    /var
swap                    27G     0K    27G     0%    /tmp
swap                    27G    16K    27G     1%    /var/run
/dev/dsk/c1t1d0s1       16G    11G   4.3G    73%    /proxy
/dev/dsk/c1t1d0s3       20G    14G   5.6G    72%    /cache1

squid版本是2.6,安裝在/proxy下
squid.conf配置如下:

http_port 10.1.1.1:8081
hierarchy_stoplist cgi-bin ?
hierarchy_stoplist -i ^https:\\ ?
acl QUERY urlpath_regex -i cgi-bin \? \.asp \.php \.jsp \.cgi
acl denyssl urlpath_regex -i ^https:\\
cache deny QUERY
cache deny denyssl
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

cache_mem 1024 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 8192 KB
maximum_object_size_in_memory 32 KB

cache_dir ufs /proxy/var/cache 12000 48 256
cache_dir ufs /cache1 15000 48 256
access_log none
cache_log /cmproxy/var/logs/cache.log
cache_store_log none
mime_table /proxy/etc/mime.conf
pid_filename /proxy/var/logs/squid.pid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320

acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80                # http
acl Safe_ports port 21                # ftp
acl Safe_ports port 443 563           # https, snews
acl Safe_ports port 70                # gopher
acl Safe_ports port 210               # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488                # gss-http
acl Safe_ports port 591                # filemaker
acl Safe_ports port 777                # multiling http
acl CONNECT method CONNECT
acl SSL_gmcc_port port 81 443 563 7001 8080 8888 9087 9088 9315
acl our_networks src 10.0.0.0/8 192.168.0.0/16
http_access allow our_networks
http_access allow manager localhost
http_access deny manager
http_access allow SSL_gmcc_port
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all
http_reply_access allow all

logfile_rotate 7
visible_hostname none
forwarded_for off
store_objects_per_bucket 50
icon_directory /proxy/share/icons
coredump_dir /cache1

fugangyun

同時在線連接數(shù)超過3K后,終端打開網(wǎng)頁極慢,且squid進程僵死,無法通過squid -k shutdown停止,即使用squid -k kill 或kill -9 進程id也無效,只有重啟機器

連接數(shù)

netstat -an|grep 8081|grep EST|wc -l
3096

等待TIME_WAIT數(shù)

netstat -an|grep TIME_WAIT|wc -l
5300

CPU占用率并不高

#sar -u 3 5

SunOS 5.10 Generic_118833-02 sun4u    03/21/2007

08:16:10    %usr    %sys    %wio   %idle
08:16:13       3       5       0      92
08:16:16       3       5       0      92
08:16:19       2       6       0      92
08:16:22       2       4       0      94
08:16:25       2       4       0      93

Average        2       5       0      93

磁盤IO使用率較高

iostat -cxn 5
                 extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d20
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d21
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d22
    0.0    0.6    0.0    0.4  0.0  0.0    0.0    9.1   0   0 c1t0d0
   18.6  128.2   70.8 1440.9  0.0  3.8    0.0   25.9   0  81 c1t1d0

內(nèi)存使用及機器負荷

prstat
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP      
   279 root     1577M 1573M cpu1     0    0   3:45:19 3.7% squid/1
   125 named      10M 7752K sleep   59    0   0:04:56 0.1% named/11
   280 nobody   1168K  840K sleep   60    0   0:04:15 0.1% unlinkd/1

Total: 26 processes, 134 lwps, load averages: 0.50, 0.55, 0.54

[ 本帖最后由 fugangyun 于 2007-3-21 09:49 編輯 ]

fugangyun

squid的信息

squidclient -p 8081 mgr:info

HTTP/1.0 200 OK
Server: squid/2.6.STABLE10
Date: Wed, 21 Mar 2007 00:23:01 GMT
Content-Type: text/plain
Expires: Wed, 21 Mar 2007 00:23:01 GMT
Last-Modified: Wed, 21 Mar 2007 00:23:01 GMT
X-Cache: MISS from none
Via: 1.0 none:8081 (squid/2.6.STABLE10)
Proxy-Connection: close

Squid Object Cache: Version 2.6.STABLE10
Start Time:     Tue, 20 Mar 2007 04:32:03 GMT
Current Time:   Wed, 21 Mar 2007 00:23:01 GMT
Connection information for squid:
        Number of clients accessing cache:      4514
        Number of HTTP requests received:       7291534
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   6122.4
        Average ICP messages per minute since start:    0.0
        Select loop called: 39501660 times, 1.809 ms avg
Cache information for squid:
        Request Hit Ratios:     5min: 46.8%, 60min: 47.8%
        Byte Hit Ratios:        5min: 14.7%, 60min: 14.5%
        Request Memory Hit Ratios:      5min: 17.4%, 60min: 21.3%
        Request Disk Hit Ratios:        5min: 13.4%, 60min: 11.9%
        Storage Swap size:      24968613 KB
        Storage Mem size:       1048296 KB
        Mean Object Size:       22.17 KB
        Requests given to unlinkd:      871980
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.19742  0.12106
        Cache Misses:          0.44492  0.30459
        Cache Hits:            0.07014  0.04277
        Near Hits:             0.30459  0.22004
        Not-Modified Replies:  0.05046  0.03427
        DNS Lookups:           0.04854  0.02809
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        71457.499 seconds
        CPU Time:       13602.398 seconds
        CPU Usage:      19.04%
        CPU Usage, 5 minute avg:        30.82%
        CPU Usage, 60 minute avg:       30.37%
        Process Data Segment Size via sbrk(): 1596379 KB
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 10096743
Memory usage for squid via mallinfo():
        Total space in arena:  1596379 KB
        Ordinary blocks:       1471660 KB 258638 blks
        Small blocks:               0 KB      0 blks
        Holding blocks:         10928 KB     10 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:  124718 KB
        Total in use:          1482588 KB 92%
        Total free:            124718 KB 8%
        Total size:            1607307 KB
Memory accounted for:
        Total accounted:       1287875 KB
        memPoolAlloc calls: 753300395
        memPoolFree calls: 746272417
File descriptor usage for squid:
        Maximum number of file descriptors:   32768
        Largest file desc currently in use:   5321
        Number of file desc currently in use: 4396
        Files queued for open:                   0
        Available number of file descriptors: 28372
        Reserved number of file descriptors:   100
        Store Disk files open:                  26
        IO loop method:                     poll
Internal Data Structures:
        1133385 StoreEntries
        143938 StoreEntries with MemObjects
        143638 Hot Object Cache Items
        1126459 on-disk objects

fugangyun

每次DOWN機都在messages里出現(xiàn),

[ID 702911 user.alert] xmalloc: Unable to allocate 49152 bytes!
WARNING: High TCP connect timeout rate! System (port 8081) may be under a SYN flood attack!

chinaglwo

把系統(tǒng)的SYN flood 打開呢

linux可以的，solaris 不太清楚

xdb1010

可能是有人用CC攻擊你的站點,耗死你服務(wù)器的資源.
提示SYN flood attack的話可能就是有SYN 洪水攻擊.
設(shè)置操作系統(tǒng)本身效果不會很大的,如果有硬件防火墻效果就能好些.

fugangyun

咨詢過SUN公司，出現(xiàn)

WARNING: High TCP connect timeout rate! System (port 8081) may be under a SYN flood attack!

是一種常規(guī)告警，并非真正的受到攻擊

且這臺服務(wù)器是在DMZ區(qū)，有CISCO PIX 535硬件防火墻

fugangyun

在squid 的FAQ找到如下信息:

xmalloc: Unable to allocate 4096 bytes!


Messages like "FATAL: xcalloc: Unable to allocate 4096 blocks of 1 bytes!" appear when Squid can't allocate more memory, and on most operating systems (inclusive BSD) there are only two possible reasons:

    * The machine is out of swap
    * The process' maximum data segment size has been reached

The first case is detected using the normal swap monitoring tools available on the platform (pstat on SunOS, perhaps pstat is used on BSD as well).

To tell if it is the second case, first rule out the first case and then monitor the size of the Squid process. If it dies at a certain size with plenty of swap left then the max data segment size is reached without no doubts.

The data segment size can be limited by two factors:

    * Kernel imposed maximum, which no user can go above
    * The size set with ulimit, which the user can control.

When squid starts it sets data and file ulimit's to the hard level. If you manually tune ulimit before starting Squid make sure that you set the hard limit and not only the soft limit (the default operation of ulimit is to only change the soft limit). root is allowed to raise the soft limit above the hard limit.

fugangyun

問題另類解決了!

SUN公司給了一個編譯好的二進制squid2.5 for solaris 10的包給我，裝好后正常

用戶連接數(shù)同時達到55000，也不會進程僵死，速度還可以

zhangweizj

連接數(shù)太大而導致squid內(nèi)存分配失敗嗎？，可惜不能看到已經(jīng)編譯好的squid的源代碼。。，看看改了什么