centos 7 下的squid與openstack下的cdn實(shí)例

cryboy2001

Contos 7 下最新squid設(shè)定

常用代理
透明代理
用戶認(rèn)證
強(qiáng)大的alc功能
Cdn介紹
Squid在cdn中的使用---反向代理
openstack下的cdn實(shí)例
bind實(shí)現(xiàn)view功能

環(huán)境
最小化安裝contos7 ，關(guān)掉selinux，打開轉(zhuǎn)發(fā)，防火墻就不關(guān)了，透明代理時(shí)要用到
兩塊網(wǎng)卡，ens6  192.168.0.207與外網(wǎng)eth0：2.2.2.2
SELINUX=disabled
echo "1" > /proc/sys/net/ipv4/ip_forward
加裝編譯環(huán)境
yum -y install make gcc cc  gcc+ gcc-c++
下載最新版squid
wget http://www.squid-cache.org/Versions/v3/3.4/squid-3.4.7.tar.gz

tar -zxvf squid-3.4.7.tar.gz
cd squid-3.4.7/
./configure --prefix=/usr/local/squid
make all
make install
ls /usr/local/squid/   #查看一下
加用戶與權(quán)限
adduser squid -d /dev/null -s /sbin/nologin
chown -R squid.squid squid
編輯配置文件
vi /usr/local/squid/etc/squid.conf

visible_hostname squid.test.com      #加主機(jī)名
pid_filename /var/run/squid.pid
cache_effective_user squid                #用戶與組
cache_effective_group squid
#下面是定義與應(yīng)用acl，功能很強(qiáng)大
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/24 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all                  #最后一句拒絕其它所有
http_port 3128                           #端口
cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256            #cache放的地方
coredump_dir /usr/local/squid/var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
檢查配置文件一下對(duì)不對(duì)
./squid -k parse
./squid  -z         #初始化
運(yùn)行
./squid



修改了配置后再運(yùn)行
./squid -k reconfigure
加防火墻與以前的iptables不一樣的
firewall-cmd --add-port=3128/tcp
客戶端設(shè)置



成功的樣子



access.log




透明代理

上面的代理要實(shí)現(xiàn)，要在客戶端設(shè)置，為了方便不用在客戶端設(shè)置，可以用透明代理
透明代理設(shè)置很簡(jiǎn)單，只在
把原來的
http_port 3128  
改成 192.168.0.207是這臺(tái)電腦的ip
http_port 192.168.0.207 3128  transparent

關(guān)鍵是要改防火墻
firewall-cmd --zone=external --change-interface=eth0
firewall-cmd --zone=internal --change-interface=ens6
firewall-cmd --zone=internal --add-forward-port=port=80:proto=tcp:toaddr=192.168.0.207:toport=3128   
firewall-cmd --zone=internal --list-all

  

客戶端設(shè)置
不用設(shè)ie代理，只要把路由指向這臺(tái)squid電腦就可以了，這可以通過dhcp設(shè)置。




cat var/logs/access.log




待續(xù)

cryboy2001

用戶認(rèn)證

因?yàn)樵谏厦鏇]加認(rèn)證模塊所以要重新編譯

  cd squid-3.4.7/
./configure --enable-basic-auth-helpers="NCSA"
make
編譯好了之后，不用現(xiàn)安裝了，直接把要用的copy過去就可以了

cp  helpers/basic_auth/NCSA/basic_ncsa_auth /usr/local/squid/bin/ncsa_auth
   chown squid.squid ncsa_auth

修改配配

vi /usr/local/squid/etc/squid.conf

auth_param basic program /usr/local/squid/bin/ncsa_auth /usr/local/squid/etc/passwd
acl authuser proxy_auth REQUIRED
http_access allow authuser
#加上以上三句，注意一定要加在
http_access deny all
這一句前面

把透明代理的改回來
#http_port 192.168.0.207 3128  transparent
http_port 3128
把前面
#acl localnet src 192.168.0.0/24
注消掉

完成的squid.conf如下

visible_hostname squid.test.com
pid_filename /var/run/squid.pid
cache_effective_user squid
cache_effective_group squid
http_port 3128
acl localnet src 10.0.0.0/8        # RFC1918 possible internal network
acl localnet src 172.16.0.0/12        # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80         # http
acl Safe_ports port 21         # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70         # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
auth_param basic program /usr/local/squid/bin/ncsa_auth /usr/local/squid/etc/passwd
acl   authuser proxy_auth REQUIRED  
http_access allow  authuser
http_access deny all
cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256
coredump_dir /usr/local/squid/var/cache/squid
refresh_pattern ^ftp:         1440        20%        10080
refresh_pattern ^gopher:        1440        0%        1440
refresh_pattern -i (/cgi-bin/|\?) 0        0%        0
refresh_pattern .         0        20%        4320

用htpasswd生成密碼文件
htpasswd -c /usr/local/squid/etc/passwd test1
htpasswd /usr/local/squid/etc/passwd test2



重新生效squid配置

./squid -k reconfigure

直接訪問出現(xiàn)



要向開始一樣改代理


改好后再訪問出現(xiàn)





用戶認(rèn)證就成功了。

cryboy2001

acl功能

在前面的兩個(gè)應(yīng)用中已用到了acl功能，acl應(yīng)用分兩步
1、定義acl 組如
acl all src 0.0.0.0/0.0.0.0    #定義一個(gè)所有的ip組合，般用在最后一句

以下是定義名為localnet的acl組，對(duì)這個(gè)組作同樣的操作
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines

下面是定義端口
acl SSL_ports port 443  #定義安全端口，不作這代理。

acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http

2、應(yīng)用acl組，用deny或者allow來做為動(dòng)作

http_access deny !Safe_ports    #加！號(hào)就是相反，拒絕不是以上定的Safe_ports組中的端口
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost

http_access deny all       #拒絕所有其它來源

常用的acl
1、不讓192.168.0.22通過代理上網(wǎng)
acl test1 src 192.168.0.22
http_access deny test1

2、允許192.168.1.0/24通過
acl test2 src 192.168.1.0/24
http_access deny test2

3、不允許訪問一個(gè)服務(wù)器

acl test3 dst 2.2.2.2
http_access deny test3

4、只有上班時(shí)間這臺(tái)電腦才可以上網(wǎng)
acl pc1 src 192.168.0.22
acl worktime time 8:00-17:00

http_access allow pc1 worktime

5、限制一臺(tái)電腦上網(wǎng)的連接數(shù)
acl pc1 src 192.168.0.22
acl maxconn maxconn 10
http_access deny  pc1 maxconn

cryboy2001

cdn架構(gòu)


一句話：通過智能dns，根據(jù)用戶來源，就近訪問，代理緩存服務(wù)器，以達(dá)到快速的目的，同時(shí)可以增加源站的安全與穩(wěn)定。
代理緩存服務(wù)器就是反向代理源站的內(nèi)容。

組成部分：源站、緩存服務(wù)器、智能dns、用戶





cdn的性能指標(biāo)：并發(fā)量、吞吐量、響應(yīng)時(shí)間、丟包率、命中率
前四個(gè)指標(biāo)與其它服務(wù)器一樣，命中率：客戶端訪問時(shí)，在cache中有，不用到源站抓起，這叫命中，否則就是不命中。
命中率高說明cdn系統(tǒng)的緩存策略越好，給源站分擔(dān)的壓力越多。

緩存的對(duì)象：靜態(tài)、動(dòng)態(tài)、流媒體等多種

下面就一般緩存方法：

客戶端----------緩存服務(wù)器--------------源站

用Squid做為緩存服務(wù)器

#cat /usr/local/squid/etc/squid.conf

visible_hostname squid.test.com
pid_filename /var/run/squid.pid
cache_effective_user squid
cache_effective_group squid
acl localnet src 10.0.0.0/8        # RFC1918 possible internal network
acl localnet src 172.16.0.0/12        # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80         # http
acl Safe_ports port 21         # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70         # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256
coredump_dir /usr/local/squid/var/cache/squid
refresh_pattern ^ftp:         1440        20%        10080
refresh_pattern ^gopher:        1440        0%        1440
refresh_pattern -i (/cgi-bin/|\?) 0        0%        0
refresh_pattern .         0        20%        4320

#主要就加了以下3句，

http_port 80 accel vhost vport                #啟用80端口，要不然squid是偵聽3128端口的
cache_peer 2.2.2.2  parent 80 0               #cache中沒命中的話就到上一級(jí)或源站
http_access allow all                                #允許所有連接 要把之前的deny all 刪除

使配置生效
#squid -k reconfigure     

看看80端口誰在用
#lsof -i :80


是squidg正使用80端口

在客戶端測(cè)試看看



查看日志

TCP_MISS 在cache中沒找到，要連接到源站



TCP_MEM在cache中找到了，直接還回



當(dāng)然以上只是最簡(jiǎn)單的測(cè)試，cdn的核心其實(shí)是智能dns，真正使用的cdn還包過各部分的集群，使用的最多的智能dns開源軟件是bind view,有空做一個(gè)完整的cdn。

yestreenstars

牛，學(xué)習(xí)了，先頂了再看

cryboy2001

用opentstack做cdn實(shí)驗(yàn)

架構(gòu)



主機(jī)名與ip



一、源主機(jī)的安裝

[root@www ~]# #yum -y install httpd

產(chǎn)生一個(gè)測(cè)試頁面

[root@www ~]# cat /var/www/html/index.html
<html>
<body>
This is a test page!
</body>
</html>
[root@www ~]#

打開防火墻

systemctl httpd.service start

systemctl start httpd.service



二、dns主機(jī)安裝

安裝bind

#yum -y install make gcc cc gcc+ gcc-c++ openssl openssl-devel perl
# wget ftp://ftp.isc.org/isc/bind9/9.10.1/bind-9.10.1.tar.gz
#tar zxvf bind-9.10.1.tar.gz
# cd bind-9.10.1

# ./configure --prefix=/usr/local/bind

#make && make install



制作配置文件

#/usr/local/bind/sbin/rndc-confgen >rndc.conf

#cat rndc.conf >rndc.key

#tail -n10 rndc.conf |head -n9 |sed -e s/#\ //g >named.conf


[root@dns bind]# cat etc/named.conf
key "rndc-key" {
algorithm hmac-md5;
secret "mHcJqq5mIIkvwZMwr5I5hg==";
};

controls {
inet 127.0.0.1 port 953
allow { 127.0.0.1; } keys { "rndc-key"; };
};

logging {
channel query_log {
file "query.log" versions 3 size 20m;
severity info;
print-time yes;
print-category yes;
};
category queries {
query_log;
};
};

options {
directory "/usr/local/bind/var";
pid-file "named.pid";
listen-on port 53 {any;};
allow-query {any;};
};

view "squid1" {
match-clients { 10.0.10.0/24; 10.0.1.0/24; };
recursion yes;

zone "." IN {
type hint;
file "named.ca";
};


zone "localhost" IN {
type master;
file "localhost.zone";
};
zone "0.0.127.in-addr.arpa" IN {
type master;
file "localhost.arpa";
};


zone "test.com" IN {
type master;
file "test1.com.zone";
allow-update {none;};
};

};

view "squid2" {
match-clients { 10.0.20.0/24;};
recursion yes;
zone "." IN {
type hint;
file "named.ca";
};

zone "localhost" IN {
type master;
file "localhost.zone";
};
zone "0.0.127.in-addr.arpa" IN {
type master;
file "localhost.arpa";
};


zone "test.com" IN {
type master;
file "test2.com.zone";
allow-update {none;};
};

};

view "other" {
match-clients { 0.0.0.0/0;};
recursion yes;
zone "." IN {
type hint;
file "named.ca";
};

zone "localhost" IN {
type master;
file "localhost.zone";
};
zone "0.0.127.in-addr.arpa" IN {
type master;
file "localhost.arpa";
};


zone "test.com" IN {
type master;
file "test.com.zone";
allow-update {none;};
};

};





相應(yīng)目錄下的zonefile

[root@dns bind]# ls var/test*
var/test1.com.zone var/test2.com.zone var/test.com.zone



[root@dns bind]# cat var/test*
$TTL 86400
@ IN SOA dns.test.com. hostmaster.test.com. (
20140930 ; serial
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum



@ IN NS dns.test.com.
dns IN A 10.0.1.18
www IN A 10.0.10.102
$TTL 86400
@ IN SOA dns.test.com. hostmaster.test.com. (
20140930 ; serial
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum


@ IN NS dns.test.com.
dns IN A 10.0.1.18
www IN A 10.0.20.101
$TTL 86400
@ IN SOA dns.test.com. hostmaster.test.com. (
20140930 ; serial
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum

@ IN NS dns.test.com.
dns IN A 10.0.1.18
www IN A 10.0.1.19
squid1 IN A 10.0.10.102
squid2 IN A 10.0.20.101

設(shè)客戶端dns
  
xp1上的dns查詢


xp2上的查詢



其它電腦上的查詢




通過以上的查詢，確認(rèn)dns view已成功配好了。

三、代理緩存安裝

看前面的squid設(shè)置內(nèi)容

#cat squid.conf

visible_hostname squid2.test.com

pid_filename /var/run/squid.pid

cache_effective_user squid

cache_effective_group squid

acl localnet src 10.0.0.0/8 # RFC1918 possible internal network

acl localnet src 172.16.0.0/12 # RFC1918 possible internal network

acl localnet src fc00::/7 # RFC 4193 local private network range

acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443

acl Safe_ports port 80 # http

acl Safe_ports port 21 # ftp

acl Safe_ports port 443 # https

acl Safe_ports port 70 # gopher

acl Safe_ports port 210 # wais

acl Safe_ports port 1025-65535 # unregistered ports

acl Safe_ports port 280 # http-mgmt

acl Safe_ports port 488 # gss-http

acl Safe_ports port 591 # filemaker

acl Safe_ports port 777 # multiling http

acl CONNECT method CONNECT

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost manager

http_access deny manager

http_access allow localnet

http_access allow localhost

cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256

coredump_dir /usr/local/squid/var/cache/squid

refresh_pattern ^ftp: 1440 20% 10080

refresh_pattern ^gopher: 1440 0% 1440

refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

refresh_pattern . 0 20% 4320

http_port 80 accel vhost vport

cache_peer 10.0.1.19 parent 80 0

http_access allow all


兩臺(tái)一樣，除主機(jī)名外其它都是一樣的


還有要打開80端口防火墻

四、客戶機(jī)上試用

在客戶機(jī)上的動(dòng)作與日志



找一個(gè)不存在的網(wǎng)頁



squid上的相關(guān)日志非常明顯



加一臺(tái)客戶端，訪問的是另一臺(tái)squid



不存在的網(wǎng)頁



日志很清楚

cu_shell

高手，，，收藏學(xué)習(xí)，，版主這個(gè)squid配置教程支持 rhel5.x 版本嗎？

action08

shell@ubuntu:~$ uname -a
Linux ubuntu 3.2.0-69-generic #103-Ubuntu SMP Tue Sep 2 05:02:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
shell@ubuntu:~$


我的應(yīng)該支持吧

cryboy2001

回復(fù) 7# cu_shell

@action08

自己編譯的，什么系統(tǒng)都支持，配置都差不多的，只是其它操作稍有不同，比如啟動(dòng)程序，防火墻等
   

reb00t

版主v587，求拜師。