|
作者：尹正杰 curl是基于URL语法在命令行方式下工作的文件传输工具，它支持FTP，FTPS，HTTP，HTTPS，GOPHER，TELNET，DICT，FILE及LDAP等协议。curl支持HTTPS认证，并且支持HTTP的POST,PUT等方法，FTP上传，kerberos认证，HTTP上传，代理服务器，cookies，用户名/密码认证，通过http代理服务器上传文件到FTP服务器等等，功能十分强大。
一.curl的常用选项：
 1 -A/--user-agent <string> 设置用户代理发送给服务器，即告诉服务器浏览器为什么
2 -basic 使用HTTP基本验证
3 --tcp-nodelay 使用TCP_NODELAY选项
4 -e/--referer <URL> 来源网址，跳转过来的网址
5 --cacert <file> 指定CA证书 (SSL)
6 --compressed 要求返回是压缩的形势，如果文件本身为一个压缩文件，则可以下载至本地
7 -H/--header <line>自定义头信息传递给服务器
8 -I/--head 只显示响应报文首部信息
9 --limit-rate <rate> 设置传输速度
10 -u/--user <user[:password]>设置服务器的用户和密码
11 -0/--http1.0 使用HTTP 1.0
二.检查测试环境 1.操作环境
1 [root@yinzhengjie ~]# cat /etc/redhat-release 2 CentOS release 6.6 (Final) 3 [root@yinzhengjie ~]# 4 [root@yinzhengjie ~]# uname -r 5 2.6.32-504.el6.x86_64 6 [root@yinzhengjie ~]# 7 [root@yinzhengjie ~]# uname -m 8 x86_64 9 [root@yinzhengjie ~]#
2.检查hosts文件 1 [root@yinzhengjie ~]# more /etc/hosts 2 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 3 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 4 5 192.168.1.105 node1.yinzhengjie.com 6 192.168.1.110 node2.yinzhengjie.com 7 192.168.1.115 node3.yinzhengjie.com 8 192.168.1.200 node4.yinzhengjie.com 9 10 192.168.1.115 www.yinzhengjie.com 11 192.168.1.115 www.yinzhengjie.gov.cn 12 192.168.1.115 www. 13 [root@yinzhengjie ~]# 14 [root@yinzhengjie ~]#
3.检查htttp服务是否启动（因为测试都是以本机测试为主，我的虚拟机自己设置成无法联网，哈哈~） 1 [root@yinzhengjie ~]# lsof -i :80 2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 3 clock-app 13013 root 22u IPv4 204927 0t0 TCP node3.yinzhengjie.com:56318->a23-33-178-8.deploy.static.akamaitechnologies.com:http (ESTABLISHED) 4 httpd 63824 root 4u IPv6 203348 0t0 TCP *:http (LISTEN) 5 httpd 63826 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 6 httpd 63827 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 7 httpd 63828 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 8 httpd 63829 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 9 httpd 63830 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 10 httpd 63831 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 11 httpd 63832 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 12 httpd 63833 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 13 httpd 63834 apache 4u IPv6 203348 0t0 TCP *:http (LISTEN) 14 [root@yinzhengjie ~]# 15 [root@yinzhengjie ~]# 16 [root@yinzhengjie ~]# 17 [root@yinzhengjie ~]# lsof -i :443 18 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 19 httpd 63824 root 8u IPv6 203356 0t0 TCP *:https (LISTEN) 20 httpd 63826 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 21 httpd 63827 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 22 httpd 63828 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 23 httpd 63829 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 24 httpd 63830 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 25 httpd 63831 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 26 httpd 63832 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 27 httpd 63833 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 28 httpd 63834 apache 8u IPv6 203356 0t0 TCP *:https (LISTEN) 29 [root@yinzhengjie ~]#
三.curl命令用法展示 我们此处仅仅展示以命令的方式来执行HTTP协议的请求的工具。 1.-I选项，只获得对方的响应首部信息； 这个参数在很多时候都很有用，比方说，我们实现了代理服务器并在代理服务器上手动自定义了一些首部的话，使用curl这个工具的“-I”选项可以很容易的探测出我们的代理服务器是否正确添加了我们自定义的首部。 1 [root@yinzhengjie ~]# curl -I http://www. 2 HTTP/1.1 200 OK 3 Date: Mon, 23 Oct 2017 15:28:07 GMT 4 Server: Apache/2.2.15 (CentOS) 5 Last-Modified: Sat, 21 Oct 2017 12:47:33 GMT 6 ETag: "bef67-20-55c0dfe410f50" 7 Accept-Ranges: bytes 8 Content-Length: 32 9 Connection: close 10 Content-Type: text/html; charset=UTF-8 11 12 You have new mail in /var/spool/mail/root 13 [root@yinzhengjie ~]#
2.-A,设置用户代理发送给服务器,即可以伪装客户端身份 1 [root@yinzhengjie ~]# curl -A testagent http://www. 2 <h1>www.</h1> 3 [root@yinzhengjie ~]# 4 [root@yinzhengjie httpd]# tailf www..log 5 192.168.1.115 - - [23/Oct/2017:08:38:58 -0700] "GET / HTTP/1.1" 200 32 "-" "testagent" ------>在日志可以看到客户端的名称被隐藏了
3.-e，伪装<URL> 来源网址，跳转过来的网址  1 [root@yinzhengjie ~]# curl -e http://www.google.com/index.html http://www. 2 <h1>www.</h1> 3 [root@yinzhengjie ~]# 4 [root@yinzhengjie httpd]# tailf www..log 5 192.168.1.115 - - [23/Oct/2017:08:38:58 -0700] "GET / HTTP/1.1" 200 32 "-" "testagent" 6 192.168.1.115 - - [23/Oct/2017:08:42:44 -0700] "GET / HTTP/1.1" 200 32 "http://www.google.com/index.html" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" ------>果不其然，显示的源连接也被伪装了 7
4.-cacert，指定CA证书 (SSL) [root@yinzhengjie ~]# curl --cacert /etc/pki/CA/cacert.pem https://www.
5.访问一个网页 1 [root@yinzhengjie ~]# curl www.baidu.com 2 <!DOCTYPE html> 3 <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1./r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> 4 [root@yinzhengjie ~]#
6.显示http response的头信息 1 [root@yinzhengjie ~]# curl -i www.baidu.com 2 HTTP/1.1 200 OK 3 Server: bfe/1.0.8.18 4 Date: Sun, 29 Oct 2017 13:25:05 GMT 5 Content-Type: text/html 6 Content-Length: 2381 7 Last-Modified: Mon, 23 Jan 2017 13:27:36 GMT 8 Connection: Keep-Alive 9 ETag: "588604c8-94d" 10 Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform 11 Pragma: no-cache 12 Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ 13 Accept-Ranges: bytes 14 15 <!DOCTYPE html> 16 <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1./r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> 17 [root@yinzhengjie ~]#
7.显示一次的http请求的通信过程
1 [root@yinzhengjie ~]# curl -v www.baidu.com 2 * About to connect() to www.baidu.com port 80 (#0) 3 * Trying 220.181.111.188... connected 4 * Connected to www.baidu.com (220.181.111.188) port 80 (#0) 5 > GET / HTTP/1.1 6 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 7 > Host: www.baidu.com 8 > Accept: */* 9 > 10 < HTTP/1.1 200 OK 11 < Server: bfe/1.0.8.18 12 < Date: Sun, 29 Oct 2017 13:26:22 GMT 13 < Content-Type: text/html 14 < Content-Length: 2381 15 < Last-Modified: Mon, 23 Jan 2017 13:27:36 GMT 16 < Connection: Keep-Alive 17 < ETag: "588604c8-94d" 18 < Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform 19 < Pragma: no-cache 20 < Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ 21 < Accept-Ranges: bytes 22 < 23 <!DOCTYPE html> 24 <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1./r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> 25 * Connection #0 to host www.baidu.com left intact 26 * Closing connection #0 27 [root@yinzhengjie ~]# 28 [root@yinzhengjie ~]# curl --trace opuput.txt www.baidu.com 29 <!DOCTYPE html> 30 <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1./r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> 31 [root@yinzhengjie ~]# 32 [root@yinzhengjie ~]# more opuput.txt
8.Curl执行GET/POST/PUT/DELETE操作
1 [root@yinzhengjie ~]# curl -X PUT www.baidu.com 2 [root@yinzhengjie ~]# curl -X DELETE www.baidu.com 3 [root@yinzhengjie ~]# curl -X POST www.baidu.com 4 [root@yinzhengjie ~]# curl -X GET www.baidu.com
9.查看帮助： [root@yinzhengjie ~]# curl --help |
|
|
