Sentry手动安装、使用手册 1 Sentry简介Apache Sentry 是Cloudera公司发布的一个Hadoop开源组件,截止目前还是Apache的孵化项目,它提供了细粒度级、基于角色的授权以及多租户的管理模式。Sentry当前可以和Hive/Hcatalog、Apache Solr 和Cloudera Impala集成,未来会扩展到其他的Hadoop组件,例如HDFS和HBase. 2 准备2.1 环境说明1 Sentry安装采用rpm包的方式. 2 Hadoop版本为hadoop-2.5.0-cdh5.3.3,Hive版本为hive-0.13.1-cdh5.3.3,Sentry版本为sentry-1.4.0-cdh5.3.3 3 Sentry下载地址: http://archive-primary./cdh5/redhat/6/x86_64/cdh/5.3.3/RPMS/noarch/
[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux Red Hat Enterprise Linux Server release 6.5 (Santiago) [root@localhost native]# Java -version Java version "1.7.0_67" Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01) Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode) 2.1.3mysql版本 [root@localhost native]# MySQL -uroot -proot-e"select version()"; Warning: Using a password onthe command line interface can be insecure. +-----------+ | version() | +-----------+ | 5.6.14 | +-----------+ 注: 1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar 2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享
3 安装3.1安装Mysql1)安装mysql相关服务 rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm时报如下错误: file/usr/share/mysql/charsets/macroman.xml from install of MySQL-server-5.6.14-1.el6.x86_64conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64 file/usr/share/mysql/charsets/swe7.xml from install ofMySQL-server-5.6.14-1.el6.x86_64 conflicts with file from packagemysql-libs-5.1.71-1.el6.x86_64 rpm -q mysql-libs-5.1.71-1.el6.x86_64 rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64 rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm ARANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER ! Youwill find that password in '/root/.mysql_secret'.(生成mysql root密码) Youmust change that password on your first connect, noother statement but 'SET PASSWORD' will be accepted. rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm service mysql start #这里密码 来自/root/.mysql_secret mysql -uroot -p9RNrbk9O
#首次执行会提示修改mysql root密码 SET PASSWORD=PASSWORD('root'); #创建Hive数据库为Hive的元数据库 create database hive; GRANT all ON hive.* TO root@'%' IDENTIFIED BY 'root'; #创建sentry数据库为sentry元数据库 create database sentry; CREATE USER sentry IDENTIFIED BY 'sentry'; GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY'sentry'; flush privileges;
3.2 安装Hive1)解压Hive压缩包,并配置环境变量 cd /root tar –zxvf hive-0.13.1-cdh5.3.3.tar.gz vi ~/.bash_profile exportHIVE_HOME=/root/hive-0.13.1-cdh5.3.3 #追加HIVE执行文件路径到PATH内 exportPATH=$PATH:$HIVE_HOME/bin 2) 拷贝Mysql驱动到Hive的lib目录下: cp /root/mysql-connector-java-5.1.31-bin.jar/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar
3)配置Hive的conf.详细的hive-site.xml配置内容见下:
3.3 安装Sentry1)安装Sentry相关服务 rpm -ivh --nodepssentry-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm rpm -ivh --nodeps sentry-hdfs-plugin-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm rpm -ivh --nodepssentry-store-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
2)替换Sentry内的Hadoop、Hive、Impala、Hbase、Zookeeper、Parquet、Avro等jar包 rm -rf /usr/lib/sentry/lib/hive*.jar rm -rf /usr/lib/sentry/lib/hadoop*.jar rm -rf /usr/lib/sentry/lib/zookeeper*.jar rm -rf /usr/lib/sentry/lib/avro*.jar rm -rf /usr/lib/sentry/lib/server/hive-beeline.jar cp ~/SentryLibs/* /usr/lib/sentry/lib/ 注:这里的Sentrylibs里的jar包是通过/usr/lib/sentry/lib里的文件名分别在Hadoop、Hive、Hbase等里的安装目录里一一找到的. mv /root/SentryLibs/hive-beeline.jar/usr/lib/sentry/lib/server/ #拷贝mysql驱动到sentry的lib目录内 cp/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar/usr/lib/sentry/lib/ #如果装了Impala,此步可忽略 rpm -ivh bigtop-utils-0.7.0+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6.noarch.rpm
3) 配置Sentry,详细配置见sentry-site.xml内容如下:
4)初始化Sentry元数据 sentry --command schema-tool --conffile/etc/sentry/conf/sentry-site.xml --dbType mysql --initSchema .......................... No rows affected (0.094 seconds) No rows affected (0.015 seconds) No rows affected (0.075 seconds) 1 row affected (0.007 seconds) Closing: 0: jdbc:mysql://localhost:3306/sentry Initialization script completed Sentry schemaTool completed
3.4 安装问题暂无. 4 配置Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr 注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr. 4.1 Hive集成Sentry配置1) 拷贝Sentry相关jar包到Hive的lib目录内 cp /usr/lib/sentry/lib/sentry*.jar$HIVE_HOME/lib/ cp /usr/lib/sentry/lib/shiro-core-*.jar$HIVE_HOME/lib/
2) Hive conf内新增Sentry-site.xml,配置内容见下:
3) 修改Hive conf内的hive-site.xml,修改成如下:
4) 验证Sentry的权限控制,详见 5 使用:
4.2 Impala集成Sentry配置1) Impala的conf里新增sentry配置,/etc/impala/conf/sentry-site.xml内容如下:
2) 修改impala的参数 vi/etc/default/impala IMPALA_CATALOG_ARGS 参数里追加 -sentry_config=/etc/impala/conf/sentry-site.xml IMPALA_SERVER_ARGS 参数里追加 -server_name=server1\ -sentry_config=/etc/impala/conf/sentry-site.xml 5 使用5.1 Hive集成Sentry权限验证1) 启动sentry服务,hive的metastore和hiveserver2服务 sentry --command service--conffile /etc/sentry/conf/sentry-site.xml nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 & nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 & #hiveserver2里新增hive和test用户,通过beeline方式访问. groupadd hive; useradd hive -ghive;passwd hive (type hive123) groupadd test; useradd test -gtest;passwd test (type test123) beeline -u "jdbc:hive2://data1:10000"-n hive -p test
2)验证整体思路及场景概述 思路: 1 HIVE用户属于管理员组,服务整个server的权限 2 Test用户创建两个数据库test_only(all权限),test_select_only(仅有select权限) 具体验证: 1 通过show databases和use database验证访问权限 2 通过test用户的use test_only能drop table,create table验证all权限 3 通过test用户的use test_select_only不能drop table验证仅有select权限
#场景一 root用户连入,尝试执行DDL操作.(无权操作) [root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n root scan complete in 17ms Connecting to jdbc:hive2://data1:10000 Connected to: Apache Hive (version 0.13.1-cdh5.3.3) Driver: Hive JDBC (version 0.13.1-cdh5.3.3) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 0.13.1-cdh5.3.3 by Apache Hive 0:jdbc:hive2://data1:10000>create database sensitive; Error: Error while compiling statement: FAILED:SemanticException No valid privileges Required privileges for this query: Server=server1->action=*;(state=42000,code=40000) 0: jdbc:hive2://data1:10000> create role admin_role; Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.SentryAccessDeniedException: Access denied to root (state=08S01,code=1) 0:jdbc:hive2://data1:10000> !q Closing: 0: jdbc:hive2://data1:10000
#场景二 hive用户连入,对用户hive、test进行授权.
[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n hive scan complete in 5ms Connecting to jdbc:hive2://data1:10000 Connected to: Apache Hive (version 0.13.1-cdh5.3.3) Driver: Hive JDBC (version 0.13.1-cdh5.3.3) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 0.13.1-cdh5.3.3 by Apache Hive 0:jdbc:hive2://data1:10000>create role admin_role; No rows affected (1.383 seconds) GRANT ALL ONSERVER server1 TO ROLE admin_role; GRANT ROLEadmin_role TO GROUP hive; create roletest_role; GRANT ALL ONDATABASE test_only TO ROLE test_role; GRANT ROLEtest_role TO GROUP test; GRANT SELECT ONDATABASE test_select_only TO ROLE test_role; #场景三 test用户连入,对数据库admin_only、test_select_only进行权限验证. beeline -u"jdbc:hive2://data1:10000" -n test 0:jdbc:hive2://data1:10000> showdatabases; +----------------+--+ | database_name | +----------------+--+ | admin_only | | default | | test_only | +----------------+--+ 3 rows selected (0.721 seconds) 0:jdbc:hive2://data1:10000>use admin_only; Error: Error while compiling statement: FAILED:SemanticException No valid privileges Required privileges for this query:Server=server1->Db=admin_only->Table=*->action=insert;Server=server1->Db=admin_only->Table=*->action=select;(state=42000,code=40000) 0:jdbc:hive2://data1:10000>use test_select_only; No rows affected (0.313 seconds) 0:jdbc:hive2://data1:10000>show tables; +--------------+--+ | tab_name | +--------------+--+ | select_only | +--------------+--+ 1 row selected (0.337 seconds) 0:jdbc:hive2://data1:10000>drop table select_only; Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:hive.metastore.execute.setugi can't be false in nonsecure mode) (state=08S01,code=1) #这里需要在HIVE服务端添加如下参数 <property> <name>hive.metastore.execute.setugi</name> <value>true</value> </property>
4 rows selected (2.9 seconds) 0:jdbc:hive2://data1:10000>use test_select_only; Norows affected (0.46 seconds) 0:jdbc:hive2://data1:10000>show tables; +--------------+--+ | tab_name | +--------------+--+ | select_only | +--------------+--+ 1 row selected (0.507 seconds) 0:jdbc:hive2://data1:10000>drop table select_only; Error: Error while compilingstatement: FAILED: SemanticException No valid privileges Required privileges for this query:Server=server1->Db=test_select_only->Table=select_only->action=*;(state=42000,code=40000) #场景四 test用户连入,对数据库test_only进行验证,拥有所有权限:可以查看、删除表 0: jdbc:hive2://data1:10000>usetest_only; No rows affected (0.819 seconds) 0:jdbc:hive2://data1:10000> showtables; +--------------+--+ | tab_name | +--------------+--+ | test_itself | +--------------+--+ 1 row selected (0.426 seconds) 0:jdbc:hive2://data1:10000>drop table test_itself; No rows affected (6.336 seconds)
0: jdbc:hive2://data1:10000> create table test_newone ( ont string); Norows affected (1.377 seconds) 0:jdbc:hive2://data1:10000>show tables; +--------------+--+ | tab_name | +--------------+--+ | test_newone | +--------------+--+ 5.2 Impala集成Sentry权限验证1) 进入impala-shell,执行权限验证 create role supervisor; grant all on server to rolesupervisor; grant role supervisor to groupimpala; GRANT ALL ON SERVER server1 TOROLE any_operation; GRANT ROLE any_operation TOGROUP hive; grant ALL ON database EDA TOtest_role; grant select ON databasepriselect TO test_role;
2) 切换至test用户测试impala权限:
6 总结仔细跟踪log,具体问题具体分析. |
|