同步操作将从 lee/ambari-flink-service 强制同步,此操作会覆盖自 Fork 仓库以来所做的任何修改,且无法恢复!!!
确定后同步将在后台操作,完成时将刷新页面,请耐心等待。
在HDP集群上轻松安装和管理Flink的Ambari服务。 Ambari service for easily installing and managing Flink on HDP clusters. ApacheFlink是一个用于分布式流和批处理数据处理的开源平台。 Apache Flink is an open source platform for distributed stream and batch data processing 关于Flink的更多细节,以及它在今天的行业中是如何使用的,在这里可以找到。 More details on Flink and how it is being used in the industry today available here: http://flink-forward.org/?post_type=session
The Ambari service lets you easily install/compile Flink on HDP 2.5.3
Limitations:
Author: Ali Bajwa
192.168.191.241 sandbox.hortonworks.com sandbox
Note that you will need to replace the above with the IP for your own VM
Connect to the VM via SSH (password hadoop)
ssh root@sandbox.hortonworks.com
To download the Flink service folder, run below
运行以下脚本,将Ambari Stack加入flink
安装git、wget
sudo yum -y install git
VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
sudo git clone https://gitee.com/sleechengn/ambari-flink-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK
#sandbox
service ambari restart
#non sandbox
sudo service ambari-server restart
On bottom left -> Actions -> Add service -> check Flink server -> Next -> Next -> Change any config you like (e.g. install dir, memory sizes, num containers or values in flink-conf.yaml) -> Next -> Deploy
By default:
On successful deployment you will see the Flink service as part of Ambari stack and will be able to start/stop the service from here:
You can see the parameters you configured under 'Configs' tab
One benefit to wrapping the component in Ambari service is that you can now monitor/manage this service remotely via REST API
export SERVICE=FLINK
export PASSWORD=admin
export AMBARI_HOST=localhost
#detect name of cluster
output=`curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' http://$AMBARI_HOST:8080/api/v1/clusters`
CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`
#get service status
curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X GET http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE
#start service
curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Start $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE
#stop service
curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE
su flink
export HADOOP_CONF_DIR=/etc/hadoop/conf
cd /opt/flink
./bin/flink run --jobmanager yarn-cluster -yn 1 -ytm 768 -yjm 768 ./examples/batch/WordCount.jar
This should generate a series of word counts
Open the YARN ResourceManager UI. Notice Flink is running on YARN
Click the ApplicationMaster link to access Flink webUI
Use the History tab to review details of the job that ran:
View metrics in the Task Manager tab:
More details on Flink and how it is being used in the industry today available here: http://flink-forward.org/?post_type=session
Stop the service via Ambari
Unregister the service
export SERVICE=FLINK
export PASSWORD=admin
export AMBARI_HOST=localhost
#detect name of cluster
output=curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' http://$AMBARI_HOST:8080/api/v1/clusters
CLUSTER=echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'
curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X DELETE http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE
#if above errors out, run below first to fully stop the service #curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE ```
rm -rf /opt/flink* rm /tmp/flink.tgz
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。