MapReduce Install
Show Source Code
module.exports = header: 'MapReduce Client Install', handler: ({options}) ->
version = null
Register
Show Source Code
@registry.register 'hconfigure', 'ryba/lib/hconfigure'
@registry.register 'hdp_select', 'ryba/lib/hdp_select'
@registry.register 'hdfs_upload', 'ryba/lib/hdfs_upload'
@registry.register ['hdfs','put'], 'ryba/lib/actions/hdfs/put'
IPTables
| Service | Port | Proto | Parameter |
|---|---|---|---|
| mapreduce | 59100-59200 | http | yarn.app.mapreduce.am.job.client.port-range |
IPTables rules are only inserted if the parameter "iptables.action" is set to "start" (default value).
Show Source Code
jobclient = options.mapred_site['yarn.app.mapreduce.am.job.client.port-range']
jobclient = jobclient.replace '-', ':'
@tools.iptables
header: 'IPTables'
if: options.iptables
rules: [
{ chain: 'INPUT', jump: 'ACCEPT', dport: jobclient, protocol: 'tcp', state: 'NEW', comment: "MapRed Client Range" }
]
Identities
Show Source Code
@system.group header: 'Group', options.hadoop_group
@system.user header: 'User', options.user
Service
Show Source Code
@call header: 'Packages', ->
@service
name: 'hadoop-mapreduce'
@hdp_select
name: 'hadoop-client'
@system.execute
shy: true
cmd: """
hdp-select versions | tail -1
"""
, (err, {status, stdout}) ->
throw err if err
version = stdout.trim()
@hconfigure
header: 'Configuration'
target: "#{options.conf_dir}/mapred-site.xml"
source: "#{__dirname}/../resources/mapred-site.xml"
local: true
properties: options.mapred_site
backup: true
uid: options.user.name
gid: options.group.name
HDFS Tarballs
Upload the MapReduce tarball inside the "/hdp/apps/$version/mapreduce" HDFS directory. Note, the parent directories are created by the "ryba/hadoop/hdfs_dn/layout" module.
Show Source Code
# @hdfs_upload
# header: 'HDFS Tarballs'
# wait: 60*1000
# id: options.hostname
# lock: '/tmp/ryba-mapreduce.lock'
# krb5_user: options.hdfs_krb5_user
@call
header: 'HDFS Tarballs'
, ->
@hdfs.put
header: 'HDFS Tarballs'
source: '/usr/hdp/current/hadoop-client/mapreduce.tar.gz'
target: "/hdp/apps/#{version}/mapreduce/mapreduce.tar.gz"
nn_url: options.nn_url
krb5_user: options.hdfs_krb5_user
mode: '444'
Ulimit
Increase ulimit for the MapReduce user. The HDP package create the following files:
cat /etc/security/limits.d/mapred.conf
mapred - nofile 32768
mapred - nproc 65536
Note, a user must re-login for those changes to be taken into account. See the "ryba/hadoop/hdfs" module for additional information.
Show Source Code
@system.limits
header: 'Ulimit'
user: options.user.name
, options.user.limits
Dependencies
Show Source Code
mkcmd = require '../../lib/mkcmd'