Apache Bigtop Puppet
Don’t install puppet using sudo apt-get install then expect the README.md instructions to work.
There are 3 steps to get to the point where you can start debugging the puppet scripts:
1)install puppet using the puppet labs repo shown below
2)copysite.csv.example to site.csv and modify the first entry to include the private IP address of the AWS node you are using. Make sure to update /etc/hostnames also.
3)Modify site.pp to see the config/site.csv file.
4)Run sudo puppet -d --modulepath=bigtop-deploy/puppet/modules bigtop-deploy/puppet/manifests/site.pp
Enter the following into puppet.list under /etc/apt/sources.list.d/puppet.list
deb lucid main
sudo apt-get update
ubuntu@ip-10-224-115-96:~$ apt-cache search puppet
etherpuppet - create a virtual interface from a remote Ethernet interface
puppetlabs-release - "Package to install Puppet Labs gpg key and apt repo"
puppetmaster-common - Puppet master common scripts
puppetmaster-passenger - Centralised configuration management - master setup to run under mod passenger
puppetmaster - Centralized configuration management - master startup and compatibility scripts
puppet-common - Centralized configuration management
puppet-dashboard - Dashboard for Puppet
puppet-el - syntax highlighting for puppet manifests in emacs
puppet-testsuite - Centralized configuration management - test suite
puppet - Centralized configuration management - agent startup and compatibility scripts
vim-puppet - syntax highlighting for puppet manifests in vim
The difference between the puppet repo and the default Ubuntu repo is the puppetlabs-release version.
Install the puppetlabs-release version. This will install ralsh which is not installed if you do >sudo apt-get install puppet.
Create 2 files, one called site.pp and one called testfile.pp
Create a blank testfile.txt using >sudo touch testfile.txt, run site.pp using >sudo puppet site.pp and you should see the permissions for testfile.txt change to 777.
We can add ensure=>present to the testifile.pp class to create testfile if it isn’t there.
For every system resources which exists on the server, puppet can associate a configuration or puppet resource object to it.
There is a resource shell in puppet called ralsh which is not available if you use sudo apt-get install puppet. You have to download and install puppet from the tar.gz files.
Ralsh is useful for listing the current system state in puppet syntax which can be used to create puppet configuration files.
ralsh –h for a list of commands. Note: ralsh doesn’t support –help like puppet
ralsh --types to list the commands for package
Puppet defines different types of resources. Type resources are things like packages, services, system parameters like hosts which are used to build node configurations.
Type Resources:
Install bigtop and use ralsh to print out the installed packages in puppet format using the >ralsh package command.
You will see an output like this:
package { 'gzip':
ensure => '1.3.12-9ubuntu1.1',
}
package { 'hadoop':
ensure => '1.0.1-1',
}
package { 'hadoop-conf-pseudo':
ensure => '1.0.1-1',
}
package { 'hadoop-datanode':
ensure => '1.0.1-1',
}
package { 'hadoop-doc':
ensure => '1.0.1-1',
}
package { 'hadoop-fuse':
ensure => '1.0.1-1',
}
package { 'hadoop-jobtracker':
ensure => '1.0.1-1',
}
package { 'hadoop-namenode':
ensure => '1.0.1-1',
}
package { 'hadoop-native':
ensure => '1.0.1-1',
}
package { 'hadoop-pipes':
ensure => '1.0.1-1',
}
package { 'hadoop-sbin':
ensure => '1.0.1-1',
}
package { 'hadoop-secondarynamenode':
ensure => '1.0.1-1',
}
package { 'hadoop-source':
ensure => '1.0.1-1',
}
package { 'hadoop-tasktracker':
ensure => '1.0.1-1',
}
This will be useful in helping us figure out how to create the node configuration files for all the bigtop services we need to specify.
ralsh service describes the running services:
service { 'networking':
ensure => 'stopped',
enable => 'false',
}
service { 'ondemand':
ensure => 'stopped',
enable => 'true',
}
service { 'oozie':
ensure => 'running',
enable => 'true',
}
service { 'plymouth':
ensure => 'running',
enable => 'false',
}
ralsh user
ralsh group
ralsh host
host { 'ip6-allhosts':
ensure => 'present',
ip => 'ff02::3',
target => '/etc/hosts',
}
host { 'ip6-allnodes':
ensure => 'present',
ip => 'ff02::1',
target => '/etc/hosts',
}
host { 'ip6-allrouters':
ensure => 'present',
ip => 'ff02::2',
target => '/etc/hosts',
}
host { 'ip6-localhost':
ensure => 'present',
host_aliases => ['ip6-loopback'],
ip => '::1',
target => '/etc/hosts',
}
host { 'ip6-localnet':
ensure => 'present',
ip => 'fe00::0',
target => '/etc/hosts',
}
host { 'ip6-mcastprefix':
ensure => 'present',
ip => 'ff00::0',
target => '/etc/hosts',
}
host { 'localhost':
ensure => 'present',
ip => '127.0.0.1',
target => '/etc/hosts',
}
Hadoop Experiments
Can we configure nodes to install and run hadoop?
q) abigtop installation starts all the daemons.
What does puppet do? It should also start the daemons?
There is no config file to add in IP addresses like master or slave.
q) how do you control file ops in puppet?
Require to create empty file.
Add attributes to the file and see if this works as a config file for a puppet object
We have to create a s3cfg object with a hardcoded accesskey and secret key.
Is this less work?
How to make .bashrc distributed across all computers?
Function Resources:
Functions execute only in the puppet master.
$hadoop_core_proxyusers = { oozie => { groups => 'root,hadoop,jenkins,oozie,users', hosts => "${hadoop_head_node},localhost,127.0.0.1" } }
Puppet Configuration:
You will see this error if you don’t have a site.csv file defined.
No match found for 'hadoop_head_node' in any data file during extlookup() at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/site.pp:43 on node ip-10-224-109-199.ec2.internal
Under bigtop-deploy/puppet/config copy site.csv.example to site.csv. Add in the FDQN for the AWS server, make sure you update /etc/hostname
ubuntu@ip-10-224-109-199:~/bigtop$ cat bigtop-deploy/puppet/config/site.csv
hadoop_head_node,ip-10-224-109-199
hadoop_storage_dirs,/data/1,/data/2,/data/3,/data/4
bigtop_yumrepo_uri,
Fix site.pp, I couldn’t get the data config directory for extlookup() to work so I hardcoded the path in:
After this the output looks like:
ubuntu@ip-10-224-109-199:~/bigtop$ sudo puppet -d --modulepath=bigtop-deploy/puppet/modules bigtop-deploy/puppet/manifests/site.pp
warning: Implicit invocation of 'puppet apply' by passing files (or flags) directly
to 'puppet' is deprecated, and will be removed in the 2.8 series. Please
invoke 'puppet apply' directly in the future.
info: Loading facts in bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_dirs.rb
info: Loading facts in bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb
info: Loading facts in bigtop-deploy/puppet/modules/kerberos/lib/facter/kadm_keytab.rb
debug: importing '/home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp' in environment production
debug: importing '/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/bigtop_util/manifests/init.pp' in environment production
debug: Automatically imported bigtop_util from bigtop_util into production
notice: Scope(Class[main]): puppet_confdir:
debug: Failed to load library 'selinux' for feature 'selinux'
debug: Puppet::Type::Package::ProviderRpm: Executing '/usr/bin/rpm --version'
debug: Puppet::Type::Package::ProviderUrpmi: Executing '/usr/bin/rpm -ql rpm'
debug: Puppet::Type::Package::ProviderYum: Executing '/usr/bin/rpm --version'
debug: Puppet::Type::Package::ProviderAptrpm: Executing '/usr/bin/rpm -ql rpm'
warning: Dynamic lookup of $hadoop_head_node at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp:19 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_head_node at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp:27 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_head_node at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp:35 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_head_node at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp:39 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_head_node at /home/ubuntu/bigtop/bigtop-deploy/puppet/manifests/cluster.pp:41 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
debug: importing '/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp' in environment production
debug: Automatically imported hadoop::datanode from hadoop/datanode into production
debug: importing '/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp' in environment production
debug: Automatically imported hadoop-hbase::server from hadoop-hbase/server into production
warning: Dynamic lookup of $auth at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:35 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
debug: Scope(Class[Hadoop::Common]): Retrieving template hadoop/core-site.xml
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml]: Bound template variables for /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml in 0.00 seconds
warning: Dynamic lookup of $hadoop_namenode_host at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:24 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_namenode_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:24 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_security_authentication at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:29 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_security_authentication at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:32 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_core_proxyusers at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:46 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_core_proxyusers at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml:47 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml]: Interpolated template /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml in 0.01 seconds
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Scope(Class[Hadoop::Common]): Retrieving template hadoop/mapred-site.xml
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml]: Bound template variables for /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml in 0.00 seconds
warning: Dynamic lookup of $hadoop_security_authentication at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:21 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_jobtracker_host at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:99 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_jobtracker_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:99 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $mapred_data_dirs at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:104 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $processorcount at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:217 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $processorcount at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:224 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_jobtracker_thrift_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:269 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_jobtracker_thrift_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:273 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_mapred_jobtracker_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:276 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_mapred_jobtracker_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:279 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_mapred_tasktracker_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:283 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_mapred_tasktracker_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml:286 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml]: Interpolated template /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml in 0.01 seconds
debug: Scope(Class[Hadoop::Common]): Retrieving template hadoop/hdfs-site.xml
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml]: Bound template variables for /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml in 0.00 seconds
warning: Dynamic lookup of $hadoop_security_authentication at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:21 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_namenode_host at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:106 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_namenode_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:106 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hdfs_data_dirs at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:111 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $namenode_data_dirs at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:116 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_dfs_namenode_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:120 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_dfs_namenode_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:123 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_dfs_datanode_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:128 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_dfs_datanode_plugins at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:131 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_namenode_thrift_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:136 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
warning: Dynamic lookup of $hadoop_namenode_thrift_port at /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml:139 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes.
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml]: Interpolated template /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml in 0.01 seconds
debug: Scope(Class[Hadoop::Common]): Retrieving template hadoop/hadoop-env.sh
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh]: Bound template variables for /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh in 0.00 seconds
debug: template[/home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh]: Interpolated template /home/ubuntu/bigtop/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh in 0.01 seconds