Tuesday, June 30, 2015

Creating puppetlab with vagrant and bash shell

#To create puppetlab environment, we need to first install virtualbox and vagrant.(Each virtualbox.org and vagrangup.com).

#Vagrantfile use bash(other shell?) shell to install update guest os as in Linux servers such as centos, debian and ubuntu.
#Here is my Vagrantfile.
# Wee need to first create vagrant box. We can obtain those boxes from "https://atlas.hashicorp.com/boxes/search". Or we can create box on our own.

I made boxes on my filesystem like belows.


whatsup@whatsup-To-be-filled-by-O-E-M ~/deploy_rails $ ls ../vg/*.box
../vg/apm.box  ../vg/centos70.box  ../vg/package.box  ../vg/ubuntu14.box

My humble article shortly explained.
http://wnapdlf.blogspot.kr/2014/12/vagrant-packaging.html

#Here is my Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.define "puppetmaster" do |puppetmaster|
   puppetmaster.vm.box = "puppetmaster"
   puppetmaster.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   puppetmaster.vm.provision "shell", inline: "echo now time to executing shell"
   puppetmaster.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   puppetmaster.vm.provision "shell", :path => "install-puppet-master.sh"
   puppetmaster.vm.network "private_network",ip:"10.1.0.2"
        #virtualbox__nat: false
   puppetmaster.vm.host_name = "puppetmaster"
      puppetmaster.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id,"--memory","1024"]
      end
   end


  config.vm.define "puppetclient1" do |puppetclient1|
   puppetclient1.vm.box = "puppetclient1"
   puppetclient1.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   puppetclient1.vm.provision "shell", :path => "install-puppet-node.sh"
   puppetclient1.vm.provision "shell", inline: "echo now time to executing shell"
   puppetclient1.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   puppetclient1.vm.provision "shell", inline: "apt-get update && apt-get upgrade"
   puppetclient1.vm.network "private_network",ip:"10.1.0.3"
   puppetclient1.vm.host_name = "puppetclient1"
      puppetclient1.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id,"--memory","1024"]
      end
   end

  config.vm.define "puppetclient2" do |puppetclient2|
   puppetclient2.vm.box = "puppetclient2"
   puppetclient2.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   puppetclient2.vm.provision "shell", :path => "install-puppet-node.sh"
   puppetclient2.vm.provision "shell", inline: "echo now time to executing shell"
   puppetclient2.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   puppetclient2.vm.provision "shell", inline: "apt-get update && apt-get upgrade"
   puppetclient2.vm.network "private_network",ip:"10.1.0.4"
   puppetclient2.vm.host_name = "puppetclient2"
      puppetclient2.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id,"--memory","256"]
        vb.cpus = 1
      end
   end



  config.vm.define "puppetclient3" do |puppetclient3|
   puppetclient3.vm.box = "puppetclient3"
   puppetclient3.vm.box_url = "file:///home/whatsup/test/package.box"
   puppetclient3.vm.provision "shell", :path => "install-puppet-node.sh"
   puppetclient3.vm.provision "shell", inline: "echo now time to executing shell"
   puppetclient3.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   puppetclient3.vm.provision "shell", inline: "apt-get update && apt-get upgrade"
   puppetclient3.vm.network "private_network",ip:"10.1.0.5"
   puppetclient3.vm.host_name = "puppetclient3"
      puppetclient3.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id,"--memory","256"]
        vb.cpus = 1
      end
   end

end



  config.vm.define "centos7-puppet-client" do |ct7|
   ct7.vm.box = "ct7"
   ct7.vm.box_url = "file:///home/whatsup/vg/centos7-puppet-client0.box"
   ct7.vm.provision "shell", :path => "install-puppet-centos7.sh"
   ct7.vm.provision "shell", inline: "echo now time to executing shell"
   ct7.vm.provision "shell", inline: "echo timezone config; timedatectl set-timezone Asia/Seoul"
   ct7.vm.network "private_network", ip:"10.1.0.6"
   ct7.vm.host_name = "centos7-puppet-client"
     ct7.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "256"]
     end
  end
 

#And shell script that are inserted into Vagrantfile are as follows.
#!/usr/bin/env bash

remove(){
  #remove previous packages
  apt-get remove --purge -y puppet*
}

check(){
   RV=$?
   if  [[ $RV != "0" ]]
   then
       echo -e "\e[31m $JOB failed please check"
       exit $RV
   else
       echo -e "\e[34m $JOB successful"
   fi

}

 

check_remove(){
   /usr/bin/which puppet
   sudo rm -rf /var/log/puppet
   JOB="puppet remove check on ubuntu14.04 trusty"
   check
}




install(){
   #install trusty puppet-package
   wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
   sudo dpkg -i puppetlabs-release-trusty.deb
   sudo apt-get update -y
   JOB="INSTALLING PUPPET REPO UBUNTU TRUSTY14.04"
   check

}

install_master(){
   sudo apt-get install puppetmaster-passenger -y
   JOB="INSTALLING PUPPET MASTER"
   check
}

#puppet package install for puppet master
remove
check_remove
install
install_master
whatsup@whatsup-To-be-filled-by-O-E-M ~/deploy_rails $ cat install-puppet-node.sh
#!/usr/bin/env bash

remove(){
  #remove previous packages
  sudo apt-get remove --purge -y puppet*
}

check(){
   RV=$?
   if  [[ $RV != "0" ]]
   then
       echo -e "\e[31m $JOB failed please check"
       exit $RV
   else
       echo -e "\e[34m $JOB successful"
   fi
}
 

check_remove(){
   /usr/bin/which puppet
   sudo rm -rf /var/log/puppet
   JOB="puppet remove check on ubuntu14.04 trusty"
   check
}


install(){
   #install trusty puppet-package
   wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
   sudo dpkg -i puppetlabs-release-trusty.deb
   sudo apt-get update -y
   JOB="INSTALLING PUPPET REPO UBUNTU TRUSTY14.04"
   check
}

install_node(){
   sudo apt-get install puppet -y
   JOB="INSTALLING PUPPET NODE"
   check
}

#puppet package install for puppet master
remove
check_remove
install
install_node

whatsup@whatsup-To-be-filled-by-O-E-M ~/deploy_rails $ cat install-puppet-centos7.sh
#!/usr/bin/env bash

remove(){
  #remove previous packages
  sudo yum remove  -y puppet*
}

check(){
   RV=$?
   if  [[ $RV != "0" ]]
   then
       echo -e "\e[31m $JOB failed please check"
       exit $RV
   else
       echo -e "\e[34m $JOB successful"
   fi
}
 

check_remove(){
   /usr/bin/which puppet
   sudo rm -rf /var/log/puppet
   JOB="puppet remove check on centos7 "
   check
}


install_repo(){
   #install centos7 puppet-package
    sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm

   sudo yum update -y
   JOB="INSTALLING PUPPET REPO centos7"
   check
}

install_puppet(){
   #install puppet with yum
   sudo yum install puppet
   JOB="INSTALLING PUPPET REPO centos7"
   check
}

###If everything prepared, vagrant up command will make up basic puppetlab###


### I wrote master and node setup on my blog###
http://wnapdlf.blogspot.kr/2015/06/puppet-master-and-node-configuration.html




Puppet master and node configuration

1. This very small lab has three servers. First is master, second is client1(puppetclient1),node2(puppetclient2).
To resolve dns, I put host information on each nodes including master.

First of all, master.

#Puppet node(client) search for first puppet.hostname.com and if that is not exist then search puppet.
#It is a must to set puppet hostname or alias to puppetmaster.
#So edit /etc/hosts file should be correct to operate puppet system correctly.

root@puppetmaster:~# vi /etc/hosts
#puppet nodes first search domain puppet if not resolved by
10.1.0.2  puppetmaster puppet
10.1.0.3  puppetclient1
10.1.0.4  puppetclient2

#Above configuration is saying that now puppet architecture is puppetmaster(puppet) and 2 nodes(client).

#Next we must tell dns_alt_names to puppet master.
root@puppetmaster:~# vi /etc/puppet/puppet.conf

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
dns_alt_names = puppet,puppetclient1,puppetclient2
environment_timeout=unlimited

#On production server, recommended configuration is as follows.
#From puppetlabs.com. For the purpose of reference.
[main]
certname = puppetmaster01.example.com
server = puppet
environment = production
runinterval = 1h
strict_variables = true
trusted

[master]
dns_alt_names = puppetmaster01,puppetmaster01.example.com,puppet,puppet.example.com
reports = puppetdb
storeconfigs_backend = puppetdb
storeconfigs = true
environment_timeout = unlimited


#puppet master receving request via 8140 port. There is also 443 port and 80 port is on.
#Later , when puppet is stablized, we better check these open port is necessary or not.

root@puppetmaster:~# netstat -tpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1697/sshd      
tcp6       0      0 :::22                   :::*                    LISTEN      1697/sshd      
tcp6       0      0 :::443                  :::*                    LISTEN      4569/apache2   
tcp6       0      0 :::8140                 :::*                    LISTEN      4569/apache2   
tcp6       0      0 :::80                   :::*                    LISTEN      4569/apache2   

#I install puppetmaster as apache passenger mode. So, starting apache2 server is a must.
root@puppetmaster:~# service apache2 stop
 * Stopping web server apache2                                                                                                                                *


###Creating puppet master certificate and MASTER CA certificate###

vagrant@puppetmaster:~$ sudo puppet master --verbose --no-daemonize
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Notice: Starting Puppet master version 3.8.1

^CNotice: Caught INT; storing stop
Notice: Processing stop



###Network check from puppetclient1 ####
root@puppetclient1:~# telnet puppet 8140
Trying 10.1.0.2...
Connected to 10.1.0.2.
Escape character is '^]'.

vagrant@puppetmaster:~$ netstat -tpln
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -              
tcp6       0      0 :::22                   :::*                    LISTEN      -              
tcp6       0      0 :::443                  :::*                    LISTEN      -              
tcp6       0      0 :::8140                 :::*                    LISTEN      -              
tcp6       0      0 :::80                   :::*                    LISTEN      -       


vagrant@puppetclient1:~$ cat /etc/puppet/puppet.conf
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY


### puppet client setting ###

vagrant@puppetclient1:~$ sudo vi /etc/hosts

127.0.0.1       localhost
127.0.1.1 puppetclient1 puppetclient1
10.1.0.2 puppetmaster puppet
10.1.0.3 puppetclient1


### puppet version check ###
vagrant@puppetclient1:~$ puppet --version
3.8.1
vagrant@puppetclient1:~$



# puppetclient1 node new key generating
root@puppetclient1:~# puppet agent --test
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for puppetclient1.bla
Info: Certificate Request fingerprint (SHA256): C4:A0:CE:12:D4:4F:42:3C:44:DA:4A:0A:6C:82:DC:3A:2E:B1:7F:1D:CC:61:3B:6B:10:15:77:33:EE:3B:21:8F
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled

#Info: Creating a new SSL certificate request for puppetclient1.bla domain name is not the intended domanin name.
#I had to remove bla from /etc/resolv.conf.


#To restart puppet agent --test, remove or backup /var/lib/puppet/ssl.
root@puppetclient1:~# rm -rf /var/lib/puppet/ssl/
root@puppetclient1:~# puppet agent --test
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Creating a new SSL key for puppetclient1.tbroad
Info: Caching certificate for ca
Info: Caching certificate_request for puppetclient1.tbroad
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled
root@puppetclient1:~# vi /etc/hosts
root@puppetclient1:~# ls /var/lib/puppet/ssl/
certificate_requests  certs  private  private_keys  public_keys

#It is ok...
#From master(puppetmaster) node


root@puppetmaster:~# puppet cert --list
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
  "puppetclient1" (SHA256) C4:A0:CE:12:D4:4F:42:3C:44:DA:4A:0A:6C:82:DC:3A:2E:B1:7F:1D:CC:61:3B:6B:10:15:77:33:EE:3B:21:8F

#Let's suppress warning
root@puppetmaster:~#

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
#templatedir=$confdir/templates

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
dns_alt_names = puppet,puppetclient1,puppetclient2
environment_timeout=unlimited

#Cleaning ssl request from master.
root@puppetmaster:~# rm -rf /var/lib/puppet/ssl/
root@puppetmaster:~# puppet cert --list
Notice: Signed certificate request for ca


#Puppet master is using passenger in apache virtual hosts file.
#When there is error message, we must check like belows if certificate file name right or not.

 root@puppetmaster:/etc/init.d# vi /etc/apache2/sites-available/puppetmaster.conf
root@puppetmaster:/etc/init.d# service apache2 restart
 * Restarting web server apache2                                                                                                                      [fail]
 * The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 23 of /etc/apache2/sites-enabled/puppetmaster.conf:
SSLCertificateFile: file '/var/lib/puppet/ssl/certs/puppetmaster' does not exist or is empty
Action 'configtest' failed.
The Apache error log may have more information.
root@puppetmaster:/etc/init.d#



root@puppetmaster:/etc/init.d# service apache2 restart
 * Restarting web server apache2                                                                                                                      [ OK ]
root@puppetmaster:/etc/init.d# suppressing  * Restarting web server apache2                                                                                                                             AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
>
> ^C
root@puppetmaster:/etc/init.d#


# This Apache 2 virtual host config shows how to use Puppet as a Rack
# application via Passenger. See
# http://docs.puppetlabs.com/guides/passenger.html for more information.

# You can also use the included config.ru file to run Puppet with other Rack
# servers instead of Passenger.

# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
ServerName puppetmaster


#Now there is no certificate request from nodes(clients?)
root@puppetmaster:/etc/init.d# puppet cert --list
root@puppetmaster:/etc/init.d#


#Again, generting agent certificate request
root@puppetclient1:~# puppet agent --test
Info: Creating a new SSL key for puppetclient1
Info: Caching certificate for ca
Info: Caching certificate_request for puppetclient1
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled
root@puppetclient1:~#


root@puppetmaster:/etc/init.d# puppet cert --list
  "puppetclient1" (SHA256) 12:3C:60:26:53:B0:7A:76:48:F8:97:31:28:36:3A:64:36:72:E7:B7:3E:6B:ED:65:0F:56:15:15:FE:3D:A8:AD
root@puppetmaster:/etc/init.d#



#Sign up nodes(puppetclient1) request.

root@puppetmaster:/etc/init.d# puppet cert sign puppetclient1
Notice: Signed certificate request for puppetclient1
Notice: Removing file Puppet::SSL::CertificateRequest puppetclient1 at '/var/lib/puppet/ssl/ca/requests/puppetclient1.pem'
root@puppetmaster:/etc/init.d#

#No error..relax.


***When regenerating puppetmaster certificate***
#There shall be a situation when we need to reset whole puppet configuration.#
#Need to be prepared.

#ssl is exist in default location: /var/lib/puppet/ssl.
root@puppetmaster:/etc/apache2# ls /var/lib/puppet/ssl/
ca  certificate_requests  certs  crl.pem  private  private_keys  public_keys

#If remove or rename apache2 passenger startup will be failed.
root@puppetmaster:/etc/apache2# mv /var/lib/puppet/ssl/ /var/lib/puppet/ssl.bak/
root@puppetmaster:/etc/apache2# service apache2 restart
 * Restarting web server apache2                                                                                                                      [fail]
 * The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 23 of /etc/apache2/sites-enabled/puppetmaster.conf:
SSLCertificateFile: file '/var/lib/puppet/ssl/certs/puppetmaster.pem' does not exist or is empty
Action 'configtest' failed.
The Apache error log may have more information.

#It is time to generate puppetmaster certificate file.
root@puppetmaster:/etc/apache2# puppet cert generate puppetmaster
Notice: Signed certificate request for ca
Notice: puppetmaster has a waiting certificate request
Notice: Signed certificate request for puppetmaster
Notice: Removing file Puppet::SSL::CertificateRequest puppetmaster at '/var/lib/puppet/ssl/ca/requests/puppetmaster.pem'
Notice: Removing file Puppet::SSL::CertificateRequest puppetmaster at '/var/lib/puppet/ssl/certificate_requests/puppetmaster.pem'
root@puppetmaster:/etc/apache2# service apache2 restart
 * Restarting web server apache2                                                                                                                      [ OK ]
root@puppetmaster:/etc/apache2#

###We must regnerate each nodes request keys?###
###Now I have only one node(puppetclient), but what if there are hundreds of servers?###
###http://docs.puppetlabs.com/puppet/4.1/reference/ssl_regenerate_certificates.html is saying we should do it^^;###


Tuesday, June 9, 2015

Vim special character input(₩,¥,€ )

When we want to input currency characters, such as dollor=$.
There is no symbol such as Korean unit ₩.

1. Input(i)
2. <ctrl>+k , then ? mark will be on screen.
3. When we want to input Korean currency Won(₩).
4. Type upper case W and =, then ₩ will be on.
5 In case of Japanese currency unit symbol, type upper case Y and -, then ¥ will be on screen. Chinese symbol is same but pronunced differently(Yen-Japanese, Yuan-Chinese)
6.Euro sign would be <ctrl>+k+=+e  €


 :digraph command will show a lot.

Below link will help more.
http://vim.wikia.com/wiki/Entering_special_characters