Pages

Saturday, November 21, 2015

Central Authentication Server

Prerequisties :

Java (JDK) is installed

Apache Tomcat is installed and running with SSL enabled.

ApacheDS is installed

Download CAS server in /tmp folder and unzip.

$ wget -O cas-server-4.0.0.zip http://github.com/Jasig/cas/releases/download/v4.0.0/cas-server-4.0.0-release.zip

Stop the Tomcat server.
$ sudo /opt/apache-tomcat/bin/shutdown.sh

Copy CAS war file to $TOMCAT_HOME/webapps

$ sudo cp /tmp/cas-server-4.0.0/modules/cas-server-webapp-4.0.0.war /opt/apache-tomcat/webapps

Start the Tomcat Server.
$ sudo /opt/apache-tomcat/bin/startup.sh

It will extract the war file  into cas-server-webapp-4.0.0 directory.

Stop the Tomcat server.
$ sudo /opt/apache-tomcat/bin/shutdown.sh

Add cas-server-support-ldap and ldaptive dependencies to /opt/apache-tomcat/webapps/cas-server-webapp-4.0.0/META-INF/maven/org.jasig.cas/cas-server-webapp/pom.xml

      <dependency>
            <groupId>org.jasig.cas</groupId>
            <artifactId>cas-server-support-ldap</artifactId>
            <version>${cas.version}</version>
       </dependency>

       <dependency>
             <groupId>org.ldaptive</groupId>
             <artifactId>ldaptive</artifactId>
             <version>1.1.0</version>
       </dependency>

Copy cas-server-support-ldap-4.0.0.jar and ldaptive-1.1.0.jar to $TOMCAT_HOME\webapps\cas-server-webapp-4.0.0\WEB-INF\lib

$ sudo cp /tmp/cas-server-4.0.0/modules/cas-server-support-ldap-4.0.0.jar /opt/apache-tomcat/webapps/cas-server-webapp-4.0.0/WEB-INF/lib

$ wget http://central.maven.org/maven2/org/ldaptive/ldaptive/1.1.0/ldaptive-1.1.0.jar
$ sudo cp /tmp/ldaptive-1.1.0.jar /opt/apache-tomcat/webapps/cas-server-webapp-4.0.0/WEB-INF/lib

Modify $TOMCAT_HOME\webapps\cas-server-webapp-4.0.0\WEB-INF\deployerConfigContext.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!--

    Licensed to Jasig under one or more contributor license
    agreements. See the NOTICE file distributed with this work
    for additional information regarding copyright ownership.
    Jasig licenses this file to you under the Apache License,
    Version 2.0 (the "License"); you may not use this file
    except in compliance with the License.  You may obtain a
    copy of the License at the following location:

      http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

-->
<!--
| deployerConfigContext.xml centralizes into one file some of the declarative configuration that
| all CAS deployers will need to modify.
|
| This file declares some of the Spring-managed JavaBeans that make up a CAS deployment.
| The beans declared in this file are instantiated at context initialization time by the Spring
| ContextLoaderListener declared in web.xml.  It finds this file because this
| file is among those declared in the context parameter "contextConfigLocation".
|
| By far the most common change you will need to make in this file is to change the last bean
| declaration to replace the default authentication handler with
| one implementing your approach for authenticating usernames and passwords.
+-->

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:c="http://www.springframework.org/schema/c"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:sec="http://www.springframework.org/schema/security"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
       http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.2.xsd
       http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.2.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
       http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">

    <!--
       | The authentication manager defines security policy for authentication by specifying at a minimum
       | the authentication handlers that will be used to authenticate credential. While the AuthenticationManager
       | interface supports plugging in another implementation, the default PolicyBasedAuthenticationManager should
       | be sufficient in most cases.
       +-->
    <bean id="authenticationManager" class="org.jasig.cas.authentication.PolicyBasedAuthenticationManager">
        <constructor-arg>
            <map>
                <!--
                   | IMPORTANT
                   | Every handler requires a unique name.
                   | If more than one instance of the same handler class is configured, you must explicitly
                   | set its name to something other than its default name (typically the simple class name).
                   -->
               <!--
                <entry key-ref="proxyAuthenticationHandler" value-ref="proxyPrincipalResolver" />
                <entry key-ref="primaryAuthenticationHandler" value-ref="primaryPrincipalResolver" />
               -->
                <entry key-ref="ldapAuthenticationHandler" value-ref="primaryPrincipalResolver" />
            </map>
        </constructor-arg>

        <!-- Uncomment the metadata populator to allow clearpass to capture and cache the password
             This switch effectively will turn on clearpass.
        <property name="authenticationMetaDataPopulators">
           <util:list>
              <bean class="org.jasig.cas.extension.clearpass.CacheCredentialsMetaDataPopulator"
                    c:credentialCache-ref="encryptedMap" />
           </util:list>
        </property>
        -->

        <!--
           | Defines the security policy around authentication. Some alternative policies that ship with CAS:
           |
           | * NotPreventedAuthenticationPolicy - all credential must either pass or fail authentication
           | * AllAuthenticationPolicy - all presented credential must be authenticated successfully
           | * RequiredHandlerAuthenticationPolicy - specifies a handler that must authenticate its credential to pass
           -->
        <property name="authenticationPolicy">
            <bean class="org.jasig.cas.authentication.AnyAuthenticationPolicy" />
        </property>
    </bean>

    <!-- Required for proxy ticket mechanism. -->
    <bean id="proxyAuthenticationHandler"
          class="org.jasig.cas.authentication.handler.support.HttpBasedServiceCredentialsAuthenticationHandler"
          p:httpClient-ref="httpClient" />

    <!--
       | TODO: Replace this component with one suitable for your enviroment.
       |
       | This component provides authentication for the kind of credential used in your environment. In most cases
       | credential is a username/password pair that lives in a system of record like an LDAP directory.
       | The most common authentication handler beans:
       |
       | * org.jasig.cas.authentication.LdapAuthenticationHandler
       | * org.jasig.cas.adaptors.jdbc.QueryDatabaseAuthenticationHandler
       | * org.jasig.cas.adaptors.x509.authentication.handler.support.X509CredentialsAuthenticationHandler
       | * org.jasig.cas.support.spnego.authentication.handler.support.JCIFSSpnegoAuthenticationHandler
       -->
    <!--
    <bean id="primaryAuthenticationHandler"
          class="org.jasig.cas.authentication.AcceptUsersAuthenticationHandler">
        <property name="users">
            <map>
                <entry key="casuser" value="Mellon"/>
            </map>
        </property>
    </bean>
    -->
    <bean id="ldapAuthenticationHandler"
      class="org.jasig.cas.authentication.LdapAuthenticationHandler"
      p:principalIdAttribute="uid"
      c:authenticator-ref="authenticator">
    <property name="principalAttributeMap">
        <map>
            <!--
               | This map provides a simple attribute resolution mechanism.
               | Keys are LDAP attribute names, values are CAS attribute names.
               | Use this facility instead of a PrincipalResolver if LDAP is
               | the only attribute source.
               -->
            <entry key="member" value="member" />
            <entry key="mail" value="mail" />
            <entry key="cn" value="cn" />
        </map>
    </property>
    </bean>

    <!-- Required for proxy ticket mechanism -->
    <bean id="proxyPrincipalResolver"
          class="org.jasig.cas.authentication.principal.BasicPrincipalResolver" />

    <!--
       | Resolves a principal from a credential using an attribute repository that is configured to resolve
       | against a deployer-specific store (e.g. LDAP).
       -->
    <bean id="primaryPrincipalResolver"
          class="org.jasig.cas.authentication.principal.PersonDirectoryPrincipalResolver" >
        <property name="attributeRepository" ref="attributeRepository" />
    </bean>

    <!--
    Bean that defines the attributes that a service may return.  This example uses the Stub/Mock version.  A real implementation
    may go against a database or LDAP server.  The id should remain "attributeRepository" though.
    +-->
    <bean id="attributeRepository" class="org.jasig.services.persondir.support.StubPersonAttributeDao"
            p:backingMap-ref="attrRepoBackingMap" />

    <util:map id="attrRepoBackingMap">
        <entry key="uid" value="uid" />
        <entry key="eduPersonAffiliation" value="eduPersonAffiliation" />
        <entry key="groupMembership" value="groupMembership" />
    </util:map>

    <!--
    Sample, in-memory data store for the ServiceRegistry. A real implementation
    would probably want to replace this with the JPA-backed ServiceRegistry DAO
    The name of this bean should remain "serviceRegistryDao".
    +-->
    <bean id="serviceRegistryDao" class="org.jasig.cas.services.InMemoryServiceRegistryDaoImpl"
            p:registeredServices-ref="registeredServicesList" />

    <util:list id="registeredServicesList">
        <bean class="org.jasig.cas.services.RegexRegisteredService"
              p:id="0" p:name="HTTP and IMAP" p:description="Allows HTTP(S) and IMAP(S) protocols"
              p:serviceId="^(https?|imaps?)://.*" p:evaluationOrder="10000001" />
        <!--
        Use the following definition instead of the above to further restrict access
        to services within your domain (including sub domains).
        Note that example.com must be replaced with the domain you wish to permit.
        This example also demonstrates the configuration of an attribute filter
        that only allows for attributes whose length is 3.
        -->
        <!--
        <bean class="org.jasig.cas.services.RegexRegisteredService">
            <property name="id" value="1" />
            <property name="name" value="HTTP and IMAP on example.com" />
            <property name="description" value="Allows HTTP(S) and IMAP(S) protocols on example.com" />
            <property name="serviceId" value="^(https?|imaps?)://([A-Za-z0-9_-]+\.)*example\.com/.*" />
            <property name="evaluationOrder" value="0" />
            <property name="attributeFilter">
              <bean class="org.jasig.cas.services.support.RegisteredServiceRegexAttributeFilter" c:regex="^\w{3}$" />
            </property>
        </bean>
        -->
    </util:list>

    <bean id="auditTrailManager" class="com.github.inspektr.audit.support.Slf4jLoggingAuditTrailManager" />

    <bean id="healthCheckMonitor" class="org.jasig.cas.monitor.HealthCheckMonitor" p:monitors-ref="monitorsList" />

    <util:list id="monitorsList">
      <bean class="org.jasig.cas.monitor.MemoryMonitor" p:freeMemoryWarnThreshold="10" />
      <!--
        NOTE
        The following ticket registries support SessionMonitor:
          * DefaultTicketRegistry
          * JpaTicketRegistry
        Remove this monitor if you use an unsupported registry.
      -->
      <bean class="org.jasig.cas.monitor.SessionMonitor"
          p:ticketRegistry-ref="ticketRegistry"
          p:serviceTicketCountWarnThreshold="5000"
          p:sessionCountWarnThreshold="100000" />
    </util:list>

<bean id="authenticator" class="org.ldaptive.auth.Authenticator"
      c:resolver-ref="dnResolver"
      c:handler-ref="authHandler" />

<!--
   | The following DN format works for many directories, but may need to be
   | customized.
   -->
<bean id="dnResolver"
      class="org.ldaptive.auth.FormatDnResolver"
      c:format="uid=%s,${ldap.authn.baseDn}" />

<bean id="authHandler" class="org.ldaptive.auth.PooledBindAuthenticationHandler"
      p:connectionFactory-ref="pooledLdapConnectionFactory" />

<bean id="pooledLdapConnectionFactory"
      class="org.ldaptive.pool.PooledConnectionFactory"
      p:connectionPool-ref="connectionPool" />

<bean id="connectionPool"
      class="org.ldaptive.pool.BlockingConnectionPool"
      init-method="initialize"
      p:poolConfig-ref="ldapPoolConfig"
      p:blockWaitTime="${ldap.pool.blockWaitTime}"
      p:validator-ref="searchValidator"
      p:pruneStrategy-ref="pruneStrategy"
      p:connectionFactory-ref="connectionFactory" />

<bean id="ldapPoolConfig" class="org.ldaptive.pool.PoolConfig"
      p:minPoolSize="${ldap.pool.minSize}"
      p:maxPoolSize="${ldap.pool.maxSize}"
      p:validateOnCheckOut="${ldap.pool.validateOnCheckout}"
      p:validatePeriodically="${ldap.pool.validatePeriodically}"
      p:validatePeriod="${ldap.pool.validatePeriod}" />

<bean id="connectionFactory" class="org.ldaptive.DefaultConnectionFactory"
      p:connectionConfig-ref="connectionConfig" />

<bean id="connectionConfig" class="org.ldaptive.ConnectionConfig"
      p:ldapUrl="${ldap.url}"
      p:connectTimeout="${ldap.connectTimeout}"
      p:useStartTLS="${ldap.useStartTLS}"
      p:sslConfig-ref="sslConfig" />

<bean id="sslConfig" class="org.ldaptive.ssl.SslConfig">
    <property name="credentialConfig">
        <bean class="org.ldaptive.ssl.X509CredentialConfig"
              p:trustCertificates="${ldap.trustedCert}" />
    </property>
</bean>

<bean id="pruneStrategy" class="org.ldaptive.pool.IdlePruneStrategy"
      p:prunePeriod="${ldap.pool.prunePeriod}"
      p:idleTime="${ldap.pool.idleTime}" />

<bean id="searchValidator" class="org.ldaptive.pool.SearchValidator" />
</beans>

Add LDAP properties in /WEB-INF/ldap.properties as follows.

#========================================
# General properties
#========================================
ldap.url=ldap://localhost:10389

# LDAP connection timeout in milliseconds
ldap.connectTimeout=3000

# Whether to use StartTLS (probably needed if not SSL connection)
ldap.useStartTLS=false

#========================================
# LDAP connection pool configuration
#========================================
ldap.pool.minSize=3
ldap.pool.maxSize=10
ldap.pool.validateOnCheckout=false
ldap.pool.validatePeriodically=true

# Amount of time in milliseconds to block on pool exhausted condition
# before giving up.
ldap.pool.blockWaitTime=3000

# Frequency of connection validation in seconds
# Only applies if validatePeriodically=true
ldap.pool.validatePeriod=300

# Attempt to prune connections every N seconds
ldap.pool.prunePeriod=300

# Maximum amount of time an idle connection is allowed to be in
# pool before it is liable to be removed/destroyed
ldap.pool.idleTime=600

#========================================
# Authentication
#========================================

# Base DN of users to be authenticated
ldap.authn.baseDn=ou=users,ou=system

# Manager DN for authenticated searches
ldap.authn.managerDN=uid=admin,ou=system,dc=example,dc=com

# Manager password for authenticated searches
ldap.authn.managerPassword=secret

# Search filter used for configurations that require searching for DNs
#ldap.authn.searchFilter=(&(uid={user})(accountState=active))
ldap.authn.searchFilter=(uid={user})

# Search filter used for configurations that require searching for DNs
#ldap.authn.format=uid=%s,ou=users,dc=example,dc=com
ldap.authn.format=%s@example.com

# A path to trusted X.509 certificate for StartTLS
ldap.trustedCert=/path/to/cert.cer

Load into spring application context by modifying  /WEB-INF/spring_configuration/propertyFileConfigurer.xml
    
<context:property-placeholder location="/WEB-INF/cas.properties,/WEB-INF/ldap.properties"/>

Start tomcat and confirm there are no errors in the $TOMCAT_HOME\logs\catalina.out log.

Open a browser to the URL http://localhost:8443/cas-server-webapp-4.0.0 and authenticate with the LDAP credentials.

Saturday, September 19, 2015

ApacheDS

Download ApacheDS in /tmp folder and install with all the defaults.

$ cd /tmp
$ wget http://apache.cs.utah.edu//directory/apacheds/dist/2.0.0-M21/apacheds-2.0.0-M21-64bit.bin
$ chmod a+x apacheds-2.0.0-M21-64bit.bin
$ sudo ./apacheds-2.0.0-M21-64bit.bin

Do you agree to the above license terms? [yes or no]
yes
Unpacking the installer...
Extracting the installer...
Where do you want to install ApacheDS? [Default: /opt/apacheds-2.0.0-M21]

Where do you want to install ApacheDS instances? [Default: /var/lib/apacheds-2.0.0-M21]

What name do you want for the default instance? [Default: default]

Where do you want to install the startup script? [Default: /etc/init.d]

Which user do you want to run the server with (if not already existing, the specified user will be created)? [Default: apacheds]

Which group do you want to run the server with (if not already existing, the specified group will be created)? [Default: apacheds]

Installing...
id: apacheds: No such user
Done.
ApacheDS has been installed successfully.

By default the LDAP server listens on port 10389 (unencrypted or StartTLS) and 10636 (SSL).

user : uid=admin,ou=system
password : secret


$ sudo service apacheds-2.0.0-M21-default start
$ sudo chkconfig --add apacheds-2.0.0-M21-default

Saturday, April 11, 2015

Ansible Install and Playbooks

$ sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm

Enable ol7_optional_latest Repository for python-jinja2 package.

$ sudo yum install ansible

Edit /etc/ansible/hosts and put remote systems in it.

mail.example.com

[webservers]
web1.example.com
web2.example.com

[dbservers]
db1.example.com
db2.example.com

Ping all the nodes

$ ansible all -m ping

Run a live command on all of the nodes

$ ansible all -a "echo hello"

Playbook for Pre-requisites on all nodes :

$ cat roles/common/tasks/main.yml
---
- hosts: all
  user: vagrant
  sudo: yes

  tasks:

  - name: sudo access to Hadoop Admins

  - name: hostname must be fully qualified domain name
    ansible edge -a hostname

  - name: DNS lookup should resolve to correct IP Address
    command: nslookup {{ ansible_hostname }}
    register: ns

  - debug: var=ns.stdout

  - name: Reverse DNS lookup should be successful
    command: nslookup `host {{ ansible_hostname }} | awk '{print $4}'`

  - name: Verify sticky bit permission set on /tmp filesystem on all nodes
    ansible edge -a 'ls -ld /tmp'

  - name: Verify /etc/hosts file entries
    ansible edge -a 'cat /etc/hosts'

  - name: Verify Network configuration

  - name: Make sure the system is up to date
    yum: name=* state=latest

  - name: Ensure NTP (for time synchronization) is installed.
    yum: name=ntp state=present

  - name: Ensure NTP is running and enable it to run at boot.
    service: name=ntpd state=started enabled=yes

  - name: Install libselinux-python
    yum: name=libselinux-python state=present

  - name: Disable SELINUX
    selinux: state=disabled

  - name: Disable iptables
    service: name=iptables state=stopped enabled=no

  - name: Disable transparent huge pages on data nodes
    lineinfile: dest=/etc/rc.d/rc.local regexp='^.*transparent_hugepage.*$' line='echo never > /sys/kernel/mm/transparent_hugepage/defrag'

  - name: Execution permission
    file: path=/etc/rc.d/rc.local mode=u+x

  - name: Set swappiness to 1 on data nodes (Default is 60)
    sysctl: name=vm.swappiness value=1 state=present

  - name: Set overcommit_memory to 1 on data nodes (Default is 0)
    sysctl: name=vm.overcommit_memory value=1 state=present

Playbook for Data nodes:

---
- hosts: dn
  user: vagrant
  sudo: yes

  vars:
    fstype: ext4
    device: /dev/sdb
    mntdir: /data1

  tasks:

    - name: Check if partition present
      shell: fdisk -l | grep {{device}} | wc -l
      changed_when: False
      register: partition_present

    - name: fdisk/create partition/format/Reserved Block Count to 1%
      shell: "echo -e 'n\np\n1\n\n\nw\n' | fdisk {{ device }} && mkfs -t {{ fstype }} {{ device }}1 && tune2fs -m 1 {{ device }}1"
      when: partition_present.stdout is defined and partition_present.stdout|int == 1

    - name: Create Directory for mount
      file: path={{ mntdir }} state=directory

    - name: Mount device
      mount: name={{ mntdir }} src={{ device }}1 fstype={{ fstype }} opts="noatime,nodiratime" passno=2 state=mounted

# - name: Make sure data disks are configured with RAID10 on non-data nodes
# - name: Make sure data disks are configured as JBODs and with no RAID on data nodes
    command: pvs
    register: pvs

  - debug: var=pvs.stdout_lines

# - name: Make sure JBOD are ext4 type and mounted on /data{1..20} mount points with noatime on data nodes
    command: df -T | grep "/data"
    command: cat /etc/fstab | grep /data

# - name: Make sure data disks won't fsck based on time or number of reboots on data nodes
    command: tune2fs -l /dev/sdb1 | grep -i 'Maximum mount count|Check interval'

# - name: Make sure data disks Reserved block count is 4882414 or 1% (Default is 5%) on data nodes
    command: tune2fs -l /dev/sdb1 | grep -i 'Reserved block count'

# - name: filesystem sizes /var/log

Playbook for JDK Install:

$ cat roles/jdk/tasks/main.yml
---
- hosts: all
  user: vagrant
  sudo: yes

  vars:
    java_rpm_url: http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm
    java_archive: /tmp/jdk-7u79-linux-x64.rpm

  tasks:

    - name: Download Java Development Kit
      command: 'wget -q -O {{ java_archive }} --no-cookies --header "Cookie: gpw_e24=http://www.oracle.com/; oraclelicense=accept-securebackup-cookie" {{ java_rpm_url }} creates={{ java_archive }}'

    - name: Install JDK on all Servers
      yum: name={{ java_archive }} state=present

Playbook for MySQL Server and JDBC Driver:

$ cat roles/mysql/tasks/main.yml

---
- hosts: edge1
  user: vagrant
  sudo: yes

  vars:
    root_pw: rootpw
    jdbc_url: http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz
    jdbc_archive: /tmp/mysql-connector-java-5.1.35.tar.gz
    jdbc_driver: mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar

    mysql_jdbc_driver: /usr/share/java/mysql-connector-java-5.1.35-bin.jar
    link_jdbc_driver: /usr/share/java/mysql-connector-java.jar


  tasks:

    - name: Install MySQL repo
      yum: name=http://repo.mysql.com/mysql-community-release-el6-5.noarch.rpm state=present

    - name: Install MySQL
      yum: name={{ item }} enablerepo=mysql56-community state=present
      with_items:
      - MySQL-python
      - mysql-community-server

    - name: Start mysqld
      service: name=mysqld state=started enabled=yes
    - name: Set root password
      mysql_user: login_user=root login_password="" name=root host={{ item }} password={{ root_pw }}
      with_items:
      - 127.0.0.1
      - ::1
      - localhost

    - name: Remove anonymous users
      mysql_user: login_user=root login_password={{ root_pw }} name="" host={{ item }} state=absent
      with_items:
      - localhost
      - "{{ ansible_nodename }}"
      - "{{ ansible_hostname }}"

    - name: Disallow root login remotely
      mysql_user: login_user=root login_password={{ root_pw }} name=root host={{ ansible_nodename }} state=absent

    - name: Remove test database and access to it
      mysql_db: login_user=root login_password={{ root_pw }} name=test state=absent

    - name: Create database
      mysql_db: login_user=root login_password={{ root_pw }} name={{ item }} encoding='utf8'
      with_items:
      - amon
      - smon
      - rman
      - hmon
      - nav
      - hive
      - hue
      - oozie

    - name: Create users
      mysql_user: login_user=root login_password={{ root_pw }} name={{ item.name }} password={{ item.password }} priv={{item.priv }} state=present
      with_items:
      - { name: 'amon', password: 'amonpw', priv: 'amon.*:ALL' }
      - { name: 'smon', password: 'smonpw', priv: 'smon.*:ALL' }
      - { name: 'rman', password: 'rmanpw', priv: 'rman.*:ALL' }
      - { name: 'hmon', password: 'hmonpw', priv: 'hmon.*:ALL' }
      - { name: 'hive', password: 'hivepw', priv: 'hive.*:ALL' }
      - { name: 'hue', password: 'huepw', priv: 'hue.*:ALL' }
      - { name: 'oozie', password: 'ooziepw', priv: 'oozie.*:ALL' }

    - name: Download JDBC Driver for MySQL on Edge Servers
      command: 'wget -q -O {{ jdbc_archive }} {{ jdbc_url }} creates={{ jdbc_archive }}'

    - name: Extract JDBC Driver
      command: 'tar -zxf {{ jdbc_archive }} -C /tmp {{ jdbc_driver }} creates={{ jdbc_driver }}'

    - name: Create /usr/share/java folder
      file: path=/usr/share/java state=directory

    - name: Copy to /usr/share/java
      copy: src=/tmp/{{ jdbc_driver }} dest=/usr/share/java

    - name: Create symbolic link
      file: src={{ mysql_jdbc_driver }} dest={{ link_jdbc_driver }} state=link

MySQL root password:

---
- name: MySQL root password
  user: vagrant
  sudo: yes
  hosts: cm

  tasks:
     - name: Create mysql root pass
       command: /usr/bin/openssl rand -base64 16
       register: mysql_root_passwd

     - debug: var=mysql_root_passwd

Playbook for Cloudera Repos :
---
- hosts: all
  user: vagrant
  sudo: yes

  vars:
    cloudera_repos:
      - http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo
      - http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo
      - http://archive-primary.cloudera.com/impala/redhat/6/x86_64/impala/cloudera-impala.repo

  tasks:

    - name: Install Cloudera repos
      get_url: url={{ item }} dest=/etc/yum.repos.d/ mode=0644
      with_items: cloudera_repos

Playbook for Cloudera Manger:

---
- hosts: edge1
  user: vagrant
  sudo: yes

  vars:
    mysql_root_pw: rootpw
    scm_pw: scmpw

  tasks:

    - name: Install Cloudera Manager Server
      yum: name=cloudera-manager-server state=latest

    - name: Prepare scm database
      command: '/usr/share/cmf/schema/scm_prepare_database.sh -h localhost -u root -p{{mysql_root_pw }} --scm-host localhost mysql scm scm {{ scm_pw }}'

    - name: Start Cloudera Manager Server
      service: name=cloudera-scm-server state=started enabled=yes

# Login Cloudera Manager Admin Console at http://edge1.example.com:7180/
# The default credentials are Username: admin Password: admin


Host variables:
---
- name: test playbook
  user: vagrant
  sudo: yes
  hosts: cm

  tasks:
#    - name: My message
#      debug: msg="{{ ansible_nodename }}"
#    - debug: var=result
    - name: Display hostname
      debug: var=hostvars[inventory_hostname]






Saturday, February 14, 2015

mongoDB Sharded Cluster



MongoDB production cluster must have three Config Servers, one or more Query Routers, and two or more Shards. Shards are either replica sets or a standalone mongod instances. The mongos instances are routers for the cluster, typically one mongos instance on each application server.

Config Servers are used to store metadata that links requested data with the shard. The application connects to Query Routers which communicates to Config Servers to determine the data location and returns the data from the appropriate shards. Shards are responsible for the actual data storage operations.

3 Config Servers (mongod metadata) : cfg0.example.com, cfg1.example.com, cfg2.example.com
2 Query Routers (mongos) : mongos0.example.com, mongos1.example.com
Shard 1 (mongod) : mongodb0.example.com, mongodb1.example.com, mongodb2.exmple.com
Shard 2 (mongod) : mongodb3.example.com, mongodb4.example.com, mongodb5.exmple.com

Disable SELINUX

# vi /etc/selinux/config
SELINUX=disabled

Add the following lines to /etc/security/limits.conf to increase limit.
mongod        soft    nproc           64000
mongod        hard    nproc           64000
mongod        soft    nofile          64000
mongod        hard    nofile          64000

Disable the usage of transparent hugepages

Add the following script lines to /etc/rc.local to improve performance.
if test -f /sys/kernel/mm/transparent_hugepage/khugepaged/defrag; then
  echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

# chmod +x /etc/rc.local

Enable NTP on all cluster machines.
# yum install ntp
# service ntpd start
# chkconfig ntpd on

Configure mongoDB repository.

$ sudo vi /etc/yum.repos.d/mongodb-enterprise.repo

[mongodb-enterprise]
name=MongoDB Enterprise Repository
baseurl=https://repo.mongodb.com/yum/redhat/$releasever/mongodb-enterprise/stable/$basearch/
gpgcheck=0
enabled=1

Disk Configuration

# fdisk /dev/sdb

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1044, default 1044):
Using default value 1044
Command (m for help): w

# mkfs.ext4 /dev/sdb1
# mkdir /data      
Add the following line to /etc/fstab
/dev/sdb1 /data ext4 noatime,nodiratime 0 2
# mount /dev/sdb1

Configure Config Servers

On each of the config servers,
Create a data directory to store metadata.
$ sudo mkdir /data/configdb      --> all config servers for journal
$ sudo chown mongod:mongod /data/configdb

Install mongoDB server.
$ sudo yum --disablerepo=* --enablerepo=mongodb-org-3.0 install -y mongodb-enterprise-server

Modify parameters in /etc/mongod.conf
configsvr=true
dbpath=/data/configdb
Comment out to listen on all interfaces.
# bind_ip=127.0.0.1


$ sudo firewall-cmd --zone=public --add-port=27019/tcp
$ sudo firewall-cmd --permanent --add-port=27019/tcp


Start mongod instance on each of the three config servers.
$ sudo service mongod start


Verify Mongod proccess by checking the contents of the log file at /var/log/mongodb/mongod.log.
[initandlisten] waiting for connections on port <port>
Where <port>  is 27019 by default.


Configure Query Routers

Install mongos, mongo shell and mongo tools on each of query routers.
$ sudo yum --disablerepo=* --enablerepo=mongodb-org-3.0 install -y mongodb-enterprise-mongos mongodb-enterprise-shell mongodb-enterprise-tools


$ sudo firewall-cmd --zone=public --add-port=27017/tcp
$ sudo firewall-cmd --permanent --add-port=27017/tcp


Start the mongos instance specifying the config servers on both Query Routers. The mongos runs on the default port 27017.

$ sudo mongos --configdb cfg0.example.com:27019,cfg1.example.com:27019,cfg2.example.com:27019

Query Routers should begin to communicate to the three configuration servers.

Create the Replica Sets

Create a data directory on each of the three shard Servers.
$ sudo yum --disablerepo=* --enablerepo=mongodb-org-3.0 install -y mongodb-enterprise-server

$ sudo mkdir /data/db
$ chown mongod:mongod /data/db

Modify parameters in /etc/mongod.conf
shardsvr=true
dbpath=/data/configdb
Comment out to listen on all interfaces.
# bind_ip=127.0.0.1
replSet=rs0

$ sudo firewall-cmd --zone=public --add-port=27018/tcp
$ sudo firewall-cmd --permanent --add-port=27018/tcp

Start mongod instance on each of the three shard servers that will deploy each member of the replica set to its own machine. 
$ sudo service mongod start

Verify Mongod proccess by checking the contents of the log file at /var/log/mongodb/mongod.log.
[initandlisten] waiting for connections on port <port>
Where <port> is 27018 by default.

From query router mongos0.example.com, connect to the replica set member on mongodb0.example.com
$ mongo --host mongodb0.example.com --port 27018 admin

Initiate the replica set.
> rs.initiate()

Verify the initial replica set configuration.
rs0:SECONDARY> rs.conf()

Add the members to the replica set.
rs0:PRIMARY> rs.add("mongodb1.example.com:27018")
{ "ok" : 1 }
rs0:PRIMARY> rs.add("mongodb2.example.com:27018")
{ "ok" : 1 }

Check the status of the replica set.
rs0:PRIMARY> rs.status()
Repeat the above steps to create another replica set rs1 on mongodb3.example.com, mongodb4.example.com, and mongodb5.example.com.

Add Shards to the Cluster

Connect to any of the Query Routers.   
$ mongo mongos0.example.com:27017/admin
mongos> show dbs
admin   (empty)
config  0.016GB
mongos> sh.status()

Add a Shard for a replica set rs0 with replica set name and a replica set member mongodb0.example.com to the cluster.
mongos> sh.addShard("rs0/mongodb0.example.com:27018")
{ "shardAdded" : "rs0", "ok" : 1 }
mongos> sh.addShard("rs1/mongodb3.example.com:27018")
{ "shardAdded" : "rs1", "ok" : 1}
mongos> sh.status()

Enable sharding for a Database.

mongos> sh.enableSharding("testdb")
{ "ok" : 1 }
OR
mongos> db.runCommand( { enablesharding: "testdb" } )
{ "ok" : 0, "errmsg" : "already enabled" }

mongos> sh.shardCollection("testdb.testCollection",{"name":1})

{ "collectionsharded" : "testdb.testCollection", "ok" : 1 }

mongos> sh.status()

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "minCompatibleVersion" : 5,

        "currentVersion" : 6,

        "clusterId" : ObjectId("553c531d71621ab16006d19c")

}
  shards:
        {  "_id" : "rs0",  "host" : "rs0/192.168.1.21:27018,192.168.1.22:27018,mongodb0.example.com:27018" }
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  2
        Last reported error:  ReplicaSetMonitor no master found for set: rs0
        Time of Reported error:  Sun Apr 26 2015 01:51:27 GMT-0400 (EDT)
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "testdb",  "partitioned" : true,  "primary" : "rs0" }
                testdb.testCollection
                        shard key: { "name" : 1 }
                        chunks:
                                rs0     1
                        { "name" : { "$minKey" : 1 } } -->> { "name" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 0)

mongos> sh.getBalancerState()
true

On query router
$ mongostat --discover
$ mongostat --host mongodb0.example.com:27018, mongodb1.example.com:27018, mongodb2.example.com:27018
If you don't see new database
mongos> use config
switched to db config
mongos> db.databases.find()
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "testdb", "partitioned" : true, "primary" : "rs0" }

Create "hashed" shard key on id field on collection.
mongos> use testdb
mongos> db.test_collection.ensureIndex( { _id : "hashed" } )
{
        "raw" : {
                "rs0/192.168.1.21:27018,192.168.1.22:27018,mongodb0.example.com:27018" : {
                        "createdCollectionAutomatically" : true,
                        "numIndexesBefore" : 1,
                        "errmsg" : "exception: bad index key pattern { _id: \"hasted\" }: Unknown index plugin 'hasted'",
                        "code" : 67,
                        "ok" : 0,
                        "$gleStats" : {
                                "lastOpTime" : Timestamp(1430087893, 1),
                                "electionId" : ObjectId("553c6a2f97bbb1cbf830be1c")
                        }
                }
        },
        "code" : 67,
        "ok" : 0,
        "errmsg" : "{ rs0/192.168.1.21:27018,192.168.1.22:27018,mongodb0.example.com:27018: \"exception: bad index key pattern { _id: \"hasted\" }: Unknown index plugin 'hasted'\" }"
}

Enable sharding for a Collection

mongos> sh.shardCollection("testdb.testCollection", { "_id": "hashed" } )
{ "ok" : 0, "errmsg" : "already sharded" }

Insert data into the collection.

mongos> use testdb
mongos> for (var i = 1; i <= 500; i++) db.testCollection.insert( { x : i } )
WriteResult({ "nInserted" : 1 })

Query Data from the Collection
mongos> db.testCollection.find()
{ "_id" : ObjectId("553d69f83dc4c576940dc59a"), "x" : 1 }
{ "_id" : ObjectId("553d69f83dc4c576940dc59b"), "x" : 2 }
{ "_id" : ObjectId("553d69f83dc4c576940dc59c"), "x" : 3 }
{ "_id" : ObjectId("553d69f83dc4c576940dc59d"), "x" : 4 }
{ "_id" : ObjectId("553d69f83dc4c576940dc59e"), "x" : 5 }
{ "_id" : ObjectId("553d69f83dc4c576940dc59f"), "x" : 6 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a0"), "x" : 7 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a1"), "x" : 8 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a2"), "x" : 9 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a3"), "x" : 10 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a4"), "x" : 11 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a5"), "x" : 12 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a6"), "x" : 13 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a7"), "x" : 14 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a8"), "x" : 15 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5a9"), "x" : 16 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5aa"), "x" : 17 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5ab"), "x" : 18 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5ac"), "x" : 19 }
{ "_id" : ObjectId("553d69f83dc4c576940dc5ad"), "x" : 20 }
Type "it" for more

Get info about specific shards.
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("553c531d71621ab16006d19c")
}
  shards:
        {  "_id" : "rs0",  "host" : "rs0/192.168.1.21:27018,192.168.1.22:27018,mongodb0.example.com:27018" }
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  2
        Last reported error:  ReplicaSetMonitor no master found for set: rs0
        Time of Reported error:  Sun Apr 26 2015 18:45:42 GMT-0400 (EDT)
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "testdb",  "partitioned" : true,  "primary" : "rs0" }
                testdb.testCollection
                        shard key: { "name" : 1 }
                        chunks:
                                rs0     1
                        { "name" : { "$minKey" : 1 } } -->> { "name" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 0)

Create the Database
> use SoccerLeague
switched to db SoccerLeague

Create a table(Collection)
> db
SoccerLeague
> db.createCollection("Teams")
{
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : Timestamp(1430088542, 1),
                "electionId" : ObjectId("553c6a2f97bbb1cbf830be1c")
        }
}

Use the Collections (tables) with a DB
> show collections
Teams
system.indexes

Drop a Collection (table) within a DB
> db.Teams.drop();
true

Insert Data into the Collection
> var a = {"name":"New York Giants", "conference":"American"}
> db.Teams.save(a)
WriteResult({ "nInserted" : 1 })

Query Data from the Collection
> db.Teams.find()
{ "_id" : ObjectId("553d6ccd3dc4c576940dc78e"), "name" : "New York Giants", "conference" : "American" }
> var b = {"name":"New York Giants", "conference":"American"}
> db.Teams.save(b)
WriteResult({ "nInserted" : 1 })
> db.Teams.insert({ name: 'Apple', product: 'iPhone', emp_no: '0'})
WriteResult({ "nInserted" : 1 })
> db.Teams.find()
{ "_id" : ObjectId("553d6ccd3dc4c576940dc78e"), "name" : "New York Giants", "conference" : "American" }
{ "_id" : ObjectId("553d6d4e3dc4c576940dc78f"), "name" : "New York Giants", "conference" : "American" }
{ "_id" : ObjectId("553d6d9e3dc4c576940dc790"), "name" : "Apple", "product" : "iPhone", "emp_no" : "0" }
> db.Teams.find().forEach(printjson)
{
        "_id" : ObjectId("553d6ccd3dc4c576940dc78e"),
        "name" : "New York Giants",
        "conference" : "American"
}
{
        "_id" : ObjectId("553d6d4e3dc4c576940dc78f"),
        "name" : "New York Giants",
        "conference" : "American"
}
{
        "_id" : ObjectId("553d6d9e3dc4c576940dc790"),
        "name" : "Apple",
        "product" : "iPhone",
        "emp_no" : "0"
}
> db.Teams.find({"name":"New York Giants"});
{ "_id" : ObjectId("553d6ccd3dc4c576940dc78e"), "name" : "New York Giants", "conference" : "American" }
{ "_id" : ObjectId("553d6d4e3dc4c576940dc78f"), "name" : "New York Giants", "conference" : "American" }
> db.Teams.find({"name":"New York Yankees"});
Multiple Update a document (row) within a Collection
mongos> db.Teams.update({"name":"New York Giants"},{"name":"New York Jets", "conference":"National"}, {multi:true});
WriteResult({
        "nMatched" : 0,
        "nUpserted" : 0,
        "nModified" : 0,
        "writeError" : {
                "code" : 9,
                "errmsg" : "multi update only works with $ operators"
        }
})

Single Update a document (row) within a Collection
> db.Teams.update({"name":"New York Giants"},{"name":"New York Jets", "conference":"National"});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

updates first row
> db.Teams.find()
{ "_id" : ObjectId("553d6ccd3dc4c576940dc78e"), "name" : "New York Jets", "conference" : "National" }
{ "_id" : ObjectId("553d6d4e3dc4c576940dc78f"), "name" : "New York Jets", "conference" : "National" }
{ "_id" : ObjectId("553d6d9e3dc4c576940dc790"), "name" : "Apple", "product" : "iPhone", "emp_no" : "0" }

Remove a document (row) within a Collection
> db.Teams.remove({"name":"New York Giants"});
WriteResult({ "nRemoved" : 0 })
> db.Teams.remove({"_id":ObjectId("553d6ccd3dc4c576940dc78e")});
WriteResult({ "nRemoved" : 1 })
> db.Teams.count()
2
> exit

Bulk Load or Script Data into the Collection
> load("LoadData.js");
> show collections

 
Copy Database
> db.copyDatabase("SoccerLeague","FootballLeague","localhost")
{
        "ok" : 0,
        "errmsg" : "couldn't connect to server localhost:27017 (127.0.0.1), conn
ection attempt failed",
        "$gleStats" : {
                "lastOpTime" : Timestamp(1430088584, 1),
                "electionId" : ObjectId("553c6a2f97bbb1cbf830be1c")
        }
}

Create User

mongos> db.createUser(
...     {
...       user: "testUser",
...       pwd: "testpw",
...       roles: [
...          { role: "readWrite", db: "testdb" }
...       ]
...     }
... )
Successfully added user: {
        "user" : "testUser",
        "roles" : [
                {
                        "role" : "readWrite",
                        "db" : "testdb"
                }
        ]
}