One-time HDFS Protocol Installation

One-time HDFS Protocol Installation

Install and configure Hadoop for use with gphdfs as follows:
  1. Install Java 1.7 or later on all Greenplum Database hosts: master, segment, and standby master.
  2. Install a compatible Hadoop distribution on all hosts. The distribution must be the same on all hosts. For Hadoop installation information, see the Hadoop distribution documentation.

    See the Greenplum Database Release Notes for information about compatible Hadoop distributions.

  3. After installation, ensure that the Greenplum system user (gpadmin) has read and execute access to the Hadoop libraries or to the Greenplum MR client.
  4. Set the following environment variables on all segments:
    • JAVA_HOME – the Java home directory
    • HADOOP_HOME – the Hadoop home directory
    For example, add lines such as the following to the gpadmin user .bashrc profile.
    export JAVA_HOME=/usr/java/default
    export HADOOP_HOME=/usr/lib/gphd

    The variables must be set in the ~gpadmin/.bashrc or the ~gpadmin/.bash_profile file so that the gpadmin user shell environment can locate the Java home and Hadoop home.

  5. Set the following Greenplum Database server configuration parameters and restart Greenplum Database.
    Table 1. Server Configuration Parameters for Hadoop Targets
    Configuration Parameter Description Default Value Set Classifications
    gp_hadoop_target_version The Hadoop target. Choose one of the following.

    cdh5

    cdh4.1

    hdp2

    gpmr-1.2

    hadoop2

    gphd-1.1 master

    session

    reload
    gp_hadoop_home With Pivotal HD, this parameter specifies the installation directory for Hadoop. For example, the default installation directory is /usr/lib/gphd.

    When using Greenplum HD 1.2 or earlier, specify the same value as the HADOOP_HOME environment variable.

    NULL master

    session

    reload

    For example, the following commands use the Greenplum Database utilities gpconfig and gpstop to set the server configuration parameters and restart Greenplum Database:
    gpconfig -c gp_hadoop_target_version -v "'hdb2'"
    gpstop -u
    For information about the Greenplum Database utilities gpconfig and gpstop, see the Greenplum Database Utility Guide.
  6. If needed, ensure that the CLASSPATH environment variable generated by the $GPHOME/lib/hadoop/hadoop_env.sh file on every Greenplum Database host contains the path to JAR files that contain Java classes that are required for gphdfs.

    For example, if gphdfs returns a class not found exception, ensure the JAR file containing the class is on every Greenplum Database host and update the $GPHOME/lib/hadoop/hadoop_env.sh file so that the CLASSPATH environment variable created by file contains the JAR file.