YARN (Yet Another Resource
Negotiator) can manage Hadoop applications like MapReduce so that
applications can reserve resources like CPU and memory so that resources
are not denied to other applications. YARN applications request resources
from a resource manager.
If YARN is already used
on the cluster, then you can configure SAS LASR Analytic Server
instances and high-performance procedures to participate in the resource
accounting that YARN performs. This enables administrators to have
a complete view of resource usage. Otherwise, having a mix of some
applications accounting for their resources with YARN and others that
are not essentially results in no management at all.
Note: Some YARN implementations
create CGroups automatically, which can interfere with CGroups that
you administer manually.
In order to integrate
with YARN, the following settings in the resource.settings
file
are used.
# The number of cores to allocate to each host's container.
export TKMPI_YARN_CORES=1
# The amount of memory in megabytes to reserve.
export TKMPI_MEMSIZE=30000
# The priority of the application if scheduler uses it.
export TKMPI_YARN_PRIORITY=2
# Length of time TKGrid should wait for the resource reservation in seconds.
export TKMPI_YARN_TIMEOUT=3600
# The queue to submit the job to.
export TKMPI_YARN_QUEUE=default
# The next setting must be on one line in the resource.settings file,
# but is split for readability.
export TKMPI_RESOURCEMANAGER="java -cp \"`$HADOOP_HOME/bin/hadoop
classpath`\" com.sas.grid.provider.yarn.tkgrid.JobLauncher
--masterMem 2000 --javaMem 500 --hostlist \$TKMPI_YARN_HOSTS \
--cores \$TKMPI_YARN_CORES --memory \$TKMPI_MEMSIZE \
--priority \$TKMPI_YARN_PRIORITY --timeout \$TKMPI_YARN_TIMEOUT
--jobname $TKMPI_APPNAME --queue $TKMPI_YARN_QUEUE"
Note: The TKMPI_YARN_HOSTS and
TKMPI_APPNAME variables are automatically set by SAS software.
The TKMPI_MEMSIZE variable
specifies the amount of memory for YARN to reserve. SAS also uses
the value to self-govern the memory allocations that are performed
by the server or high-performance procedure.
YARN must be configured
on the cluster. The JobLauncher
class starts
a YARN application to request the specified resources on each machine
so that YARN knows it cannot allocate those resources to other applications.
If YARN does not grant the resource in the time-out period (TKMPI_YARN_TIMEOUT),
then the initializing of the server or job fails.
If the State field
for an application indicates RUNNING and
the Progress percentage is at 50%, then the
resources are reserved and the application is running. This is the
normal state.
If the State field
indicates ACCEPTED and
the Tracking UI field indicates UNASSIGNED,
then the application is not running. Check if you are near or at capacity.