how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

Question

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory

according to:

yarn.nodemanager.resource.cpu-vcores: Set to the appropriate number in yarn-site.xml on all the nodes. This is strictly dependent on the type of workloads running in a cluster, but the general recommendation is that admins set it to be equal to the number of physical cores on the machine.

we set the yarn.nodemanager.resource.cpu-vcores to 16

but what is the right value for - yarn.scheduler.maximum-allocation-vcores ?

yarn.scheduler.maximum-allocation-vcores: This is the maximum allocation for every container request at the Resource Manager, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.

Dose it mean 80% from total core or something between 50% to 100% of total CPU CORE ?

jhop · Answer

This blog: https://qubole.zendesk.com/hc/en-us/articles/115002396546-Reference-Default-yarn-scheduler-maximum-allocation-vcores

...claims:

The default value for yarn.scheduler.maximum-allocation-vcores in Qubole is set to twice the number of CPUs. This oversubscription assumes that CPUs are not always running a thread, and hence assigning more cores enables maximum CPU utilization.

If you're not comfortable with that, you could set yarn.scheduler.maximum-allocation-vcores = 80% * (total CPU cores) * 2. After making any setting changes, monitor your cluster/node CPU utilization for a while under load. 70%-80% utilization should be about right.

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

One Answer

Add your own answers!

Ask a Question