Advanced GPU configuration (Optional)

vGPU Scheduling Policies

NVIDIA AI Enterprise provides three GPU scheduling options to accommodate a variety of QoS requirements of customers. However, since Enterprise AI workloads are typically long-running operations, it is recommended to implement the Fixed Share or Equal Share scheduler for optimal performance.

  • Fixed share scheduling always guarantees the same dedicated quality of service. The fixed share scheduling policies guarantee equal GPU performance across all vGPUs sharing the same physical GPU.

  • Equal share scheduling provides equal GPU resources to each running VM. As vGPUs are added or removed, the share of GPU processing cycles allocated changes accordingly, resulting in performance to increase when utilization is low, and decrease when utilization is high.

  • Best effort scheduling provides consistent performance at a higher scale and therefore reduces the TCO per user. The best effort scheduler leverages a round-robin scheduling algorithm that shares GPU resources based on actual demand, resulting in optimal utilization of resources. This results in consistent performance with optimized user density. The best effort scheduling policy best utilizes the GPU during idle and not fully utilized times, allowing for optimized density and a good QoS.

Additional information regarding GPU scheduling can be found here.

RmPVMRL Registry Key

The RmPVMRL registry key sets the scheduling policy for NVIDIA vGPUs.

Note

You can change the vGPU scheduling policy only on GPUs based on the Pascal, Volta, Turing, and Ampere architectures.

Type

Dword

Contents

Value

Meaning

0x00 (default)

Best effort scheduler

0x01

Equal share scheduler with the default time slice length

0x00TT0001

Equal share scheduler with a user-defined time slice length TT

0x11

Fixed share scheduler with the default time slice length

0x00TT0011

Fixed share scheduler with a user-defined time slice length TT

Examples

The default time slice length depends on the maximum number of vGPUs per physical GPU allowed for the vGPU type.

Maximum Number of vGPUs

Default Time Slice Length

Less than or equal to 8

2 ms

Greater than 8

1 ms

TT

  • Two hexadecimal digits in the range 01 to 1E set the time slice length in milliseconds (ms) for the equal share and fixed share schedulers. The minimum length is 1 ms, and the maximum length is 30 ms.

  • If TT is 00, the length is set to the default length for the vGPU type.

  • If TT is greater than 1E, the length is set to 30 ms.

Examples

This example sets the vGPU scheduler to equal share scheduler with the default time slice length.

RmPVMRL=0x01

This example sets the vGPU scheduler to equal share scheduler with a time slice that is 3 ms long.

RmPVMRL=0x00030001

This example sets the vGPU scheduler to a fixed share scheduler with the default time slice length.

RmPVMRL=0x11

This example sets the vGPU scheduler to a fixed share scheduler with a time slice 24 (0x18) ms long.

RmPVMRL=0x00180011

Changing the vGPU Scheduling Policy for All GPUs

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine. On all supported hypervisors, you can use a secure shell (SSH) for this purpose. Set the RmPVMRL registry key to the value that sets the GPU scheduling policy needed.

  2. In the VMware vSphere SSH CLI, use the esxcli set command.

    # esxcli system module parameters set -m nvidia -p "NVreg_RegistryDwords=RmPVMRL=value"
    

    Note

    Where <value> is the value that sets the vGPU scheduling policy you want, for example:

    • 0x00 - Equal Share Scheduler with the default time slice length

    • 0x00030001 - Equal Share Scheduler with a time slice of 3 ms

    • 0x011 - Fixed Share Scheduler with the default time slice length

    • 0x00180011 - Fixed Share Scheduler with a time slice of 24 ms (0x18)

    The default time slice length depends on the maximum number of vGPUs per physical GPU allowed for the vGPU type.

    Maximum Number of vGPUs

    Default Time Slice Length

    Less than or equal to 8

    2 ms

    Greater than 8

    1 ms

  3. Reboot your hypervisor host machine.

Changing the vGPU Scheduling Policy for Select GPUs

Perform this task in your hypervisor command shell:

  1. Open a command shell as the root user on your hypervisor host machine. On all supported hypervisors, you can use a secure shell (SSH) for this purpose.

  2. Use the lspci command to obtain the PCI domain and bus/device/function (BDF) of each GPU for which you want to change the scheduling behavior.

    • Pipe the output of lspci to the grep command to display information only for NVIDIA GPUs.

      # lspci | grep NVIDIA
      

      The NVIDIA GPUs listed in this example have the PCI domain 0000 and BDFs 85:00.0 and 86:00.0.

      0000:85:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [M60] (rev a1)
      0000:86:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [M60] (rev a1)
      
  3. Use the module parameter NVreg_RegistryDwordsPerDevice to set the pci and RmPVMRL registry keys for each GPU.

    • Use the esxcli set command:

      # esxcli system module parameters set -m nvidia -p "NVreg_RegistryDwordsPerDevice=pci=pci-domain:pci-bdf;RmPVMRL=value [;pci=pci-domain:pci-bdf;RmPVMRL=value...]"
      

      For each GPU, provide the following information:

      • pci-domain

        • The PCI domain of the GPU.

      • pci-bdf

        • The PCI device BDF of the GPU.

      • value

        • 0x00 - Sets the vGPU scheduling policy to Equal Share Scheduler with the default time slice length.

        • 0x00030001 - Sets the vGPU scheduling policy to Equal Share Scheduler with a time slice that is 3ms long.

        • 0x011 - Sets the vGPU scheduling policy to Fixed Share Scheduler with the default time slice length.

        • 0x00180011 - Sets the vGPU scheduling policy to Fixed Share Scheduler with a time slice of 24 ms (0x18) long.

      • For all supported values, see RmPVMRL Registry Key.

  4. Reboot your hypervisor host machine.

Restoring Default vGPU Scheduler Settings

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine. On all supported hypervisors, you can use a secure shell (SSH) for this purpose.

  2. Unset the RmPVMRL registry key by setting the module parameter to an empty string.

    # esxcli system module parameters set -m nvidia -p "module-parameter="
    

    module-parameter

    The module parameter to set, which depends on whether the scheduling behavior was changed for all GPUs or select GPUs:

    • For all GPUs, set the NVreg_RegistryDwords module parameter.

    • For select GPUs, set the NVreg_RegistryDwordsPerDevice module parameter.

    For example, to restore default vGPU scheduler settings after they were changed for all GPUs, enter this command:

    # esxcli system module parameters set -m nvidia -p "NVreg_RegistryDwords="
    
  3. Reboot your hypervisor host machine.

Page Retirement and ECC

NVIDIA AI Enterprise supports ECC and dynamic page retirement. This feature will “retire” bad frame buffer memory cells by retiring the page the cell belongs to. Dynamic page retirement is done automatically for cells that are degrading in quality. This feature can improve the longevity of an otherwise good board and is thus a vital resiliency feature on supported products, especially in HPC and enterprise environments. Retiring of pages may only occur when ECC is enabled. However, once a page has been retired, it will permanently be blacklisted, even if ECC is later disabled. Refer to the NVIDIA Developer Zone page retirement documentation for more information.

These page retirement and ECC features are offered on all GPUs that are supported on NVIDIA AI Enterprise.

Disabling and Enabling ECC Memory

Specific GPUs that support NVIDIA vGPU software support error-correcting code (ECC) memory. ECC memory improves data integrity by detecting and handling double-bit errors. However, not all GPUs, vGPU types, and hypervisor software versions support ECC memory with NVIDIA vGPU.

On GPUs that support ECC memory with NVIDIA vGPU, ECC memory is supported with C-series vGPUs, but not with A-series and B-series vGPUs. On physical GPUs that do not have HBM2 memory, the amount of frame buffer usable by vGPUs is reduced. All types of vGPU are affected, not just vGPUs that support ECC memory.

The effects of enabling ECC memory on a physical GPU are as follows:

  • ECC memory is exposed as a feature on all supported vGPUs on the physical GPU.

  • In VMs that support ECC memory, ECC memory is enabled, with the option to disable ECC in the VM.

  • ECC memory can be enabled or disabled for individual VMs. Enabling or disabling ECC memory in a VM does not affect the amount of frame buffer usable by vGPUs.

This task depends on whether you are changing ECC memory settings for a physical GPU or a vGPU.

  • For a physical GPU, perform this task from the hypervisor host.

  • For a vGPU, perform this task from the VM to which the vGPU is assigned.

    Note

    ECC memory must be enabled on the physical GPU on which the vGPUs reside.

Disabling ECC Memory

Before you begin, ensure that NVIDIA Virtual GPU Manager is installed on your hypervisor. If you are changing ECC memory settings for a vGPU, ensure that the NVIDIA vGPU software graphics driver is installed in the VM to which the vGPU is assigned. Please refer to Chapter 10 – how to create your first VM if you do not already have a VM to work with.

  1. Use nvidia-smi to list the status of all physical GPUs or vGPUs and check for ECC noted as enabled.

    # nvidia-smi -q
    
    ==============NVSMI LOG==============
    
    Timestamp                           : Mon Jul 13 18:36:45 2020
    Driver Version                      : 450.55
    
    Attached GPUs                       : 1
    GPU 0000:02:00.0
    
    [...]
    
        Ecc Mode
            Current                     : Enabled
            Pending                     : Enabled
    
    [...]
    
  2. Change the ECC status to off for each GPU for which ECC is enabled.

    • If you want to change the ECC status to off for all GPUs on your host machine or vGPUs assigned to the VM, run this command:

      # nvidia-smi -e 0
      
    • If you want to change the ECC status to off for a specific GPU or vGPU, run this command:

      # nvidia-smi -i id -e 0
      

    id is the index of the GPU or vGPU as reported by nvidia-smi. This example disables ECC for the GPU with index 0000:02:00.0.

    # nvidia-smi -i 0000:02:00.0 -e 0
    
  3. Reboot the host or restart the VM.

  4. Confirm that ECC is now disabled for the GPU or vGPU.

    # nvidia—smi —q
    
    ==============NVSMI LOG==============
    
    Timestamp                           : Mon Jul 13 18:37:53 2020
    Driver Version                      : 450.55
    
    Attached GPUs                       : 1
    GPU 0000:02:00.0
    [...]
    
        Ecc Mode
            Current                     : Disabled
            Pending                     : Disabled
    
    [...]
    

Enabling ECC Memory

If ECC memory is suitable for your workloads and is supported by your hypervisor software and GPUs, but is disabled on your GPUs or vGPUs, enable it. This task depends on whether you are changing ECC memory settings for a physical GPU or a vGPU. - For a physical GPU, perform this task from the hypervisor host. - For a vGPU, perform this task from the VM to which the vGPU is assigned.

Note

ECC memory must be enabled on the physical GPU on which the vGPUs reside.

Before you begin, ensure that NVIDIA Virtual GPU Manager is installed on your hypervisor. If you are changing ECC memory settings for a vGPU, ensure that the NVIDIA vGPU software graphics driver is installed in the VM to which the vGPU is assigned.

  1. Use nvidia-smi to list all physical GPUs or vGPUs’ status and check for ECC noted as disabled.

  2. Change the ECC status to on for each GPU or vGPU for which ECC is enabled.

    • If you want to change the ECC status to on for all GPUs on your host machine or vGPUs assigned to the VM, run this command:

      # nvidia-smi -e 1
      
    • If you want to change the ECC status to on for a specific GPU or vGPU, run this command:

      # nvidia-smi -i id -e 1
      

      id is the index of the GPU or vGPU as reported by nvidia-smi.

      This example enables ECC for the GPU with index 0000:02:00.0.

      # nvidia-smi -i 0000:02:00.0 -e 1
      
  3. Reboot the host or restart the VM.

  4. Confirm that ECC is now enabled for the GPU or vGPU.

    # nvidia—smi —q
    
    ==============NVSMI LOG==============
    
    Timestamp                           : Mon Jul 13 18:37:53 2020
    Driver Version                      : 450.55
    
    Attached GPUs                       : 1
    GPU 0000:02:00.0
    [...]
    
        Ecc Mode
            Current                     : Enabled
            Pending                     : Enabled
    
    [...]