1 | Page

Information on resource allocation within NetBackup.

Scope:

This document will outline how resources are allocated within NetBackup and configuration options that are available to change the behavior of the NetBackup Resource Broker (nbrb). In-depth behavior of nbrb will be outlined, as it is the primary process responsible for resource allocation. Other processes will be covered only as they affect resource allocation.

This document will not cover troubleshooting techniques. That will be covered in a different document.

The areas of the application that will be covered as follows:

–For the purpose of this document, resource allocation will be defined as the point in time a job appears queued in the activity monitor, until the point that a job has started writing.

–The behavior outlined will apply to NetBackup versions 6.5.2 and later releases.

–The focus will be on a scheduled backup job.

–The differences between bptm parent and child will not be covered.

–The portion of the functional overview covered, is as follows:

Definitions:

bpcd: The NetBackup client daemon. This listens for connections on all NetBackup servers and clients. In 6.x, the default behavior is that connections from the master/media server(s) are made through vnetd. The vnetd daemon then makes a connection to bpcd to initiate whatever action needs to be completed.

bpcompatd: NetBackup compatibility daemon. Used by nbjm to initiate calls to media servers.

bpbrm: NetBackup backup and restore manager. This runs on media servers when backups or restores are running. The nbjm process will connect with bpbrm, via bpcompatd to vnetd to bpcd, to spawn the bptm process during backup and restores. The bpbrm process will send job manger (bpjobd) updates via nbjm. For mpx backups, bpbrm is started in MPX mode. A backup is then added to this mpx group by sending the “START BACKUP” message to bpbrm.

bptm:NetBackup tape manager. This runs on media servers and is typically initiated by bpbrm. Some of the many things this process is responsible for are: writing data to disk or tape, initiating tape actions such as load/unload, reporting drive states to nbrb and establishing a pipe with the client bpbkar process in order to pass data to the media.

nbemm: NetBackup Enterprise Media Manager. Originator ID (OID) 111 This process will run on the entity that is known as the emm server, typically this is the master server. This process has several subcomponents, such as mds (media and device selection component, OID 143) and da (the device allocator for shared drives, OID 144).

nbrb: NetBackup Resource Broker. Originator ID (OID) 118. This process is responsible for allocating disk and tape resources to jobs requiring them. It interacts with the nbemm components such as mds (the media selection component of emm), da (device allocator) ,fsm (Fiber Transport service manager) and dsm (disk service manager)frequently to query and update allocation status within the database.

nbjm: NetBackup Job Manager. Originator ID (OID) 117. This process is responsible for passing job updates to the jobs database (bpjobd) and passing resource requests to nbrb and resource updates to bptm.

vnetd: This daemon listens for connections on all NetBackup servers and clients.

PBX: Private Branch Exchange. Originator ID (OID 103), product ID 50936. PBX listens on port 1556. It is responsible for establishing connections on the master and media servers for certain functions.

Specific examples of when each of the above processes/daemons are utilized will be outlined later in this document. Neither the examples above, nor the ones outlined later on in this document are a full list of scenarios, but will cover many of the steps and actions required to allow resource allocation to occur.

Process flow explained:

When a job is scheduled by the NetBackup Policy Execution Manager, Originator ID (OID) 116, and it appears in the activity monitor, the job has reached a state where it has been passed to nbjm. The nbjm process will then add the request to the nbrb inbound queue, for resource allocation fulfillment. The job may appear in the activity monitor before or after the request is sent to nbrb, depending on timing of the receipt of the job. Job attribute updates are sent to bpjobd from nbjm every 10 seconds.

The job will remain in the inbound queue until nbrb breaks from the current allocation cycle. Breaks in the evaluation cycle can occur for several reasons:

–The nbrb process has reviewed all allocations that are waiting.

–The nbrb.conf file parameter SECONDS_FOR_EVAL_LOOP_RELEASE time has been met (the default is 180 seconds at 6.5.4 and later versions).

–If a high priority request [span request, subsequent requests of a dup job, and requests with a pending condition] is received and the nbrb.conf parameter BREAK_EVAL_ON_DEMAND is set to “1” (enabled.)

If nbrb is not currently in the middle of an evaluation cycle, an evaluation cycle will start as soon as nbjm places the job in the inbound queue.

Once nbrb reads the request from the inbound queue, it is placed on the nbrb queue of requests that it is currently reviewing. The process will review specific resource requirements for the job. Resource allocations will be reviewed and an allocation assigned for each of the following:

–A client allocation, after reviewing the maximum number of jobs allowed per client.

–A policy allocation, after reviewing the maximum of jobs allowed per policy.

–One or more storage unit allocations (which include media and drive information) based on the request type.

  • First checks if it can join an existing MPX group. If it can, the request is satisfied internally without going to MDS. Other checks may be made to MDS at this time outside of the allocation.
  • If the request cannot join an existing MPX group, nbrb checks its cache to see if it is worth sending the request to MDS. If nbrb's cache indicates that all the drives in the storage unit are in use, MDS is also going to return 'retry later'. Why to send it to MDS then?
  • If, as per nbrb cache, it is okay to send the request to MDS, nbrb will send the request to nbrb.

Each of the above are separate allocations and if a limit is exceeded or otherwise not available (such as if all the drives are busy or down), the resource allocation will not succeed and will be retried at a later time.

Once nbrb has successfully allocated resources for a backup job, the resource allocations are passed to nbjm via the nbrb outbound queue.

The nbjm process reads the information from the nbrb outbound queue the job will appear as active in the activity monitor within a short period of time. The nbjm process will initiate a connection to the bpbrm process on the media server, typically via vnetd to bpcd communication, and bpbrm in turn will initiate the bptm process.

When bptm is started, it initiates a connection to the nbjm process on the master server in order to get the media and drive information for the backup job. This communication is initiated through PBX, and an ORB connection will be established between the bptm and nbjm processes.

At the time that nbjm receives the request from bptm, it will add the media/drive information to the outbound Callback queue.

*** Note: There is only one nbjm Callback queue used for this purpose, all resources requests are passed to the media server bptm processes in a first-in-first-out order.

The activity monitor will now show the drive as mounting, positioning and eventually writing.

If the backup encounters an end of media (EOM) during the course of writing the backup, bptm will notify nbjm via PBX, and will wait for additional resource and drive information to be allocated. The current drive resource will not be released completely until a new drive/media is allocated. This ensures that the if no other drives are available, the current backup job should be able to use the same drive.

At the end of a backup job, bptm will send an update to nbjm via PBX, that it is finished writing. The media server bpbrm process will send a job end message to nbjm. The nbjm process will send an update to nbrb to free the resources and unload the drive

The unloading of drives will always take place at the end of an evaluation cycle. The release of resources will take place at the beginning of each cycle. The nbrb process will update nbemm as drives become available, are unloaded, etc.

An in-depth look at the nbrb process

The nbrb, or NetBackup Resource Broker, is the daemon/service responsible for handling resource requests for NetBackup.

The nbrb process runs through an evaluation cycle periodically to review requests, unload media, release resources, etc.

When the allocation cycle starts, nbrb sorts the list of jobs waiting for allocations, and looks at each item in the list sequentially. It either:

–Allocates the resources and crosses it off the list

–Or marks it for later evaluation.

The resource broker (nbrb) maintains a set of queues for the requests and allocations:

–A queue for requests it receives, referred to as the inbound queue.

–A queue for requests it is currently evaluating.

–A queue for requests processed that need to be transferred to the media server, referred to as the outbound queue.

The process exposes a CORBA interface to nbjm and it uses nbemm's exposed CORBA interfaces for the purposes of passing resource requests and updates.

It may be helpful to note that nbjm is nbrb’s client, and nbrb is emm(mds)’s client.

It uses a timer to schedule tasks to include the start of an evaluation cycle, real releases, bptm report drives process, etc. The default timers are as follows:

–Every 300 seconds nbrb will wake up and look for entities that require servicing, such as releases, new requests, etc. We haven't seen a need to change this ever.

–Every 10 minutes nbrb will initiate a bptm report drives process.

  • Reports the actual status of the drives. Failure can cause media servers to be marked offline in emm.

For each evaluation cycle, nbrb completes the following, in this order:

–Read in the nbrb.conf file or configuration settings contained in the emm database (depending on version).

–Add new jobs from the inbound job queue.

–Release allocations: These are allocations that are already marked to be released for jobs that have completed.

–Mark released drives as available.

–Build the drive/stu/stugrp cache (the evaluation loop timer is started here).

–Evaluate requests in prioritized areas:

  • span requests and subsequent resource requests for duplication jobs already active and requests that are allocated but have a pending condition associated with them.
  • these are known as “high priority requests”
  • Everything else.
  • User type backup jobs will be sorted to the top of the lists after the high priority requests.

–Look for drives to unload [the drives that are released and are not used for MEDIA_UNLOAD_DELAY seconds], start unloading drives.

Common items that impact the evaluation cycle:

There are manyvariables that will impact the time it takes nbrb to allocate resources for a specific job. The most common of those items, along with very basic details about them are listed below:

–Tuning parameters specific to nbrb.

  • This will be discussed in detail in the next topic.

–Tuning parameters specific to emm.

  • There is only one write DB connection.
  • Tuning parameters implemented in emm.conf will not increase the number of threads allowed to write
  • More read/update threads are not better in many circumstances.
  • High numbers of threads configured in emm.conf will not help performance of allocations and may hinder them as each emm thread may consume 200 to 400 MB of memory each.

–The storage unit group configuration may slow down the allocation cycle. How much of an impact will depend on the algorithm used and the number of storage units within the group and their availability. In general:

  • The more storage units in a group the longer the allocation will take.
  • It is not recommended to have more than5 storage units within a group.
  • The algorithm selected will also impact the time it takes to allocate resources.

* For more information on the algorithms available for storage units, , see the NetBackup Administration Guide, for 6.5, volume 1, Chapter 5 “Storage units, unit groups, and lifecycle policies”.

–Duplication jobs, such as vault, SLP and disk staging jobs will take longer to allocate due to the requirement of more than one physical resource.

–Policy configuration variables that impact allocations:

  • The number of backups allowed per policy and client
  • The number of copies.
  • The type of backup
  • Application backups such as oracle will spawn a dummy allocation prior to starting the backup.
  • Schedules that have a variety of different mpx and retention level settings.
  • This can cause drives to not be used at full capacity as ‘one off’ backups consume drive resources.

–Poor disk i/o.

  • The nbrb process uses SQL database heavily for storing its state (allocations, drive info, and other stuff). If the disk I/O is slow, the allocation cycle will slow down.
  • If logging levels are set to a high level, and there are many logs being generated, this may slow down the allocation cycle as the processes cache data to be written to the logs. In general the log level should be set to the default level which is 1. If the customer is facing any issue and symptoms show that this is because of RB, the debug log level should be set to 4. In some extreme cases, where we think that the problem could be because of slow SQL operations, the debug level should be set to 5.
  • nbjm updates bpjobd via bpcompatd connection every 10 seconds.The bpjobd process is single threaded; if bpjobd updates are slow, job updates will not get populated to the activity monitor in a timely manner and jobs may appear to be queued, but actually have resources and be active.

–Hardware issues

  • Library or drive issues cause problems with allocations.
  • If a robot is down or having sporadic communication issues, mounts/unmounts may take substantially longer or may have to be retried many times before succeeding.
  • If a drive is unavailable, nbrb will need confirmation that the drive has been unloaded prior to allowing it to be reallocated. At times this may take a substantial amount of time.

–Media servers being marked offline.

  • Media servers can be marked offline for several reasons:
  • EMM does not get heartbeat from the media server (vmd/ltid).
  • bpstsupdate is spawned every 5 minutes to poll any disk drives. If this fails, disk allocations will not occur until a status is received.
  • Nbrb polls drives every 10 minutes from the media servers to get current status. If nbrb doesn’t receive a response or the report drives fails, the media server will be marked offline.

–Communication issues.

  • Corba transient errors can occur if media and master servers are unable to communicate effectively. This can slow down unloads, updates and a variety of other issues.

–“Home grown” scripts that are passing invalid parameters can cause load on the system and emm database. This can cause delays if there are enough emm update or queries being run.

– Insufficient resources, such as available memory.

–Unneeded entries in bp.conf that will trigger host name lookups.

  • CONNECT_OPTIONS
  • These are only for media server related connections. Client systems should not be listed. They should be configured in the Master Server  Host properties  Client attributes tab and NOT the Master Server  Host properties  Firewall tab. *
  • Invalid SERVER or MEDIA server entries.

–Clients listed in the Master Server  Host properties  Client attributes tab that are not resolvable.

**For more information on the Client attributes and Firewall tabs in Master Server  Host properties, see the NetBackup Administration Guide, for 6.5, volume 1, Chapter 7, “Host Properties”.

–SQL DB, legacy catalogs and logs can get pretty big. It is highly recommended to keep them on separate disk spindles. Often these files are kept on fancy SAN disks enclosures. If that is the case, please make sure they are configured properly. To measure the performance, focus on both read and write to the disks. We have seen at least two cases where these files were on a fancy SAN disk. The read operations were super fast but the writes were very slow.

–A large emm transaction log (NBDB.log file). Sizes of the NBDB.log will vary based on the number of transactions that occur in the period since the last backup. If the transaction log grows to large (too large is qualified as 1 to 2GB in size), allocations will slow down.