Showing posts with label GPFS. Show all posts
Showing posts with label GPFS. Show all posts

Sunday, 9 March 2014

Troubleshooting GPFS Issues

In this article ,we are going to discuss about most general methods of  GPFS issues troubleshooting.
 TROUBLESHOOTING GPFS ISSUES

When you got GPFS issue?

Got a problem? Don’t panic! 
Check for possible basic problems: 
  • Is Network OK? 
  • Check status of the cluster: “mmgetstate–a” 
  • Check status of NSDs: “mmlsdisk fsname” 
Take a 5 min break  
  • In major cases GPFS will recover by it self without need of any intervention from the administrator 
If not recovered 
  • Ensure that you are the only person who is doing the work! 
  • check gpfslogs (first on cluster manager, then on FS manager, then on NSD servers) 
  • check syslog(/var/log/messages) for eventual errors 
  • Check disks availability (mmlsdisk fsname
  • Consult “Problem determination guide” 

Some usefull commands:

  • “mmfsadm dump waiters” will help to find long lasting processes 
  • “mmdiag --network|grep pending” helps to individuate non-responsive node 
  • “mmdiag --iohist” lists last 512 I/O operations performed by GPFS on current node (helps to find malfunctioning disk) 
  • “gpfs.snap” will garter all logs and configurations from all nodes in the cluster 
  • the first thing to send to IBM support when opening service reques

GPFS V3.4 Problem Determination Guide:

NFS stale file handle:

When a GPFS mount point is in the "NFS stale file handle" status, example 
[root@um-gpfs1 root]# df 
Filesystem 1K-blocks Used Available Use% Mounted on !
/dev/gpfs_um1 8125032448 8023801088 101231360 99% /storage/gpfs_um
df: `/storage/gpfs_um': Stale NFS file handle 
Then check if there is any NSD with status "down" 
[root@um-gpfs1 root]# mmlsdisk gpfs_um 
disk driver sector failure holds holds 
name type size group metadata data status availability 
------------ -------- ------ ------- -------- ----- ------------- ------------ 
disk21 nsd 512 4015 yes yes ready up !
disk22 nsd 512 4015 yes yes ready down !
disk23 nsd 512 4015 yes yes ready down !
disk24 nsd 512 4013 yes yes ready up !
restart the NSDs (important: do it for all NSD with status "down" in one command): 
[root@um-gpfs1 root]# mmchdisk gpfs_um start -d "disk21;disk24”
re-mount filesystems

Recovery of GPFS configuration:

If a node of the cluster lost its configuration (has been re-installed) but still present as member of this cluster
(“mmgetstate” lists it in “unknown” state) use this command to recover the node: 
/usr/lpp/mmfs/bin/mmsdrrestore -p diskserv-san-5 -R /usr/bin/scp

Checking existing NSD:

  • If get this warning while creating new nsd Disk descriptor xxx system refers to an existing NSD 
Use this command to verify if this device is actually used in one of the file systems 
mmfsadm test readdescraw /dev/emcpowerax

Thursday, 6 March 2014

How to Collect a GPFS snap

Question:

How do I collect a gpfs snap for review by  IBM GPFS support?

Answer

The command to collect GPFS Snap is  "gpfs,snap" , this is very generic but very helpful.

# gpfs.snap

Note:  The  purpose of above command is to collect the  snap on one node in the cluster,preferably the problematic one.

If the cluster is experiencing problems as a whole use -a flag

# gpfs.snap -a

Note1:  -a flag only recommended for smaller clusters as it collects the logs for all of the nodes in the cluster.

Note2: -d  option can  specify the Output Directory but  the default is /tmp/gpfs.snapOut.

==> Another recommendation is collect an mmfsadm dump waiters

/usr/lpp/mmfs/bin/mmfsadm dump waiters > gpfs.waiters
/usr/lpp/mmfs/bin/mmfsadm dump all > gpfs.dump.all
/usr/lpp/mmfs/bin/mmfsadm dump kthreads > gpfs.dump.kthreads

==> Another file you need to check is log file /var/adm/ras/mmfs.log.latest

This information will greatly assist development in reviewing performance or problematic issues in GPFS.

Friday, 9 August 2013

Building a Two-Node IBM GPFS Cluster on IBM AIX

Summary:  This article is a step-by-step guide for deploying a two-node IBM® General Parallel File System (IBM GPFS™) V3.5 cluster on IBM AIX® 7.1.

Overview

The purpose of this article is to provide a step-by-step guide for installing and configuring a simple two-node GPFS cluster on AIX. The following diagram provides a visual representation of the cluster configuration.

Figure 1. Visual representation of the cluster configuration

 

GPFS

GPFS provides a true "shared file system" capability, with excellent performance and scalability. GPFS allows concurrent access for a group of computers to a common set of file data over a common storage area network (SAN) infrastructure, a network, or a mix of connection types. GPFS provides storage management, information lifecycle management tools, and centralized administration and allows for shared access to file systems from remote GPFS clusters providing a global namespace.

GPFS offers data tiering, replication, and many other advanced features. The configuration can be as simple or complex as you want.

Preparing the AIX environment for GPFS

We'll assume that you have already purchased the necessary licenses and software for GPFS. With a copy of the GPFS software available, copy the GPFS file sets to each of the AIX nodes on which you need to run GPFS.

In this article, each partition was built with AIX version 7.1, Technology Level 2, Service Pack 1:
# oslevel -s
7100-02-01-1245
Each AIX system is configured with seven SAN disks. One disk is used for the AIX operating system (rootvg) and the remaining six disks are used by GPFS.
# lspv
hdisk0          00c334b6af00e77b                    rootvg          active
hdisk1          none                                none
hdisk2          none                                none
hdisk3          none                                none
hdisk4          none                                none
hdisk5          none                                none
hdisk6          none                                none
The SAN disks (to be used with GPFS) are assigned to both nodes (that is, they are shared between both partitions). Both AIX partitions are configured with virtual Fibre Channel adapters and access their shared storage through the SAN, as shown in the following figure.

Figure 2. Deployment diagram


The following attributes, shown in the table below, were changed for each hdisk, using the chdev command.

AIX device name Size in GB AIX disk device type Algorithm queue_depth reserve_policy
hdisk0 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk1 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk2 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk3 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk4 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk5 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve
hdisk6 50 Hitachi MPIO Disk VSP round_robin 32 no_reserve

The lsattr command can be used to verify that each attribute is set to the correct value:
# lsattr -El hdisk6 –a queue_depth –q algorithm –a reserve_policy
algorithm       round_robin        Algorithm                        True
queue_depth     32                 Queue DEPTH                      True
reserve_policy  no_reserve         Reserve Policy                   True
The next step is to configure Secure Shell (SSH) so that both nodes can communicate with each other. When building a GPFS cluster, you must ensure that the nodes in the cluster have SSH configured correctly so that they do not require password authentication. This requires the configuration of Rivest-Shamir-Adleman algorithm (RSA) key pairs for the root users SSH configuration. This configuration needs to be configured in both directions, to all nodes in the GPFS cluster.

The mm commands in GPFS require authentication in order for them to work. If the keys are not configured correctly, the commands will prompt for the root password each time and the GPFS cluster might fail. A good way to test this is to ensure that the ssh command can work unhindered by a request for the roots password.

You can refer to the step-by-step guide for configuring SSH keys on AIX:

You can confirm that the nodes can communicate with each other (unhindered) using SSH with the following commands on each node:
aixlpar1# ssh aixlpar1a date
aixlpar1# ssh aixlpar2a date
aixlpar2# ssh aixlpar2a date
aixlpar2# ssh aixlpar1a date
With SSH working, configure the WCOLL (Working Collective) environment variable for the root user. For example, create a text file that lists each of the nodes, one per line:
# vi /usr/local/etc/gfps-nodes.list
aixlpar1a
aixlpar2a

Copy the node file to all nodes in the cluster.

Add the following entry to the root users .kshrc file. This will allow the root user to execute commands on all nodes in the GPFS cluster using the dsh or mmdsh commands.
export WCOLL=/usr/local/etc/gfps-nodes.list
The root users PATH should be modified to ensure that all GPFS mm commands are available to the system administrator. Add the following entry to the root user's .kshrc file.
export PATH=$PATH:/usr/sbin/acct:/usr/lpp/mmfs/bin
The /etc/hosts file should be consistent across all nodes in the GPFS cluster. Each IP address for each node must be added to/etc/hosts on each cluster node. This is recommended, even when Domain Name System (DNS) is configured on each node. For example:
# GPFS_CLUSTER1 Cluster - Test
# # GPFS Admin network  - en0
10.1.5.110  aixlpar1a aixlpar1
10.1.5.120  aixlpar2a aixlpar2
# # GPFS Daemon - Private Network  – en1   
10.1.7.110   aixlpar1p
10.1.7.120   aixlpar2p

Installing GPFS on AIX

Now that the AIX environment is configured, the next step is to install the GPFS software on each node. This is a very straight forward process.

We will install GPFS version 3.5 (base-level file sets) and then apply the latest updates to bring the level up to 3.5.0.10. There are only three file sets to install. You can use System Management Interface Tool (SMIT) or the installp command to install the software.
aixlpar1 : /tmp/cg/GPFS/gpfs35_aix # inutoc .
aixlpar1 : /tmp/cg/GPFS/gpfs35_aix # ls -ltr
total 123024
-rw-r--r--    1 root     system       175104 Jun  7 2012  gpfs.msg.en_US
-rw-r--r--    1 root     system       868352 Jun  7 2012  gpfs.docs.data
-rw-r--r--    1 root     system     61939712 Jun  7 2012  gpfs.base
-rw-r--r--    1 root     system         3549 Apr 26 16:37 .toc
aixlpar1 : /tmp/cg/GPFS/gpfs35_aix # install –Y –d . ALL
Repeat this operation on the second node.
You can verify that the base-level GPFS file sets are installed by using the lslpp command:
# lslpp -l | grep -i gpfs
  gpfs.base                  3.5.0.0  COMMITTED  GPFS File Manager
  gpfs.msg.en_US             3.5.0.0  COMMITTED  GPFS Server Messages - U.S.
  gpfs.base                  3.5.0.0  COMMITTED  GPFS File Manager
  gpfs.docs.data             3.5.0.0  COMMITTED  GPFS Server Manpages and
The latest GPFS updates are installed next. Again, you can use SMIT (or installp) to update the file sets to the latest level. The lslpp command can be used to verify that the GPFS file sets have been updated.
aixlpar1 : /tmp/cg/gpfs_fixes_3510 # inutoc .
aixlpar1 : /tmp/cg/gpfs_fixes_3510 # ls -ltr
total 580864
-rw-r--r--    1 30007    bin       910336 Feb  9 00:10 U858102.gpfs.docs.data.bff
-rw-r--r--    1 30007    bin       47887360 May  8 08:48 U859646.gpfs.base.bff
-rw-r--r--    1 30007    bin       99655680 May  8 08:48 U859647.gpfs.gnr.bff
-rw-r--r--    1 30007    bin       193536 May  8 08:48 U859648.gpfs.msg.en_US.bff
-rw-r--r--    1 root     system    4591 May 10 05:15 changelog
-rw-r--r--    1 root     system    3640 May 10 05:42 README
-rw-r-----    1 root     system    55931 May 15 10:23 GPFS-3.5.0.10-power-AIX.readme.html
-rw-r-----    1 root     system    148664320 May 15 10:28 GPFS-3.5.0.10-power-AIX.tar
-rw-r--r--    1 root     system    8946 May 15 14:48 .toc
aixlpar1 : /tmp/cg/gpfs_fixes_3510 # smitty update_all
COMMAND STATUS
Command: OK            stdout: yes           stderr: no
Before command completion, additional instructions may appear below.
[MORE...59]
Finished processing all filesets.  (Total time:  18 secs).
+-----------------------------------------------------------------------------+
                        Pre-commit Verification...
+-----------------------------------------------------------------------------+
Verifying requisites...done
Results...
SUCCESSES
---------
  Filesets listed in this section passed pre-commit verification
  and will be committed.
  Selected Filesets
  -----------------
  gpfs.base 3.5.0.10                          # GPFS File Manager
  gpfs.msg.en_US 3.5.0.9                      # GPFS Server Messages - U.S. ...
  << End of Success Section >>
+-----------------------------------------------------------------------------+
                          Committing Software...
+-----------------------------------------------------------------------------+
installp: COMMITTING software for:
        gpfs.base 3.5.0.10
Filesets processed:  1 of 2  (Total time:  18 secs).
installp: COMMITTING software for:
        gpfs.msg.en_US 3.5.0.9
Finished processing all filesets.  (Total time:  18 secs).
+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
gpfs.msg.en_US              3.5.0.9         USR         APPLY       SUCCESS
gpfs.base                   3.5.0.10        USR         APPLY       SUCCESS
gpfs.base                   3.5.0.10        ROOT        APPLY       SUCCESS
gpfs.base                   3.5.0.10        USR         COMMIT      SUCCESS
gpfs.base                   3.5.0.10        ROOT        COMMIT      SUCCESS
gpfs.msg.en_US              3.5.0.9         USR         COMMIT      SUCCESS
aixlpar1 : /tmp/cg/gpfs_fixes_3510 # lslpp -l gpfs\*
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  gpfs.base                 3.5.0.10  COMMITTED  GPFS File Manager
  gpfs.msg.en_US             3.5.0.9  COMMITTED  GPFS Server Messages - U.S.
                                                 English
Path: /etc/objrepos
  gpfs.base                 3.5.0.10  COMMITTED  GPFS File Manager
Path: /usr/share/lib/objrepos
  gpfs.docs.data             3.5.0.3  COMMITTED  GPFS Server Manpages and
                                                 Documentation
Repeat the update on the second node.

Configuring the GPFS cluster

Now that GPFS is installed, we can create a cluster across both AIX systems. First, we create a text file that contains a list of each of the nodes and their GPFS description and purpose. We have chosen to configure each node as a GPFS quorum manager. Each node is a GPFS server. If you are unsure of how many quorum managers and GPFS servers are required in your environment, refer to the GPFS Concepts, Planning, and Installation document for guidance.
aixlpar1 : /tmp/cg # cat gpfs-nodes.txt
aixlpar2p:quorum-manager:
aixlpar1p:quorum-manager:
The cluster is created using the mmcrcluster command.* The GPFS cluster name is GPFS_CLUSTER1. The primary node (orNSD server; discussed in the next section) is aixlpar1p and the secondary node is aixlpar2p. We have specified that ssh andscp will be used for cluster communication and administration.
aixlpar1 : /tmp/cg # mmcrcluster –C GPFS_CLUSTER1 -N /tmp/cg/gpfs-nodes.txt -p 
aixlpar1p -s aixlpar2p -r /usr/bin/ssh -R /usr/bin/scp
Mon Apr 29 12:01:21 EET 2013: mmcrcluster: Processing node aixlpar2
Mon Apr 29 12:01:24 EET 2013: mmcrcluster: Processing node aixlpar1
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: Propagating the cluster configuration data to all
affected nodes.  This is an asynchronous process.

*Note: To ensure that GPFS daemon communication occurs over the private GPFS network, during cluster creation, we specified the GPFS daemon node names (that is, host names ending with p). There are two types of communication to consider in a GPFS cluster, administrative commands and daemon communication. Administrative commands use remote shell (ssh, rsh, or other) and socket-based communications. It is considered as a best practice to ensure that all GPFS daemon communication is performed over a private network. Refer to the GPFS developerWorks wiki for further information and discussion on GPFS network configuration considerations and practices.

To use a separate network for administration command communication, you can change the "Admin node name" using the mmchnode command. In this example, the separate network address is designated by "a" (for Administration) at the end of the node name, aixlpar1a for example.
# mmchnode -admin-interface=aixlpar1p -N aixlpar1a
# mmchnode -admin-interface=aixlpar2p -N aixlpar2a
The mmcrcluster command warned us that not all nodes have the appropriate GPFS license designation. We use the mmchlicense command to assign a GPFS server license to both the nodes in the cluster.
aixlpar1 : / # mmchlicense server --accept  -N aixlpar1a,aixlpar2a
The following nodes will be designated as possessing GPFS server licenses:
aixlpar2a
aixlpar1a
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
The cluster is now configured. The mmlscluster command can be used to display cluster information.
# mmlscluster
GPFS cluster information
========================
  GPFS cluster name:         GPFS_CLUSTER1.aixlpar1p
  GPFS cluster id:           8831612751005471855
  GPFS UID domain:           GPFS_CLUSTER.aixlpar1p
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
  Primary server:    aixlpar1p
  Secondary server:  aixlpar2p
 Node  Daemon node name  IP address      Admin node name  Designation
----------------------------------------------------------------------
   1   aixlpar2p       10.1.7.120   aixlpar2a      quorum-manager
   2   aixlpar1p       10.1.7.110   aixlpar1a      quorum-manager
At this point, you can use the mmdsh command to verify that the SSH communication is working as expected on all GPFS nodes. This runs a command on all the nodes in the cluster. If there is an SSH configuration problem, this command highlights the issues.
aixlpar1 : / # mmdsh date
aixlpar1:  Mon Apr 29 12:05:47 EET 2013
aixlpar2:  Mon Apr 29 12:05:47 EET 2013
aixlpar2 : / # mmdsh date
aixlpar1:  Mon Apr 29 12:06:41 EET 2013
aixlpar2:  Mon Apr 29 12:06:41 EET 2013

Configuring Network Shared Disks

GPFS provides a block-level interface over TCP/IP networks called the Network Shared Disk (NSD) protocol. Whether using the NSD protocol or a direct attachment to the SAN, the mounted file system looks the same to the users and application (GPFS transparently handles I/O requests).

A shared disk cluster is the most basic environment. In this configuration, the storage is directly attached to all the systems in the cluster. The direct connection means that each shared block device is available concurrently to all of the nodes in the GPFS cluster. Direct access means that the storage is accessible using a Small Computer System Interface (SCSI) or other block-level protocol using a SAN.

The following figure illustrates a GPFS cluster where all nodes are connected to a common Fibre Channel SAN and storage device. The nodes are connected to the storage using the SAN and to each other using a local area network (LAN). Data used by applications running on the GPFS nodes flows over the SAN, and GPFS control information flows among the GPFS instances in the cluster over the LAN. This configuration is optimal when all nodes in the cluster need the highest performance access to the data.

Figure 3. Overview diagram of the GPFS cluster


The mmcrnsd command is used to create NSD devices for GPFS. First, we create a text file that contains a list of each of the hdisk names, their GPFS designation (data, metadata, both*), and the NSD name.
hdisk1:::dataAndMetadata::nsd01::
hdisk2:::dataAndMetadata::nsd02::
hdisk3:::dataAndMetadata::nsd03::
hdisk4:::dataAndMetadata::nsd04::
hdisk5:::dataAndMetadata::nsd05::
hdisk6:::dataAndMetadata::nsd06::
Note:Here you can specify the nodes in second field , if you want to limit the nsd disks  
to be created on specific nodes.Like  
hdisk1:aixlpar1p,aixlpar3p::dataAndMetadata::nsd01::

*Note: Refer to the GPFS Concepts, Planning, and Installation document for guidance on selecting NSD device usage types.

Then, run the mmcrnsd command to create the NSD devices.
# mmcrnsd -F /tmp/cg/gpfs-disks.txt
mmcrnsd: Processing disk hdisk1
mmcrnsd: Processing disk hdisk2
mmcrnsd: Processing disk hdisk3
mmcrnsd: Processing disk hdisk4
mmcrnsd: Processing disk hdisk5
mmcrnsd: Processing disk hdisk6
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
The lspv command now shows the NSD name associated with each AIX hdisk.
# lspv
hdisk0          00c334b6af00e77b                    rootvg          active
hdisk1          none                                nsd01
hdisk2          none                                nsd02
hdisk3          none                                nsd03
hdisk4          none                                nsd04
hdisk5          none                                nsd05
hdisk6          none                                nsd06
The mmlsnsd command displays information for each NSD, in particular which GPFS file system is associated with each device. At this point, we have not created a GPFS file system. So each disk is currently free. You'll notice that under NSD servers each device is shown as directly attached. This is expected for SAN-attached disks.
# mmlsnsd
 File system   Disk name    NSD servers
---------------------------------------------------------------------------
 (free disk)   nsd01        (directly attached)
 (free disk)   nsd02        (directly attached)
 (free disk)   nsd03        (directly attached)
 (free disk)   nsd04        (directly attached)
 (free disk)   nsd05        (directly attached)
 (free disk)   nsd06        (directly attached)

GPFS file system configuration

Next, the GPFS file systems can be configured. The mmcrfs command is used to create the file systems. We have chosen to create two file systems; /gpfs and /gpfs1. The /gpfs (gpfs0) file system will be configured with a GPFS block size of 256K (the default) and /gpfs1 (gpfs1) with a block size of 1M*. Both file systems are configured for replication (-M2 –R2). The /tmp/cg/gpfs-disk.txt file is specified for /gpfs and /tmp/cg/gpfs1-disk.txt for /gpfs1. These files specify which NSD devices are used for each file system during creation.

*Note: Choose your block size carefully. It is not possible to change this value after the GPFS device has been created.
# cat /tmp/cg/gpfs-disk.txt
nsd01:::dataAndMetadata:-1::system
nsd02:::dataAndMetadata:-1::system
nsd03:::dataAndMetadata:-1::system
# cat /tmp/cg/gpfs1-disk.txt
nsd04:::dataAndMetadata:-1::system
nsd05:::dataAndMetadata:-1::system
nsd06:::dataAndMetadata:-1::system
# mmcrfs /gpfs gpfs0 -F/tmp/cg/gpfs-disks.txt -M2 -R 2
# mmcrfs /gpfs1 gpfs1 -F/tmp/cg/gpfs1-disks.txt -M2 -R 2 –B 1M
The mmlsnsd command displays the NSD configuration per file system. NSD devices 1 to 3 are assigned to the gpfs0 device and devices 4 to 6 are assigned to gpfs1.
# mmlsnsd
 File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs0         nsd01        (directly attached)
 gpfs0         nsd02        (directly attached)
 gpfs0         nsd03        (directly attached)
 gpfs1         nsd04        (directly attached)
 gpfs1         nsd05        (directly attached)
 gpfs1         nsd06        (directly attached)
Both GPFS file systems are now available on both nodes.
aixlpar1 : / # df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           1.00      0.89   12%     5211     3% /
/dev/hd2           3.31      0.96   71%    53415    18% /usr
/dev/hd9var        2.00      1.70   16%     5831     2% /var
/dev/hd3           2.00      1.36   33%      177     1% /tmp
/dev/hd1           2.00      2.00    1%      219     1% /home
/proc                 -         -    -         -     -  /proc
/dev/hd10opt       1.00      0.79   21%     3693     2% /opt
/dev/local         1.00      0.97    3%      333     1% /usr/local
/dev/loglv         1.00      1.00    1%       54     1% /var/log
/dev/tsmlog        1.00      1.00    1%        7     1% /var/tsm/log
/dev/hd11admin      0.12      0.12    1%       13     1% /admin
/dev/optIBMlv      2.00      1.99    1%       17     1% /opt/IBM
/dev/gpfs1       150.00    147.69    2%     4041     3% /gpfs1
/dev/gpfs0       150.00    147.81    2%     4041     7% /gpfs
The mmdsh command can be used here to quickly check the file system status on all the nodes.
aixlpar1 : / # mmdsh df -g | grep gpfs
aixlpar2:  /dev/gpfs0       150.00    147.81    2%     4041     7% /gpfs
aixlpar2:  /dev/gpfs1       150.00    147.69    2%     4041     3% /gpfs1
aixlpar1:  /dev/gpfs1       150.00    147.69    2%     4041     3% /gpfs1
aixlpar1:  /dev/gpfs0       150.00    147.81    2%     4041     7% /gpfs
If more detailed information is required, the mmdf command can be used.
aixlpar1 : /gpfs # mmdf gpfs0 --block-size=auto
disk      disk size  failure holds    holds                 free                free
name      group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- ------------
Disks in storage pool: system (Maximum disk size allowed is 422 GB)
nsd01       50G       -1 yes      yes          49.27G ( 99%)          872K ( 0%)
nsd02       50G       -1 yes      yes          49.27G ( 99%)          936K ( 0%)
nsd03       50G       -1 yes      yes          49.27G ( 99%)          696K ( 0%)
           -------------                       -------------------- -------------------
(pool total) 150G                              147.8G ( 99%)        2.445M ( 0%)
           =============                       ==================== ===================
(total)          150G                          147.8G ( 99%)        2.445M ( 0%)
Inode Information
-----------------
Number of used inodes:            4040
Number of free inodes:           62008
Number of allocated inodes:      66048
Maximum number of inodes:        66048
aixlpar1 : /gpfs # mmdf gpfs1 --block-size=auto
disk       disk size  failure holds   holds             free            free
name       group metadata data        in full blocks    in fragments
--------------- ------------- -------- -------- ----- -------------------- 
Disks in storage pool: system (Maximum disk size allowed is 784 GB)
nsd04        50G       -1 yes      yes          49.55G ( 99%)        1.938M ( 00%)
nsd05        50G       -1 yes      yes          49.56G ( 99%)          992K ( 0%)
nsd06        50G       -1 yes      yes          49.56G ( 99%)        1.906M ( 00%)
          -------------                         -------------------- -------------------
(pool total) 150G                               148.7G ( 99%)        4.812M ( 00%)
          =============                         ==================== ===================
(total)           150G                          148.7G ( 99%)        4.812M ( 00%)
Inode Information
-----------------
Number of used inodes:            4040
Number of free inodes:          155704
Number of allocated inodes:     159744
Maximum number of inodes:       159744

Node quorum with tiebreaker disks

Tiebreaker disks are recommended when you have a two-node cluster or you have a cluster where all of the nodes are SAN-attached to a common set of logical unit numbers (LUNs) and you want to continue to serve data with a single surviving node. Typically, tiebreaker disks are only used in two-node clusters. Tiebreaker disks are not special NSDs; you can use any NSD as a tiebreaker disk.

In this example, we chose three (out of six) NSD devices as tiebreaker disks. We stopped GPFS on all nodes and configured the cluster accordingly.
# mmshutdown -N
# mmchconfig tiebreakerDisks="nsd01;nsd03;nsd05"
# mmstartup -a

Cluster daemon status

There are two GPFS daemons (processes) that remain active while GPFS is active (mmfsd64 and runmmfs).
# ps -ef | grep mmfs
root 4784176 5505220 0   May 20 - 0:27 /usr/lpp/mmfs/bin/aix64/mmfsd64
root 5505220       1 0   May 20 - 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs
You can use the mmgetstate command to view the status of the GPFS daemons on all the nodes in the cluster.
# mmgetstate -aLs
 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state  Remarks
------------------------------------------------------------------------------------
       1      aixlpar2a       1*        2          2       active      quorum node
       2      aixlpar1a       1*        2          2       active      quorum node
 Summary information
---------------------
Number of nodes defined in the cluster:            2
Number of local nodes active in the cluster:       2
Number of remote nodes joined in this cluster:     0
Number of quorum nodes defined in the cluster:     2
Number of quorum nodes active in the cluster:      2
Quorum = 1*, Quorum achieved

Thursday, 4 July 2013

HPSS for GPFS

HPSS:

The High Performance Storage System (HPSS) is IBM's highly scalable Hierarchical Storage Management (HSM) System. HPSS is intended to be used by IBM's high-end HPC customers, with storage requirements in the tens of millions, hundreds of millions, and even the billion file range. HPSS is capable of concurrently accessing hundreds of tapes for extremely high aggregate data transfer rates, and can easily meet otherwise unachievable total storage bandwidth and capacity requirements. HPSS can stripe files across multiple tapes, which ensures high bandwidth data transfers of huge files. HPSS provides for stewardship and access of many petabytes of data stored on robotic tape libraries.
HPSS Scales

GPFS:

The IBM General Parallel File System (GPFS) is a true distributed, clustered file system. Multiple servers are used to manage the data and metadata of a single file system. Individual files are broken into multiple blocks and striped across multiple disks, and multiple servers, which eliminates bottlenecks. Information Lifecycle Management (ILM) policy scans are also distributed across multiple servers, which allows GPFS to quickly scan the entire file system, identifying files that match a specific criteria. Shortly after we showed the Billion File Demo at the international conference on high performance computing (SC07), the Almaden Research Center showed that a pre-GA version of GPFS is capable of scanning a single GPFS file system, containing a billion files, in less than 15 minutes!
GPFS Scales
GPFS/HPSS Interface (GHI):

HPSS can now be used to automatically HSM manage GPFS disk resources. GPFS customers can now store petabytes of data on a file system with terabytes of high performance disks. HPSS can also be used to capture a point-in-time backup of your GPFS cluster and file systems. In the event of a catastrophic failure, HPSS can be used to restore your GPFS cluster and file systems. The GPFS high performance ILM policy scans are used to:
  • Identify new files, or files that changed, so the data can be migrated to tape;
  • Identify older, unused files that no longer need to remain on disk;
  • Identify files that users need to bulk-stage back to GPFS for future processing;
  • Capture GPFS cluster information; and
  • Capture GPFS file system structure and file attributes.
GPFS + HPSS:

The ILM policy scan results are sent to the Scheduler. The Scheduler distributes the work to the I/O Managers (IOM), and the GPFS data are copied to HPSS in parallel. For those files that are no longer active, holes are then punched into GPFS files to free up GPFS disk resources. The continuous movement of GPFS files to HPSS tape, and the freeing of GPFS disk resources is an automated process that is transparent to the GPFS user. If the GPFS user should access a file that is only on HPSS tape, the file will automatically stage back to GPFS, so the user can access the file.

The GPFS/HPSS Interface continuously copies GPFS file data to HPSS. When the time comes to perform a backup, only those files that have not yet been copied to HPSS tape are migrated. Therefore, it is NOT necessary to recapture all of the file data at each backup.

GHI to HPSS
Small File Aggregation:

Most GPFS file systems are made up of small files about 90% of the files use 10% of the disk resources. Traditionally, copying small files to tape diminishes your tape drive performance. The GPFS/HPSS Interface copies small files from GPFS to HPSS by grouping many small files into much larger aggregates. Small file aggregation is completely configurable, but 10,000 GPFS files were placed into each HPSS aggregate at the SC07 Billion File Demo. Large aggregates allow data to stream to the tape drive, which yields higher tape transfer rates.
Aggregation


That's why we say...
GPFS + HPSS = Extreme Storage Scalability!

Tuesday, 21 May 2013

Upgrading the GPFS cluster on AIX

Upgrading the GPFS cluster on AIX

  • All the GPFS nodes should be upgrade at same time.
  • Make sure the Application is stopped fully.
  • Before starting with OS Upgrade all the GPFS file system should be unmounted. If there are any application process running, and those process using the GPFS file systems, we cannot unmount the GPFS file systems.
  • Before the OS upgrade starts, the GPFS cluster should be stopped.

1) View the cluster information

Before starting the OS Upgrade starts complete below steps.
# mmlscluster                                  
Example output:
GPFS cluster information
========================
  GPFS cluster name:         HOST.test1-gpfs
  GPFS cluster id:           13882565243868289165
  GPFS UID domain:           HOST.test1-gpfs
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
  Primary server:    test1-gpfs
  Secondary server:  test2-gpfs
Node  Daemon node name            IP address       Admin node name             Designation
-----------------------------------------------------------------------------------------------
   1   test1-gpfs            192.168.199.137  test1-gpfs            quorum
   2   test3-gpfs            192.168.199.138  test3-gpfs            quorum
   3   test2-gpfs            192.168.199.139  test2-gpfs            quorum

2) View all the gpfs file systems

# mmlsfs all                                    
Example output:

File system attributes for /dev/gpfs1001:
=========================================
flag value            description
---- ---------------- -----------------------------------------------------
-f  131072           Minimum fragment size in bytes
-i  512              Inode size in bytes
-I  32768            Indirect block size in bytes
-m  1                Default number of metadata replicas
-M  2                Maximum number of metadata replicas
-r  1                Default number of data replicas
-R  2                Maximum number of data replicas
-j  cluster          Block allocation type
-D  nfs4             File locking semantics in effect
-k  all              ACL semantics in effect
-a  -1               Estimated average file size
-n  64               Estimated number of nodes that will mount file system
-B  4194304          Block size
-Q  user;group;fileset Quotas enforced
     none             Default quotas enabled
-F  1000000          Maximum number of inodes
-V  10.01 (3.2.1.5)  File system version
-u  yes              Support for large LUNs?
-z  no               Is DMAPI enabled?
-L  4194304          Logfile size
-E  yes              Exact mtime mount option
-S  no               Suppress atime mount option
-K  whenpossible     Strict replica allocation option
-P  system           Disk storage pools in file system
-d  gpfs1nsd;gpfs2nsd;gpfs3nsd;gpfs4nsd  Disks in file system
-A  yes              Automatic mount option
-o  none             Additional mount options
-T  /sasmart         Default mount point
File system attributes for /dev/gpfs1002:
=========================================
flag value            description
---- ---------------- -----------------------------------------------------
-f  131072           Minimum fragment size in bytes
-i  512              Inode size in bytes
-I  32768            Indirect block size in bytes
-m  1                Default number of metadata replicas
Standard input -M  2                Maximum number of metadata replicas
-r  1                Default number of data replicas
-R  2                Maximum number of data replicas
-j  cluster          Block allocation type
-D  nfs4             File locking semantics in effect
-k  all              ACL semantics in effect
-a  -1               Estimated average file size
-n  64               Estimated number of nodes that will mount file system
-B  4194304          Block size
-Q  user;group;fileset Quotas enforced
     none             Default quotas enabled
-F  1000000          Maximum number of inodes
-V  10.01 (3.2.1.5)  File system version
-u  yes              Support for large LUNs?
-z  no               Is DMAPI enabled?
-L  4194304          Logfile size
-E  yes              Exact mtime mount option
-S  no               Suppress atime mount option
-K  whenpossible     Strict replica allocation option
-P  system           Disk storage pools in file system
-d  gpfs5nsd       Disks in file system
-A  yes              Automatic mount option
-o  none             Additional mount options
-T  /sasplex1        Default mount point
File system attributes for /dev/gpfs1003:
=========================================
flag value            description
---- ---------------- -----------------------------------------------------
-f  131072           Minimum fragment size in bytes
-i  512              Inode size in bytes
-I  32768            Indirect block size in bytes
-m  1                Default number of metadata replicas
-M  2                Maximum number of metadata replicas
-r  1                Default number of data replicas
-R  2                Maximum number of data replicas
-j  scatter          Block allocation type
-D  nfs4             File locking semantics in effect
-k  all              ACL semantics in effect
-a  -1               Estimated average file size
-n  64               Estimated number of nodes that will mount file system
-B  4194304          Block size
Standard input -Q  user;group;fileset Quotas enforced
     none             Default quotas enabled
-F  1000000          Maximum number of inodes
-V  10.01 (3.2.1.5)  File system version
-u  yes              Support for large LUNs?
-z  no               Is DMAPI enabled?
-L  4194304          Logfile size
-E  yes              Exact mtime mount option
-S  no               Suppress atime mount option
-K  whenpossible     Strict replica allocation option
-P  system           Disk storage pools in file system
-d gpfs6nsd;gpfs7nsd;gpfs8nsd;gpfs9nsd;gpfs10nsd;gpfs11nsd;gpfs12nsd;gpfs13nsd;gpfs14nsd;gpfs15nsd;gpfs16nsd;gpfs17nsd;gpfs1
8nsd;gpfs19nsd;gpfs20nsd;gpfs21nsd;gpfs22nsd;
-d
gpfs23nsd;gpfs24nsd;gpfs25nsd;gpfs26nsd;gpfs27nsd;gpfs28nsd;gpfs29nsd;gpfs30nsd;gpfs31nsd;gpfs32nsd;gpfs33nsd;gpfs34nsd;g
pfs35nsd;gpfs36nsd;gpfs37nsd;gpfs38nsd;gpfs39nsd;
-d
gpfs40nsd;gpfs41nsd;gpfs42nsd;gpfs43nsd;gpfs44nsd;gpfs45nsd;gpfs46nsd;gpfs47nsd;gpfs48nsd;gpfs49nsd;gpfs50nsd;gpfs51nsd;g
pfs52nsd;gpfs53nsd;gpfs54nsd;gpfs55nsd;gpfs56nsd;
-d
gpfs57nsd;gpfs58nsd;gpfs59nsd;gpfs60nsd;gpfs61nsd;gpfs62nsd;gpfs63nsd;gpfs64nsd;gpfs65nsd;gpfs66nsd;gpfs67nsd;gpfs68nsd;g
pfs69nsd  Disks in file system
-A  yes              Automatic mount option
-o  none             Additional mount options
-T  /app1            Default mount point
File system attributes for /dev/gpfs1004:
=========================================
flag value            description
---- ---------------- -----------------------------------------------------
-f  131072           Minimum fragment size in bytes
-i  512              Inode size in bytes
-I  32768            Indirect block size in bytes
-m  1                Default number of metadata replicas
-M  2                Maximum number of metadata replicas
-r  1                Default number of data replicas
-R  2                Maximum number of data replicas
-j  cluster          Block allocation type
-D  nfs4             File locking semantics in effect
-k  all              ACL semantics in effect
-a  -1               Estimated average file size
-n  64               Estimated number of nodes that will mount file system
-B  4194304          Block size
-Q  user;group;fileset Quotas enforced
     none             Default quotas enabled
Standard input -F  1000000          Maximum number of inodes
-V  10.01 (3.2.1.5)  File system version
-u  yes              Support for large LUNs?
-z  no               Is DMAPI enabled?
-L  4194304          Logfile size
-E  yes              Exact mtime mount option
-S  no               Suppress atime mount option
-K  whenpossible     Strict replica allocation option
-P  system           Disk storage pools in file system
-d  gpfs70nsd      Disks in file system
-A  yes              Automatic mount option
-o  none             Additional mount options
-T  /sasuserhome     Default mount point

3) View the gpfs filesystem mounted on number of nodes

# mmlsmount all                                          
Example output:
File system gpfs1001 is mounted on 3 nodes.
File system gpfs1002 is mounted on 3 nodes.
File system gpfs1003 is mounted on 3 nodes.
File system gpfs1004 is mounted on 3 nodes.
Standard input: END

4) Check the existing gpfs cluster version

# lslpp -l |grep -i gpfs
Example output:
  gpfs.base                 3.2.1.18  APPLIED    GPFS File Manager
  gpfs.msg.en_US            3.2.1.11  APPLIED    GPFS Server Messages - U.S.
  gpfs.base                 3.2.1.18  APPLIED    GPFS File Manager
  gpfs.docs.data             3.2.1.1  APPLIED    GPFS Server Manpages and

5) unmount all gpfs filesystems

# mmumount all -N test1-gpfs,test3-gpfs,test2-gpfs
Example output:
Wed May 11 00:05:35 CDT 2011: 6027-1674 mmumount: Unmounting file systems ...

6) Verify all gpfs file system are un mounted

# mmlsmount all                            
Example output:
File system gpfs1001 is not mounted.
File system gpfs1002 is not mounted.
File system gpfs1003 is not mounted.
File system gpfs1004 is not mounted.

7) Stop the gpfs cluster

# mmshutdown -a                      
Example output:
Wed May 11 00:08:22 CDT 2011: 6027-1341 mmshutdown: Starting force unmount of GPFS file systems
Wed May 11 00:08:27 CDT 2011: 6027-1344 mmshutdown: Shutting down GPFS daemons
test3-gpfs:  Shutting down!
test2-gpfs:  Shutting down!
test3-gpfs:  'shutdown' command about to kill process 516190
test2-gpfs:  'shutdown' command about to kill process 483444
test1-gpfs:  Shutting down!
test1-gpfs:  'shutdown' command about to kill process 524420
test1-gpfs:  Master did not clean up; attempting cleanup now
test1-gpfs:  Wed May 11 00:09:28.423 2011: GPFS: 6027-311 mmfsd64 is shutting down.
test1-gpfs:  Wed May 11 00:09:28.424 2011: Reason for shutdown: mmfsadm shutdown command timed out
test1-gpfs:  Wed May 11 00:09:28 CDT 2011: mmcommon mmfsdown invoked.  Subsystem: mmfs  Status: down
test1-gpfs:  Wed May 11 00:09:28 CDT 2011: 6027-1674 mmcommon: Unmounting file systems ...
Wed May 11 00:09:33 CDT 2011: 6027-1345 mmshutdown: Finished

8) Verify any cluster process running

# ps -ef|grep -i gpfs                      
After GPFS cluster is stopped, then proceed with patching/upgrade.
Once the OS patching/upgrade complete, then upgrade the GPFS .Make sure the GPFS file systems are not in mounted state.

9) mount the NIM directory on /mnt

# mount wydainim010:/export/ibm _lpp /mnt

10) Change dir to the GPFS new version filesets location

# /mnt/gpfs/3.3/3.3.0.12                           

11) Use update_all cmd to update the file sets do preview only

# smitty update _all                                    
Example output:
INPUT device / directory for software                  .
* SOFTWARE to update_update_allPREVIEW only?
(update operation will NOT occur)                        yes       =====> Select yes for Preview
COMMIT software updates?                               no         =====> Select no for COMMIT filesets
SAVE replaced files?                                         yes       =====> Select yes here
AUTOMATICALLY install requisite software?       yes+
EXTEND file systems if space needed?              yes+
VERIFY install and check file sizes?                   no+
DETAILED output?                                            no+
Process multiple volumes?                                 yes+
ACCEPT new license agreements?                     yes       =====> Accept new licence agreement
Preview new LICENSE agreements?                   no+
If everything is fine in PREVIEW stage, proceed with upgrading the GPFS filesets bye selecting PREVIEW as no

12) Now verify the GPFS filesets version

# lslpp -l |grep -i gpfs              
Example output:
  gpfs.base                  3.3.0.8  APPLIED    GPFS File Manager
  gpfs.msg.en_US             3.3.0.5  APPLIED    GPFS Server Messages - U.S.
  gpfs.base                  3.3.0.8  APPLIED    GPFS File Manager
  gpfs.docs.data             3.3.0.1  APPLIED    GPFS Server Manpages and
Then continue with EMC upgrade and once the emc upgrade is done.
Make sure all the pv's are available in all nodes after the EMC upgrade.
Start the GPFS cluster.

13) Starts the GPFS cluster

# mmstartup -a                                            
Example output:
Wed May 11 06:09:32 CDT 2011: 6027-1642 mmstartup: Starting GPFS ...

14) Check the GPFS cluster

# mmlscluster                                
Example output:
GPFS cluster information
========================
  GPFS cluster name:         HOST.test1-gpfs
  GPFS cluster id:           13882565243868289165
  GPFS UID domain:           HOST.test1-gpfs
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
  Primary server:    test1-gpfs
  Secondary server:  test2-gpfs
Node  Daemon node name            IP address       Admin node name             Designation
-----------------------------------------------------------------------------------------------
   1   test1-gpfs            192.168.199.137  test1-gpfs            quorum
   2   test3-gpfs            192.168.199.138  test3-gpfs            quorum
   3   test2-gpfs            192.168.199.139  test2-gpfs            quorum

15) Check the GPFS cluster state on all nodes

# mmgetstate -a                                            
Example output:
Node number  Node name        GPFS state
------------------------------------------
       1      test1-gpfs active
       2      test3-gpfs active
       3      test2-gpfs active

16) Check all the filesystems

# mmlsfs all                                       

17) Mount all the gpfs fil systems

# mmmount all -a                       
Wed May 11 06:13:16 CDT 2011: 6027-1623 mmmount: Mounting file systems ...

18) Check the file systems are mounted on all nodes

# mmlsmount all                        
Example output:
File system gpfs1001 is mounted on 3 nodes.
File system gpfs1002 is mounted on 3 nodes.
File system gpfs1003 is mounted on 3 nodes.
File system gpfs1004 is mounted on 3 nodes.

19) Verify the GPFS cluster configuration information

# mmlsconfig                                
Example output:
Configuration data for cluster HOST.test1-gpfs:
----------------------------------------------------------
clusterName HOST.test1-gpfs
clusterId 13882565243868289165
clusterType lc
autoload yes
minReleaseLevel 3.2.1.5
dmapiFileHandleSize 32
maxblocksize 4096K
pagepool 1024M
maxFilesToCache 5000
maxStatCache 40000
maxMBpS 3200
prefetchPct 60
seqDiscardThreshhold 10240000000
worker1Threads 400
prefetchThreads 145
adminMode allToAll
File systems in cluster HOST.test1-gpfs:
---------------------------------------------------
/dev/gpfs1001
/dev/gpfs1002
/dev/gpfs1003
/dev/gpfs1004

Thursday, 9 May 2013

Adding a disk to GPFS filesystem Dynamically

Objectives:

* Add a disk to a storage pool online
* Re-balance existing data in the file system

Prerequisite:

1. GPFS cluster is up and running
2. GPFS File System is mounted.
3. Device to add is ready (hdiskxx)

Step 1: Add a disk to the existing file system

1. Verify that GPFS is running and the file system is mounted using the mmgetstate command and the df command

   a. The mmgetstate command will show the status of the nodes in the cluster.
   > mmgetstate -a
   b. The df command will display the mounted GPFS file system.
      df

2. Create a disk descriptor file /gpfs-to-extend/data/adddisk.txt for the new disk using the format

    #DiskName:serverlist::DiskUsage:FailureGroup:DesiredName:StoragePool
    hdiskxx:::dataOnly::nsd5:pool1

3. Use the mmcrnsd command to create the NSD

    > mmcrnsd -F /gpfs-to-extend/data/adddisk.txt

4. Verify the disk has been created using the mmlsnsd command

   > mmlsnsd

   The disk you just added should show as a (free disk)

5. Add the new NSD to the fs1 file system using the mmadddisk command

    > mmadddisk fs1 -F /gpfs-to-extend/data/adddisk.txt

6. Verify the NSD was added using the mmdf command

    > mmdf fs1

disk                disk size  failure holds    holds              free KB             free KB
name                    in KB    group metadata data        in full blocks        in fragments
————— ————- ——– ——– —– ——————– ——————-
Disks in storage pool: system
nsd1                 20971520       1 yes      yes        20873984 (100%)           284 ( 0%)
nsd2                 20971520       2 yes      yes        20873984 (100%)           202 ( 0%)
————-                         ——————– ——————-
(pool total)         41943040                              41747968 (100%)           486 ( 0%)
Disks in storage pool: pool1
nsd3                 20971520       1 no       yes        20969408 (100%)            62 ( 0%)
nsd4                 20971520       2 no       yes        20969408 (100%)            62 ( 0%)
nsd5                 20971520      -1 no       yes        20969408 (100%)            62 ( 0%)
————-                         ——————– ——————-
(pool total)         62914560                              62908224 (100%)           186 ( 0%)
=============                         ==================== ===================
(data)              104857600                             104656192 (100%)           672 ( 0%)
(metadata)           41943040                              41747968 (100%)           486 ( 0%)
=============                         ==================== ===================
(total)             104857600                             104656192 (100%)           672 ( 0%)
Inode Information
—————–
Number of used inodes:            4045
Number of free inodes:           78131
Number of allocated inodes:      82176
Maximum number of inodes:        82176

Step 2: Re-balancing the data

In some cases you may wish to have GPFS re-balance existing data over the new disks that were added to the file system. Often it is not necessary to manually re-balance the data across the new disks. New data that is added to the file system is correctly striped. Re-striping a large file system requires a large number of insert and delete operations and may affect system performance. Plan to perform this task when system demand is low.

1. To re-balance the existing data in the file system use the mmrestripefs command.
     > mmrestripefs fs1 -b
2. Use the mmdf command to view the utilization of each disk.

Wednesday, 17 April 2013

General Parallel File System (GPFS)–quick Sheet

Below you'll find a complete list of all the GPFS commands, the same are located under /usr/lpp/mmfs/bin, enjoy!

Managing the cluster

CommandDescription
mmcrclusterCreates GPFS Cluster

Usage: mmcrfs Device {?DiskDesc[;DiskDesc...]? | -F DescFile} [-A {yes | no | automount}] [-D {nfs4 | posix}] [-B BlockSize] [-E {yes | no}] [-j {cluster | scatter}] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-L LogFileSize] [-m DefaultMetadataReplicas] [-M MaxMetadataReplicas] [-n NumNodes] [-N NumInodes[:NumInodesToPreallocate]] [-Q {yes | no}] [-r DefaultDataReplicas] [-R MaxDataReplicas] [-S {yes | no}] [-t DriveLetter] [-T MountPoint] [-v {yes | no}] [-z {yes | no}] [--version Version]
mmlsclusterDisplays GPFS cluster configuration information

Usage: mmlscluster [--cnfs]
mmlsconfigDisplays the configuration data for a GPFS cluster

Usage: mmlsconfig <attribute>
mmgetstateDisplays the state of the GPFS daemon on one or more nodes

Usage: mmgetstate [-L] [-s] [-v] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmaddnodeAdds nodes to a GPFS cluster

Usage: mmaddnode -N {NodeDesc[,NodeDesc...] | NodeFile}
mmchlicenseDesignates appropriate GPFS licenses

Usage: mmchlicense {server | client} [--accept] -N {Node[,Node...] | NodeFile | NodeClass}
mmlslicenseDisplays information about the GPFS node licensing designation.

Usage: mmlslicense [-L]
mmchnodeChanges node attributes

Usage: mmchnode change-options -N{Node[,Node...] | NodeFile | NodeClass} Or,
Usage: mmchnode {-S Filename | --spec-file=Filename}
mmdelnodeDeletes nodes from a GPFS cluster

Usage: mmdelnode {-a | -N Node[,Node...] | NodeFile | NodeClass}
mmchcluster or mmchconfigChanges the GPFS cluster configuration data
mmlsmgrDisplays the file system manager node

Usage: mmlsmgr [Device [Device...]] Or,Usage: mmlsmgr -C ClusterName Or,Usage: mmlsmgr -c
mmchmgrChanges the file system manager node

Usage: mmchmgr {Device | -c} [Node]
mmstartupStarts GPFS cluster

Usage: mmstartup [-a | -N {Node[,Node...] | NodeFile | NodeClass}] [-E EnvVar=value ...]
mmshutdownStops GPFS cluster

Usage: mmshutdown [-t UnmountTimeout ] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmrefreshPlaces the most recent GPFS cluster configuration data files on the specified nodes
mmsdrrestoreRestores the latest GPFS system files on the specified nodes.
mmsdrbackupPerforms a backup of the GPFS configuration data.
mmbackupconfigBacking up file system configuration information

Usage: mmbackupconfig Device -o OutputFile
mmrestoreconfigRestores file system configuration information

Usage: mmrestoreconfig Device -i InputFile [-I {test | yes | continue }] [-Q {yes | no}] [-W NewDeviceName] [-z {yes | no}] Or,Usage:mmrestoreconfig Device -i InputFile -F configFile Or,
Usage: mmrestoreconfig Device -i InputFile -I continue

Managing file systems

CommandDescription
mmcrfsCreates a file system

Usage: mmcrfs Device {?DiskDesc[;DiskDesc...]? | -F DescFile} [-A {yes | no | automount}] [-D {nfs4 | posix}] [-B BlockSize] [-E {yes | no}] [-j {cluster | scatter}] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-L LogFileSize] [-m DefaultMetadataReplicas] [-M MaxMetadataReplicas] [-n NumNodes] [-N NumInodes[:NumInodesToPreallocate]] [-Q {yes | no}] [-r DefaultDataReplicas] [-R MaxDataReplicas] [-S {yes | no}] [-t DriveLetter] [-T MountPoint] [-v {yes | no}] [-z {yes | no}] [--version Version]
mmmountMounts a file system

Usage: mmmount {Device | DefaultMountPoint | DefaultDriveLetter | all | all_local | all_remote} [-o MountOptions] [-a | -N {Node[,Node...] | NodeFile | NodeClass}] Or,Usage: mmmount Device {MountPoint | DriveLetter } [-o MountOptions] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmumountUmounts a file system

Usage: mmumount {Device | MountPoint | DriveLetter | all | all_local | all_remote} [-f ] [-a | -N {Node[,Node...]| NodeFile | NodeClass}] Or, 
Usage: mmumount Device -f -C {all_remote | ClusterName} [-N Node[,Node...]]
mmdelfsDeletes a file system

Usage: mmdelfs Device [-p]
mmdfQueries available file space on a GPFS file system

Usage: mmdf Device [-d | -F | -m] [-P PoolName]
mmlsmountDetermines which nodes have a file system mounted

Usage: mmlsmount {Device | all | all_local | all_remote} [-L ] [-C {all | all_remote | ClusterName[,ClusterName...] } ]
mmfsckChecks and repairs a file system

Usage: mmfsck Device [-n | -y] [-c | -o] [-t Directory] [ -v | -V] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmlsfsListing file system attributes

Usage: mmlsfs {Device | all | all_local | all_remote} [-A] [-a] [-B] [-D] [-d] [-E] [-F] [-f] [-I] [-i] [-j] [-k] [-K][-L] [-M] [-m] [-n] [-o] [-P] [-Q] [-R] [-r] [-S] [-t] [-T] [-u] [-V] [-z]
mmchfsModifies file system attributes

Usage: mmchfs Device [-A {yes | no | automount}] [-E {yes | no}] [-D {nfs4 | posix}] [-F MaxNumInodes[:NumInodesToPreallocate]] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-m DefaultMetadataReplicas] [-n NumberOfNodes] [-o MountOptions] [-Q {yes | no}] [-r DefaultDataReplicas] [-S {yes | no} ] [-T Mountpoint] [-t DriveLetter] [-V {full | compat}] [-z {yes | no}] Or, 
Usage: mmchfs Device [-W NewDeviceName]
mmlsattrQuerying file replication attributes

Usage: mmlsattr [-l] [-L] FileName [FileName...]
mmchattrChanging file replication attributes

Usage: mmchattr [-m MetadataReplicas] [-M MaxMetadataReplicas] [-r DataReplicas] [-R MaxDataReplicas] [-P DataPoolName] [-D {yes | no} ] [-I {yes | defer}] [-i {yes | no}] [-l] Filename [Filename...]
mmrestripefsRestriping a GPFS file system

Usage: mmrestripefile {-b | -m | -p | -r} {-F FilenameFile | Filename [Filename...]}
mmdefragfsQuerying and reducing file system fragmentation

Usage: mmdefragfs Device [-i] [-u BlkUtilPct] [-P PoolName] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmbackupBacking up a file system

Usage: mmbackup {Device | Directory} [-f] [-g GlobalWorkDirectory] [-N {Node[,Node...] | NodeFile | NodeClass}] [-s LocalWorkDirectory] [-S SnapshotName] [-t {full | incremental}]

Managing snapshots

CommandDescription
mmcrsnapshotCreates a snapshot of an entire GPFS file system at a single point in time

Usage: mmcrsnapshot Device Directory
mmdelsnapshotDeletes a GPFS snapshot

Usage: mmdelsnapshot Device Directory [-N {Node [,Node...] | NodeFile | NodeClass}]
mmrestorefsRestores a file system from a GPFS snapshot

Usage: mmrestorefs Device Directory [-c]
mmlssnapshotDisplays GPFS snapshot information for the specified file system

Usage: mmlssnapshot Device [-d] [-Q]
mmsnapdirCreates and deletes invisible directories that connect to the snapshots of a GPFS file system, and changes the name of the snapshots subdirectory

Usage: mmsnapdir Device {[-r | -a] [-s SnapDirName]} Or,
Usage: mmsnapdir Device [-q]

Managing filesets

CommandDescription
mmcrfilesetCreates a GPFS fileset

Usage: mmcrfileset Device FilesetName [-t Comment]
mmchfilesetChanges the attributes of a GPFS fileset

Usage: mmchfileset Device {FilesetName | -J JunctionPath} {[-j NewFilesetName] | [-t NewComment]}
mmlsfilesetDisplays status and attributes of GPFS filesets

Usage: mmlsfileset Device {[[Fileset [,Fileset ...]] [-J Junction [,Junction ...]] | -F FileName} [-L] [-d] [-i]
mmdelfilesetDeletes a GPFS fileset

Usage: mmdelfileset Device FilesetName [-f]
mmrestripefileRebalances or restores the replication factor of the specified files

Usage: mmrestripefs Device {-b | -m | -p | -r | -R} [-N {Node[,Node...] | NodeFile | NodeClass}] [-P PoolName]
mmlinkfilesetCreates a junction that references the root directory of a GPFS fileset

Usage: mmlinkfileset Device FilesetName [-J JunctionPath]
mmunlinkfilesetRemoves the junction to a GPFS fileset

Usage: mmunlinkfileset Device {FilesetName | -J JunctionPath} [-f]

Managing disks

CommandDescription
mmcrnsdCreates network shared disk servers

Usage: mmcrnsd -F DescFile [-v {yes |no}]
mmlsnsdDisplaying disks in a GPFS cluster

Usage: mmlsnsd [-a | -F | -f Device | -d ?DiskName[;DiskName...]? ] [-L | -m | -M | -X] [-v]
mmdelnsdDeletes NSDs from the GPFS cluster

Usage: mmdelnsd {?DiskName[;DiskName...]? | -F DiskFile}
mmadddiskAdding disks to a file system

Usage: mmadddisk Device {?DiskDesc[;DiskDesc...]? | -F DescFile} [-a] [-r] [-v {yes | no}] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmdeldiskDeleting disks to a file system

Usage: mmdeldisk Device {?DiskName[;DiskName...]? | -F DescFile} [-a] [-c] [-m | -r | -b] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmrpldiskReplacing disks in a GPFS file system

Usage: mmrpldisk Device DiskName {DiskDesc | -F DescFile} [-v {yes | no}] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmlsdiskDisplaying GPFS disk states

Usage: mmlsdisk Device [-d ?DiskName[;DiskName...]?] [-e] [-L] Or,Usage: mmlsdisk Device [-d ?DiskName[;DiskName...]?] {-m | -M}
mmchdiskChanging GPFS disk states and parameters

Usage: mmchdisk Device {suspend | resume | stop | start | change} -d ?DiskDesc [;DiskDesc...]? | -F {DescFile} [-N {Node [,Node...] | NodeFile | NodeClass}] Or, 
Usage: mmchdisk Device {resume | start} -a [[-N {Node[,Node...] | NodeFile | NodeClass}]
mmchnsdChanging your NSD configuration

Usage: mmchnsd {?DiskDesc[;DiskDesc...]? | -F DescFile}
mmnsddiscoverRediscovers paths to the specified network shared disks

Usage: mmnsddiscover [-a | -d ?Disk[;Disk...]? | -F DiskFile] [-C ClusterName] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmcrvsdCreates virtual shared disks for use by GPFS

Usage: mmcrvsd [-f FanoutNumber] [-y] [-c] -F DescFile

Managing quotas

CommandDescription
mmcrfs or mmchfs -Q yesEnabling and disabling GPFS quota management
mmedquotaExplicitly establishing and changing quotas

Usage: mmedquota {-u [-p ProtoUser] User...| -g [-p ProtoGroup] Group... | -j [-p ProtoFileset ] Device:Fileset... | -d {-u User...| -g Group... | -j Device:Fileset...} | -t {-u | -g | -j}}
mmcheckquotaChecking quotas

Usage: mmcheckquota [-v] [-N {Node[,Node...] | NodeFile | NodeClass}] {Device [Device...] | -a} Or, 
Usage: mmcheckquota {-u UserQuotaFilename | -g GroupQuotaFileName | -j FilesetQuotaFilename} Device
mmdefquotaonEnabling default quota

Usage: mmdefquotaon [-u] [-g] [-j] [-v] [-d] {Device [Device... ] | -a}
mmdefedquotaDefine default quota

Usage: mmdefedquota {-u | -g | -j} Device
mmdefquotaoffDisable default quota

Usage: mmdefquotaoff [-u] [-g] [-j] [-v] {Device [Device...] | -a}
mmlsquotaListing quotas

Usage: mmlsquota [-u User | -g Group | -j Fileset] [-v | -q] [-e] [-C ClusterName] [Device [Device...]] Or, 
Usage: mmlsquota -d {-u | -g | -j} [-C ClusterName] [Device [Device...]]
mmquotaonActivating quota limit checking

Usage: mmquotaon [-u] [-g] [-j] [-v] {Device [Device...] | -a}
mmquotaoffDeactivating quota limit checking

Usage: mmquotaoff [-u] [-g] [-j] [-v] {Device [Device ...] | -a}
mmrepquotaCreating file system quota reports

Usage: mmrepquota [-e] [-g] [-q] [-u] [-n] [-v] [-j] {Device [Device...] | -a}

Managing ACLs

CommandDescription
mmputaclSetting traditional GPFS access control lists

Usage: mmputacl [-d] [-i InFilename] Filename
mmgetaclDisplaying traditional GPFS access control lists

Usage: mmgetacl [-d] [-o OutFilename] [-k {nfs4 |posix | native}] Filename
mmeditaclChanging traditional GPFS access control lists

Usage: mmeditacl [-d] [-k {nfs4 | posix | native}] Filename
mmdelaclDeleting traditional GPFS access control lists

Usage: mmdelacl [-d] Filename

Managing Remote Cluster

CommandDescription
mmauthManages secure access to GPFS file systems

Usage: mmauth genkey {new | commit | propagate [-N {Node[,Node...] | NodeFile | NodeClass}]} Or, 
Usage: mmauth add RemoteClusterName -k KeyFile -l CipherList Or, 
Usage: mmauth update RemoteClusterName -C NewClusterName -k KeyFile [-l CipherList] Or, 
Usage: mmauth delete {RemoteClusterName | all } Or,
Usage: mmauth grant {RemoteClusterName | all } -f { Device | all } [-a {rw | ro} ] [-r {uid:gid | no}] Or, 
Usage: mmauth deny {RemoteClusterName | all } -f { Device | all } Or,Usage: mmauth show [RemoteClusterName | all]
mmexportfsExports a file system into the cluster

Usage: mmexportfs {Device | all} -o ExportFilesysData
mmimportfsImports a file system into the cluster

Usage: mmimportfs {Device | all} -i ImportfsFile [-S ChangeSpecFile]
mmremoteclusterManages information about remote clusters

Usage: mmremotecluster add RemoteClusterName [-n ContactNodes] [-k KeyFile] Or, 
Usage: mmremotecluster update RemoteClusterName [-C NewClusterName] [-n ContactNodes] [-k KeyFile] Or,Usage:mmremotecluster delete {RemoteClusterName | all} Or,
Usage: mmremotecluster show [RemoteClusterName | all]
mmremotefsManages the information needed for mounting remote GPFS file systems

Usage: mmremotefs add Device -f RemoteDevice -C RemoteClusterName [-T MountPoint] [-t DriveLetter] [-A {yes | no | automount}] [-o MountOptions] Or, 
Usage: mmremotefs delete {Device | all | -C RemoteClusterName} Or, 
Usage: mmremotefs show [Device | all | -C RemoteClusterName] Or,
Usage: mmremotefs update Device [-f RemoteDevice] [-C RemoteClusterName] [-T MountPoint] [-t DriveLetter] [-A {yes | no | automount}] [-o MountOptions]

Miscellaneous

CommandDescription
mmaddcallbackRegisters a user-defined command into the GPFS system

Usage: mmaddcallback CallbackIdentifier --command CommandPathname --event Event[,Event...] [--priority Value][--async | --sync [--timeout Seconds] [--onerror Action]] [-N {Node[,Node...] | NodeFile | NodeClass}] [--parms ParmeterString ...]
mmdelcallbackDeletes one or more user-defined callbacks from the GPFS system

Usage: mmdelcallback CallbackIdentifier[,CallbackIdentifier...]
mmlscallbackLists callbacks that are currently registered in the GPFS system

Usage: mmlscallback [CallbackIdentifier[,CallbackIdentifier...]]
mmapplypolicyDeletes files or migrates file data between storage pools within a single file system in accordance with policy rules.

Usage: mmapplypolicy {Device | Directory} [-A IscanBuckets] [-a IscanThreads] [-B MaxFiles] [-D yyyy-mm-dd [@hh:mm[:ss]]] [-e] [-f FileListPrefix] [-g GlobalWorkDirectory] [-I {yes | defer | test | prepare}] [-i InputFileList] [-L n] [-M  name=value...] [-m ThreadLevel] [-N {all | mount | Node[,Node...] | NodeFile | NodeClass}] [-n DirThreadLevel] [-P PolicyFile] [-q] [-r FileListPathname...] [-S SnapshotName] [-s LocalWorkDirectory]
mmchpolicyEstablishes policy rules for a GPFS file system

Usage: mmchpolicy Device PolicyFileName [-t DescriptiveName ] [-I {yes | test} ]
mmlspolicyDisplays policy information

Usage: mmlspolicy Device [-L]
mmpmonMonitors GPFS performance on a per-node basis

Usage: mmpmon [-i CommandFile] [-d IntegerDelayValue] [-p] [-r IntegerRepeatValue] [-s] [-t IntegerTimeoutValue]
mmtracectlSets up and enables GPFS tracing

Usage: mmtracectl {--start | --stop | --off | --set} [--trace={io | all | def | ?Class Level [Class Level ...]?] [--trace-recycle={off | local | global | globalOnShutdown}] [--aix- trace-buffer-size=BufferSize] [--tracedev-buffer-size=BufferSize] [--tracedev-compression-level=Level] [--tracedev-overwrite-buffersize= BufferSize] [--tracedev-write-mode={blocking | overwrite}] [--trace-file-size=FileSize] [--trace-dispatch={yes | no}] [-N {Node [,Node...] | NodeFile | NodeClass}]
gpfs.snapTools to gather preliminary data when a GPFS problem is encountered

Usage: gpfs.snap [-c "CommandString"] [-d OutputDirectory] [-z] [-a | -N {Node[,Node...] | NodeFile | NodeClass}] [--check-space | --no-check-space | --check-space-only] [--exclude-aix-disk-attr] [--exclude-aix-lvm] [--exclude-net] [--exclude-merge-logs] [--mmdf]
mmfsadmExtracts data from GPFS without using locking, so that it can collect the data in the event of locking errors

Usage: TBD
mmfileidDetermine which files are located on areas of a disk that are damaged or considered to be suspect

Usage: mmfileid Device {-d DiskDesc|-F DescFile} [-o OutputFile] [-f NumThreads] [-t Directory] [-N {Node[,Node...]  | NodeFile | NodeClass}]
mmfsctlIssues a file system control request

Usage: mmfsctl Device {suspend | resume} Or,Usage: mmfsctl Device {exclude | include} {-d ?DiskName[;DiskName...]? | -F DiskFile | -G FailureGroup} Or,
Usage: mmfsctl Device syncFSconfig {-n RemoteNodesFile | -C remoteClusterName} [-S SpecFile]
dshConcurrently runs commands on multiple nodes and hardware devices.

Usage: TBD

Specifying nodes as input to GPFS commands

Many GPFS commands accept a node or multiple nodes as part of their input, using the -N flag. Nodes can be specified to GPFS commands in a variety of ways:
Node A representation of an individual node, which can be any of these:

  • Short GPFS administration node interface name.
  • Long GPFS administration node interface name.
  • Short GPFS daemon node interface name.
  • Long GPFS daemon node interface name.
  • IP address corresponding to the GPFS daemon node interface.
  • GPFS node number.
Node - Node
A node range, indicated by specifying two node numbers separated by a hyphen (-), with the first node number being less than or equal to the second node number. For example, node range 3-8 specifies the nodes with node numbers 3, 4, 5, 6, 7, and 8.
NodeClass
A set of nodes that share a particular characteristic. These node classes are known to GPFS:

  • all All of the nodes in the GPFS cluster.
  • clientnodes: All nodes that do not participate in file system administration activities.
  • managernodes: All nodes in the pool of nodes from which file system managers and token managers are selected.
  • mount: For commands involving a file system, all of the local nodes on which the file system is mounted
  • nonquorumnodes: All of the non-quorum nodes in the GPFS cluster.
  • nsdnodes: All of the NSD server nodes in the GPFS cluster.
  • quorumnodes: All of the quorum nodes in the GPFS cluster.
NodeFile
A file that contains a list of nodes. A node file can contain individual nodes or node ranges.
Logs, traces and dumps
GPFS writes both operational messages and error data to the MMFS log file. The MMFS log can be found in the /var/adm/ras directory on each node.

The MMFS log file is named mmfs.log.date.nodeName, where date is the time stamp when the instance of GPFS started on the node and nodeName is the name of the node. The latest mmfs log file can be found by using the symbolic file name /var/adm/ras/mmfs.log.latest.

The MMFS log from the prior startup of GPFS can be found by using the symbolic file name /var/adm/ras/mmfs.log.previous. All other files have a timestamp and node name appended to the file name.

The error log (errpt) contains information about several classes of events or errors. These classes are:

  • “MMFS_ABNORMAL_SHUTDOWN”
  • “MMFS_DISKFAIL”
  • “MMFS_ENVIRON”
  • “MMFS_FSSTRUCT”
  • “MMFS_GENERIC”
  • “MMFS_LONGDISKIO”
  • “MMFS_QUOTA”
  • “MMFS_SYSTEM_UNMOUNT”
  • “MMFS_SYSTEM_WARNING”
Configuration Files
/var/mmfs/gen/mmfsNodeDataContains GPFS cluster configuration data pertaining to the node.
/var/mmfs/gen/mmsdrfsContains a local copy of the mmsdrfs file found on the primary and secondary GPFS cluster configuration server nodes.
/var/mmfs/gen/mmfs.cfgContains GPFS daemon startup parameters.
GPFS port usage 
The nodes in a GPFS cluster communicate with each other using the TCP/IP protocol. The port number used by the main GPFS daemon (mmfsd) is controlled with the tscTcpPort configuration parameter. The default port number is 1191.