Some utility scripts that are handy when managing pureScale or DPF cluster.

runall scripts

Use runall script to commands on all hosts. This way, it is easier to run commands in a single command rather than to type many different lines.

#!/bin/bash

NODES=/home/db2psc/sqllib/db2nodes.cfg
SSH="/usr/bin/ssh -q -o StrictHostKeyChecking=no"
if [ -f $NODES ] ; then
 servers=$(awk '{print $2}' $NODES)
else
 servers=localhost
fi

for server in $servers
do
 echo Running "$@" on server $server
 $SSH $server "$@"
done

runscp script

Sometime, when you build a golden image for one node and need to copy file to all other hosts, runscp is handy to copy same file on all servers.

#!/bin/bash

NODES="node02 node04 node05 node06 node07"
SCP="/usr/bin/scp -q -o StrictHostKeyChecking=no"

if [[ $# -ne 1 ]] ; then
 echo Usage: runscp fullpathname
 exit 1
fi

file=$1

for server in $NODES
do
 path=${file%/*}
 echo Copying $file to $server
 $SCP $file $server:$path
done

ssh test on cluster

To make sure that password less ssh is working from each host to each other host. The ssh test should be done for short names, FQDN and IP addresses. Change nodes accordingly for short names, FQDN and IP addresses to test for all 3 cases. This solves many pureScale issues that you might come across.

[root@node01 setup]# cat ip.txt
192.168.100.171 node01.zinox.com node01 
192.168.100.172 node02.bcbsfl.com node02 
192.168.100.173 node03.bcbsfl.com node03 
192.168.100.174 node04.bcbsfl.com node04 

[root@node01 setup]# cat testssh
#!/bin/bash

IPFILE=ip.txt
SSH="/var/db2/db2ssh/db2locssh -o ConnectTimeout=10 -q -o StrictHostKeyChecking=no"

while read a b c
do
 a1[index]=$a
 a2[index]=$b
 a3[index]=$c
 ((index++))
done < $IPFILE

for h1 in ${a1[@]}
do
 for h2 in ${a1[@]}
 do
 echo -n "Testing from $h1 to $h2 "
 $SSH $h1 "$SSH $h2 hostname"
 done
done

for h1 in ${a2[@]}
do
 for h2 in ${a2[@]}
 do
 echo -n "Testing from $h1 to $h2 "
 $SSH $h1 "$SSH $h2 hostname -s"
 done
done

for h1 in ${a3[@]}
do
 for h2 in ${a3[@]}
 do
 echo -n "Testing from $h1 to $h2 "
 $SSH $h1 "$SSH $h2 hostname -s"
 done
done

• The Db2 Warehouse client container provides a toolkit that has very nice utility scripts that can be downloaded from http://ibm.biz/db2warehousetools .
• The list of tools available in the tool kit are shown as below:
  • dbsql – This is like Db2 CLP but based upon Netezza command tool nzsql.
• Scripts available in the tool are shown below. You can test these commands by providing -d switch with the name of the database alias that was generated in the previous section.
• db_ddl_database – It shows the script that was used to create a database. Use the alias name generated.
• db_ddl_function – To generate the DDL used to define user defined functions.
• db_ddl_procedure – To generate DDL for user created stored procedures.
• db_ddl_security – To generate DDL for security objects.
• db_ddl_sequence – To generate DDL for sequences
• db_ddl_synonym – To generate DDL for synonyms
• db_ddl_table – To generate DDL for user defined tables
• db_ddl_view – To generate DDL for user defined views.
• db_genstats – To generate statistics on tables that do not have up-to-date statistics.
• db_getstats – To get statistics on all tables or for a specific table.
• db_migrate – Migrate Netezza database to Db2 Warehouse
• db_size – Get the amount of storage space used by each table
• db_tables – List all tables in a database with other useful information
• db_ddl_diff – Reports any DDL differences between two schemas within the database
• db_ddl_grant_group – Dumps out the SQL DDL statement that was used to grant access to a group
• db_ddl_grant_user: Dumps out the SQL/DDL statement that was used to grant access to a user
• db_ddl_object: Dumps out the SQL DDL statements that were used to create any object of any type
• db_plan: Analyzes query plans and generates a report of important details in the specified plan
• db_skew: Exports the details of database skew for all the tables in a specified database

Using HPU to unload data from source and transferring that data over sockets and using native LOAD to load the data. Also, partitioning is changing.

A simple script that counts the number of rows and rate at which the LOAD is going.

#!/bin/bash
prevRows=0
while true
do
rows=$(db2 list utilities show detail | grep -E 'Completed Work.*rows' | awk '{sum+=$4} END {print sum}')
start_time="$(date -u +%s)"
if [[ "$rows" == "" ]] ; then
echo "No rows. Exiting ..."
break;
else
rateRows="$(bc <<< "scale=2; ($rows-$prevRows)/5")"
printf "$(date +'%Y-%m-%d %H:%M:%S') Total Rows %'9.0f Rate %9.2f per seconds\n" \
$rows $rateRows
fi
sleep 5
end_time="$(date -u +%s)"
prevRows=$rows
done

A sample output:

[dbpemon@node0101-fab - dashDB hpu]$ ./monrows
2017-12-06 08:53:09 Total Rows 2,258,737,188 Rate 812133.20 per seconds
2017-12-06 08:53:15 Total Rows 2,263,078,727 Rate 868307.80 per seconds
2017-12-06 08:53:21 Total Rows 2,268,224,157 Rate 1029086.00 per seconds
2017-12-06 08:53:27 Total Rows 2,272,421,340 Rate 839436.60 per seconds
2017-12-06 08:53:33 Total Rows 2,277,457,985 Rate 1007329.00 per seconds
2017-12-06 08:53:39 Total Rows 2,281,807,463 Rate 869895.60 per seconds
2017-12-06 08:53:45 Total Rows 2,285,964,980 Rate 831503.40 per seconds

The size of the database can be determined by calling

CALL GET_DBSIZE_INFO(?, ?, ?, 0);

When we call above Db2 procedure, it also populates information in SYSTOOLS.STMG_DBSIZE_INFO table.

We can write the following query to properly format the results and get the info in bytes, KB, MB, GB and TB.

select distinct 'D' AS Object,
'BLUDB' as Name
,DB_SIZE AS Bytes
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1024.0 ) , 
   '999,999,999,999' ),1,15) AS KB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1048576.0 ) , 
   '999,999,999' ),1,15) AS MB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1073741824.0 ) , 
   '999,999.0'),1,15) AS GB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1099511627776.0 ) , 
   '999.0'),1,15) AS TB
,substr(CURRENT USER,1,12) as OWNER
from SYSTOOLS.STMG_DBSIZE_INFO;

If we we want similar information for all tables in TPCDS schema, then the following query provides the similar results per table.

select t.tabtype AS Object,
substr(RTRIM(t.TABSCHEMA) ||'.'|| t.TABNAME,1,50) as Name
,sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE)*1024 AS Bytes
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1.0 ) , '999,999,999,999' ),1,15) AS KB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1024.0 ) , '999,999,999' ),1,15) AS MB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1048576.0 ) , '999,999.0'),1,15) AS GB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1073741824.0 ) , '999.0'),1,15) AS TB
,substr(tb.OWNER,1,12) OWNER
from
TABLE (SYSPROC.ADMIN_GET_TAB_INFO('TPCDS', '')) AS T
,syscat.tables tb
where
t.TABNAME=tb.TABNAME and t.TABSCHEMA=tb.TABSCHEMA and
t.tabschema not in ('SYSCAT','SYSIBM','SYSIBMADM','SYSPUBLIC','SYSSTAT','SYSTOOLS')
GROUP BY t.tabschema, t.tabname, t.tabtype, tb.owner;

Now, you can also combine both of the above queries in a single statement but remember to call first CALL GET_DBSIZE_INFO(?, ?, ?, 0) before running the query.

-- Call GET_DBSIZE_INFO(?, ?, ?, 0) to refresh SYSTOOLS.STMG_DBSIZE_INFO
CALL GET_DBSIZE_INFO(?, ?, ?, 0);
select * from (
select distinct 'D' AS Object,
'BLUDB' as Name
,DB_SIZE AS Bytes
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1024.0 ) , 
        '999,999,999,999' ),1,15) AS KB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1048576.0 ) , 
        '999,999,999' ),1,15) AS MB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1073741824.0 ) , 
        '999,999.0'),1,15) AS GB
,substr(TO_CHAR ( nvl( DB_SIZE ,0 ) / ( 1099511627776.0 ) , 
        '999.0'),1,15) AS TB
,substr(CURRENT USER,1,12) as OWNER
from SYSTOOLS.STMG_DBSIZE_INFO, syscat.tablespaces
UNION ALL
select t.tabtype AS Object,
substr(RTRIM(t.TABSCHEMA) ||'.'|| t.TABNAME,1,50) as Name
,sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE)*1024 AS Bytes
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1.0 ) , 
        '999,999,999,999' ),1,15) AS KB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
     XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1024.0 ) , 
         '999,999,999' ),1,15) AS MB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
        XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1048576.0 ) , 
           '999,999.0'),1,15) AS GB
,substr(TO_CHAR ( nvl( sum(DATA_OBJECT_P_SIZE+LONG_OBJECT_P_SIZE+LOB_OBJECT_P_SIZE+
        XML_OBJECT_P_SIZE+COL_OBJECT_P_SIZE) ,0 ) / ( 1073741824.0 ) , 
           '999.0'),1,15) AS TB
,substr(tb.OWNER,1,12) OWNER
from
TABLE (SYSPROC.ADMIN_GET_TAB_INFO('TPCDS', '')) AS T
,syscat.tables tb
where
t.TABNAME=tb.TABNAME and t.TABSCHEMA=tb.TABSCHEMA and
t.tabschema not in ('SYSCAT','SYSIBM','SYSIBMADM','SYSPUBLIC','SYSSTAT','SYSTOOLS')
GROUP BY t.tabschema, t.tabname, t.tabtype, tb.owner)
ORDER BY bytes;

The sample output for a TPCDS schema that I populated with 1 TB of data.

OBJECT NAME                                               BYTES                KB              MB              GB              TB              OWNER
------ ----------------------------- -------------------- --------------- --------------- --------------- --------------- ------------
T      TPCDS.SHIP_MODE                             262144              25            0            .0          .0          DBPEMON
T      TPCDS.WAREHOUSE                             262144              25            0            .0          .0          DBPEMON
T      TPCDS.PROMOTION                             262144              25            0            .0          .0          DBPEMON
T      TPCDS.CALL_CENTER                           262144              25            0            .0          .0          DBPEMON
T      TPCDS.REASON                                262144              25            0            .0          .0          DBPEMON
T      TPCDS.DBGEN_VERSION                         262144              25            0            .0          .0          DBPEMON
T      TPCDS.INCOME_BAND                           262144              25            0            .0          .0          DBPEMON
T      TPCDS.STORE                                 262144              25            0            .0          .0          DBPEMON
T      TPCDS.WEB_SITE                              262144              25            0            .0          .0          DBPEMON
T      TPCDS.WEB_PAGE                              262144              25            0            .0          .0          DBPEMON
T      TPCDS.HOUSEHOLD_DEMOGRAPHICS                524288              51            1            .0          .0          DBPEMON
T      TPCDS.CATALOG_PAGE                         2097152            2,04            2            .0          .0          DBPEMON
T      TPCDS.TIME_DIM                            10616832           10,36           10            .0          .0          DBPEMON
T      TPCDS.DATE_DIM                            10878976           10,62           10            .0          .0          DBPEMON
T      TPCDS.ITEM                                13631488           13,31           13            .0          .0          DBPEMON
T      TPCDS.CUSTOMER_ADDRESS                    31588352           30,84           30            .0          .0          DBPEMON
T      TPCDS.CUSTOMER                            78905344           77,05           75            .1          .0          DBPEMON
T      TPCDS.CUSTOMER_DEMOGRAPHICS              141688832          138,36          135            .1          .0          DBPEMON
T      TPCDS.INVENTORY                          327417856          319,74          312            .3          .0          DBPEMON
T      TPCDS.WEB_RETURNS                       9750183936        9,521,66        9,299           9.1          .0          DBPEMON
T      TPCDS.CATALOG_RETURNS                  21585068032       21,079,16       20,585          20.1          .0          DBPEMON
T      TPCDS.STORE_RETURNS                    32669171712       31,903,48       31,156          30.4          .0          DBPEMON
T      TPCDS.WEB_SALES                       137349038080      134,129,92      130,986         127.9          .1          DBPEMON
T      TPCDS.CATALOG_SALES                   274693619712      268,255,48      261,968         255.8          .2          DBPEMON
T      TPCDS.STORE_SALES                     377689210880      368,837,12      360,193         351.8          .3          DBPEMON
D      BLUDB                                 856484638720      836,410,78      816,807         797.7          .8          DBPEMON

  26 record(s) selected.

Source: Of course, I did not write these SQLs and I took it from db_size.

Usually we run uptime command in Unix to find the uptime of the server. But, what about DB2 database? How long it had been working?

db2 “select db_conn_time from table (mon_get_database(-2))”

The db2pd -d <database> – –> will also tell the uptime of the database. Note dash after the database name.

db2pd – ==> will tell instance uptime. Note dash after db2pd for instance uptime.

For example: I get an error creating a table or accessing a table space.

create table tpcds.vikram 
( 
c1 int not null primary key, 
tx_date date not null, 
c2 char(10), 
c3 timestamp(9) default current timestamp implicitly hidden, 
c4 smallint default current member implicitly hidden 
) 
distribute by hash (c1) 
partition by range (tx_date) 
   (starting ('1/1/1998') ENDING ('12/31/2030') every 1 month)
;
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL0290N Table space access is not allowed. SQLSTATE=55039
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL0290N Table space access is not allowed. SQLSTATE=55039

And, I then do db2 list tablespaces or db2pd -d BLUDB -tablespaces command to find out the table space whose state is not good. Please note that db2 list tablespace will only show the table spaces on that database partition only.

So, that is why I like to use this SQL (by using -2), I get all partitions and find out the table space whose state is not normal.

$ db2 "select tbsp_id,
>     varchar(tbsp_name, 30) as tbsp_name, DBPARTITIONNUM,
>     varchar(tbsp_state, 40) as tbsp_state
>   from table(mon_get_tablespace('',-2)) as t"

TBSP_ID              TBSP_NAME                      DBPARTITIONNUM TBSP_STATE
-------------------- ------------------------------ -------------- ----------------
                   1 TEMPSPACE1                                  2 NORMAL
                   2 USERSPACE1                                  2 QUIESCED_SHARE
                   5 MONSPACE                                    2 NORMAL
                   7 OTHTS                                       2 NORMAL
                   8 TS_EDW_001D16                               2 NORMAL

So, we now know that USERSPACE1 is in QUIESCED STATE. But, which object in this table space is a cause of this state.

$ db2 "select substr(t.tabschema,1,15) schema,
substr(t.tabname,1,30) table,
substr(tbsp_name, 1, 12) tbsp_name,
substr(quiescer_auth_id,1,10) authid,
quiescer_application_handle application_handle,
quiescer_state state, dbpartitionnum, member
from table(mon_get_tablespace_quiescer(-2)) q, syscat.tables t
where t.tbspaceid = q.quiescer_ts_id
and t.tableid = q.quiescer_obj_id
order by dbpartitionnum"

SCHEMA   TABLE          TBSP_NAME    AUTHID     APPLICATION_HANDLE STATE DBPARTITIONNUM MEMBER
-------- -------------- ------------ ---------- ------------------ ----- -------------- ------
TPCDS     CATALOG_SALES   USERSPACE1   DB2INST1                  0 SHARE              2      2

So, we now know that TPCDS.CATALOG_SALES table state is causing table space USERSPACE1 to be in QUIESCED SHARE state and this is in database partition number 2.

$ db2 terminate
DB20000I  The TERMINATE command completed successfully.
$ db2 "SET CLIENT CONNECT_DBPARTITIONNUM 2"
DB20000I  The SET CLIENT command completed successfully.
$ db2 connect to bludb

   Database Connection Information

 Database server        = DB2/LINUXPPC64LE 11.1.9.0
 SQL authorization ID   = DBPEMON
 Local database alias   = BLUDB

$ db2 quiesce tablespaces for table tpcds.catalog_sales reset
DB20000I  The QUIESCE TABLESPACES command completed successfully.

Now, we have brought out this table out of quiesced mode, confirm with the SQL on MON_GET_TABLESPACE_QUIESCER again.

But, we also need to make sure that we revert back our connection to the coordinator node.

Please make sure that you login as same user who is holding the quiesce state which in this case was db2inst1.

How to find the total number of database partitions and range (data) partitions held by a table?

Here is the query:

select t.tabschema SCHEMA, 
       t.tabname TABLE, 
       (select count(*) 
	    from   syscat.dbpartitiongroupdef pgs 
		where  pgs.dbpgname = ts.dbpgname) DB_PARTITIONS,
count(ts.tbspace) DATA_PARTITIONS
from syscat.tables t,
     syscat.tablespaces ts,
     syscat.datapartitions d
where ts.tbspaceid = d.tbspaceid
and   t.tabschema = d.tabschema
and   t.tabname = d.tabname
and   t.tabschema = 'schemaName'
and   t.tabname = 'tableName'
group by t.tabschema, 
         t.tabname, 
		 ts.dbpgname;
SCHEMA          TABLE                          DB_PARTITIONS DATA_PARTITIONS
--------------- ------------------------------ ------------- ---------------
TPCDS           CATALOG_SALES                             36             192