Prerequisites
Before you begin to work with an IBM IMS database as a source in Qlik Replicate, make sure that all of the prerequisites have been met.
z/OS platform
-
Java 17
-
z/OS versions 2.5 and higher
-
IMS versions 15.x
Metadata design platform (Windows/Linux)
If the IMS segment metadata is not exclusively sourced from IMS Catalog, the following tools are needed:
-
IBM Developer for z/OS
-
IMS Explorer for Development (part of IBM Explorer for z/OS)
Replicate prerequisites
-
Download the IMS universal drivers jar 15.166 or later from https://mvnrepository.com/artifact/com.ibm.ims/udb and copy it to <Replicate-installation-folder>endpoint_srv/externals.
Information noteReplicate requires the non-redistributable IBM IMS Java client. -
Restart the Qlik Replicate Server service.
Installing the Qlik IMS Server on z/OS
The section describes how to install the Qlik IMS Server on z/OS.
Step 1: Download the Qlik IMS Server kit
Download the R4IMS kit file from Qlik and transfer it to a z/OS Unix directory.
To download the file:
-
Go to Product Downloads.
-
Select Qlik Data Integration.
-
In the Product list, select Qlik Replicate.
-
In the Release list, select your Qlik Replicate version.
-
In the Release Number list, select Initial Release.
-
In the Download Link column, click the QlikReplicate_<version>_R4IMS-<version>.tar.gz link to start your download.
The contents of the kits are summarized in the following table:
| Path | Description |
|---|---|
| server/lib/ | Required server jars |
| server/ims-server-<version>.jar | Qlik IMS server jar |
| server/logging.properties | Qlik IMS server logging properties file |
| server/server.jcl | Sample Qlik IMS server started task procedure |
| server/ims_server.sh | Shell script to run the Qlik IMS server |
| exit/logcrims.jcl | Sample JCL to create a logstream |
| exit/R4IMSEXI | Qlik LOGWRT exit for IMS |
| exit/r4irepl.jcl | Sample JCLto configure and rebuild the Qlik LOGWRT exit for IMS |
Step 2: Extract the QlikIMS Server kit
-
Extract the R4IMS-<version>.tar file to a directory of your choice. These instructions will assume it was extracted to /u/qlik/ims.
$ mkdir -p /u/qlik/ims $ tar -xvf R4IMS-<version>.tar -C /u/qlik/ims -
Set the owner and group of the files using chown -R to the desired owner.
$ chown <user>:<group> -R /u/qlik/ims -
Make the ims_server.sh executable:
$ chmod 744 /u/qlik/ims/server/ims_server.sh
Step 3: Download the IBM IMS client for Java
Download the IMS universal drivers jar 15.166 or later from https://mvnrepository.com/artifact/com.ibm.ims/udb and copy it to the /u/qlik/ims/server/lib directory.
Step 4: Create and configure a TLS server certificate for the Qlik IMS Server
The Qlik IMS Server serves its client using HTTPS; therefore, a trusted TLS certificate need to be configured. The certificate generated in the examples below is a self-signed certificate. This is completely secure since a matching public certificate will be configured in the Replicate IMS source endpoint, so that it can trust the server. The TSO commands in the examples below assume that the z/OS system is using the IBM RACF security system. For other cases, consult the vendor documentation.
-
Create a self-signed TLS certificate for the server
RACDCERT GENCERT SUBJECTSDN(CN('<host-name>')) SIZE(384) NISTECC WITHLABEL('qlik.ims.hostcert') RACDCERT ADDRING(qlik.ims.server.keyring) RACDCERT CONNECT(LABEL('qlik.ims.hostcert') RING(qlik.ims.server.keyring) USAGE(PERSONAL)) SETROPTS RACLIST(DIGTCERT, DIGTRING) REFRESH -
Export the public TLS certificate to a .pem file using the following TSO commands:
RACDCERT EXPORT(LABEL('qlik.ims.hostcert')) DSN('<HLQ>.qlik.ims.hostcert') FORMAT(CERTB64) OPUT '<HLQ>.qlik.ims.hostcert' '<filename>.pem' TEXT -
Copy the .pem file to the Replicate machine as a text file. It will be used by the endpoint to authenticate the server.
Step 5: Create and configure a client self-signed TLS certificate
The Replicate IMS source endpoint identifies and authenticates itself to the Qlik IMS Server using a TLS certificate.
The commands below create a private and public certificate using OpenSSL on a Linux/Windows machine, and show how to configure the public certificate on the z/OS machine:
-
Create a private key and a matching public certificate, as follows:
$ openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 -keyout qlik-ims-client-key.pem -out qlik-ims-client-cert.pem -sha256 -days 365 -nodesThe certificate expires in 365 days. It will need to be replaced before it expires to ensure that the server remains trusted by the client.
-
Export the certificate as pkcs12 format for use in the Replicate IMS source endpoint configuration, as follows:
$ openssl pkcs12 -export -out qlik-ims-client-cert.p12 -inkey qlik-ims-client-key.pem -in qlik-ims-client-cert.pemYou will be prompted to provide a password for the certificate. Make sure you have it saved somewhere safe so it can be used in the Replicate IMS source endpoint configuration as well.
-
Copy the qlik-ims-client-cert.pem file to the z/OS machine as text. Then copy the file to a dataset by running the following command from a UNIX shell:
$ cp qlik-ims-client-cert.pem "//'<HLQ>.qlik.ims.clntcert'"
Step 6: Configure change capture in IMS: Create a z/OS logstream
Replicate captures changes to IMS segments using a LOGWRT EXIT function which writes change records to a dedicated z/OS System Logger logstream. Replicate reads the changes from the logstream via the Qlik IMS Server.
RACDCERT ADD('<HLQ>.qlik.ims.clntcert') WITHLABEL('qlik.ims.client.cert') TRUST
RACDCERT ADDRING(qlik-ims-client-keyring)
RACDCERT CONNECT (LABEL('qlik.ims.client.cert') RING(qlik-ims-client-keyring) USAGE(CERTAUTH))
SETROPTS RACLIST(DIGTCERT, DIGTRING) REFRESH Before setting up the EXIT function, you need to create a z/OS logstream. If an existing ARC LOGWRT EXIT function is configured for the IMS database (for use with the older ARC-based IMS source replication endpoint), you can skip this step and the following steps dealing with configuring change capture in IMS, and the existing logstream can be used as-is. The kit contains a sample job file /u/qlik/ims/server/logcrims.jcl, as follows:
//LOGCRIMS JOB 'QLIK',MSGLEVEL=(1,1),
//CLASS=A,NOTIFY=&SYSUID
//STEP1 EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DATA TYPE(LOGR) REPORT(NO)
DEFINE LOGSTREAM NAME(QLIK.IMS.LGSTRM)
DASDONLY(YES)
LS_SIZE(1000)
STG_SIZE(1000)
HIGHOFFLOAD(80)
LOWOFFLOAD(50)
AUTODELETE(YES)
EXPDT(5) /* Verify record retention period in days */
MAXBUFSIZE(64000)
HLQ(IXGLOGR) /* Update IXGLOGR with a valid HLQ on your site */
// To use the sample job file, copy it into a member in your PROCLIB concatenation, for example, LOGCRIMS:
$ cp qlik-ims-client-cert.pem "//'<HLQ>.qlik.ims.clntcert'"
You should edit the job according to your needs. Note the following:
-
You can change the logstream name in the sample (QLIK.IMS.LGSTRM) as needed.
-
The logstream must be configured to automatically delete old change records, otherwise it with fill up the available storage. This is specified using the AUTODELETE(YES) parameter, and the EXPDT(<retention-days>) parameter, which specifies to how long to retain records in the logstream.
-
If your logstreams are located under a different HLQ than IXGLOGR, update the HLQ to the actual location.
After saving your changes, submit the job to create the logstream.
Step 7: Configure change capture in IMS: Customize the LOGWRT EXIT function
The program /u/qlik/ims/exit/R4IMSEXI is a Qlik-provided LOGWRT EXIT function that writes IMS log data to a z/OS logstream. Before using it, it needs to be customized to use a specific logstream name (along with other parameters). To customize the LOGWRT EXIT function, Qlik provides a sample job file at /u/qlik/ims/exit/r4irepl.jcl:
// JOB
//* ADD A JOBCARD ABOVE
//COMP EXEC PGM=ASMA90
//SYSIN DD *
************* DO NOT MAKE ANY CHANGES BEFORE LINE 60
MACRO
&NAME R4IMAC &STRMNAME=QLIK.IMS.LGSTRM, X
&BLOCKNUM=100, X
&ENABLE=YES, X
&DATASHARING=NO, X
&SEQUENCE=YES, X
&DEBUG=NO
DC CL8'&SYSDATE'
DC CL5'&SYSTIME'
DC CL26'&STRMNAME'
AIF (T'&BLOCKNUM NE 'N').BLKNUMERROR BLOCKNUM IS NOT NUMBER
AIF (&BLOCKNUM GT 999).BLKNUMLARGE BLOCKNUM IS > 999
DC H'&BLOCKNUM'
AIF ('&ENABLE(1)' EQ 'NO').CHKDS
AIF ('&ENABLE(1)' EQ 'YES').CHKDS
MNOTE 8,'ENABLE SHOULD BE YES OR NO'
MEXIT
.CHKDS ANOP
DC CL3'&ENABLE'
AIF ('&DATASHARING(1)' EQ 'NO').CHKSEQ
AIF ('&DATASHARING(1)' EQ 'YES').CHKSEQ
MNOTE 8,'DATASHARING SHOULD BE YES OR NO'
MEXIT
.CHKSEQ ANOP
DC CL3'&DATASHARING'
AIF ('&SEQUENCE(1)' EQ 'NO').CHKDBG
AIF ('&SEQUENCE(1)' EQ 'YES').CHKDBG
MNOTE 8,'SEQUENCE SHOULD BE YES OR NO'
MEXIT
.CHKDBG ANOP
DC CL3'&SEQUENCE'
AIF ('&DEBUG(1)' EQ 'NO').EXITOK
AIF ('&DEBUG(1)' EQ 'YES').EXITOK
MNOTE 8,'DEBUG SHOULD BE YES OR NO'
MEXIT
MEXIT
.LENGTHERROR ANOP
MNOTE 8,'LOGSTREAM NAME GREATER THAN 26'
MEXIT
.BLKNUMERROR ANOP
MNOTE 8,'BLOCKNUM IS NOT NUMERIC'
MEXIT
.BLKNUMLARGE ANOP
MNOTE 8,'BLOCKNUM IS GREATER THAN 999'
MEXIT
.EXITOK ANOP
DC CL3'&DEBUG'
MEXIT
MEND
R4IPARMS CSECT
R4IPARMS AMODE 31
R4IPARMS RMODE ANY
PRINT GEN,ON
****** DO NOT MAKE ANY CHANGES ABOVE
* Below, make any changes to the EXIT defaults:
*
* STRMNAME should be set to the name of the log stream used to capture
* changes.
*
* ENABLE should be set to YES in most cases. If you want to disable the
* EXIT it, set it to NO.
*
* DATASHARING should be set to yes if you are in an IMS data-sharing
* environment.
*
* DEBUG should be set to YES if you want to log the EXIT's actions
* Please note that in a busy system this will cause a large output
R4IMAC STRMNAME=QLIK.IMS.LGSTRM, <-THE NAME OF THE LOGSTREAMX
ENABLE=YES, <- SET TO NO TO DISABLE EXIT X
DATASHARING=NO, <- SET TO YES EHEN DATASHARING X
DEBUG=NO < SET TO YES FOR DEBUG LOGGING
END
//SYSLIN DD DSN=&&OBJ(R4IPARMS),DISP=(,PASS),SPACE=(TRK,(1,1,1)),
// UNIT=VIO
//SYSPRINT DD SYSOUT=*
// IF (RC = 0) THEN
//LINK EXEC PGM=HEWL
//* change IMS.RESLIB to the name of a RESLIB dataset
//SYSLMOD DD DSN=IMS.RESLIB,DISP=SHR <=Verify that this is
//RESLIB DD DSN=IMS.RESLIB,DISP=SHR <=where the EXIT resides
//SYSPRINT DD SYSOUT=*
//SYSLIB DD DSN=*.COMP.SYSLIN,DISP=(OLD,DELETE)
//SYSLIN DD *
INCLUDE RESLIB(R4IMSEXI)
REPLACE -IMMED,R4IPARMS(R4IPARMS)
ENTRY R4IMSEXI
NAME R4IMSEXI(R)
// ENDIF To customize the LOGWRT EXIT function
-
Copy the provided file /u/qlik/ims/exit/r4irepl.jcl into a member in your PROCLIB concatenation, for example, R4IREPL:
$ cp /u/qlik/ims/exit/r4irepl.jcl "//<HLQ>.PROCLIB(R4IREPL)'" -
Copy the provided exit function to the IMS RESLIB (or any other PDS in the IMS control region STEPLIB concatenation). For example:
$ cp -X /u/qlik/ims/exit/R4IMSEXI //’IMS.RESLIB’ -
Edit the R4IREPL job, and change the following items:
-
Change STRMNAME (logstream name) parameter of the R4IMAC macro to be the logstream you defined earlier.
-
If needed, change IMS.RESLIB to where the EXIT resides.
-
-
Submit the R4IREPL job to update the EXIT with the desired configuration.
Step 8: Configure change capture in IMS – Installing the LOGWRT EXIT
This step describes how to update the IMS environment to use the R4IMSEXI EXIT function.
-
Either create or update your DFSDFxxx in the IMS PROCLIB as follows:
<SECTION=USER_EXITS> EXITDEF=(TYPE=LOGWRT, /* LOGWRT USER EXIT */ EXITS=(R4IMSEXI)) /* EXIT LIST */If you do not have a DFSDFxxx member loaded at IMS startup, add this line to your DFSPByyy member:
DFSDF=xxx
Where xxx is the suffix you gave to your DFSDF member.
Information noteIf you already have another (third party) LOGWRT in use, code the EXITS statements as follows: EXITS=(R4IMSEXI,YOUREXIT)The exits will be called in the specified order.
-
Recycle IMS, or if applicable, dynamically load the EXIT using the following IMS type-2 command:
REFRESH USEREXIT TYPE(LOGWRT)
Note that If you change the R4IMSEXI EXIT (as described above) after it is used by IMS, the changes will be triggered the next time the EXIT is invoked by a captured segment change. When this happens, you will see something like this in the IMS control region job log:
Logstream name is: QLIK.IMS.LGSTRM
Number of blocks: 100
Sequencing : YES
Enabled : YES
Datasharing : NO
Debug: YESR4ZI020I current runtime parameters are: 054
If you change the logstream name, the following message may appear (possibly several times):
R4ZI035D Qlik LOGWRT Logger RC=8 - will waitThis is normal and expected.
Step 9: Adjust the relevant IMS database DBDs to capture
-
For any IMS database from which you want to capture changes using Replicate, you have to update its DBD to make it use the Qlik provided LOGWRT EXIT function (DFSFLGX0).
To do this, add the following parameter to the DBD macro:
EXIT= (*, KEY, NOPATH, DATA, LOG, (CASCADE, KEY, NODATA, NOPATH))Information noteThe value NODATA in the cascade above indicates that no data is returned from non-key fields. If a parent record is deleted, the child records are also deleted, but the data is returned only for the key field. In this case, the non-key fields for the child records will be empty. -
Recompile the DBD and the corresponding PSB and ACB objects, then restart the IMS Control Region.
Step 10: Configure and start the Qlik IMS Server
-
The Qlik IMS Server runs as a z/OS started task. The kit contains a sample job file /u/qlik/ims/server/server.jcl which needs to be copied into a member in your PROCLIB concatenation, for example, QLIKIMS:
$ cp /u/qlik/ims/server/server.jcl //'<HLQ>.PROCLIB(QLIKIMS)' -
Edit the QLIKIMS member and set up the configuration parameters in the SRVPARAM DD card.
The configuration parameters are:
- server.address: The listening interface for the Qlik IMS server (defaults to 0.0.0.0 which means listening on all interfaces).
- server.port: The listening port of the Qlik IMS server (defaults to 50051).
- server.maxParallelRequests: The total number of requests the server can handle in parallel (defaults to 100).
- server.mtls.keyring.userId: The name of the user that owns the keyring referred to in server.mtls.keyring.name.
- server.mtls.keyring.name: The name of the keyring that is used as a Key Manager with the certificate chain of the server.
- client.mtls.keyring.userId: The name of the user that owns the keyring referred to in client.mtls.keyring.name.
- client.mtls.keyring.name: The name of the keyring that is used as a Trust Manager for the server to validate the client's certificate
The following is an example of a server job file QLIKIMS:
//QLIKIMS PROC
//STEP1 EXEC PGM=BPXBATCH,PARM='SH /u/qlik/ims/ims_server.sh'
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD *
_BPX_SHAREAS=YES
/*
//SRVPARAM DD *
# Listening TCP address (default 0.0.0.0)
server.address = 0.0.0.0
# Listening TCP port (default 50051)
server.port = 50051
# UserID of the server keyring
server.mtls.keyring.userId = <SERVER-KEYRING-USER-ID>
# Name of the server keyring
server.mtls.keyring.name = qlik.ims.server.keyring
# UserID of the client keyring
client.mtls.keyring.userId = <CLIENT-RING-USER-ID>
# Name of the client keyring
client.mtls.keyring.name = qlik.ims.client.keyring
// PEND Server logging
To diagnose errors such as TLS certificate problems, change the level of the logger in the server from INFO to FINE.
You should only do this if you encounter internal errors, and you should revert to normal logging as soon as possible to avoid flooding SYSOUT.
To change the logging level, edit the logging.properties file and set both the .level and java.util.logging.ConsoleHandler.level to FINE.
After changing the logging level, you must restart the server.