Mainframe Conversion - Related Documents

JCLcnv1demo.htm - DEMO conversions, sample JCL, scripts, executions JCLcnv2real.htm - comprehensive instructions for REAL conversions JCLcnv3aids.htm - conversion AIDS (cross-refs,tips,mass changes,etc) JCLcnv4gdg.htm - GDG file handler from UV Software DATAcnv1.htm - Data file conversion - Original Documentation DATAcnv2.htm - Data File Conversion - Comprehensive & Complete <--This Doc**

DATAcnv2 - Contents by Part

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

DATAcnv2 - Contents by Part - continued

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

DATAcnv2 - Contents by Part - continued

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_1. Data File Conversion - Comprehensive & Complete

Part 1 - Introduction & Overview

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A1. Data File Conversion

Introduction

Here in JCLcnv1demo.htm#Part_7 we will demonstrate data file conversions, using our simplified set of test/demo files. See JCLcnv2real.htm#Part_3 & Part 4 for more complex problems that you might encounter in real data file conversions.

Note that DATAcnv1.htm presents the simplest data conversion methods for A one file at a time. The 'heart' of all methods is to use the COBOL copybooks to generate uvcopy jobs that translate to ASCII & then restore packed/binary fields (same for both mainframe & unix/linux COBOLs). For a preview of six sample copybooks & the generated data conversion jobs, see pages '3C0' - 3C6.

The generated uvcopy jobs will be named the same as the COBOL copybooks & the embedded input/output datafilenames are also coded with the copybooknames. This is OK for low volume datafile conversions, since it is simple to edit the generated jobs to insert the correct datafilenames.

But for high volume datafile conversions, we can eliminate multiple edits by creating a 'control-file' to relate the datafilenames to the copybooknames. Utility 'uvdata52' reads the control-file & replicates the conversion job (named from the copybook) for each datafilename in the control-file, which allows for the fact that there may be multiple datafiles for any 1 copybook.

Following the datafilename & copybookname, our control-file includes file info in keyword format for file-type, record-sizes, generations,& Indexed file key locations. This info can be extracted from the mainframe LISTCAT reports, COBOL copybooks, the JCL, and other sources such as excel spreadsheets that some sites may have. The uvdata52 utility will use this info to correct the assumptions made for information lacking in the copybooks.

sample control-file

 ar.customer.armaster   cpy=armaster typ=IDXf1  rca=00256 rcm=00256 rcs=00256 gdg=___ key=000,006
 ar.customer.arsales    cpy=arsales  typ=RSF    rca=00064 rcm=00064 rcs=00064 gdg=___ key=___,___
 ar.customer.artrans    cpy=custran3 typ=IDXf8v rca=00120 rcm=00249 rcs=00245 gdg=___ key=___,___
 ar.customer.phone.list cpy=________ typ=RSF    rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=citytax1 typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=___ key=___,___
 city.tax.payments      cpy=citytax2 typ=IDXf8v rca=00080 rcm=00128 rcs=00128 gdg=___ key=___,___
 gl.account.glmaster_   cpy=glmaster typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=003 key=___,___
 gl.account.gltran_     cpy=gltran   typ=RSF    rca=00080 rcm=00080 rcs=00080 gdg=004 key=___,___
 vendor.master.names    cpy=vendorm  typ=IDXf8v rca=00019 rcm=00041 rcs=00064 gdg=___ key=___,___
 vendor.master.payments cpy=vendorpm typ=IDXf8v rca=00019 rcm=00062 rcs=00064 gdg=___ key=___,___

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A2. Data File Conversion

master control-file all info for files to be converted

It is best if you can create one comprehensive control-file of all datafilenames that need to be transferred & converted to initialize applications on unix/linux. If your site has a spreadsheet with this info, you can export it to a .csv file, & we can show you how to modify a uvcopy utility to convert the .csv file into the 'keyword-coded control-file' required by our datafile conversion methods.

Or, if you do not have a spreadsheet list of files that need to be transferred, you can create the control-file with our 'mvsfiles5A' script that analyzes all JCL to determine the 'Essential Files' required to initialize applications on Unix/Linux. Then utility 'xmvs2ctl1' will convert the essential files report into the control file required for datafile conversions.

updating control-file with keyword info

Regardless of how you create the initial control-file of filenames to be transferred, you can append/update the keyword coded info with aditional info available from the mainframe LISTCAT reports &/or the control-file extracted during JCL conversions. The only info that you need to enter manually should be the copybookname & all other info can be determined automatically. See more explanations about file info sources at '2A3' - 2A5 & '2B1' - 2C4.

We also recommend that you add a 'group-code' to your control-file to identify logical groups of datafiles to be transferred together. For example: grp=ar, grp=ap, grp=gl, etc. Later (on page '5B2'), we will show you how to select the control-file entries for the next group of files to be transferred.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A3. Data File Conversion

generating JCL to FTP files from mainframe to unix

This selected list could be used to generate the FTP JCL to be transferred to the mainframe & executed to FTP the file group to Unix/Linux. For this method we had to generate different JCL steps for Fixed-Length & Variable-Length files since the FTP must be coded with the 'RDW' option for variable-length, but not for fixed-length.

The better method is use the newer 'genF2Vjcl' utility to generate JCL for the mainframe to convert all files to Variable-Length before we run FTP with RDW on unix to get the files. This way we can know the record-sizes for all files immediately when we receive them on unix. We will know the Fixed-Length files because the minimum & maximum record-sizes will be the same.

The demos in this documentation https://uvsoftware.ca/datacnv2.htm do not assume that the recommended master control-file of all files required for transfer has been created. These demos assume we knew 10 datafiles desired and have FTP'd them to Unix/Linux.

We then create our initial control file by reading the directory of filenames & appending keywords for file info. For these demos, we will assume that only the variable-length files were FTP'd with RDW & demonstrate how we can fill in record-sizes for fixed lengthe files from control files extracted from mainframe LISTCAT reports or from the control files extracted from the JCL.

Studying the test/demos here will give you a good understanding of these data conversion procedures. Even better, if you have the Vancouver Utilities, you can run the demos using the test files provided.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A4. Data File Conversion

uvcopy - powerful conversion tool

'uvcopy' is essential to converting & running mainframe applications on Unix/Linux systems, because it can do many things not possible with the standard unix/linux utilities such as:

1. uvcopy can process packed decimal fields.
   
   
2. uvcopy can process data files with fixed length record files and without the 'Line-Feeds' required by most unix/linux system utilities.
   
   
3. uvcopy can process Indexed files, compatible with Micro Focus COBOL
   
   
4. uvcopy can address record fields by column position, similar to many mainframe utilities. Very few unix utilities provide column addressing.
   
   
5. uvcopy can process both types of binary fields - 'Big-End' in mainframe files & 'Little-End' used on Intel systems. uvcopy easily converts between the 2 formats as required for conversions & ongoing file maintenance on the unix/linux system since COBOL retains Big-End format in data files.
   
   

uvcopy provides the power of assembler without the complexity. The uvcopy language is based on the IBM 360/ZOS assembler instruction set, which means many mainframers will be familiar with it. Since it is interpretive, no compiles are required - just update the parameter file of instructions & re-run.

uvcopy has over 90 instructions documented at uvcopy3.htm. Many of these are powerful functions such as scan/replace, converting between fixed format & delimited files such as csv or pipe delimited, translating between EBCDIC & ASCII or UPPER & lower-case, table lookups, etc.

uvcopy allows you to manipulate data on unix/linux systems more powerfully than you could on the mainframe - process packed decimal fields, Indexed files, variable length files, fixed length records without linefeeds, column addressing.

uvcopy provides the capabilities on unix/linux to replace various mainframe utilities such as IDCAMS, IEBGENER, FILEAID, EASYTRIEVE, QUIKJOB, etc.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A5. Data File Conversion

Converting to/from Delimited Files

This DATAcnv2.htm documents converting flat files from the mainframe to unix/linux, resulting in exactly the same record layout, but preserving packed/binary/signed fields, which would be destroyed by simple translate.

If you need to convert files to delimited format for loading database tables, please see the following jobs in DATAcnv1.htm#Part_4.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B1. Data File Conversion

setup directories for test/demo conversions

We will assume that you have the Vancouver Utilities installed as documented at https://uvsoftware.ca/jclcnv1demo.htm#1A1 - 1D9.

We also assume that you have setup your user login account as documented at https://uvsoftware.ca/jclcnv1demo.htm#3C1 - 3C2.

We also assume that you have setup conversion directories as documented at https://uvsoftware.ca/jclcnv1demo.htm#3D1 using scripts mvslibsdirs, mvsdatadirs, & cnvdatadirs, and you have run script 'copymvsctls' to copy control files to cnvdata/.

You may have run the COBOL & JCL conversion test/demos as documented at https://uvsoftware.ca/jclcnv1demo.htm#Part_3 & Part 4, but these are not required if you are only interested in the data conversion demos.

We will repeat here only the instructions to setup the data conversion directories & copy the test files required to run the data conversion demos here in https://uvsoftware.ca/datacnv2.htm.

 #1. Login userxx --> /home/userxx/

 #2. mkdir cnvdata        <-- make superdir for conversion subdirs
     =============

 #3. cd cnvdata --> /home/userxx/cnvdata
     ==========

 #3a. cdc       <-- alternate way to 'cd cnvdata' using alias
      ===           alias cdc='cd $CNVDATA' --> /home/userxx/cnvdata/
                  - alias's & CNVDATA,etc defined in profiles

 #4. cnvdatadirs  <-- script to setup data conversion subdirs
     ===========

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B2. Data File Conversion

directories for data-file conversions

 /home/userxx/       <-- homedir OK for demos, but not high volume data conversions
 :-----cnvdata          <-- data conversion superdir $CNVDATA
 :     :--*--cpys         - COBOL copybooks
 :     :--*--ctl          - control file directory
 :     :     :--*--cat0   - LISTCAT reports from mainframe
 :     :     :-----cat1   - file types,recsizes,generations extracted to text file
 :     :     :-----cat2   - translate filenames to lower, GDG files to UV format
 :     :--*--d0ebc        - EBCDIC files from mainframe for conversion
 :     :-----d1ebc        - may need to convert variable-length to fixed for conversion
 :     :-----d2asc        - files converted to ASCII (with same record layout)
 :     :-----d3ebc        - back to EBCDIC for return to mainframe or prove conversions
 :     :-----d4pipe       - data files converted to '|' pipe delimited format
 :     :-----maps         - cobmaps (record layouts) generated from copybooks
 :     :-----pfx1         - uvcopy jobs to convert EBCDIC to ASCII (gen from cobmaps)
 :     :-----pfx2         - jobs copied here for manual changes for multiple Record Types
 :     :-----pfx3         - jobs renamed here for the datafilename (vs copybookname)
 :     :                    datafilenames,filetypes,recsizes inserted from control-file
 :     :-----pfx4         - jobs copied here, replicating & inserting GDG filenames
 :     :                    depending on generations present in d1ebc/...
 :     :-----pfx5         - jobs to convert ASCII back to EBCDIC to verify conversions
 :     :                    by comparing to original EBCDIC with uvcmpRDW1
 :     :-----pfp1         - uvcopy jobs to convert to pipe delimited (from copybooks)
 :     :-----rptcmp       - comparison reports stored here by uvcmpRDW1/uvcmp2
 :     :-----tmp          - tempdir for various tempfiles, to keep working directory
 :     :                    clean (subdirs only, no tempfile clutter)

 #5. cp $UV/mvstest/mvstest2/cnvdata1/cpys/* cpys   <-- demo copybooks
     ============================================

 #6. cp $UV/mvstest/mvstest2/cnvdata1/d0ebc/* d0ebc <-- demo datafiles
     ==============================================     to be converted

 #7. cp $UV/ctl/cat0/* ctl/cat0   <-- LISTCAT datafile info from the mainframe
     ==========================     - listcat001 & listcat002

 #8. cp $UV/ctl/lrecl0 ctl <-- additional file from mainframe, with record-sizes
     =====================     that were omitted on LISTCAT for NON-VSAM files

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C1. Data File Conversion

FTP datafiles from Mainframe to Unix/Linux

Before FTP'ing files from mainframe to Unix/Linux, we strongly recommend converting the Fixed-Length files to Variable-Length so we can use the RDW option. Then, when we receive the files on unix/linux, we will immediately know the record-sizes of all files & also know which were originally Fixed because their minimum & maximum record sizes will be the same. Here is a skeleton JCL that could be used to convert Fixed to Variable on the mainframe.

 //* JCL to convert Fixed-Length to Variable-Length
 //SORT001   EXEC PGM=SORT
 //SYSUDUMP  DD SYSOUT=K,HOLD=YES
 //SYSOUT    DD SYSOUT=*
 //SYSPRINT  DD SYSOUT=*
 //SORTIN    DD DSN=AAAAA.BBBBB.CCC.DDDDD,
 //             DISP=(SHR,KEEP,KEEP)
 //SORTOUT   DD DSN=AAAAA.BBBBB.CCC.DDDDD.X,
 //             DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
 //             SPACE=(TRK,(0500,0500),RLSE),
 //             DCB=(RECFM=VB,LRECL=32000)
 //SYSIN     DD *
    SORT FIELDS=COPY
    OUTFIL FNAMES=SORTOUT,FTOV
 /*
 //*Note - 'FTOV' means Fixed TO Variable

You could then run FTP with RDW on the mainframe with the following JCL, or you could run FTP on Unix/Linux as shown on the next page.

 //* JCL to FTP Variable-Length with RDW to Unix/Linux
 //ACHP01B   EXEC PGM=FTP,PARM='(TRAN TRTAUGE1'
 //OUTPUT    DD  SYSOUT=*
 //SYSPRINT  DD  SYSOUT=*
 //INPUT     DD  *
 30.252.99.11 21 (exit
 userid password
 cd /p2/cnv1/cnvdata/d0ebc/
 pwd
 type i
 LOCSITE RDW
 PUT 'AAAAA.BBBBB.CCC.DDDDD.X' AAAAA.BBBBB.CCC.DDDDD
 QUIT
 /*

Also see pages '5C1' - 5C4, which show how you can automatically generate the JCL to convert Fixed to Variable from a control-file list of filenames to be transferred to Unix/Linux. 'genF2Vjcl1' is a Vancouver Utility which runs on unix/linux, so you need to FTP the generated JCL to the mainframe for execution.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C2. Converting Mainframe Data Files from EBCDIC to ASCII

FTP datafiles from Mainframe to Unix/Linux

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. rm -f d0ebc/*  <-- remove any old files if starting a new group
     =============      of files to be converted

 #2. cd d0ebc       <-- change into the data subdir
     ========

 #3. ftp xxx.xxx.xxx.xxx   <-- FTP from mainframe IP#
     ===================
 #3a. userid--> .....      <-- enter userid
 #3b. passwd--> .....      <-- enter password
 #3c. BINARY               <-- ensure BINARY transfer
 #3d. LITERAL SITE RDW     <-- required for variable length files
 #3e. cd ...               <-- change to data files directory ??
 #3f. prompt               <-- turn prompt off for mget
 #3g. mget *               <-- get all files in directory
 #3g. mget *.X             <-- OR all files flagged for FTP

 #4. chmod 664 *          <-- all files should be 664
     ===========

Notes re the 'RDW' option

Option 'RDW' prefixes each record with the record size in binary, which means we can know the record-size immediately on the Unix/Linux system without having to get that info from a control-file (still need control-file for copybooks,etc).

It would be nice if we could FTP all files with the RDW option, but we found that it works only for variable-length files, so the solution is to convert the Fixed-Length-Files to Variable-Length-Files before FTP.

converting 'FLR' to 'VLR' on mainframe prior to FTP

See 'genF2Vjcl' on page '5C1' - 5C5 to generate mainframe JCL to copy files to variable-length format guided by a control-file of datafilenames to be transferred in the next desired group. We would of course, FTP the generated JCL to the mainframe for execution.

When we receive the files on unix/linux, we will know which files were Fixed-Length on the mainframe, because all records will have the same size in the RDW prefix vs the Variable-Length files where minimum & maximum record-sizes will be different.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C3. Converting Mainframe Data Files from EBCDIC to ASCII

Re-Naming files as required

On some systems FTP'd files may be enclosed in single quotes, may be UPPER case, may have embedded blanks or '$' signs. ON unix we need to remove single quotes, remove any embedded blanks or '$' signs, & lower case is strongly recommended.

 #5. mv  \'XXX.XXX.XXX\'  xxx.xxx.xxx   <-- the hard way
     ================================     - see the easy ways below

It would be very slow & tedious if you had to manually rename the files using 'mv' commands when you have hundreds of files to be renamed. Vancouver Utilities has many 'rename' scripts to make mass changes to filenames easy.

 #5a. cd ..           <-- change back out of the d0ebc/ subdir
      =====

 #5a. renameL d0ebc   <-- easy way to translate names to Lower case
      ==============
 #5b. renameQQ d0ebc  <-- easy way to remove Quotes from All files in subdir
      =============
 #5c. renameBU d0ebc  <-- easy way to convert Blanks to '_' Underscores
      ==============
 #5d. renameDU d0ebc  <-- easy way to convert '$' signs to '_' Underscores
      ==============

VU JCL/script filename standards

• lower case
• convert any embedded '$' & '#' to '_'s
• modify GDG G1234V00, etc to '_001234', or resequence to '_000001'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C4. Converting Mainframe Data Files from EBCDIC to ASCII

using demo files vs FTP from mainframe

Here are the demo files we will use to demonstrate datafile conversion. These were copied into $CNVDATA/d0ebc/... on page '1B2'.

 -rw-rw-r--. 1 mvstest2 apps  8192 Mar 20 09:15 AR.CUSTOMER.ARMASTER
 -rw-rw-r--. 1 mvstest2 apps  1280 Mar 20 09:15 AR.CUSTOMER.ARSALES
 -rw-rw-r--. 1 mvstest2 apps  6068 Mar 20 09:15 AR.CUSTOMER.ARTRANS
 -rw-rw-r--. 1 mvstest2 apps  2560 Mar 20 09:15 AR.CUSTOMER.PHONE.LIST
 -rw-rw-r--. 1 mvstest2 apps  1280 Mar 20 09:15 CITY.TAX.ASSESS
 -rw-rw-r--. 1 mvstest2 apps  1680 Mar 20 09:15 CITY.TAX.PAYMENTS
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar 20 09:15 GL.ACCOUNT.GLMASTER.G0001V00
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar 20 09:15 GL.ACCOUNT.GLMASTER.G0002V00
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar 20 09:15 GL.ACCOUNT.GLMASTER.G0003V00
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar 20 09:15 GL.ACCOUNT.GLTRAN.G0001V00
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar 20 09:15 GL.ACCOUNT.GLTRAN.G0002V00
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar 20 09:15 GL.ACCOUNT.GLTRAN.G0003V00
 -rw-rw-r--. 1 mvstest2 apps   249 Mar 20 09:15 VENDOR.MASTER.NAMES
 -rw-rw-r--. 1 mvstest2 apps   745 Mar 20 09:15 VENDOR.MASTER.PAYMENTS

Renaming mainframe files for conversion

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. renameL d0ebc    <-- rename all files to lower-case
     =============

Renaming GDG files

Utility 'mkcopyd0d1' makes a script to copy datafiles from d0ebc/* to d1ebc/..., lowercasing & renaming GDG files, depending on User OPtion uop=g0/g1/g2.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C5. Converting Mainframe Data Files from EBCDIC to ASCII

make script to copy/rename GDG files

 #2. uvcopy mkcopyd0d1,fild1=d0ebc,filo1=sf/copyd0d1,uop=g0
     ======================================================
     - make script to copy files from d0ebc/ to d1ebc/, lowercasing,&
       renaming GDG files depending on User OPtion uop=g0/g1/g2
     - uop=g0 is the default, but you could change at the prompt

 #2. ksh sf/copyd0d1 d0ebc d1ebc   <-- execute script (created above)
     ===========================

output files in d1ebc/ for conversion to d2asc/

 -rw-rw-r--. 1 mvstest2 apps  8192 Mar  6 17:21 ar.customer.armaster
 -rw-rw-r--. 1 mvstest2 apps  1280 Mar  6 17:21 ar.customer.arsales
 -rw-rw-r--. 1 mvstest2 apps  6068 Mar  6 17:21 ar.customer.artrans
 -rw-rw-r--. 1 mvstest2 apps  2560 Mar 15 14:54 ar.customer.phone.list
 -rw-rw-r--. 1 mvstest2 apps  1280 Mar  6 17:21 city.tax.assess
 -rw-rw-r--. 1 mvstest2 apps  1680 Mar  6 17:21 city.tax.payments
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar  6 17:21 gl.account.glmaster_000123
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar  6 17:21 gl.account.glmaster_000124
 -rw-rw-r--. 1 mvstest2 apps 13952 Mar  6 17:21 gl.account.glmaster_000125
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar  6 17:21 gl.account.gltran_000456
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar  6 17:21 gl.account.gltran_000457
 -rw-rw-r--. 1 mvstest2 apps  1600 Mar  6 17:21 gl.account.gltran_000458
 -rw-rw-r--. 1 mvstest2 apps   249 Mar  6 17:21 vendor.master.names
 -rw-rw-r--. 1 mvstest2 apps   745 Mar  6 17:21 vendor.master.payments

Notes about Re-Named files

1. The mainframe GDG suffixes '.G0001V00' are converted to '_000001' for the GDG file conventions used in Vancouver Utility JCL conversions to shell scripts.
   
   
2. We will later generate conversion jobs from the COBOL copybooks for each file.
   
   
3. For GDG files, we will generate only 1 job for each GDG set of files and then replicate that job for the number of generations exisitng in the d1ebc/ directory.
   
   
4. This saves having to possibly make manual changes to multiple jobs for multi-record-type files, or incorrect record sizes, etc.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C6. Converting Mainframe Data Files from EBCDIC to ASCII

using 'uvhd' to investigate DATA files

'uvhd' is an invaluable tool for displaying any kind of data-file in vertical hexadecimal - 3 lines per 64 byte segment - characters on the 1st, hex zones on the 2nd,& hex digits on the 3rd.

uvhd is especially useful for EBCDIC files using option 'a' to show the character line translated to ASCII, while retaining the EBCDIC values for the zones & digits on the 2nd & 3rd line.

Here is an example using uvhd to display the vendor.master.names file using option 'a' & option 'z4' (for RDW variable length records):

 uvhd d1ebc/vendor.master.names z4a
 ==================================

 rec#=1 rsize=23 fptr=0 fsize=249 records=8
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 M  Dell Inc.
             0100CCDDFF4D44C8994C984
             0700453310040045330953B

 rec#=2 rsize=30 fptr=23 fsize=249 records=8
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          23 ....HP0010 M  Hewlett Packard
             0100CDFFFF4D44C8A98AA4D8898984
             0E0087001004008563533071321940

 rec#=3 rsize=45 fptr=53 fsize=249 records=8
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
          53 ....IBM010 M  International Business Machines
             0200CCDFFF4D44C9A8998A899894CAA898AA4D888898A
             0D0092401004009535951396513024295522041389552

notes re RDW variable length files

1. Each record has a 4 byte prefix with the record size in binary in the 1st 2 bytes & nulls in the 3rd & 4th bytes (unused).
   
   
2. The record sizes for the 1st 3 records shown above (expressed in horizontal hex) are: x'0017', x'001E', x'002D'.
   
   
3. Converting to Decimal these are: 1*16+7=23, 1*16+14=30, 2*16+13=45
   
   
4. Of course uvhd has already done this for you & displayed the record-sizes rsize=23, rsize=30, rsize=45
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_2. Data File Conversion - Comprehensive & Complete

Part 2 - Creating the Conversion Control-File

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A1. Data File Conversion

Control-File for High-Volume Data Conversion

 We strongly recommend creating a control-file for high volume data conversions.
 When we run 'uvdata51' to generate data-conversion-jobs from the COBOL copybooks,
 the generated jobs have the same name as the copybooks. We then run 'uvdata52'
 with the control-file to relate the datafilenames to copybooknames and replicate
 the data-conversion-jobs with the datafilenames inserted in place of copybooknames.
 The control-file can also supply file info not readily available in the copybooks
 - such as file-type, indexed key locations,& perhaps record-sizes if we wish to
 change from the FTP/RDW record size.

test/demo control-file

 ar.customer.armaster  cpy=armaster typ=IDXf1  rca=0256 rcm=0256 rcs=0256 gdg=___ key=000,006
 ar.customer.arsales   cpy=arsales  typ=RSF    rca=0064 rcm=0064 rcs=0064 gdg=___ key=___,___
 ar.customer.artrans   cpy=custran3 typ=IDXf8v rca=0120 rcm=0249 rcs=0245 gdg=___ key=___,___
 ar.customer.phone.lst cpy=________ typ=RSF    rca=____ rcm=____ rcs=____ gdg=___ key=___,___
 city.tax.assess       cpy=citytax1 typ=RSF    rca=0128 rcm=0128 rcs=0128 gdg=___ key=___,___
 city.tax.payments     cpy=citytax2 typ=IDXf8v rca=0080 rcm=0128 rcs=0128 gdg=___ key=___,___
 gl.account.glmaster_  cpy=glmaster typ=RSF    rca=0128 rcm=0128 rcs=0128 gdg=003 key=___,___
 gl.account.gltran_    cpy=gltran   typ=RSF    rca=0080 rcm=0080 rcs=0080 gdg=004 key=___,___
 vendor.master.names   cpy=vendorms typ=IDXf8v rca=0019 rcm=0041 rcs=0064 gdg=___ key=___,___
 vendor.master.payment cpy=vendorpm typ=IDXf8v rca=0019 rcm=0062 rcs=0064 gdg=___ key=___,___

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A2. Data File Conversion

Low Volume Data Conversion

As noted above, we recommend taking the time to initially create a control-file if you have high-volumes of datafiles to be transferred & converted. Several methods are suggested to help automate creation of the control-file, in this Part_2.

BUT, if you only have a few files to convert, you could skip this Part_2, & proceed to Part_3, where you can run uvdata51 to generate data-conversion-jobs. You could then edit the generated jobs to insert the datafilenames in place of the copybooknames.

OR, you could manually edit/create the control file for the few files desired for example, if you only needed to convert the 1st 3 files, you could research the file info,& edit the control file as follows:

 vi ctl/datacpy52
 ================

 ar.customer.armaster  cpy=armaster typ=IDXf1  rca=0256 rcm=0256 rcs=0256 gdg=___ key=000,006
 ar.customer.arsales   cpy=arsales  typ=RSF    rca=0064 rcm=0064 rcs=0064 gdg=___ key=___,___
 ar.customer.artrans   cpy=custran3 typ=IDXf8v rca=0120 rcm=0249 rcs=0245 gdg=___ key=___,___

To save substantial editing I recommend you run datactl61 as on page '2D2' to create a skeleton control-file with the filenames you have FTP'd to d1ebc/...

 ar.customer.armaster  cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.arsales   cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.artrans   cpy=____________ typ=RDWz4  rca=00120 rcm=00249 rcs=_____ gdg=___ key=___,___

datactl61 can fill-in the min/max record sizes for variable length files FTP'd with the RDW option. If you had converted fixed-length files to variable-length on the mainframe before FTP/RDW as recommended on page '1C1' then datactl61 could generate the control file as follows:

 ar.customer.armaster  cpy=____________ typ=RSF    rca=00256 rcm=00256 rcs=_____ gdg=___ key=___,___
 ar.customer.arsales   cpy=____________ typ=RSF    rca=00064 rcm=00064 rcs=_____ gdg=___ key=___,___
 ar.customer.artrans   cpy=____________ typ=RDWz4  rca=00120 rcm=00249 rcs=_____ gdg=___ key=___,___

Now, you only need to add the copybooknames, fill-in rcs=... (copybook size), & the key location/length for armaster.

Then, you can skip the rest of Part 2 & proceed to Part 3 page '3B1' where you will skip #1 (copies ctl/datactl63 to ctl/datacpy52), since we have created datacpy52 manually & dont need control-file creation jobs datactl62 & 63.

You will run #2 - #5 on page '3B1' & '3B2', and your manually created control-file will be input to 'uvdata52' on page '3B3'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A3. Data File Conversion

Sources for Control-File information

1. LISTCAT reports - from the mainframe may be transferred to unix/linux & script 'catdata50' run to extract record-sizes, file-types, Indexed-key-locations,& number of GDG generations. See sample LISTCAT report listed on '2B2'
   
   
2. COBOL copybooks - can be used by script 'cpyrcs1A' to create Indexed file 'cpyrcs1I', to supply record-sizes for datafiles in case LISTCAT does not include some datafiles. Need to precede with utility 'xrefcpy51' to create Indexed file 'ctlcpy1I' to relate the datafilename to the copybookname. The input could be a .csv file exported from a user spreadsheet OR created by editing ,copybooknames onto the directory list of datafiles to be converted.
   
   
3. Excel spreadsheets - some sites have spreadsheets that include file info (record-sizes, file-types, Indexed-key-locations,& no of GDG generations). You can convert your spreadsheet into the required keyword coded format, using a uvcopy job similar to the sample provided. You can copy/rename sample uvcopy job $UV/pf/IBM/xlsdata51 & modify for your .csv file format. Also see sample corresponding input file $UV/ctl/xlsdata51.csv Note that the output file is written into ctl/cat2/... from which all files are combined into one control file ctl/datacat51.
   
   
4. JCL conversion - leaves a compatible control-file at $RUNLIBS/ctl/datactl53, contains info extracted from the mainframe JCL (record-sizes, file-types,& Indexed-key-locations) from output file definitions. File info will be missing for files that are never declared as outputs anywehere in all JCL. If you want to include the JCL file info into the DATA conversion control- file, you would copy $RUNLIBS/ctl/datactl53 into ctl/cat2/... from which all files are combined into one control file ctl/datacat51. You probably should not include the JCL file info with the DATA control-file if you have complete LISTCAT control files.
   
   
5. Editor - you may create text files in the required keyword format to supply info (rec-sizes,file-types,key-locations,generations) for files that you notice have missing or incorrect info on your initial conversions. Add any number of such files with any filenames desired to $CNVDATA/ctl/cat2 & rerun the 'catdata50' script which will include info from all ctl/cat0,1,2 subdirs into the data conversion control file $CNVDATA/ctl/datacat51, which is then loaded into Indexed file datacat52I, whose info is combined with other info by utility ctldata63 into final Indexed file $CNVDATA/ctl/datactl63, which is the input into utility 'uvdata52' which completes the data conversion jobs by inserting datafilenames (vs copybooknames) & adding file info that may be unavailable in the copybooks.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A4. Data File Conversion

Identifying files for transfer to Unix/Linux

Here are suggestions how you might determine the essential files that you need to transfer from the mainframe to initiate conversions & operations on the unix/linux system. You would want to avoid transferring the many temporary, inter-step, inter-job files that are not required to initiate your applications on the new system.

1. Existing documentation (spreadsheets) that you may have. As well as determining datafiles you need to transfer, you can convert them into the control-file keyword format required for the datafile conversions. See page '5B4' & sample uvcopy job '$UV/pf/IBM/xlsdata51'
   
   
2. Essential files report - see https://uvsoftware.ca/jclcnv2real.htm#1P1 - 1P4. Script 'mvsfiles5A' reads all JCL and creates a report of the essential files required to initiate applications on the new system, dropping temporary & intermediate files. The final report of essential files is created at $RUNLIBS/testlibs/xmvsA/mvsfiles6. Page 1P4 shows how to convert that into a control-file, but the keyword info would be based on the JCL (vs the LISTCAT info which may be more accurate for data conversion).
   
   

2a. The Essential files report generated from All JCL could be reasonably accurate if jobnames were assigned in alpha sequence, so filenames in new jobs are from previous jobs, or are in fact new files for the whole system. Reports generated for 1 JCL at a time are 100% accurate, since thre is no confusion whether a filename is the 1st instance in the JCL or a subsequent instance (input from a previous step).

3. Manually creating a list of files to be transferred based on your knowledge. (if all else fails).
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A5. Data File Conversion

Transfer datafiles in logical groups

You will probably want to transfer your datafiles in logical groups for testing one application at a time. After FTP'ing a logical group, you can create the control-file to convert that group, from the directory of FTP'd datafiles (see operating instructions beginning at '2B1').

If you do create one master control-file (from an existing spreadsheet or the 'Essential Files Report'), I suggest you append another keyword on each file to identify the logical FTP group (for example grp=ap,ar,gl,etc).

Then you could run 'uvcopy keep1' (as shown on page '5B2'), to select the next desired logical group of files for FTP.

Generating FTP JCL automatically

See JCLcnv2real.htm#4D1 - 4D5 to automatically generate the mainframe JCL required to FTP the selected group of data files. Utility 'genftpjcl1' runs only on unix/linux, so you must transfer the generated JCL back to the mainframe for execution.

This assumes that you have created that master control-file (mentioned above). There is a big advantage if the control-file accurately identifies variable- length record files (when rca=... is non-zero & different than rcm=...). The 'genftpjcl1' utility can then generate the proper FTP JCL to transfer variable-length files vs fixed-length files.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B1. Prep to Create the data conversion Control-File

Alternative if only a few datafiles

If you only have a few files to convert, you could skip this Part_2, & proceed to Part_3, to run uvdata51 to generate data-conversion-jobs. You could then edit the generated jobs to insert the datafilenames in place of the copybooknames.

OR (better) run just 'uvcopy ctldata61' on page '2D2' to create the control file picking up record-sizes from files FTP'd with option 'RDW'. ctldata61 will code minimum recsize in rca=... & maximum recsize in rcm=... You can then edit ctl/datactl61 to insert the copybookname in cpy=____________ & the copybook record-size on rcs=_____

Pre-Requisites Creating the Control-File

1. EBCDIC Data Files to be converted transferred & stored in subdir d0ebc/...
   
   
2. LISTCAT reports transferred & stored in subdir ctl/cat0/... & script 'catdata50' run to create Indexed file 'ctl/datacat52I' of keyword datafile info (record-sizes,file-type,GDGs,IndexedFileKeys)
   
   
3. Copybook/record-size cross-ref Indexed file stored in ctl/cpyrcs1I - created by script 'cpyrcs1A' #4 on page '2C1')
   
   
4. Datafile/copybook cross-ref .csv file stored in ctl/datacopyxref - exported from user spreadsheet if available, if not it can be created from datafilenames edited with copybooknames
   
   
5. JCL conversion control-file could also be used for DATA conversion, especially if you did not have complete info from the LISTCAT reports.
   
   
6. Miscellaneous control files created with the editor to supply missing info (record-sizes, file-types, GDGs, Indexed Key Locations)
   
   

   1. EBCDIC Data Files

7. EBCDIC Data Files to be converted must be FTP'd from mainframe to Unix/Linux into $CNVDATA/d0ebc/... ($CNVDATA might be /home/userxx/cnvdata/) All files should be FTP'd Binary & Variable-Length files must be FTP'd with the 'RDW' option to store record length in binary in record headers.
   
   

      ar.customer.armaster     <-- test files provided for these demos
      ar.customer.arsales
      ar.customer.artrans
      ar.customer.phone.list
      city.tax.assess
      city.tax.payments
      gl.account.glmaster_
      gl.account.gltran_
      vendor.master.names
      vendor.master.payments

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B2. Prep to Create the data conversion Control-File

2. Extracting file-info from LISTCAT reports

If mainframe LISTCAT reports are available, you can transfer them to unix & run the 'catdata50' script to extract: file types, GDG generations, & Indexed key locations (start,length).

2a. FTP mainframe LISTCAT reports must be stored in cnvdata/ctl/cat0/... May have multiple reports & filenames do not matter, since all files will be processed from ctl/cat0/* to ctl/cat1/... & ctl/cat2/..., and then combined into 1 file in ctl/datacat52 (text file) & ctl/datacat52I (Indexed file). See more LISTCAT info at https://uvsoftware.ca/jclcnv2real.htm#3C1

sample LISTCAT - ctl/cat0/listcat001

 0CLUSTER ------- AR.CUSTOMER.ARMASTER
     ASSOCIATIONS
       DATA-----AR.CUSTOMER.ARMASTER.DATA
       INDEX----AR.CUSTOMER.ARMASTER.INDEX
       KEYLEN-----------------6  AVGLRECL-----------256  BUFSPACE---------14366
       RKP--------------------0  MAXLRECL-----------256  EXCPEXIT--------(NULL)
       REC-TOTAL-------------32  SPLITS-CI------------0  EXCPS--------------495

 0CLUSTER ------- AR.CUSTOMER.ARSALES
       KEYLEN-------0  AVGLRECL--------64  BUFSPACE------4096    CISIZE--------2048
       RKP----------0  MAXLRECL--------64  EXCPEXIT------(NULL)  CI/CA----------270
       REC-TOTAL---20  SPLITS-CI------0    EXCPS----------8

 0GDG BASE ------ GL.ACCOUNT.GLMASTER
       KEYLEN-------0  AVGLRECL-------128  BUFSPACE------4096    CISIZE--------2048
       RKP----------0  MAXLRECL-------128  EXCPEXIT------(NULL)  CI/CA----------270
       LIMIT------------------3            SCRATCH               NOEMPTY
     ASSOCIATIONS
       NONVSAM--GL.ACCOUNT.GLMASTER.G0123V00
       NONVSAM--GL.ACCOUNT.GLMASTER.G0124V00
       NONVSAM--GL.ACCOUNT.GLMASTER.G0125V00

 0GDG BASE ------- GL.ACCOUNT.GLTRAN
       KEYLEN-------0  AVGLRECL--------80  BUFSPACE------4096    CISIZE--------2048
       RKP----------0  MAXLRECL--------80  EXCPEXIT------(NULL)  CI/CA----------270
       LIMIT------------------4            SCRATCH               NOEMPTY
     ASSOCIATIONS
       NONVSAM--GL.ACCOUNT.GLTRAN.G0456V00
       NONVSAM--GL.ACCOUNT.GLTRAN.G0457V00
       NONVSAM--GL.ACCOUNT.GLTRAN.G0458V00
       NONVSAM--GL.ACCOUNT.GLTRAN.G0459V00

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B3. Prep to Create the data conversion Control-File

sample LISTCAT - ctl/cat0/listcat002

 0CLUSTER ------- AR.CUSTOMER.MASTER
       KEYLEN-----------------6  AVGLRECL-----------256  BUFSPACE---------14366
       RKP--------------------0  MAXLRECL-----------256  EXCPEXIT--------(NULL)
       REC-TOTAL-------------32  SPLITS-CI------------0  EXCPS--------------495

 0CLUSTER ------- VENDOR.MASTER.NAMES
       KEYLEN-------0  AVGLRECL-------64   BUFSPACE------4096    CISIZE--------2048
       RKP----------0  MAXLRECL-------256  EXCPEXIT------(NULL)  CI/CA----------270
       REC-TOTAL-5428  SPLITS-CI------0    EXCPS----------8

 0CLUSTER ------- PY.PAYROLL.MASTER
       KEYLEN-----6    AVGLRECL--256       BUFSPACE-----14336    CISIZE--6144
       RKP--------0    MAXLRECL--512       EXCPEXIT----(NULL)    CI/CA----105
       REC-TOTAL-15946 SPLITS-CI--0        EXCPS-178347

notes re LISTCAT reports from mainframe

1. FTP ASCII from mainframe & store in $CNVDATA/ctl/cat0
   
   
2. There can be any number of LISTCAT reports from various catalogs & the filenames do not matter - all files will be combined by script 'catdata50' (see step #2d on page '2B4').
   
   

 ctl/cat0/* -----> ctl/cat1/* -----> ctl/cat2/* -----> ctl/datacat51 --> ctl/datacat52I
         catdata51         catdata52          catcat51              uvcp

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B4. Prep to Create the data conversion Control-File

LISTCAT missing record-sizes for NON-VSAM files

2b. If the LISTCAT reports do not pick up record-size info for NON-VSAM files, a mainframe expert can create a special report with recsizes for NON-VSAM files. FTP this separate file into $CNVDATA/ctl/lrecl0.

           DATA SET NAME                          VOLUME  ORG RECFM LRECL  BLKSZ
      AR.CUSTOMER.ARSALES                         SAG001   PO   FB     64   6400
      GL.ACCOUNT.GLTRAN.G1234V00                  SAG102   PS   FB     80   8000
      PY.PAYROLL.TIME.CARDS                       SAG203   PS   FB    128  12800

Then run utility 'lrecl0cat2' to convert to keyword format in ctl/cat1/lrecl2, from where it will be combined with other LISTCAT files into ctl/cat2 & then reduced to 1 entry per filename by utility 'catcat51' outputting 1 combined file to ctl/datacat52I - Indexed file adding keyword info to the data conversion controlfile ctl/datacpy52, which is the primary input to utility 'uvdata52' which inserts datafilenames into the data conversion jobs.

 #2c. uvcopy lrecl0cat2,fili1=ctl/lrecl0,filo1=ctl/cat1/lrecl2
      ========================================================
      - no need to run 'lecl0cat2' here, since included in 'catdata50' script below

      ar.customer.arsales             rca=00064 rcm=00064 rcs=00064 typ=RSF
      gl.account.gltran_              rca=00080 rcm=00080 rcs=00080 typ=RSF    gdg=___
      py.payroll.time.cards           rca=00128 rcm=00128 rcs=00128 typ=RSF

catdata50 script combines all LISTCAT & other control files

 #2d. catdata50 all <-- run script 'catdata50' to process all files from ctl/cat0/...
      =============     to ctl/cat1/..., then to ctl/cat2/..., then combine into
                        one file ctl/datacat52 & ctl/datacat52I (Indexed file).
                      - see 'catdata50' script listed at '9X3'

 ctl/cat0/* -----> ctl/cat1/* -----> ctl/cat2/* -----> ctl/datacat51 --> ctl/datacat52I
         catdata51         catdata52          catcat51              uvcp

 ar.customer.armaster      rca=00256 rcm=00256 rcs=00256 typ=IDXf1  gdg=___ key=000,006
 ar.customer.arsales       rca=00064 rcm=00064 rcs=00064 typ=RSF    gdg=___ key=_______
 ar.customer.master        rca=00256 rcm=00256 rcs=00256 typ=IDXf1  gdg=___ key=000,006
 gl.account.glmaster_      rca=_____ rcm=_____ rcs=_____ typ=RSF    gdg=003 key=_______
 gl.account.gltran_        rca=00080 rcm=00080 rcs=00080 typ=RSF    gdg=004 key=_______
 py.payroll.master         rca=00256 rcm=00512 rcs=00512 typ=IDXf8v gdg=___ key=000,006
 py.payroll.time.cards     rca=00128 rcm=00128 rcs=00128 typ=RSF    gdg=___ key=_______
 vendor.master.names       rca=00064 rcm=00256 rcs=00256 typ=IDXf8v gdg=___ key=_______

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C1. Prep to Create the data conversion Control-File

3. Creating cross-ref of copybooknames & record-sizes

 #3. cpyrcs1A cpys maps ctl  <-- script create cross-ref copybooknames & record-sizes
     ======================    - see 'cpyrcs1A' script listed at '9X4'

      cpys -------> maps ---------> ctl/cpyrcs1 --------> ctl/cpyrcs1I
           cobmap1        cpyrcs1                uvsort

Script 'cpyrcs1A' runs the following utilities

 3a. uvcopyx cobmap1 - convert copybooks to record layouts (maps)
 3b. uvcopy cpyrcs1  - converts copybook maps to ctl/cpyrcs1
 3c. uvsort          - load ctl/cpyrcs1 to Indexed file ctl/cpyrcs1I

     armaster                      rcs=00256 pbz=p__   01=armaster
     arsales                       rcs=00064 pbz=__z   01=arsales
     citytax1                      rcs=00128 pbz=pbz   01=citytax1rec
     citytax2                      rcs=00128 pbz=pbz   01=citytax2
     custran3                      rcs=00245 pbz=p__   01=custran3
     glmaster                      rcs=00128 pbz=p__   01=glmaster
     gltran                        rcs=00080 pbz=__z   01=gltran
     vendormas                     rcs=00064 pbz=___   01=vendormas
     vendorpaymas                  rcs=00064 pbz=___   01=vendorpaymas

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C2. Prep to Create the data conversion Control-File

4. Store Existing Datafile/Copybook cross-ref

Utility 'ctldata62' (step #4 on page '2F1') requires an Indexed file cross- reference of datafilenames & copybooknames (ctl/ctlcpy1I).

You may already have a datafilename/copybookname crossreference - perhaps an excel spreadsheet. If so & if the 1st 2 fields are datafilename & copybookname, you can export a .csv file & store it as ctl/datacopyxref. Then utility 'xrefcpy51' (step #3 on page '2E2') will load it into the Indexed file 'ctlcpy1I' required by step #4 'ctldata62'

 #4. cp .../... ctl/datacopyxref
     ===========================
     - copy/rename your existing .csv spreadsheet for use by xrefcpy51 step #3.
     - 1st field datafilename, 2nd copybookname, OK to have extra fields

If you do not have a datafilename/copybookname cross-reference, then you can make it using the optional steps #2a & #2b on following page '2E1'. Steps #2a & #2b will be omitted if you have the cross-reference & store it in ctl/datacopyxref for step #3 'xrefcpy51' on page '2E2'.

We did not have an existing spreadsheet datafile/copybook crossref, so we made the .csv file version using optional steps #4a & #4b below and the result will be as follows:

 #4a. cp tmp/datacopyxref ctl/ - copy tmp/file to ctl/...
      ======================== - written to tmp/... in case already in ctl/...

 #4b. vi ctl/datacopyxref - edit/append copybooknames onto datafilenames
      ==================  - separated by a ',' comma OR 1 or more spaces

      ar.customer.armaster,armaster
      ar.customer.arsales,arsales
      ar.customer.artrans,custran3
      ar.customer.customer.phone.list,       <-- no copybook available
      city.tax.assess,citytax1
      city.tax.payments,citytax2
      gl.account.glmaster_,glmaster
      gl.account.gltran_,gltran
      vendor.master.names          ,vendormas    <-- OK to leave spaces
      vendor.master.payments        vendorpaymas <-- OK to omit commas

 #4c. uvcopy xrefcpy51,fili1=ctl/datacopyxref,filo1=ctlcpy1I
      ======================================================
      - convert csv file from spreadsheet of datafilenames & copybooknames
        into Indexed file for lookup by datafilename to get copybookname
      - separates at the ',' & offsets right for 44 byte key lookup

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C3. Prep to Create the data conversion Control-File

adding JCL control-file info to Data control-file

JCL conversion control-file could also be used for DATA conversion, especially if you did not have complete info from the LISTCAT reports.

Here is a sample from the JCL conversion control-file. It will contain an entry for every filename found in the JCL when converted to shell scripts. Many entries will not match the much smaller set of datafiles selected for datafile transfer & conversion.

 ar.customer.master.armaster  cpy=________ rca=00256 rcs=00256 gdg=___ typ=RSF  key=___,___
 ar.customer.nameadrs.list120 cpy=________ rca=00133 rcs=00133 gdg=___ typ=LSTt key=___,___
 ar.customer.arsales          cpy=________ rca=00080 rcs=00080 gdg=___ typ=RSF  key=___,___
 gl.account.acntlist_         cpy=________ rca=00133 rcs=00133 gdg=___ typ=LSTt key=___,___
 gl.account.glmaster_         cpy=________ rca=00128 rcs=00128 gdg=___ typ=RSF  key=___,___
 gl.account.gltran_           cpy=________ rca=00080 rcs=00080 gdg=___ typ=RSF  key=___,___

If you decide to include the JCL conrol-file, copy from testlibs/ctl/datactl53I to cnvdata/ctl/. If not you must make an empty Indexed file as shown below.

 #5. cp ../testlibs/ctl/datactl53I ctl/
     ==================================
     - copy JCL control-file for inclusion in DATA conversion control-file

              ----- OR -----

 #5a. makeISF0 ctl/datactl53I 191 0,70
      ================================
      - make empty Indexed file to avoid missing file error

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C4. Prep to Create the data conversion Control-File

5. Editor created control-files

You could do this 'AFTER THE 1ST CONVERSION WHEN YOU NOTICE MISSING OR INCORRECT INFO' (record-sizes, file-types, GDGs, Indexed Key Locations). You would add these editor created extra control files to ctl/cat2.

Enter the filename on the left & the desired keywords on the right in any sequence with at least 1 space between filename & keywords. You can name these extra files anything you like, for example:

 #6. vi ctl/cat2/newinfo
     ===================

     data.file.aaa  rcs=80          <-- at least 1 space between filename & keywords
     data.file.bbb  cpy=copybkaa    <-- does not matter if you code .cpy extension
     data.file.ccc  cpy=copybkbb rcs=4000
     :w    <-- write out

Then you would rerun script 'catdata50', which combines all control files in ctl/cat2/* into ctl/datacat52I, used by ctldata63 to create the final control file ctl/datactl63, which is copied to ctl/datacpy52 for input to uvdata52 which completes the data conversion jobs with datafilename & file characteristics.

can omit filenames on uvcopy jobs below

You can omit entering the filenames for most of the uvcopy jobs below. The filenames shown are the defaults & are normally correct. You will be prompted to allow change, but just hit enter to accept the default. For example:

 #1. uvcopy ctldata61,fild1=d0ebc,filo1=ctl/datactl61,filo2=tmp/datacopyxref
     =======================================================================
     - filenames shown are the defaults & are correct

 #1. uvcopy ctldata61   <-- omit filenames, same as above (easier to enter)
     ================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D1. Operating Instructions to create the control file

Verify Pre-Requisites completed

1. Data files to be converted transferred & stored in subdir d0ebc/...
   
   
2. LISTCAT reports transferred & stored in subdir ctl/cat0/... & script 'catdata50' run to create Indexed file 'ctl/datacat52I' of keyword datafile info (record-sizes,file-type,GDGs,IndexedFileKeys)
   
   
3. Copybook/record-size cross-ref Indexed file stored in ctl/cpyrcs1I - created by script 'cpyrcs1A' #4 on page '2C1')
   
   
4. Datafile/copybook cross-ref .csv file stored in ctl/datacopyxref - exported from user spreadsheet if available, if not it can be created from datafilenames edited with copybooknames
   
   
5. JCL conversion control-file could also be used for DATA conversion, especially if you did not have complete info from the LISTCAT reports.
   
   
6. Miscellaneous control files created with the editor to supply missing info (record-sizes, file-types, GDGs, Indexed Key Locations)
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D2. Operating Instructions to create the control file

create initial control file from datafilenames in directory

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. uvcopy ctldata61,fild1=d1ebc,filo1=ctl/datactl61,filo2=tmp/datacopyxref
     =======================================================================
   - create control-file from the datafilenames in d0ebc/...

 ar.customer.armaster   cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.arsales    cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.artrans    cpy=____________ typ=RDWz4  rca=00120 rcm=00249 rcs=_____ gdg=___ key=___,___
 ar.customer.phone.list cpy=____________ typ=RSF    rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.payments      cpy=____________ typ=RDWz4  rca=00080 rcm=00128 rcs=_____ gdg=___ key=___,___
 gl.account.glmaster_   cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 gl.account.gltran_     cpy=____________ typ=_____  rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 vendor.master.names    cpy=____________ typ=RDWz4  rca=00019 rcm=00041 rcs=_____ gdg=___ key=___,___
 vendor.master.payments cpy=____________ typ=RDWz4  rca=00019 rcm=00062 rcs=_____ gdg=___ key=___,___

• also writes 2nd file of datafilenames only for possible use making
  a cross-ref file by ',' appending copybooknames on the right side
  (or user may already have an excel spreadsheet of datafiles & copybooks)

      ar.customer.armaster     <-- 2nd output ctl/datacopyxref
      ar.customer.arsales        - datafilenames only to add copybooknames
      ar.customer.artrans        - if you don't already have this crossref
      ar.customer.phone.list
      city.tax.assess
      city.tax.payments
      gl.account.glmaster_
      gl.account.gltran_
      vendor.master.names
      vendor.master.payments

Alternative if only a few datafiles

If you have only a few datafiles, you could run just 'uvcopy ctldata61' to create the control file picking up record-sizes from files FTP'd with option 'RDW'. ctldata61 will code minimum recsize in rca=... & maximum recsize in rcm=... You can then edit ctl/datactl61 to insert the copybookname in cpy=____________ & the copybook record-size on rcs=_____

Then you could skip the rest of this Part 2 & proceed to Part 3, where you will run generate the data conversion jobs with cobmap1, uvdata51,& uvdata52.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2E1. Operating Instructions to create the control file

create datafile/copybook crossref

 #2. Create datafilename/copybookname cross-reference
     ================================================
     - from tmp/datacopyxref OR you may already have a spreadsheet that
       you have exported to a .csv file for input to 'xrefcpy51' #3 below.

      Note: if you do not have the .csv file, then do #2a & #2b will create it
      for the current group of datafiles. You only need to do #2a & #2b
      the 1st time you do each group of datafiles since ctldata61 writes
      datacopyxref to tmp/.. & does not overwrite your edited ctl/datacopyxref.

      This saves repeating the edits if you re-convert the same group of datafiles
      And you could add the tmp/datacopyxref to existing ctl/datacopyxref for
      when you convert new groups of datafiles, building up ctl/datacopyxref.

 #2a. cp tmp/datacopyxref ctl/ - copy tmp/file to ctl/...
      ======================== - was written to tmp/ in case you already had

 #2b. vi ctl/datacopyxref - edit/append copybooknames onto datafilenames
      ==================  - separated by a ',' comma OR a blank OR blanks
      - ',' separator allows using an existing .csv spreadsheet alternative

      ar.customer.armaster,armaster        <-- edited with copybooknames
      ar.customer.arsales,arsales
      ar.customer.artrans,custran3
      ar.customer.phone.list,              <-- no copybook available
      city.tax.assess,citytax1
          --- or you may leave spaces between filename & copybook ---
      city.tax.payments                    ,citytax2
      gl.account.glmaster_                 ,glmaster
      gl.account.gltran_                   ,gltran
          --- or you may omit commas & leave spaces between filename & copybook ---
      vendor.master.names                   vendormas
      vendor.master.payments                vendorpaymas

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2E2. Operating Instructions to create the control file

load Indexed file of datafile/copybook crossref

 #3. uvcopy xrefcpy51,fili1=ctl/datacopyxref,filo1=ctl/ctlcpy1I
     ==========================================================
     - convert csv file from spreadsheet of datafilenames & copybooknames
       into Indexed file for lookup by datafilename to get copybookname
     - separates at the ',' & offsets right for 44 byte key lookup

      ar.customer.armaster                         cpy=armaster
      ar.customer.arsales                          cpy=arsales
      ar.customer.artrans                          cpy=custran3
      ar.customer.phone.list                       cpy=____________
      city.tax.assess                              cpy=citytax1
      city.tax.payments                            cpy=citytax2
      gl.account.glmaster_                         cpy=glmaster
      gl.account.gltran_                           cpy=gltran
      vendor.master.names                          cpy=vendormas
      vendor.master.payments                       cpy=vendorpaymas

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F1. Operating Instructions to create the control file

insert copybooknames & record-sizes into control-file

 #4. uvcopy ctldata62,fili1=ctl/datactl61,filo1=ctl/datactl62
                     ,filr2=ctl/cpyrcs1I,filr3=ctl/ctlcpy1I
     ========================================================
     - insert copybooknames & record-sizes into control-file
     - copybooknames from ctl/cpycpy1I & record-sizes from ctl/cpyrcs1I

 ar.customer.armaster   cpy=armaster typ=_____ rca=00256 rcm=00256 rcs=00256 gdg=___ key=___,___
 ar.customer.arsales    cpy=arsales  typ=_____ rca=00064 rcm=00064 rcs=00064 gdg=___ key=___,___
 ar.customer.artrans    cpy=custran3 typ=RDWz4 rca=00120 rcm=00249 rcs=00245 gdg=___ key=___,___
 ar.customer.phone.list cpy=________ typ=RSF   rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=citytax1 typ=_____ rca=00128 rcm=00128 rcs=00128 gdg=___ key=___,___
 city.tax.payments      cpy=citytax2 typ=RDWz4 rca=00080 rcm=00128 rcs=00128 gdg=___ key=___,___
 gl.account.glmaster_   cpy=glmaster typ=_____ rca=00128 rcm=00128 rcs=00128 gdg=___ key=___,___
 gl.account.gltran_     cpy=gltran   typ=_____ rca=00080 rcm=00080 rcs=00080 gdg=___ key=___,___
 vendor.master.names    cpy=vendorm  typ=RDWz4 rca=00019 rcm=00041 rcs=00064 gdg=___ key=___,___
 vendor.master.payments cpy=vendorpm typ=RDWz4 rca=00019 rcm=00062 rcs=00064 gdg=___ key=___,___

update control file with LISTCAT & JCL info

 #5a. cp ctl/datactl62 ctl/datactl63
      ==============================
      - copy file & skip #5 'uvcopy ctldata63' when no LISTCAT control files

 #5b. makeISF0 ctl/datacat52I 191 0,44
      ================================
      - make empty control file for LISTCAT info

 #5c. makeISF0 ctl/datactl53I 191 0,70
      ================================
      - make empty control file for JCL info

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F2. Operating Instructions to create the control file

update control file with LISTCAT info

 #5. uvcopy ctldata63,fili1=ctl/datactl62,filo1=ctl/datactl63
                     ,filr2=ctl/datacat52I,filr3=ctl/datactl53I
     ==========================================================
     - update control file with LISTCAT & JCL info
     - copy updating keyword info from Indexed file ctl/datacat52I

 ar.customer.armaster   cpy=armaster typ=IDXf1  rca=00256 rcm=00256 rcs=00256 gdg=___ key=000,006
 ar.customer.arsales    cpy=arsales  typ=RSF    rca=00064 rcm=00064 rcs=00064 gdg=___ key=___,___
 ar.customer.artrans    cpy=custran3 typ=IDXf8v rca=00120 rcm=00249 rcs=00245 gdg=___ key=___,___
 ar.customer.phone.list cpy=________ typ=RSF    rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=citytax1 typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=___ key=___,___
 city.tax.payments      cpy=citytax2 typ=IDXf8v rca=00080 rcm=00128 rcs=00128 gdg=___ key=___,___
 gl.account.glmaster_   cpy=glmaster typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=003 key=___,___
 gl.account.gltran_     cpy=gltran   typ=RSF    rca=00080 rcm=00080 rcs=00080 gdg=004 key=___,___
 vendor.master.names    cpy=vendorm  typ=IDXf8v rca=00019 rcm=00041 rcs=00064 gdg=___ key=___,___
 vendor.master.payments cpy=vendorpm typ=IDXf8v rca=00019 rcm=00062 rcs=00064 gdg=___ key=___,___

Notes re file typ=... & rca,rcs,rcb

1. During conversion in $CNVDATA, file types are either 'typ=RSF' for fixed lth OR 'typ=RDWz4' for variable length files
   
   
2. VarLth 'typ=RDWz4' set when ctldata61 detects different record-sizes in d0ebc (any variable-length files must be FTP'd with option RDW)
   
   
3. File Types in final control-file ctl/datactl63 changed to indicate the file type intended for the fully converted datafile when copied to $RUNDATA 'typ=IDXF1' for fixed-length Indexed files 'typ=IDXf8v' for variable-length files (Indexed if keylth > 0)
   
   
4. rca=... & rcm=... are set when ctldata61 reads datafiles in d0ebc/... rcm=... is the minimum record-length found & rcm=... is the maximum rcs=... is the record-length from the copybook/cobmap
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F3. Operating Instructions to create the control file

check control file for missing/incorrect info

We have just created the control-file required to perform the datafile conversions. Before you proceed to use it to generate the data conversion jobs, I suggest you check the keyword coded info for missing copybooks, record-sizes, file-types, etc. You might grep for 'cpy=___' as follows:

 grep 'cpy=___' ctl/datactl63     <-- search for missing copybooks
 ============================

 ar.customer.phone.list cpy=________ typ=RSF  rca=_____ rcm=_____ rcs=_____ gdg=___ key=__,__

For our control-file, we only get 1 entry where the copybook was missing & this is OK, because the file is all character (no packed/binary/signed fields). 'uvdata52' will use the 'skeleton2' uvcopy job template which will translate the entire record correctly. But, you should do the research to determine what the record size should be & update the control file 'rcs=_____'

 grep 'rcs=___' ctl/datactl63     <-- also search for missing record-sizes
 ============================

 vi ctl/datactl63      <-- update any missing copybooks & record-sizes
 ================

 ar.customer.phone.list cpy=________ typ=RSF  rca=_____ rcm=_____ rcs=00080 gdg=___ key=__,__

In this case, you only need to update the 'rcs=_____' since that is what is used when creating the uvcopy jobs.

If you have a lot of missing record sizes, you might be able to create more LISTCAT reports on the mainframe to cover those files. FTP (ASCII mode) to unix/linux, store in ctl/cat0,& rerun script 'catdata50' to include the new info with the previous info in ctl/cat2/...

Or if you have converted the JCL, transfer the JCL conversion control file from $RUNLIBS/ctl/datactl53I (.dat & .idx) to $CNVDATA/ctl/cat2/... & rerun script 'catdata50'.

And/Or you can edit additional text control files & add to ctl/cat2 to be combined with the other control-files, always resulting in 1 entry per datafilename with combined keyword coded info on the right.

It does not matter what you call the files & you only need to code the keywords that you wish to correct, for example, if you wanted to add the record-size for a few data files you could edit as follows:

 vi ctl/cat2/newinfo
 ===================

 data.file.aaa  rcs=80          <-- at least 1 space between filename & keywords
 data.file.bbb  cpy=copybkaa    <-- does not matter if you code .cpy extension
 data.file.ccc  cpy=copybkbb rcs=4000
 :w     <-- write out & re-run ctldata61-63 to re-create control file

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F4. Operating Instructions to create the control file

script to re-run ctldata61 thru ctldata63

To make re-runs easier we provide script 'ctldata6A'

 #1-5. ctldata6A all   <-- run or re-run utilities ctldata61 thru ctldata63
       =============     - see 'ctldata6A' script listed at '9X5'

 #5a. diff ctl/datacpy52 ctl/datacpy52
      ================================
      - see what changes have been made, before overwriting old control-file

 #6. cp ctl/datactl63 ctl/datacpy52
     ==============================
     - copy/rename to agree with 'uvdata52' & earlier documentation.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_3. Data File Conversion - Comprehensive & Complete

Part 3 - Generating the Data Conversion Jobs

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A1. Generating Data-Conversion-Jobs from copybooks & control-file

Pre-Requisites to Generating jobs

 #1. COBOL copybooks stored in $CNVDATA/cpys/...

 #2. Data-Conversion-Control-File generated in $CNVDATA/ctl/datacpy52
     See '2B1' - '2C4' operating instructions to create the DCCF

 #4. cp pfx2 pfx2.old  <-- save any previous conversion jobs for these files
     ================    - to recover any edit code for record-type tests

 #5. rm -f maps/* pfx1/* pfx2/* pfx3/* pfx4/*
     ========================================
     - clear any old files from previous job creations

 #6. cp ctl/datactl63 ctl/datacpy52
     ==============================
     - copy/rename control file for job completion by 'uvdata52' on '3B3'
     - datactl63 renamed to datacpy52 to agree with earlier job & doc.

control-file created on page '2F2'

 ar.customer.armaster   cpy=armaster typ=IDXf1  rca=00256 rcm=00256 rcs=00256 gdg=___ key=000,006
 ar.customer.arsales    cpy=arsales  typ=RSF    rca=00064 rcm=00064 rcs=00064 gdg=___ key=___,___
 ar.customer.artrans    cpy=custran3 typ=IDXf8v rca=00120 rcm=00249 rcs=00245 gdg=___ key=___,___
 ar.customer.phone.list cpy=________ typ=RSF    rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=citytax1 typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=___ key=___,___
 city.tax.payments      cpy=citytax2 typ=IDXf8v rca=00080 rcm=00128 rcs=00128 gdg=___ key=___,___
 gl.account.glmaster_   cpy=glmaster typ=RSF    rca=00128 rcm=00128 rcs=00128 gdg=003 key=___,___
 gl.account.gltran_     cpy=gltran   typ=RSF    rca=00080 rcm=00080 rcs=00080 gdg=004 key=___,___
 vendor.master.names    cpy=vendorm  typ=IDXf8v rca=00019 rcm=00041 rcs=00064 gdg=___ key=___,___
 vendor.master.payments cpy=vendorpm typ=IDXf8v rca=00019 rcm=00062 rcs=00064 gdg=___ key=___,___

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A2. Generating Data-Conversion-Jobs from copybooks & control-file

Low Volume Alternative

Part_2 recommended taking the time to initially create a control-file if you have high-volumes of datafiles to be transferred & converted. Several methods were suggested to help automate creation of the control-file in Part_2.

BUT, if you only have a few files to convert, you could skip Part_2, & come here to Part_3, where you can run uvdata51 to generate data-conversion-jobs. You can then edit the generated jobs to insert the datafilenames in place of the copybooknames.

OR, you could manually edit/create the control file for the few files desired for example, if you only needed to convert the ar.customer.armaster file & the ar.customer.arsales file, you could edit a 2 line control file:

 vi ctl/datacpy52
 ================

 ar.customer.armaster  cpy=armaster typ=IDXf1  rca=0256 rcm=0256 rcs=0256 gdg=___ key=000,006
 ar.customer.arsales   cpy=arsales  typ=RSF    rca=0064 rcm=0064 rcs=0064 gdg=___ key=___,___

Now, you would skip #1 (copies ctl/datactl63 to ctl/datacpy52), since we have created datacpy52 manually & dont want to use ctl/datacpy63 which was the high volume control file generated in Part_2.

Your manually created control-file will be input to 'uvdata52' on page '3B3'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B1. Generating Data Conversion Jobs - Operating Instructions

Generating Data Conversion Jobs - Operating Instructions

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. cp ctl/datactl63 ctl/datacpy52
     ==============================
     - copy/rename control file for job completion by 'uvdata52' on '3B3'
     - repeated here from #6 above since it is crucial

generate 'cobmap's from COBOL copybooks

 #2. uvcopyx cobmap1 cpys maps uop=q0i7
     ==================================
     - convert COBOL copybooks to cobmaps (record layouts)

generate 'data-conversion-jobs' from cobmaps

 #3. uvcopyx uvdata51 maps pfx1 uop=q0i7
     ===================================
     - generate uvcopy data conversion jobs from copybookmaps
     - codes fili1=... & filo1=... & jobnames sames as copybooknames

 #4. cp pfx1/* pfx2/   <-- copy all pfx1 jobs to pfx2
     ===============       before changes required for some file types

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B2. Generating Data Conversion Jobs - Operating Instructions

modify jobs for files with multi-record-types

It is better to make any changes required, before we run uvdata52 which can generate multiple jobs for copybooks with multiple datafiles. At this point, you only need to edit 1 job for each copybook needing changes vs having to edit multiple jobs after uvdata52 for multiple datafiles.

You might grep the copybook/maps for 'redefines' & insert record-type test code (for redefined records not fields) to branch to the appropriate bank of auto-generated instructions to preserve the packed/binary fields & fix zoned signs for that record type.

 #5a. grep 'redefines' maps/*   <-- search for 'redefined records'
      =======================

 #5b. vi pfx2/...         <-- modify jobs with multi-record-type files,
      ===========             variable length records,& occurs depending on

 #5c. cp pfx2/* pfx2save  <-- save conversion jobs with Record-Type test code
      ==================    - to recover R/T test code in case jobs regenerated

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B3. Generating Data Conversion Jobs - Operating Instructions

 #6. cp $UV/pf/IBM/skeleton2 pfx2
     ============================
     - copy uvcopy job skeleton for datafiles without copybooks, presuming that
       files without copybooks have no packed/binary fields or zoned signs
     - might modify control file to typ=LST (LineFeeds OK if no packed/binary)

pfx1/... & pfx2/... jobnames same as copybooknames

 -rw-rw-r--. 1 mvstest2 apps  831 Mar 17 11:34 armaster
 -rw-rw-r--. 1 mvstest2 apps  813 Mar 17 11:34 arsales
 -rw-rw-r--. 1 mvstest2 apps  899 Mar 17 11:34 citytax1
 -rw-rw-r--. 1 mvstest2 apps 1057 Mar 17 11:34 citytax2
 -rw-rw-r--. 1 mvstest2 apps 1125 Mar 17 11:34 custran3
 -rw-rw-r--. 1 mvstest2 apps  769 Mar 17 11:34 glmaster
 -rw-rw-r--. 1 mvstest2 apps  762 Mar 17 11:34 gltran
 -rw-rw-r--. 1 mvstest2 apps  718 Mar 17 11:34 vendormas
 -rw-rw-r--. 1 mvstest2 apps  727 Mar 17 11:34 vendorpaymas

replicate jobs inserting datafilenames vs copybooknames

 #7. uvcopy uvdata52,fili1=ctl/datacpy52,fild2=pfx2,fild3=pfx3,uop=q0i7
     ==================================================================
     - copies pfx2/* to pfx3/... inserting datafilenames (vs copybooknames)
       on the fili1=... & filo1=... I/O filename declarations
     - also renames the jobs for the datafilenames (vs copybooknames)
       since could have multiple datafiles for same copybook (esp. for GDG)
     - also inserts/updates file characteristics from keywords in control file
       (record-size, file-type, Indexed-key-locations)

pfx3/... jobs renamed same as datafilenames

 -rw-rw-r--. 1 mvstest2 apps  868 Mar 17 12:16 ar.customer.armaster
 -rw-rw-r--. 1 mvstest2 apps  849 Mar 17 12:16 ar.customer.arsales
 -rw-rw-r--. 1 mvstest2 apps 1163 Mar 17 12:16 ar.customer.artrans
 -rw-rw-r--. 1 mvstest2 apps 1073 Mar 17 12:16 ar.customer.phone.list
 -rw-rw-r--. 1 mvstest2 apps  921 Mar 17 12:16 city.tax.assess
 -rw-rw-r--. 1 mvstest2 apps 1089 Mar 17 12:16 city.tax.payments
 -rw-rw-r--. 1 mvstest2 apps  806 Mar 17 12:16 gl.account.glmaster_
 -rw-rw-r--. 1 mvstest2 apps  797 Mar 17 12:16 gl.account.gltran_
 -rw-rw-r--. 1 mvstest2 apps  754 Mar 17 12:16 vendor.master.names
 -rw-rw-r--. 1 mvstest2 apps  766 Mar 17 12:16 vendor.master.payments

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B4. Generating Data Conversion Jobs - Operating Instructions

replicating jobs for GDG files with multiple generations

 #8. uvcopy uvdata53,fild1=pfx3,fild2=pfx4,fild3=d1ebc
     =================================================
     - copy pfx3/* to pfx4/... replicating conversion jobs for GDG files
       with multiple generations & inserting filenames depending on GDG
       members found in d1ebc/...

pfx4/ jobs replicated from pfx3/

       ar.customer.armaster
       ar.customer.arsales
       ar.customer.artrans
       ar.customer.phone.list
       city.tax.assess
       city.tax.payments
       gl.account.glmaster_000123  <-- multiple jobs replicated from 1 in pfx3
       gl.account.glmaster_000124    - depending on datafile generations present
       gl.account.glmaster_000125
       gl.account.gltran_000456
       gl.account.gltran_000457
       gl.account.gltran_000458
       vendor.master.names
       vendor.master.payments

GDG jobs replicated for generations present

1. We generated only 1 job for each GDG set, to minimize changes for missing copybooks, record-sizes, etc
   
   
2. Compare the pfx4/... jobs to pfx3/... (listed on prior page), and to the datafilenames (listed on page '1C4').
   
   
3. Before we document the Execution of the generated jobs in pfx4/* (on pages '4A1' - 4C1), we will show you the jobs generated from the copybooks for 6 of the datafiles (on following pages '3C0' - 3C6).
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C0. Generating Data Conversion Jobs - Sample Results

Sample copybooks & uvcopy jobs generated from them

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C1. Generating Data Conversion Jobs - Sample Results

maps/armaster - copybook/map to generate conversion job

  01 armaster.
      10 cm-cust          pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-nameadrs.
         20 cm-name       pic  x(25).                         0010 0034  025
         20 cm-adrs       pic  x(25).                         0035 0059  025
         20 cm-city       pic  x(16).                         0060 0075  016
         20 filler001     pic  x.                             0076 0076  001
         20 cm-prov       pic  x(2).                          0077 0078  002
         20 filler002     pic  x.                             0079 0079  001
         20 cm-postal     pic  x(10).                         0080 0089  010
      10 cm-telephone     pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

pfx1/armaster - uvcopy job generated from copybook/map

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=armaster <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=JOBNAME   <-- datafilename in pfx3/...'
 fili1=?d1ebc/armaster,rcs=00256,typ=RSF
 filo1=?d2asc/armaster,rcs=00256,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00256),a0       move rec to outarea before field prcsng
        tra    b0(00256)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b120(60),a120           pns cm-thisyr-sales
        mvc    b180(60),a180           pns cm-lastyr-sales
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx3/ar.customer.master - re-named for datafile vs copybook

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=armaster <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=ar.customer.master   <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/ar.customer.armaster,rcs=00256,typ=RSF
 filo1=?d2asc/ar.customer.armaster,rcs=00256,typ=RSF
 #filo1=... copied to $RUNDATA: rca=00256 rcm=00256 rcs=00256 typ=IDXf1  key=000,006
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00256),a0       move rec to outarea before field prcsng
        tra    b0(00256)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b120(60),a120           pns cm-thisyr-sales
        mvc    b180(60),a180           pns cm-lastyr-sales
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Copying from $CNVDATA/d2asc/... to $RUNDATA/data1/...

After we have executed our conversion jobs (page '4B1') the converted data files will be in d2asc/... and we need to copy them over to $RUNDATA/data1/... for testing. During that copy, we will modify file types as required.

1. For conversions in $CNVDATA, all variable-length files are typ=RDWz4, and we will convert them to 'typ=IDXf8v' for Micro Focus COBOL.
   
   
2. For conversions in $CNVDATA, all files are sequential (non-indexed) and we will convert them to Indexed Micro Focus COBOL compatible. We will use 'typ=IDXf1' for fixed-length files & 'typ=IDXf8v' for variable length files.
   
   
3. Note that uvcopy or uvcp uses 'isk1=...' for Indexed Sequential Keyed vs 'key=___,___' that we used in our control-file, because uvsort uses 'key1=...' for sort keys.
   
   

Utility 'mkcopy2data' (on page '4E1') generates a script of uvcp commands to copy the files & the uvcp command for ar.customer.armaster will be:

 uvcp "fili1=$CNVDATA/d2asc/ar.customer.armaster,typ=RSF,rcs=256,\
       filo1=$RUNDATA/data1/ar.customer.armaster,typ=ISXf1,rcs=256,isk1=0(6)"
 ============================================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C2. Generating Data Conversion Jobs - Sample Results

maps/citytax2 - copybook/map to generate conversion job

 * citytax2 cobol copybook for redefined multi record type fi
 * tax record (type 'T')
  01  citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x.                        0009 0009  001
      10 name1                 pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler002             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

 * monthly payment record (type 'P')
  01 mthpayrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler003             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
      10 mthpayments           pic  s9(9) comp-3 occurs 12.   0020 0024  005pns 9
      10 filler004             pic  x(48).                    0080 0127  048
 *RCSZ=00128

 * file header record (type 'H')
  01 headerrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler005             pic  x.                        0009 0009  001
      10 year-end              pic  x(30).                    0010 0039  030
      10 filler006             pic  x(88).                    0040 0127  088
 ************** end of citytax2 copybook ********************
 *RCSZ=00128

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx1/citytax2 - uvcopy job generated from copybook/map

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=citytax2 <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=JOBNAME   <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
        skp    put1
 #      ---            insert redef typ__
 typ__  mvc    b20(60),a20             pns mthpayments
        skp    put1
 #      ---            insert redef typ__
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

modifying jobs for Multiple Record Types

The citytax2 copybook & city.tax.payments datafile has 3 record types identified in column 9 'H' for file Header, 'T' for Tax assessment record, & 'P' for Payment record).

We need to insert code to test the record-type & branch to the corresponding auto-generated instructions for that record-type.

 vi pfx2/citytax2   <-- edit job to insert record type test code
 ================     - see results below

We perform the edits for multi-record types on the jobs in pfx2 where they are still named for the copybooks (before uvdata52 replicates & renames them for the datafiles). This could save a lot of editing - because there may be multiple datafiles for any one copybook - especially for GDG files.

But below, we will show the edited job after uvdata52 has renamed the job for the datafile & inserted the datafilename on the fili1=... & filo1=...

 vi pfx2/city.tax.payments
 =========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx3/city.tax.assess - After inserting Record-Type test code

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=citytax2   <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=city.tax.payments <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/city.tax.payments,rcs=00128,typ=RDWz4
 filo1=?d2asc/city.tax.payments,rcs=00128,typ=RDWz4
 #filo1=... copied to $RUNDATA: rca=00080 rcm=00128 rcs=00128 typ=IDXf8v key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed & signed
        cmc    b8(1),'H'           Header rec (no packed or signed)
        skp=   put1
        cmc    b8(1),'T'           Tax rec ?
        skp=   tax1
        cmc    b8(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(80)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-------------------------------------------------------------------------
 tax1   mvc    b88(4),a88              bns post-date    <-- 'tax1' label inserted
        mvc    b92(15),a92             pns land-value
        trt    b107(9),$trtsea          ns maint-tax
        skp    put1
 #      ---            insert redef typ__
 pay1   mvc    b20(60),a20             pns mthpayments  <-- 'pay1' label inserted
        skp    put1
 #      ---            insert redef typ__
 put1   put    filo1,b0                                  <-- use 'put1' for Header
        skp    loop
 eof    cls    all
        eoj

Notes re Record Type test code inserted

1. We code --> cmc b8(1),'H' <-- to test for R/T 'H' at offset 8 which is 'zero relative' vs column 9 'one relative'.
   
   
2. We test record-types in area 'b' after the translate to ASCII (vs area 'a' before translate) since that is easier to code In area 'a' (EBCDIC) we would need to code --> cmc a8(1),x'C8' (hexadecimal for 'H' in EBCDIC).
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C3. Generating Data Conversion Jobs - Sample Results

maps/custran3 - copybook/map to generate conversion job

 * custran3 - copybook for customer master & transaction file
 * - variable length 0 to 4 sets of group occurs depending on
  01 custran3.
      10 cmt-num           pic  9(6).                         0000 0005  006 n  6
      10 cmt-delete        pic  x(4).                         0006 0009  004
      10 cmt-name          pic  x(25).                        0010 0034  025
      10 cmt-adrs          pic  x(25).                        0035 0059  025
      10 cmt-city          pic  x(16).                        0060 0075  016
      10 filler001         pic  x.                            0076 0076  001
      10 cmt-prov          pic  x(2).                         0077 0078  002
      10 filler002         pic  x.                            0079 0079  001
      10 cmt-postal        pic  x(10).                        0080 0089  010
      10 cmt-phone         pic  x(12).                        0090 0101  012
      10 cmt-contact       pic  x(16).                        0102 0117  016
      10 cmt-count         pic  s9(3) comp-3.                 0118 0119  002pns 3
 *BGNOCCURSM:c-p:00005*00025=00125:00120-00244:1:00118:02pns:
      10 cmt-trans occurs 0 to 5 depending on cmt-count.
         15 cmt-tran-type  pic  x.                            0120 0120  001
         15 cmt-tran-ref   pic  9(4).                         0121 0124  004 n  4
         15 cmt-tran-amt   pic  s9(7)v99 comp-3 occurs 4.     0125 0129  005pns 9
 * min lth 120 if no transctns, max 245 if all 5 transctns
 *ENDOCCURSM:c-p:00005*00025=00125:00120-00244:1:00118:02pns:
 *RCSZ=00245

Notes re: 'occurs' coding

1. Here we have a base record of 120 bytes & an occurs 0 to 5 of 25 bytes containing 5 bytes of character data & 20 bytes of packed (4*5).
   
   
2. 'occurs' requires no manual changes & no extra code generation, except when there are mixed character & packed/binary/zonedsigns within the occurs group.
   
   
3. See the extra code generated below & the explanatory notes following.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx1/custran3 - uvcopy job generated from copybook/map

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=custran3 <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=JOBNAME   <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/custran3,rcs=00245,typ=RSF
 filo1=?d2asc/custran3,rcs=00245,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00245),a0       move rec to outarea before field prcsng
        tra    b0(00245)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b118(2),a118            pns cmt-count
 #BGNOCCURSM:c-p:00005*00025=00125:00120-00244:1:00118:02pns:
        mvn    $ra,0
        mvn    $rj,0
 man012 nop
        add    $rj,00001
        cmn    $rj,00005
        skp>   man019
        mvc    ba125(20),aa125     005*020pns cmt-tran-amt
        add    $ra,00025
        skp    man012
 man019 nop
 #ENDOCCURSM:c-p:00005*00025=00125:00120-00244:1:00118:02pns:
 #
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Notes re: 'occurs' coding

1. 'occurs' requires no manual changes & no extra code generation, unless there are mixed data types within the occurs (packed/binary/zoned signs and character fields).
   
   
2. The inner occurs 'comp-3 occurs 4' generates the --> mvc ba125(20),aa125 <-- 4 packed 5 byte fields = 20 bytes, handled with just the 1 instruction.
   
   
3. The outer occurs does have mixed packed & character data, so a code loop is automatically generated. Within each 25 byte outer occurs the 20 bytes packed is preserved with the 'mvc' & no action is require for the 5 byte character field (already translated to ASCII by the 'tra' at the top).
   
   
4. The 'occurs depending on 0 to 5' of 25 bytes means the records could vary from 120 to 245 bytes, but we need no code to allow for this, because file typ=RDWz4 controls the length of the records. It won't matter if the code processes data beyond the data actually present since that area will not be written out.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C4. Generating Data Conversion Jobs - Sample Results

maps/glmaster - copybook/map to generate conversion job

  01  glmaster.
      10 glm-status             pic  x(00001).                0000 0000  001
      10 glm-acnt1              pic  9(00005).                0001 0005  005 n  5
      10 glm-filler1            pic  x(00007).                0006 0012  007
      10 glm-acnt2              pic  9(00005).                0013 0017  005 n  5
      10 glm-description        pic  x(00030).                0018 0047  030
      10 glm-balance            pic s9(00011) comp-3.         0048 0053  006pns11
      10 glm-lastupdate         pic  9(00008).                0054 0061  008 n  8
      10 glm-filler2            pic  x(00066).                0062 0127  066
 *RCSZ=00128

pfx1/glmaster - uvcopy job generated from copybook/map

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=glmaster <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=JOBNAME   <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/glmaster,rcs=00128,typ=RSF
 filo1=?d2asc/glmaster,rcs=00128,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b48(6),a48              pns glm-balance
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx3/gl.account.master - re-named for datafile vs copybook

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=glmaster <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=gl.account.glmaster_  <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/gl.account.glmaster_,rcs=00128,typ=RSF
 filo1=?d2asc/gl.account.glmaster_,rcs=00128,typ=RSF
 #filo1=... copied to $RUNDATA: rca=00128 rcm=00128 rcs=00128 typ=RSF    key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b48(6),a48              pns glm-balance
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Notes re GDG files

1. Jobs for GDG files generated by 'uvdata52' in pfx3/... are identified by a trailing underscore, see above:
   
   

fili1=?d1ebc/gl.account.glmaster_,rcs=00128,typ=RSF filo1=?d2asc/gl.account.glmaster_,rcs=00128,typ=RSF

2. Following utility 'uvdata53' copies pfx3/* jobs to pfx4/... & jobs with GDG files will be replicated & have the actual GDG filenames inserted depending on the actual filenames found in d1ebc/...
   
   
3. Then the jobs are executed converting files from d1ebc/... to d2asc/...
   
   
4. Then utility 'mkcopy2data' generates a script to copy files from $CNVDATA/d2asc to $RUNDATA/data1.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C5. Generating Data Conversion Jobs - Sample Results

pfx2/skeleton2 - uvcopy job template used when NO copybook

 # uvcopy skeleton2 translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='copybook=MISSING - using skeleton2 OK if no packed fields'
 opr='jobname=JOBNAME - missing copybook'
 uop=q0,was=a8192b8192  #recsize will be supplied from control file
 fili1=?d1ebc/FILEIN,rcs=RCSZI,typ=RSF
 filo1=?d2asc/FILEOUT,rcs=RCSZO,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(RCSZI),a0       move rec to outarea before field prcsng
        tra    b0(RCSZI)          translate entire outarea to ASCII
 #-----------------------------------------------------------------------
 #      ---                <--- insert R/T tests for redefined records
 #      ---                <--- insert mvc's for packed & trt's for signed
 #-----------------------------------------------------------------------
 put1   put    filo1,b0($rv8192)  # rgstr 'v' holds recsize from last get
        skp    loop
 eof    cls    all
        eoj

skeleton2 job used when no copybook

1. When no copybook is coded for a datafile in the control file, uvdata52 will use the 'skeleton2' template as the input (shown above).
   
   
2. uvdata52 will replace the FILEIN/FILEOUT with the datafilename as shown in the output (below).
   
   
3. If the control file has not been updated with a record-size, the output job will code the record sizes as '00000' on the fili1, filo1,& two instructions 'mvc' & 'tra' (see below).
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx3/ar.customer.phone.list - re-named for datafile vs copybook

 # uvcopy skeleton2 translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='copybook=MISSING - using skeleton2 OK if no packed fields'
 opr='jobname=ar.customer.phone.list - missing copybook'
 uop=q0,was=a8192b8192  #recsize will be supplied from control file
 fili1=?d1ebc/ar.customer.phone.list,rcs=00000,typ=RSF
 filo1=?d2asc/ar.customer.phone.list,rcs=00000,typ=RSF
 #filo1=... copied to $RUNDATA: rca=00000 rcm=00000 rcs=00000 typ=RSF    key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00000),a0       move rec to outarea before field prcsng
        tra    b0(00000)          translate entire outarea to ASCII
 #-----------------------------------------------------------------------
 #      ---                <--- insert R/T tests for redefined records
 #      ---                <--- insert mvc's for packed & trt's for signed
 #-----------------------------------------------------------------------
 put1   put    filo1,b0($rv8192)  # rgstr 'v' holds recsize from last get
        skp    loop
 eof    cls    all
        eoj

skeleton2 job record-size 00000

1. When you see jobs with rcs=00000 (caused by missing copybook), you should research to see if there are any packed/binary/signed fields & find the record-size.
   
   
2. If there are packed/binary/signed fields, you could should create a copybook perhaps by extracting a record layout from a COBOL program or using the 'signscan2' utility to scan the datafile & create a copybook. See signscan2 at DATAcnv1.htm#Part_1.
   
   
3. If there are no packed/binary/signed fields, you could simply insert the record-sizes manually & proceed to the execution.
   
   
4. Or if there are a lot of missing copybooks & record-sizes, you should do the research & update the control-file with the missing info. Then re-run the generation jobs as documented on page '3B1' - 3B4.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C6. Generating Data Conversion Jobs - Sample Results

maps/vendorpaymas - copybook/map to generate conversion job

 * record types for uvhdcob multi record-type handling
 *type=7(1),='M','vendormas'
 *type=7(1),='P','vendorpay'
  01  vendorpaymas      pic  x(64).                           0000 0063  064
  02  vendormas redefines vendorpaymas.
      10 venmas-num     pic  x(6).                            0000 0005  006
      10 venmas-type    pic  x(4).                            0006 0009  004
      10 venmas-name    pic  x(54).                           0010 0063  054
  02  vendorpay redefines vendorpaymas.
      10 venpay-num     pic  x(6).                            0000 0005  006
      10 recpay-type    pic  x(4).                            0006 0009  004
      10 venpay-date    pic  x(10).                           0010 0019  010
      10 venpay-inv     pic  x(10).                           0020 0029  010
      10 venpay-amt     pic  s9(9).                           0030 0038  009 ns 9
      10 filler001      pic  x(1).                            0039 0039  001
      10 venpay-item    pic  x(24).                           0040 0063  024
  02  badrecordtype redefines vendorpaymas.
      10 unidentified-record-1st30  pic  x(30).               0000 0029  030
 *RCSZ=00064

pfx1/vendorpaymas - uvcopy job generated from copybook/map

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='jobname=JOBNAME   <-- datafilename in pfx3/...'
 opr='copybook=vendorpaymas <-- copybookname in pfx1/ & pfx2/'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/vendorpaymas,rcs=00064,typ=RSF
 filo1=?d2asc/vendorpaymas,rcs=00064,typ=RSF
 #filo1=... copied to $RUNDATA: rca=_____ rcm=_____ rcs=_____ typ=______ key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00064),a0       move rec to outarea before field prcsng
        tra    b0(00064)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        trt    b30(9),$trtsea           ns venpay-amt
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

pfx3/vendorpaymas - re-named for datafile vs copybook

 # uvcopy job translate EBCDIC to ASCII, preserve packed/binary, fix zoned signs
 opr='copybook=vendorpaymas <-- copybookname in pfx1/ & pfx2/'
 opr='jobname=vendor.master.payments  <-- datafilename in pfx3/...'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/vendor.master.payments,rcs=00064,typ=RDWz4
 filo1=?d2asc/vendor.master.payments,rcs=00064,typ=RDWz4
 #filo1=... copied to $RUNDATA: rca=00019 rcm=00062 rcs=00064 typ=IDXf8v key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00064),a0       move rec to outarea before field prcsng
        tra    b0(00064)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed & signed
        cmc    b7(1),'M'           Master record (no packed or signed)
        skp=   put1            <-- skip to label 'put1' bypassing the 'trt'
        cmc    b7(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(64)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-------------------------------------------------------------------------
 pay1   trt    b30(9),$trtsea           ns venpay-amt  <-- 'pay1' label inserted
 #
 put1   put    filo1,b0        <-- label 'put1' used when no packed/binary/signed
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_4. Data File Conversion - Comprehensive & Complete

Part 4 - Executing the Data Conversion Jobs

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A1. Executing the Data-Conversion-Jobs

Pre-Requisites to Executing the Jobs

1a. EBCDIC datafiles have been FTP'd binary from mainframe into d0ebc/... If variable length FTP option 'RDW' must be used as shown on page '1C1'.

1b. OR much better - FTP all files with option 'RDW' so we can immediately know the record-sizes on Unix/Linux - because 'uvcopy ctldata61' will extract the record-size from the RDW prefix & enter into ctl/datactl61 rca=minimum & rcm=maximum. We will know Fixed-Length files from Variable- Length files because the minimum & maximum record-size will be the same.

2. d0ebc/* copied to d1ebc/... by 'copyd0d1' to rename mainframe GDG files to UV GDG conventions - lowercase & mainframe GDG suffixes converted from .G1234V00 to UV GDG conventions _123456 trailing underscore & 6 digits. See these procedures on pages '1C1' - 1C5.
   
   
3. We created the control-file relating datafilenames to copybooknames & keyword coded with record-sizes, file-types, Indexed keys, etc. See those procedures on pages '2D1' - 2F4.
   
   
4. Initial Data Conversion Jobs have been generated into $CNVDATA/pfx1 from the copybooks & the control-file using procedures on pages '3A1' - 3B4.
   
   
5. We copied the conversion jobs from pfx1/* to pfx2/ on '3B1' & modified as required inserting record-type test code for files with multiple record types. See examples on pages '3C2' & '3C6'
   
   
6. We copied pfx2/* to pfx2save/... on page '3B2' to save our Record-Type test coding in case jobs regenerated overwriting our R/T test code.
   
   
7. We ran 'uvdata52' on page '3B3' to copy the jobs from pfx2/* to pfx3/... inserting the datafilenames (vs the copybooknames) & replicating for possible multiple files for one copybook.
   
   
8. Finally, we used 'uvdata53' on page '3B4' to copy pfx3/* to pfx4/... replicating conversion jobs for GDG files. We create only 1 conversion job for GDG files to minimize any manual changes for missing copybooks, record sizes, file types, etc. Now we replicate it changing filenames for multiple generations of a particular GDG set.
   
   

                      rename GDG     EBCDIC-ASCII      copy to test
 mainframe ----> d0ebc --------> d1ebc ---------> d2asc -----------> $RUNDATA/data1
           FTP         copyd0d1          pfx4/*          copy2data1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A2. Executing the Data-Conversion-Jobs

directories for data-file conversions

 /home/userxx/       <-- homedir OK for demo, but not for high volume data conversions
 :-----cnvdata          <-- data conversion superdir $CNVDATA
 :     :-----cpys         - COBOL copybooks
 :     :-----ctl          - control file directory
 :     :-----d0ebc        - EBCDIC files from mainframe for conversion
 :     :--*--d1ebc        - renamed lowercase & UV GDG file conventions
 :     :--*--d2asc        - files converted to ASCII (with same record layout)
 :     :-----d3ebc        - back to EBCDIC for return to mainframe or prove conversions
 :     :-----d4pipe       - data files converted to '|' pipe delimited format
 :     :-----maps         - cobmaps (record layouts) generated from copybooks
 :     :-----pfx1         - uvcopy jobs to convert EBCDIC to ASCII (gen from cobmaps)
 :     :-----pfx2         - copied here for manual changes for multiple Record Types
 :     :-----pfx3         - renamed here for the datafilename (vs copybookname)
 :     :                    datafilenames,filetypes,recsizes inserted from control-file
 :     :--*--pfx4         - copied here, replicating & inserting GDG filenames
 :     :                    depending on generations present in d1ebc/...
 :     :-----pfp1         - uvcopy jobs to convert to pipe delimited (from copybooks)
 :     :-----pfy1         - uvcopy jobs to convert ASCII back to EBCDIC (prove convert OK)

data subdirs for test/demo executions

 /home/userxx/       <-- homedir OK for demo, but not for high volume data
 :-----testdata         <-- data dirs (defined as $RUNDATA in JCL/scripts)
 :     :--*--data1       - datafiles
 :     :-----ctl         - GDG control file
 :     :-----joblog      - programmer debug log files
 :     :-----jobmsgs     - status msgs from JCL/scripts
 :     :-----jobtimes    - job/step times date stamped history files
 :     :-----jobtmp      - temporary files for SYSIN instream data
 :     :-----sysout      - SYSOUT printer files
 :     :-----tmp         - tmp subdir for uvsort & misc use

                      rename GDG     EBCDIC-ASCII      copy to test
 mainframe ----> d0ebc --------> d1ebc ---------> d2asc -----------> $RUNDATA/data1
           FTP         copyd0d1          pfx4/*          copy2data1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4B1. Executing the Data-Conversion-Jobs

Executing the Data-Conversion-Jobs

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. uvcopyxx 'pfx4/*'    <-- execute all jobs to convert all files
     =================        from d1ebc/* to d2asc/...

console log - converting files from d1ebc/ to d2asc

 uvcopyxx - execute uvcopy for 14 jobs in directory pfx4
  - with option i31 to inhibit prompts for options&filenames
 170327:203148:ar.customer.ar: EOF fili01 rds=32 size=8192: d1ebc/ar.customer.armaster
 170327:203148:ar.customer.ar: EOF filo01 wrts=32 size=8192: d2asc/ar.customer.armaster
 170327:203148:ar.customer.ar: EOF fili01 rds=20 size=1280: d1ebc/ar.customer.arsales
 170327:203148:ar.customer.ar: EOF filo01 wrts=20 size=1280: d2asc/ar.customer.arsales
 170327:203148:ar.customer.ar: EOF fili01 rds=32 size=6068: d1ebc/ar.customer.artrans
 170327:203148:ar.customer.ar: EOF filo01 wrts=32 size=6068: d2asc/ar.customer.artrans
 FATAL ERR - 0 lth, missing op# 1, invalid tag(3 alpha b4 numeric)
         mvc    b0(00000),a0       move rec to outarea before field prcsng
 170327:203148:city.tax.asses: EOF fili01 rds=10 size=1280: d1ebc/city.tax.assess
 170327:203148:city.tax.asses: EOF filo01 wrts=10 size=1280: d2asc/city.tax.assess
 170327:203148:city.tax.payme: EOF fili01 rds=16 size=1680: d1ebc/city.tax.payments
 170327:203148:city.tax.payme: EOF filo01 wrts=16 size=1680: d2asc/city.tax.payments
 170327:203148:gl.account.glm: EOF fili01 rds=109 size=13952: d1ebc/gl.account.glmaster_000123
 170327:203148:gl.account.glm: EOF filo01 wrts=109 size=13952: d2asc/gl.account.glmaster_000123
 170327:203148:gl.account.glm: EOF fili01 rds=109 size=13952: d1ebc/gl.account.glmaster_000124
 170327:203148:gl.account.glm: EOF filo01 wrts=109 size=13952: d2asc/gl.account.glmaster_000124
 170327:203148:gl.account.glm: EOF fili01 rds=109 size=13952: d1ebc/gl.account.glmaster_000125
 170327:203148:gl.account.glm: EOF filo01 wrts=109 size=13952: d2asc/gl.account.glmaster_000125
 170327:203148:gl.account.glt: EOF fili01 rds=20 size=1600: d1ebc/gl.account.gltran_000456
 170327:203148:gl.account.glt: EOF filo01 wrts=20 size=1600: d2asc/gl.account.gltran_000456
 170327:203148:gl.account.glt: EOF fili01 rds=20 size=1600: d1ebc/gl.account.gltran_000457
 170327:203148:gl.account.glt: EOF filo01 wrts=20 size=1600: d2asc/gl.account.gltran_000457
 170327:203148:gl.account.glt: EOF fili01 rds=20 size=1600: d1ebc/gl.account.gltran_000458
 170327:203148:gl.account.glt: EOF filo01 wrts=20 size=1600: d2asc/gl.account.gltran_000458
 170327:203148:vendor.master.: EOF fili01 rds=8 size=249: d1ebc/vendor.master.names
 170327:203148:vendor.master.: EOF filo01 wrts=8 size=249: d2asc/vendor.master.names
 170327:203148:vendor.master.: EOF fili01 rds=16 size=745: d1ebc/vendor.master.payments
 170327:203148:vendor.master.: EOF filo01 wrts=16 size=745: d2asc/vendor.master.payments

Error - Record size missing

From the console log above, you can see that a record-size was missing. It does not tell you for which job/file, but we do see the job/files prior to & following the error & can determine that the error was for 'ar.customer.phone.list'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4C1. Executing the Data-Conversion-Jobs

Correcting jobs & re-running

From the control-file (listed on page '2F2'), you can see that the record-size was missing for ar.customer.phone.list & now you need to do the research, to find the record-size, & correct the uvcopy job:

The conversion job for this file was listed following '3C5'. Since there was no copybook, a job was generated from the 'skeleton2' template.

 #2. vi pfx4/ar.customer.phone.list  <-- correct conversion job for the file
     ==============================

 # uvcopy skeleton2 translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='copybook=MISSING - using skeleton2 OK if no packed fields'
 opr='jobname=ar.customer.phone.list - missing copybook'
 uop=q0,was=a8192b8192  #recsize will be supplied from control file
 fili1=?d1ebc/ar.customer.phone.list,rcs=00000,typ=RSF
 filo1=?d2asc/ar.customer.phone.list,rcs=00000,typ=RSF
 #filo1=... copied to $RUNDATA: rca=00000 rcm=00000 rcs=00000 typ=RSF    key=___,___
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00000),a0       move rec to outarea before field prcsng
        tra    b0(00000)          translate entire outarea to ASCII
 #-----------------------------------------------------------------------
 #      ---                <--- insert R/T tests for redefined records
 #      ---                <--- insert mvc's for packed & trt's for signed
 #-----------------------------------------------------------------------
 put1   put    filo1,b0($rv8192)  # rgstr 'v' holds recsize from last get
        skp    loop
 eof    cls    all
        eoj

You need to replace '00000's with the correct record-size on 4 lines above (the '#filo1=...' line comment does not matter). You might use 'uvhd' to investigate the datafile d1ebc/ar.customer.phone.list as shown on the next page.

Executing data-conversion-job for 1 file

 #3. uvcopy pfx4/ar.customer.phone.list <-- execute uvcopy job to convert the file
     ==================================     from d1ebc/... to d2asc/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D1. Executing the Data-Conversion-Jobs

using 'uvhd' to determine datafile record-size

 uvhd d1ebc/ar.customer.phone.list r256a
 =======================================
 - option 'r256' is the default when we do not know the record-size
 - option 'a' is required to translate the character line of EBCDIC files
   so we can read the data, but still see true hex values fr zones & digits

 rec#=1 rsize=256 fptr=0 fsize=2560 records=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140  EVERGREEN MOTORS LTD.      250-754-5531  LARRY WRENCH
             FFFFFF44CECDCDCCD4DDEDDE4DEC4444444FFF6FFF6FFFF44DCDDE4EDCDCC444
             1301400055597955504636920334B00000025007540553100319980695538000
          64                .132588  GEECOE GENERATOR SERVICES  604-278-4488
             4444444444444440FFFFFF44CCCCDC4CCDCDCEDD4ECDECCCE44FFF6FFF6FFFF4
             000000000000000A132588007553650755591369025959352006040278044880
         128  HARRY LIGHT                   .139923  JOHNSTONE BOILER & TANKS
             4CCDDE4DCCCE44444444444444444440FFFFFF44DDCDEEDDC4CDCDCD454ECDDE
             0819980397830000000000000000000A13992300168523655026935900031522
         192    604-320-1845  GEORGE BROWN                  .142175  LILLY EL
             444FFF6FFF6FFFF44CCDDCC4CDDED4444444444444444440FFFFFF44DCDDE4CD
             00060403200184500756975029665000000000000000000A1421750039338053

Determining Record-Size with uvhd

• Look for repeating data to determine the record-size.
• You can see it is '80' since the data starts to repeat at byte #16
  of the 2nd line + 64 bytes from the 1st line = 80 bytes.
• You could double-check as follows:

 uvhd d1ebc/ar.customer.phone.list r80a
 =======================================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140  EVERGREEN MOTORS LTD.      250-754-5531  LARRY WRENCH
             FFFFFF44CECDCDCCD4DDEDDE4DEC4444444FFF6FFF6FFFF44DCDDE4EDCDCC444
             1301400055597955504636920334B00000025007540553100319980695538000
          64                .
             4444444444444440
             000000000000000A
 rec#=1 rsize=80 fptr=0 fsize=2560 records=32

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D2. Executing the Data-Conversion-Jobs

Debugging data-conversion-jobs

You should not have a problem with the auto-generated uvcopy jobs, but you might have problems with multi-record-type files where you need to insert code to test the record types & branch to the appropriate bank of auto- generated instructions to preserve packed/binary fields.

We will illustrate a debug session using the conversion job for pfx4/city.tax.payments, listed on the 3rd page of '3C2' after we have added the code to test record types.

 In debug mode, uvcopy executes an instruction & prompts for user entry.
 - null entry to execute next instruction & prompt again
 - see https://uvsoftware.ca/uvcopy2.htm#J2 for all debug commands
 - here we will use 'x' display hexadecimal & 'p' display character
 - 't' is a good command to run until a specified Tag (label) is reached

 uvcopy pfx4/city.tax.payments,rop=d    <-- Run OPtion 'd' to debug
 ===================================

      0        opn.. all
 debug>                           <-- null entry execute next instrn & reprompt
    128 loop   get.. fili1,a0
 debug>                             - will omit some prompts to save space here
    304        skp>. eof
    432        mvc.. b0(00128),a0   - move area 'a' to area 'b' (128 bytes)
    560        tra.. b0(00128)      - translate area 'b'
    688        cmc.. b8(1),'H'      - test for Header record (vs Tax record)
    816        skp=. put1           - if byte 8 = 'H', skip to label 'put1'
 debug>
   2608 put1   put.. filo1,b0
 debug>
 debug> x a0                      <-- 'x' display area 'a' from byte 0 (64 default)
        ====                          in vertical hexadecimal (since EBCDIC)

                         1         2         3         4         5         6
               0123456789012345678901234567890123456789012345678901234567890123
         ----->@@@@@@@@.@........@..k@....@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
               44444444C4C88898894FF64FFFF4444444444444444444444444444444444444
               000000008045354259031B019980000000000000000000000000000000000000

 debug> p b0                      <-- 'p' display area 'b' from byte 0 (64 default)
        ====                        - area 'b' has been translated & we can read

                         1         2         3         4         5         6
               0123456789012345678901234567890123456789012345678901234567890123
         ----->        H December 31, 1998
 debug>
   2784        skp.. loop

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 debug>                           <-- will omit some prompts to save space
    128 loop   get.. fili1,a0
    304        skp>. eof
    432        mvc.. b0(00128),a0
    560        tra.. b0(00128)
    688        cmc.. b8(1),'H'
    816        skp=. put1
    944        cmc.. b8(1),'T'      - test for Tax record (vs Header & Trailer)
   1072        skp=. tax1           - if byte 8 = 'T', skip to label 'tax1'
 debug>
   1840 tax1   mvc.. b88(4),a88
   1968        mvc.. b92(15),a92
   2096        trt.. b107(9),$trtsea
   2224        skp.. put1
 debug>
   2608 put1   put.. filo1,b0

 debug> x a0
        ====
                         1         2         3         4         5         6
               0123456789012345678901234567890123456789012345678901234567890123
         ----->.........@....@.....@@@@@@@@@@@@@@@....@.....@....@@@@@@@@@@....
               FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
               1013014030168508559800000000000000018150266550961400000000005153

 debug> p b0
        ====
                         1         2         3         4         5         6
               0123456789012345678901234567890123456789012345678901234567890123
         ----->10130140T JOHN HENRY               1815 BOWEN ROAD VANC

 debug> x b64
        =====
               0123456789012345678901234567890123456789012345678901234567890123
         ----->OUVER           BCV9S1H1..C...W.........qq.000149061970530
               455452222222222244535343004D005880008000770333333333333333222222
               F5652000000000002369318101320072C0047C0111C000149061970530000000

 debug> t tax1                    <-- 't' to run until we reach Tag/label 'tax1'
        ======
   1840 tax1   mvc.. b88(4),a88
 debug>                           <-- could display areas as above
 debug> q                           - but we will now Quit

 170414:114530:city.tax.payme: EOF fili01 rds=2 size=1680: d1ebc/city.tax.payments
 170414:114530:city.tax.payme: EOF filo01 wrts=1 size=84: d2asc/city.tax.payments

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E1. Executing the Data-Conversion-Jobs

copy/create final output file types desired

                   $CNVDATA/d2asc ---------------> $RUNDATA/data1
                                     copy2data1
          /home/userxx/cnvdata/d2asc ----------> /home/userxx/testdata/data1

1. We will create the desired file-types during the copy from the conversion directories to the testdata directories.
   
   
2. During the conversion files are either 'typ=RSF' sequential fixed length OR 'typ=RDWz4' for variable-record-length files.
   
   
3. The control-file codes variable length as 'typ=IDXf8v', but that is for the final output file type desired when copied to testdata/data1/...
   
   
4. The control-file codes Indexed files as 'typ=IDXf1' if fixed-length OR 'typ=IDXf8v' for variable-length. Both would have a key length (key=000,006 for example). These codes are for the final output file when copied from $CNVDATA/d2asc to $RUNDATA/data1.
   
   
5. We will use utility 'mkcopy2data1' to generate a script to perform file-type conversions as files are copied from $CNVDATA/d2asc to $RUNDATA/data1.
   
   

   generate script to copy/create file-types desired

 #3. uvcp "fili1=ctl/datactl63,typ=LST,rcs=191,filo1=ctl/datactl63I,typ=ISF,isk1=0(44)"
     ==================================================================================
     - First create Indexed control-file for mkcopy2data1 lookup by datafilename

 #4. uvcopy mkcopy2data1,fild1=d2asc,fili2=ctl/datactl63I,filo1=sf/copy2data1
     ========================================================================
     - generate a script to copy files from $CNVDATA/d2asc to $RUNDATA/data1
       creating the final output file-types desired as coded in control-file.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E2. Executing the Data-Conversion-Jobs

 #5. vi sf/copy2data1       <-- investigate output script
     ================         - could modify if/as desired

 # sf/copy2data1 - script to copy files from d2asc/* to data1/. (later to \$RUNDATA/data1/.)
 #        - creating final output file-types desired as coded in control-file
 # job=mkcopy2data1 site=UV_Software date=2017/03/29_07:30:47
 uvcp "fili1=d2asc/ar.customer.armaster,rcs=00256,typ=RSF,
       filo1=data1/ar.customer.armaster,rcs=00256,typ=IDXf1,isk1=000(006)"
 uvcp "fili1=d2asc/ar.customer.arsales,rcs=00064,typ=RSF,
       filo1=data1/ar.customer.arsales,rcs=00064,typ=RSF"
 uvcp "fili1=d2asc/ar.customer.artrans,rcs=00245,typ=RDWz4,
       filo1=data1/ar.customer.artrans,rcs=00245,typ=IDXf8v"
 uvcp "fili1=d2asc/city.tax.assess,rcs=00128,typ=RSF,
       filo1=data1/city.tax.assess,rcs=00128,typ=RSF"
 uvcp "fili1=d2asc/city.tax.payments,rcs=00128,typ=RDWz4,
       filo1=data1/city.tax.payments,rcs=00128,typ=IDXf8v"
 uvcp "fili1=d2asc/gl.account.glmaster_000123,rcs=00128,typ=RSF,
       filo1=data1/gl.account.glmaster_000123,rcs=00128,typ=RSF"
 uvcp "fili1=d2asc/gl.account.glmaster_000124,rcs=00128,typ=RSF,
       filo1=data1/gl.account.glmaster_000124,rcs=00128,typ=RSF"
 uvcp "fili1=d2asc/gl.account.glmaster_000125,rcs=00128,typ=RSF,
       filo1=data1/gl.account.glmaster_000125,rcs=00128,typ=RSF"
 uvcp "fili1=d2asc/gl.account.gltran_000456,rcs=00080,typ=RSF,
       filo1=data1/gl.account.gltran_000456,rcs=00080,typ=RSF"
 uvcp "fili1=d2asc/gl.account.gltran_000457,rcs=00080,typ=RSF,
       filo1=data1/gl.account.gltran_000457,rcs=00080,typ=RSF"
 uvcp "fili1=d2asc/gl.account.gltran_000458,rcs=00080,typ=RSF,
       filo1=data1/gl.account.gltran_000458,rcs=00080,typ=RSF"
 uvcp "fili1=d2asc/vendor.master.names,rcs=00064,typ=RDWz4,
       filo1=data1/vendor.master.names,rcs=00064,typ=IDXf8v"
 uvcp "fili1=d2asc/vendor.master.payments,rcs=00064,typ=RDWz4,
       filo1=data1/vendor.master.payments,rcs=00064,typ=IDXf8v"

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E3. Executing the Data-Conversion-Jobs

execute the created script

 #6. ksh sf/copy2data1 all  <-- execute the script to copy files
     =====================      creating final output file-types desired

 170329:125605:uvcp: uvcp fili1=d2asc/ar.customer.armaster,filo1=data1/ar.customer.armaster
 170329:125605:uvcp: EOF fili01 32 rds, 8192 size; d2asc/ar.customer.armaster
 170329:125605:uvcp: EOF filo01 32 wrts, 0 size; data1/ar.customer.armaster
 170329:125605:uvcp: DISAM.dat: recs=32,rcsz=256,keys=1; data1/ar.customer.armaster
 170329:125605:uvcp: uvcp fili1=d2asc/ar.customer.arsales,filo1=data1/ar.customer.arsales
 170329:125605:uvcp: EOF fili01 20 rds, 1280 size; d2asc/ar.customer.arsales
 170329:125605:uvcp: EOF filo01 20 wrts, 1280 size; data1/ar.customer.arsales
 170329:125605:uvcp: uvcp fili1=d2asc/ar.customer.artrans,filo1=data1/ar.customer.artrans
 170329:125605:uvcp: EOF fili01 32 rds, 6068 size; d2asc/ar.customer.artrans
 170329:125605:uvcp: EOF filo01 32 wrts, 6180 size; data1/ar.customer.artrans
 170329:125605:uvcp: uvcp fili1=d2asc/city.tax.assess,filo1=data1/city.tax.assess
 170329:125605:uvcp: EOF fili01 10 rds, 1280 size; d2asc/city.tax.assess
 170329:125605:uvcp: EOF filo01 10 wrts, 1280 size; data1/city.tax.assess
 170329:125605:uvcp: uvcp fili1=d2asc/city.tax.payments,filo1=data1/city.tax.payments
 170329:125605:uvcp: EOF fili01 16 rds, 1680 size; d2asc/city.tax.payments
 170329:125605:uvcp: EOF filo01 16 wrts, 1808 size; data1/city.tax.payments
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.glmaster_000123,filo1=data1/gl.account.glmaster_000123
 170329:125605:uvcp: EOF fili01 109 rds, 13952 size; d2asc/gl.account.glmaster_000123
 170329:125605:uvcp: EOF filo01 109 wrts, 13952 size; data1/gl.account.glmaster_000123
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.glmaster_000124,filo1=data1/gl.account.glmaster_000124
 170329:125605:uvcp: EOF fili01 109 rds, 13952 size; d2asc/gl.account.glmaster_000124
 170329:125605:uvcp: EOF filo01 109 wrts, 13952 size; data1/gl.account.glmaster_000124
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.glmaster_000125,filo1=data1/gl.account.glmaster_000125
 170329:125605:uvcp: EOF fili01 109 rds, 13952 size; d2asc/gl.account.glmaster_000125
 170329:125605:uvcp: EOF filo01 109 wrts, 13952 size; data1/gl.account.glmaster_000125
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.gltran_000456,filo1=data1/gl.account.gltran_000456
 170329:125605:uvcp: EOF fili01 20 rds, 1600 size; d2asc/gl.account.gltran_000456
 170329:125605:uvcp: EOF filo01 20 wrts, 1600 size; data1/gl.account.gltran_000456
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.gltran_000457,filo1=data1/gl.account.gltran_000457
 170329:125605:uvcp: EOF fili01 20 rds, 1600 size; d2asc/gl.account.gltran_000457
 170329:125605:uvcp: EOF filo01 20 wrts, 1600 size; data1/gl.account.gltran_000457
 170329:125605:uvcp: uvcp fili1=d2asc/gl.account.gltran_000458,filo1=data1/gl.account.gltran_000458
 170329:125605:uvcp: EOF fili01 20 rds, 1600 size; d2asc/gl.account.gltran_000458
 170329:125605:uvcp: EOF filo01 20 wrts, 1600 size; data1/gl.account.gltran_000458
 170329:125606:uvcp: uvcp fili1=d2asc/vendor.master.names,filo1=data1/vendor.master.names
 170329:125606:uvcp: EOF fili01 8 rds, 249 size; d2asc/vendor.master.names
 170329:125606:uvcp: EOF filo01 8 wrts, 372 size; data1/vendor.master.names
 170329:125606:uvcp: uvcp fili1=d2asc/vendor.master.payments,filo1=data1/vendor.master.payments
 170329:125606:uvcp: EOF fili01 16 rds, 745 size; d2asc/vendor.master.payments
 170329:125606:uvcp: EOF filo01 16 wrts, 860 size; data1/vendor.master.payments

Next, you should check the files for corect conversions, using uvhd & uvhdcob as shown on the following pages '4F1' - 4I2.

Then you can copy the data files to $RUNDATA/data1 for testing with the demo JCL/scripts & COBOL programs as documented in JCLcnv1demo.htm#Part_4.

 #7. cp $CNVDATA1/data1/* $RUNDATA/data1/
     ====================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F1. Checking Data File Conversions

checking data file conversions with 'uvhd'

First we will check data file conversions with 'uvhd' (vertical hexadecimal dumps of data records) & later we will check with 'uvhdcob' which shows the COBOL copybook names beside the data field contents.

You can also convert the files back to EBCDIC & compare to the original FTP. See https://uvsoftware.ca/datacnv1.htm#Part_9. Utility 'uvdata55' will copy conversion jobs from pfx4/* to pfx5/... reversing the translate (see 'uvdata55' documented at https://uvsoftware.ca/datacnv1.htm#9A2)

Then use utility 'uvcmp1' to compare the 2 EBCDIC files showing any mismatched record pairs in character & hexadecimal with differences highlighted by '*'s (see page https://uvsoftware.ca/datacnv1.htm#9B2).

uvhd check ar.customer.armaster - EBCDIC file

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. uvhd d1ebc/ar.customer.armaster r256a  <-- check EBCDIC file
     =====================================
     - option 'r' (r256) for Record size
     - option 'a' to translate character line1 to ASCII so we can read
     - zones & digits on lines 2&3 (of each 64 byte segment) show EBCDIC values

     records=32  rsize=256 fsize=8192
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
             130140000055597955504636920334B000018150266550961400000000005151
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ...V@...
             CDD4444444444CC4EFEFCF4444FFF6FFF6FFFF4DCDDE4EDCDCC4444401357000
             946000000000023059218100002500754055310319980695538000000246C000
         128 .........W.....-.........X'..f.....m*..................V'...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ......f.........c..............f..............f.C 19950531
             000000890000000088000000001800068000130000000069C4FFFFFFFF444444
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F2. checking data file conversions with 'uvhd'

uvhd check ar.customer.armaster - ASCII file

 #2. uvhd data1/ar.customer.armaster r257  <-- check ASCII file
     ====================================    - Indexed IDXFORMAT1 (fixed-length)
     - option 'r257' Recsize, ONE greater for delete flag at end each record
     - option 'a' omitted since now already ASCII
     - zones & digits on lines 2&3 (of each 64 byte segment) show ASCII values

     records=32  rsize=257 fsize=8224
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ..4V|...
             4442222222222442535343222233323332333324455525544442222201357000
             9DF00000000002306931810000250D754D55310C12290725E38000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ...............................f.....<........f.C 19950531
             0000008900000000880000000018000680001300000000694233333333222222
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000
         256 .
             0
             A

checking EBCDIC to ASCII conversion

1. The 'a' option (show ASCII translation of EBCDIC file) of uvhd allows you to compare the character fields visually.
   
   
2. You can see the hex zones & digits (vertically below the characters) show the true data values for EBCDIC & ASCII. For example the 1st byte in the EBCDIC record '1' is x'F1' in EBCDIC, but x'31' in ASCII & the 10th byte 'E' is x'C5' in EBCDIC, but x'45' in ASCII.
   
   
3. The important thing is that the packed fields are unchanged as required both for COBOL on the mainframe & for Micro Focus or AIX COBOL on unix/linux. Note that the data conversion jobs translate the whole record, but then preserve the Packed/Binary fields by moving them back guided by the copybook.
   
   
4. For example, the 1st packed field is at zero relative bytes 120-124, and is x'001234567C' for both EBCDIC & ASCII. To calculate the position of byte 120 on the 2nd line, subtract 1st line 64 bytes (120 - 64 = 56) and look down from the scale at 56 to the 2nd line.
   
   
5. You will see that the characters above the packed fields are '...V@' for EBCDIC, but '..V4|' for ASCII - because the translations of packed field contents just happen to result in some readable characters, but most are unreadable & shown as '.'s.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4G1. checking data file conversions with 'uvhd'

uvhd check city.tax.payments - EBCDIC file

 #1. uvhd d1ebc/city.tax.payments z4a
     ================================
     - option 'z4' RDW variable length files with 4 byte record prefixes
     - option 'a' to translate character line (zones/digits show EBCDIC values)

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 .T..        H December 31, 1998
             050044444444C4C88898894FF64FFFF444444444444444444444444444444444
             0400000000008045354259031B01998000000000000000000000000000000000
          64
             44444444444444444444
             00000000000000000000
 rec#=1 rsize=84 fptr=0 fsize=1680 records=16

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          84 .d..10130140T JOHN HENRY               1815 BOWEN ROAD
             0800FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444
             0400101301403016850855980000000000000001815026655096140000000000
          64 VANCOUVER           BCV9S1H1..CK..Wb....g......000149061970530
             ECDCDEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFFFFFFFF44
             515364559000000000002359218101320072C0047C0111C00014906197053000
         128
             4444
             0000
 rec#=2 rsize=132 fptr=84 fsize=1680 records=16

                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         216 .T..10130140P       1999...A....A....A....A....A....A....A....A.
             0500FFFFFFFFD4444444FFFF0004200042000420004200042000420004200042
             0400101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C
          64 ...A....A....A....A.
             00042000420004200042
             0001C0001C0001C0001C

1. First 4 bytes of each record is the RDW prefix with record-size in binary in 1st 2 bytes (bytes 3&4 unused nulls)
   
   

• RDW of 1st record is x'0054' = 5*16 +4 = 84 (80 bytes data & 4 byte prefix)
• record type 'H' (Header) byte 8 of data (byte 12 with RDW prefix)

• RDW of 2nd record is x'0084' = 8*16 +4 = 132 (128 bytes data & 4 byte prefix)
• record type 'T' (Tax-record) byte 8 of data (byte 12 with RDW prefix)

• RDW of 3rd record is x'0054' = 5*16 +4 = 84 (80 bytes data & 4 byte prefix)
• record type 'P' (Payment) byte 8 of data (byte 12 with RDW prefix)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4G2. checking data file conversions with 'uvhd'

uvhd check city.tax.payments - ASCII file

 #2. uvhd data1/city.tax.payments v5
     ===============================
     - option 'v5' Micro Focus COBOL variable length files with 2 byte record
       prefixes (4 if record-size > 4095) & 128 byte File-Header record
     - option 'v4' would show only the data records (default if just 'v')

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 0~......170329072520.................>..........................
             3700000033333333333300000000000000000300000000000000000008000800
             0E00000017032907252000000000000000000E01000000001000000000000000
          64 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000

 rec#=1 rsize=128 fptr=0 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 @P        H December 31, 1998
             4522222222424666666723322333322222222222222222222222222222222222
             0000000000804535D252031C0199800000000000000000000000000000000000
          64                   ..
             22222222222222222200
             00000000000000000000

 rec#=2 rsize=84 fptr=128 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         212 @.10130140T JOHN HENRY               1815 BOWEN ROAD          VA
             4833333333524444244455222222222222222333324454425444222222222254
             001013014040AF8E085E29000000000000000181502F75E02F14000000000061
          64 NCOUVER           BCV9S1H1..C...W.........qq.000149061970530
             44455452222222222244535343004D0058800080007703333333333333332222
             E3F5652000000000002369318101320072C0047C0111C0001490619705300000
         128   ..
             2200
             0000

 rec#=3 rsize=132 fptr=212 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
         344 @P10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,..
             4533333333522222223333000420004200042000420004200042000420004200
             00101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C00
          64 .A,...A,...A,...A,..
             04200042000420004200
             01C0001C0001C0001C00

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4H1. checking data file conversions with 'uvhd'

Notes re: Micro Focus COBOL variable length records

1. The Micro Focus Variable length prefixes uses the 1st nibble (half byte) for a record ID x'4_' identifies a DATA record (vs x'3_' for the File Header)
   
   
2. The record size is in the next 3 nibbles, for city.tax.payments above: System file header x'_080' = 128 bytes, User file Header x'_050' = 80 bytes, Tax record x'_080' = 128 bytes, Payment record x'_050' = 80 bytes
   
   
3. Note that record-size in the prefix does not include the prefix-size (unlike RDW files) & does not include record padding to bring slot size up to a multiple of 4 bytes
   
   
4. For example record #2 user Header record slot size is 84 bytes - 2 byte header, 80 bytes data,& 2 bytes padding x'0000' at end record.
   
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I1. Checking Data File Conversions

checking data file conversions with 'uvhdcob'

Now we will check data file conversions with 'uvhdcob' which shows the COBOL copybook field-names beside the data field contents. uvhdcob makes it easy to see if anything has gone wrong in the conversion, such as the wrong COBOL copybook, which would cause the data field contents to be misaligned with the field-names.

For uvhdcob, you must specify the COBOL copybook as the 2nd argument following the datafilename & any options as the 3rd argument. For uvhdcob, you do not need to specify option 'r' (Record-size) since that is known from the copybook.

uvhdcob check ar.customer.armaster - EBCDIC file

 #1. uvhdcob d1ebc/ar.customer.armaster maps/armaster a
     ==================================================
     - can omit option 'r' (Record-size) since known from copybook
     - need option 'a' to translate character line to Ascii

 now=20170329:1051 uvhdcob d1ebc/ar.customer.armaster m25a
 version=20161001 copybook=maps/armaster
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0
 rec#       1           occurs  bgn  end typ <------ data (hex if typ=p/b) ----->
 001 armaster                      0    0
 002 cm-cust                       0    5 n  130140
 003 cm-delete                     6    9
 004 cm-name                      10   34    EVERGREEN MOTORS LTD.
 005 cm-adrs                      35   59    1815 BOWEN ROAD
 006 cm-city                      60   75    NANAIMO
 007 filler001                    76   76
 008 cm-prov                      77   78    BC
 009 filler002                    79   79
 010 cm-postal                    80   89    V9S1H1
 011 cm-telephone                 90  101    250-754-5531
 012 cm-contact                  102  119     LARRY WRENCH
 013 cm-thisyr-sales      1/12   120  124pns 001234567C
 014 cm-lastyr-sales      1/12   180  184pns 001234567D
 015 filler003                   240  255    C 19950531
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I2. checking data file conversions with 'uvhdcob'

uvhdcob check ar.customer.armaster - ASCII file

 #2. uvhdcob data1/ar.customer.armaster maps/armaster  <-- check ASCII file
     ================================================    - Indexed IDXFORMAT1
     - uvhdcob needs no options for fixed length ASCII files

 now=20170329:1052 uvhdcob data1/ar.customer.armaster.dat m25
 version=20161001 copybook=maps/armaster
 rec#=1 rcount=32 rsize=257 fsize=8224 fptr=0
 rec#       1           occurs  bgn  end typ <------ data (hex if typ=p/b) ----->
 001 armaster                      0    0
 002 cm-cust                       0    5 n  130140
 003 cm-delete                     6    9
 004 cm-name                      10   34    EVERGREEN MOTORS LTD.
 005 cm-adrs                      35   59    1815 BOWEN ROAD
 006 cm-city                      60   75    NANAIMO
 007 filler001                    76   76
 008 cm-prov                      77   78    BC
 009 filler002                    79   79
 010 cm-postal                    80   89    V9S1H1
 011 cm-telephone                 90  101    250-754-5531
 012 cm-contact                  102  119     LARRY WRENCH
 013 cm-thisyr-sales      1/12   120  124pns 001234567C
 014 cm-lastyr-sales      1/12   180  184pns 001234567D
 015 filler003                   240  255    C 19950531
 rec#=1 rcount=32 rsize=257 fsize=8224 fptr=0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_5. Data File Conversion - Comprehensive & Complete

Part 5 - Optional/Alternative Procedures

 5B4. Alternative source for the Master control file
      - some sites may have a spreadsheet of essential files & characteristics
        that can be converted into the control-file for datafile conversion.
     - see uvcopy job 'xlsdata51' that can be modified for your format

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A1. Data File Conversion

Concise Operating Instructions - for Re-Runs

Here is a concise summary of the 'Operating Instructions' for Data File conversion, without much explanation. You should use these after you have become very familiar with these procedures & ONLY for RE-CONVERSIONS of the same batch of datafiles.

 There are several initial (one-time) operations that you would miss or
 run when not necessary if you follow these blindly. These instructions
 assume all the datafiles, copybooks,& initial control-files are in place.
 Especially note pre-requisites to creating the control-file at '2B1'
 - very briefly reviewed here.

Pre_requisites creating control-file

1. d0ebc - datafiles FTP'd BINARY from mainframe, RDW for var-lth files
2. d1ebc - copied from d0ebc by copyd0d1 renaming GDG files to UV standards
3. cpys - copybooks FTP'd ASCII from mainframe
4. ctl/cat0/ - LISTCAT reports from mainframe (record-sizes,file-types,etc)
5. ctl/lrecl0 - special catalog reports with recsizes for Non VSAM files
6. ctl/datacopyxref - datafilename/copybookname cross-reference - exported from existing spreadsheet - OR created on 1st conversion from datafiles in d0ebc & your knowledge
7. ctl/datactl53I - may include control-file from JCL conversion - if you don't have good info from LISTCAT reports
8. ctl/cat2/... - can edit additional control-files to supply missing info
   

   can omit filenames on uvcopy jobs below

You can omit entering the filenames for most of the uvcopy jobs below. The filenames shown are the defaults & are normally correct. You will be prompted to allow change, but just hit enter to accept the default. For example:

 #1. uvcopy ctldata61,fild1=d0ebc,filo1=ctl/datactl61,filo2=tmp/datacopyxref
     =======================================================================
     - filenames shown are the defaults & are correct

 #1. uvcopy ctldata61   <-- omit filenames, same as above (easier to enter)
     ================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A2. Concise Operating Instructions - for Re-Runs

Concise Instructions - Creating Data Conversion Control File

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. uvcopy ctldata61,fild1=d0ebc,filo1=ctl/datactl61,filo2=tmp/datacopyxref
     =======================================================================
     - create control-file from the datafilenames in d0ebc/...

Low Volume conversions - skip to '5A5'

If you only have a few files to convert, you might simply FTP/RDW to d0ebc, copyd0d1 '1C5' to d1ebc, run just ctldata61 above to create a skeleton control file with datafilenames & fill in the blanks.

If you had converted all files to variable length (see mainframe JCL on '1C1') before FTP with RDW option, then datactl61 would fill in the record sizes. Here is example for the 1st 3 files:

 ar.customer.armaster  cpy=____________ typ=RSF    rca=00256 rcm=00256 rcs=_____ gdg=___ key=___,___
 ar.customer.arsales   cpy=____________ typ=RSF    rca=00064 rcm=00064 rcs=_____ gdg=___ key=___,___
 ar.customer.artrans   cpy=____________ typ=RDWz4  rca=00120 rcm=00249 rcs=_____ gdg=___ key=___,___

For Low-Volume conversions & manually edited control files - skip to '5A5' -->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A3. Concise Operating Instructions - for Re-Runs

create control-file for High Volume conversions

 #2. For re-runs, we assume datafilename/copybookname cross-ref already created
     ==========================================================================
     either from tmp/datacopyxref 1st conversion of current batch of datafiles
     OR you already had a spreadsheet that you exported to a .csv file
     AND have already copied it to ctl/datacopyxref.

 #3. uvcopy xrefcpy51,fili1=ctl/datacopyxref,filo1=ctlcpy1I
     ======================================================
     - convert csv file from spreadsheet of datafilenames & copybooknames
       into Indexed file for lookup by datafilename to get copybookname

 #4. uvcopy ctldata62,fili1=ctl/datactl61,filo1=ctl/datactl62
                     ,filr2=ctl/cpyrcs1I,filr3=ctl/ctlcpy1I
     ========================================================
     - insert record-sizes & copybooknames into control-file
       from Indexed files ctl/cpycpy1I & ctl/cpyrcs1I

 #5. uvcopy ctldata63,fili1=ctl/datactl62,filo1=ctl/datactl63
                     ,filr2=ctl/datacat52I,filr3=ctl/datactl53I
     ==========================================================
     - update control file with LISTCAT info from Indexed file ctl/datacat52I

 #1-5. ctldata6A all   <-- run or re-run utilities ctldata61 thru ctldata63
       =============

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A4. Concise Operating Instructions - for Re-Runs

 #6. cp ctl/datactl63 ctl/datacpy52
     ==============================
     - copy/rename control file to match 'uvdata52' expectation.
     - might first diff the files to confirm control file improvements

 #7. vi ctl/datacpy52  <-- check ctlfile for missing copybooks & record-sizes
     ================    - if many, update ctlfile,& re-create

 #8. Update/Add control-files to improve datafile conversions
     ========================================================
     - transfer more LISTCATs from mainframe & rerun script 'catdata50'
     - transfer/add more copybooks from mainframe & rerun script 'cpyrcs1A'
     - update your datafile/copybook crossref ctl/datacopyxref
     - might add JCL control-file testlibs/ctl/datactl53I to cnvdata/ctl/
       (if you already converted JCL & did not have any/much LISTCAT info)
     - If any updates --> restart the conversion from #1 ctldata61 above.

Generating Data-Conversion-Jobs from copybooks & control-file

These concise Operating Instructions are intended for re-conversions of the same batch of datafiles/copybooks to uvcopy data convert jobs. For the 1st conversion, follow the detailed instructions '3A1' - 3B4.

pre-Requisites to Generating jobs

1. COBOL copybooks stored in $CNVDATA/cpys/...
   
   
2. Data-Conversion-Control-File created & stored at $CNVDATA/ctl/datacpy52 See '2B1' - '2F4' operating instructions to create
   
   

 2a. cp ctl/datactl63 ctl/datacpy52
     ==============================
     - copy control file renaming as expected by 'uvdata52' below

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A5. Concise Operating Instructions - for Re-Runs

Concise Instructions - to generate Data Conversion Jobs

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. cp -r pfx2 pfx2old   <-- save any prior conversion uvcopy jobs
     ==================       for recovery of record-type test code
     - you may have invested considerable time inserting R/T test code
     - that you can recover from pfx2old/... at #6 edits below

 #2. rm -f maps/* pfx1/* pfx2/* pfx3/* pfx4/*
     ========================================
     - remove any old outputs from all subdirs except cpys/ input

generate 'cobmap's from COBOL copybooks

 #3. uvcopyx cobmap1 cpys maps uop=q0i7
     ==================================
     - convert COBOL copybooks to cobmaps (record layouts)

generate 'data-conversion-jobs' from cobmaps

 #4. uvcopyx uvdata51 maps pfx1 uop=q0i7
     ===================================
     - generate uvcopy data conversion jobs from copybookmaps
     - codes fili1=... & filo1=... & jobnames sames as copybooknames

 #5. cp pfx1/* pfx2/   <-- copy all pfx1 jobs to pfx2
     ===============       before changes required for some file types

 #6. vi pfx2/...       <-- modify jobs with multi-record-type files,
     ===========           variable length records,& occurs depending on

 #7. cp $UV/pf/IBM/skeleton2 pfx2
     ============================
     - copy uvcopy job skeleton for datafiles without copybooks, presuming that
       files without copybooks have no packed/binary fields or zoned signs
     - might modify control file to typ=LST (LineFeeds OK if no packed/binary)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A6. Concise Operating Instructions - for Re-Runs

replicate jobs inserting datafilenames vs copybooknames

 #8. uvcopy uvdata52,fili1=ctl/datacpy52,fild2=pfx2,fild3=pfx3,uop=q0i7
     ==================================================================
     - copies pfx2/* to pfx3/... inserting datafilenames (vs copybooknames)
       on the fili1=... & filo1=... I/O filename declarations
     - also renames the jobs for the datafilenames (vs copybooknames)
       since could have multiple datafiles for same copybook (esp. for GDG)
     - also inserts/updates file characteristics from keywords in control file
       (record-size, file-type, Indexed-key-locations)

 #9. vi pfx3/...   <-- spot check results
     ============    - might need to update control file & reconvert

replicate jobs for GDG files with multiple generations

 #10. uvcopy uvdata53,fild1=pfx3,fild2=pfx4,fild3=d1ebc
     =================================================
     - copy pfx3/* to pfx4/... replicating conversion jobs for GDG files
       with multiple generations & inserting filenames depending on GDG
       members found in d1ebc/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A7. Concise Operating Instructions - for Re-Runs

Pre-Requisites to Executing the Jobs

1. EBCDIC datafiles have been FTP'd binary & RDW if variable length into d0ebc/... & copied to d1ebc/... by 'copyd0d1' to rename mainframe GDG files to UV GDG conventions, see '1C1' - 1C5.
   
   
2. We created the control-file relating datafilenames to copybooknames & keyword coded with record-sizes, file-types, Indexed keys, etc. See those procedures on pages '2D1' - 2F4.
   
   
3. Initial Data Conversion Jobs have been generated into $CNVDATA/pfx1 from the copybooks & the control-file using procedures on pages '3A1' - 3B4.
   
   
4. We copied the conversion jobs from pfx1/* to pfx2/ on '3B1' & modified as required inserting record-type test code for files with multiple record types. See examples on pages '3C2' & '3C6'
   
   
5. We copied pfx2/* to pfx2save/... on page '3B2' to save our Record-Type test coding in case jobs regenerated overwriting our R/T test code.
   
   
6. We ran 'uvdata52' on page '3B3' to copy the jobs from pfx2/* to pfx3/... inserting the datafilenames (vs the copybooknames) & replicating for possible multiple files for one copybook.
   
   
7. Finally, we used 'uvdata53' on page '3B4' to copy pfx3/* to pfx4/... replicating conversion jobs for GDG files.
   
   

                      rename GDG     EBCDIC-ASCII      copy to test
 mainframe ----> d0ebc --------> d1ebc ---------> d2asc -----------> $RUNDATA/data1
           FTP         copyd0d1          pfx4/*          copy2data1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A8. Concise Operating Instructions - for Re-Runs

Executing the Data-Conversion-Jobs

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. uvcopyxx 'pfx4/*'    <-- execute all jobs to convert all files
     =================        from d1ebc/* to d2asc/...

Correcting jobs & re-running

 #2a. vi pfx4/ar.customer.phone.list  <-- correct conversion job for the file
      ==============================

See this example on page '4C1' - the record-size was missing from the control file resulting in record-size 00000 in the generated job. We needed to research and insert the correct record-size & rerun the job. We might be able to determine the correct record-size using 'uvhd' as follows;

 uvhd d1ebc/ar.customer.phone.list r256a
 =======================================

• option 'r256' is the default when we do not know the record-size
• option 'a' is required to translate the character line of EBCDIC files
  so we can read the data, but still see true hex values for zones & digits
• Look for repeating data to determine the record-size.

re-run corrected conversion job

 #2b. uvcopy pfx4/ar.customer.phone.list  <-- re-run just 1 job for 1 file
      ==================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A9. Concise Operating Instructions - for Re-Runs

copy/create final output file types desired

                   $CNVDATA/d2asc ---------------> $RUNDATA/data1
                                     copy2data1
          /home/userxx/cnvdata/d2asc ----------> /home/userxx/testdata/data1

1. We will create the desired file-types during the copy from the conversion directories to the testdata directories.
   
   
2. During the conversion files are either 'typ=RSF' sequential fixed length OR 'typ=RDWz4' for variable-record-length files.
   
   
3. The control-file codes variable length as 'typ=IDXf8v', but that is for the final output file type desired when copied to testdata/data1/...
   
   
4. The control-file codes Indexed files as 'typ=IDXf1' if fixed-length OR 'typ=IDXf8v' for variable-length. Both would have a key length (key=000,006 for example). These codes are for the final output file when copied from $CNVDATA/d2asc to $RUNDATA/data1.
   
   
5. We will use utility 'mkcopy2data1' to generate a script to perform file-type conversions as files are copied from $CNVDATA/d2asc to $RUNDATA/data1.
   
   

   generate script to copy/create file-types desired

 #3. uvcp "fili1=ctl/datactl63,typ=LST,rcs=191,filo1=ctl/datactl63I,typ=ISF,isk1=0(44)"
     ==================================================================================
     - First create Indexed control-file for mkcopy2data1 lookup by datafilename

 #4. uvcopy mkcopy2data1,fild1=d2asc,fili2=ctl/datactl63I,filo1=sf/copy2data1
     ========================================================================
     - generate a script to copy files from $CNVDATA/d2asc to $RUNDATA/data1
       creating the final output file-types desired as coded in control-file.

 #5. vi sf/copy2data1       <-- investigate output script
     ================         - could modify if/as desired

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A10. Concise Operating Instructions - for Re-Runs

execute the created script

 #6. ksh sf/copy2data1 all  <-- execute the script to copy files
     =====================      creating final output file-types desired

Next, you should check the files for corect conversions, using uvhd & uvhdcob as shown on pages '4F1' - 4I2.

 #7. uvhd d1ebc/ar.customer.armaster r256a  <-- check EBCDIC file
     =====================================

Then you can copy the data files to $RUNDATA/data1 for testing with the demo JCL/scripts & COBOL programs as documented in JCLcnv1demo.htm#Part_4.

 #8. cp $CNVDATA1/data1/* $RUNDATA/data1/
     ====================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B1. Converting Data Files in Logical Groups

Create control-file from files already FTP'd

The conversions documented so far assume that we/you have somehow determined the 1st/next group of datafiles to be converted & have FTP'd them to $CNVDATA/d0ebc.

Then we created the control-file from the files existing in d0ebc, using utility ctldata61, which reads the d0ebc directory & writes an initial control file with the filenames on the left & mostly empty keywords on the right, our example:

 ar.customer.armaster   cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.arsales    cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 ar.customer.artrans    cpy=________ typ=RDWz4 rca=00120 rcm=00249 rcs=_____ gdg=___ key=___,___
 ar.customer.phone.list cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.assess        cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 city.tax.payments      cpy=________ typ=RDWz4 rca=00080 rcm=00128 rcs=_____ gdg=___ key=___,___
 gl.account.glmaster_   cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 gl.account.gltran_     cpy=________ typ=_____ rca=_____ rcm=_____ rcs=_____ gdg=___ key=___,___
 vendor.master.names    cpy=________ typ=RDWz4 rca=00019 rcm=00041 rcs=_____ gdg=___ key=___,___
 vendor.master.payment  cpy=________ typ=RDWz4 rca=00019 rcm=00062 rcs=_____ gdg=___ key=___,___

Utillity 'ctldata61' recognizes the unique record prefixes for RDW files & reads thru the file to determine the smallest & largest records for rca=... & rcm=... It would be great if we could FTP all files with RDW, so we could know the record sizes for Fixed-Length files as well as variable-length files.

Creating a Master Control-File Before transferring files

It would be better if we could initially create a complete control-file with all filenames to be converted & with a group code (grp=ar,grp=gl,etc) that can be used to select the next group of files to be converted.

Essential Files report extracted from JCL

If you transfer all your mainframe JCL to unix/linux, you can run script 'mvsfiles5A' to analyze the JCL & select the 'Essential Files' that need to be transferred to initialize your applications on Unix/Linux. See JCLcnv2real.htm#1P1 - 1P4. Utility 'xmvsctl51' on page '1P4 will convert the Essential Files report to a control file, for example:

 uvcopy xmvs2ctl1,fili1=xmvsA/mvsfiles6,filr2=ctl/datactl53I,filo1=ctl/mvsfiles8
 ===============================================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B1. Converting Data Files in Logical Groups

sample input xmvsA/mvsfiles6

 jar200   002   CAR200   CUSTMAS  S       0256 p    * AR.CUSTOMER.ARMASTER
 icetool3 001   ICETOOL  IN1      S                 * AR.CUSTOMER.ARSALES
 jgl100   001   CGL100   ACCTMAS  S     0 0128 p    * GL.ACCOUNT.GLMASTER_
 jgl200   001   SORT     SORTIN   O                 * GL.ACCOUNT.GLTRAN_

sample OUTput ctl/mvsfiles8

 ar.customer.armaster  cpy=________ rcs=00256 gdg=___ typ=RSF  key=___,___
 ar.customer.arsales   cpy=________ rcs=00064 gdg=___ typ=RSF  key=___,___
 gl.account.glmaster_  cpy=________ rcs=00128 gdg=___ typ=RSF  key=___,___
 gl.account.gltran     cpy=________ rcs=00080 gdg=___ typ=RSF  key=___,___

If the record-sizes are not in xmvsA/mvsfiles6, they are picked up from ctl/datactl53I Indexed file of file info created for JCL conversion. Other info will be picked up from ctl/datacat52I when you select & input these control files into the procedures on page '2F2'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B3. Converting Data Files in Logical Groups

Adding a group code

 vi ctl/mvsfiles8   <-- could add group code with editor
 ================     - or in this case, use a utility to transfer
                        1st 2 bytes of filename into grp=__ & then fine tune

 ar.customer.armaster  cpy=________ rcs=00256 gdg=___ typ=RSF  grp=ar key=___,___
 ar.customer.arsales   cpy=________ rcs=00064 gdg=___ typ=RSF  grp=ar key=___,___
 gl.account.glmaster_  cpy=________ rcs=00128 gdg=___ typ=RSF  grp=gl key=___,___
 gl.account.gltran     cpy=________ rcs=00080 gdg=___ typ=RSF  grp=gl key=___,___

Above, we have inserted grp=ar or grp=gl & could use that code to select the control-file entries for the next desired group of datafiles to be converted. Our sample file here has only 4 entries for illustration, but there would probably be hundreds or thousands in the mvsfiles6 report from a mainframe.

selecting ctlfile entries for next group of files

 uvcopy keep1,fili1=ctl/mvsfiles8,filo1=ctl/datactl51,arg1="grp=ar"
 ====================================================================
 - use the'keep1' uvcopy utility to select desired entries
 - write to ctl/datactl61 input to utility 'ctldata62' on page '2F1' to
   add copybooknames from ctl/ctlcpyI & any missing record-sizes from
   ctl/cpyrcs1I

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B4. Converting Data Files in Logical Groups

Alternative - control-file from spreadsheet

Some sites may have a spreadsheet of their 'Essential Files' that we can convert into the keyword format required for these procedures. For example:

 DSN,copybook,Organization,Reclen,Block,RecFormat,Allocated,USED,Extents,created,Referenced
 #---------------------------------------------------------------------------------------
 ar.customer.armaster,armaster,PS,256,25600,FB,800,50,1,9/30/2014,3/14/2015,
 ar.customer.arsales,arsales,PS,64,6400,FB,750,25,1,8/25/2014,2/17/2015,
 gl.account.master,glmaster,PS,128,12800,FB,750,25,1,8/25/2014,2/17/2015,
 gl.account.gltran,gltran,PS,80,8000,FB,750,25,1,8/25/2014,2/17/2015,
 vendor.master.names,vendormas,PS,100,10000,VB,750,25,1,8/25/2014,2/17/2015,
 vendor.master.payments,vendorpaymas,PS,100,10000,VB,750,25,1,8/25/2014,2/17/2015,

We made a uvcopy job 'xlsdata51' to extract the data from the sppreadsheet .csv into our required keyword format. See our uvcopy job 'xlsdata51' listed on page '9Y15'. You could modify it depending on the format of your Excel spreadsheet. Run as follows:

 uvcopy xlsdata51,fili1=ctl/dataxls51.csv,filo1=ctl/cat2/dataxls51
 =================================================================
 uvcopy xlsdata51   <-- same as above, files default as shown
 ================

spreadsheet info converted to keyword format

 ar.customer.armaster    cpy=armaster     rca=00256 rcm=00256 rcs=00256 data=ps
 ar.customer.arsales     cpy=arsales      rca=00064 rcm=00064 rcs=00064 data=ps
 gl.account.master       cpy=glmaster     rca=00128 rcm=00128 rcs=00128 data=ps
 gl.account.gltran       cpy=gltran       rca=00080 rcm=00080 rcs=00080 data=ps
 vendor.master.names     cpy=vendormas    rca=00100 rcm=00100 rcs=00100 data=ps
 vendor.master.payments  cpy=vendorpaymas rca=00100 rcm=00100 rcs=00100 data=ps

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C1. Data File Conversion

converting 'FLR' to 'VLR' on mainframe prior to FTP

Here is an optional procedure that was recommended on page '1C1' where we FTP'd the mainframe files to Unix/Linux. It would be nice if all files could be FTP'd with the RDW option, so we can know the record-sizes immediately on unix/linux, but the RDW option will not convert the Fixed- Length-Files to Variable-Length-Files during the FTP.

But we could convert FLR to VLR prior to the FTP with RDW option for all. When we receive the files on unix/linux, we will know files that were Fixed-Length on the mainframe, because all record will have the same size in the RDW prefix. Here is a skeleton JCL that could be used to convert Fixed to Variable.

JCL to convert Fixed-Length to Variable-Length

 //* JCL to convert Fixed-Length to Variable-Length
 //SORT001   EXEC PGM=SORT
 //SYSUDUMP  DD SYSOUT=K,HOLD=YES
 //SYSOUT    DD SYSOUT=*
 //SYSPRINT  DD SYSOUT=*
 //SORTIN    DD DSN=AAAAA.BBBBB.CCC.DDDDD,
 //             DISP=(SHR,KEEP,KEEP)
 //SORTOUT   DD DSN=AAAAA.BBBBB.CCC.DDDDD.X,
 //             DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
 //             SPACE=(TRK,(0500,0500),RLSE),
 //             DCB=(RECFM=VB,LRECL=32000)
 //SYSIN     DD *
    SORT FIELDS=COPY
    OUTFIL FNAMES=SORTOUT,FTOV
 /*
 //*Note - 'FTOV' means Fixed TO Variable

JCL to FTP Variable-Length with RDW to Unix/Linux

 //* JCL to FTP Variable-Length with RDW to Unix/Linux
 //ACHP01B   EXEC PGM=FTP,PARM='(TRAN TRTAUGE1'
 //OUTPUT    DD  SYSOUT=*
 //SYSPRINT  DD  SYSOUT=*
 //INPUT     DD  *
 30.252.99.11 21 (exit
 userid password
 cd /p2/cnv1/cnvdata/d0ebc/
 pwd
 type i
 LOCSITE RDW
 PUT 'AAAAA.BBBBB.CCC.DDDDD.X' AAAAA.BBBBB.CCC.DDDDD
 QUIT
 /*

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C2. replicate JCL to convert FLR to VLR prior to FTP

Alternative FTP running on Unix/Linux

We could put all files in 1 directory on the mainframe & then FTP from the unix/linux system with the following FTP commands:

 ftp xxx.xxx.xxx.xxx  <-- FTP from mainframe IP#
 userid--> .....      <-- enter userid
 passwd--> .....      <-- enter password
 BINARY               <-- ensure BINARY transfer
 LITERAL SITE RDW     <-- for variable length, work for FIXED length ?
 cd ...               <-- change to data files directory ??
 prompt
 mget *               <-- FTP all files

utility to generate JCL to convert FLR to VLR

The recommended plan is to initially create a master control file of all files that need to be transferred, with a group-code to identify logical groups of files that we will transfer together for the next system to be tested on Unix/Linux.

From each group of selected control-file entries, we can automatically generate the JCL to convert the FLR files to VLR while copying all files in the group to a directory from which the unix/linux system can get the files via FTP/RDW. The auto-generated JCL would be FTP'd to the mainframe for execution.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C3. converting 'FLR' to 'VLR' on mainframe prior to FTP

replicate JCL to convert FLR to VLR prior to FTP

'genF2Vjcl1' replicates mainframe JCL to convert Fixed-Length-Files to Variable-Length-Files prior to FTP to unix with RDW option, so that we can know all record sizes when received on the unix/linux system. We will recognize fixed-length-files when the minimum & maximum record sizes are the same.

genF2Vjcl1 is a Vancouver Utility which runs on Linux & the replicated JCL can be FTP'd text mode to the mainframe for execution. genF2Vjcl1 is driven by a control-file list of the mainframe datafilenames for which the JCL is to be generated.

sample control-file for genF2Vjcl1

       ar.customer.armaster
       ar.customer.arsales
       ar.customer.artrans
       ar.customer.phone.list
       city.tax.assess
       city.tax.payments
       gl.account.glmaster_
       gl.account.gltran_
       vendor.master.names
       vendor.master.payments

The filenames will be translated to UPPER case for the mainframe. GDG files are identified by a trailing '_' underscore which will be converted to (0) to get the latest generation of that file on the mainframe.

directories for data-file conversions

 /home/userxx/       <-- homedir OK for demos, but not high volume data conversions
 :-----cnvdata/         <-- data conversion superdir $CNVDATA
 :     :--*--ctl/         - control file directory
 :     :  :--*--datajcl60 - control file of datafilenames
 :     :-----d0ebc/       - EBCDIC files from mainframe for conversion
 :     :-----d1ebc/       - GDG files renamed for VU (trailing underscore)
 :     :-----d2asc/       - files converted to ASCII
 :     :--*--jclF2V/      - mainframe JCL generated by 'genF2Vjcl1'
 :     :-----maps/        - cobmaps (record layouts) generated from copybooks
 :     :-----pfx1/        - uvcopy jobs to convert EBCDIC to ASCII (gen from cobmaps)
 :     :-----pfx2/        - copied here for manual changes for multiple Record Types
 :     :-----pfx3/        - renamed here for the datafilename (vs copybookname)
 :     :-----pfx4/        - copied here, replicating & inserting GDG filenames

genF2Vjcl1 needs only the control file ctl/datajcl60 (mainframe filenames) and the output subdir jclF2V/ to write the generated JCL, but we show you the other subdirs used to convert the datafiles after we receive them on unix/linux.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C4. replicate JCL to convert FLR to VLR prior to FTP

genF2Vjcl1 operating instructions

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. mkdir jclF2V      <-- make subdir to receive gnerated JCL
     ============

 #2. uvcopy genF2Vjcl1,fili1=ctl/datactl60,fild2=jclF2V
     ==================================================
     - replicate mainframe JCL to convert FLR to VLR prior to FTP to unix
     - JCL templates are embedded in the 'genF2Vjcl1' uvcopy job

sample output

 //FLR2VLR  JOB (AAAAAA,A0000),'USERXX',
 //         CLASS=X,MSGLEVEL=(1,1),NOTIFY=&SYSUID,MSGCLASS=Y
 //* replicate mainframe FTP JCL to convert Fixed-Length-Files
 //* to Variable-Length-Files prior to FTP to unix with RDW option
 //*------------------------------------------------------------------
 //SORT0001  EXEC PGM=SORT
 //SYSUDUMP  DD SYSOUT=K,HOLD=YES
 //SYSOUT    DD SYSOUT=*
 //SYSPRINT  DD SYSOUT=*
 //SORTIN    DD DSN=AR.CUSTOMER.ARMASTER,
 //             DISP=(SHR,KEEP,KEEP)
 //SORTOUT   DD DSN=AR.CUSTOMER.ARMASTER.X,
 //             DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
 //             SPACE=(TRK,(0500,0500),RLSE),
 //             DCB=(RECFM=VB,LRECL=8192)
 //SYSIN     DD *
    SORT FIELDS=COPY
    OUTFIL FNAMES=SORTOUT,FTOV
 /*
 //*------------------------------------------------------------------
 //SORT0007  EXEC PGM=SORT
 //SYSUDUMP  DD SYSOUT=K,HOLD=YES
 //SYSOUT    DD SYSOUT=*
 //SYSPRINT  DD SYSOUT=*
 //SORTIN    DD DSN=GL.ACCOUNT.GLMASTER(0),
 //             DISP=(SHR,KEEP,KEEP)
 //SORTOUT   DD DSN=GL.ACCOUNT.GLMASTER_.X
 //             DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
 //             SPACE=(TRK,(0500,0500),RLSE),
 //             DCB=(RECFM=VB,LRECL=8192)
 //SYSIN     DD *
    SORT FIELDS=COPY
    OUTFIL FNAMES=SORTOUT,FTOV
 /*

Above are the JCLs generated for 2 of our demo files (ar.customer.armaster & gl.account.glmaster_) 1 regular file & 1 GDG file. The '.X' suffixes are required here to make the output names different than the input names. The '.X' suffixes will be removed after FTP to unix/linux.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C5. replicate JCL to convert FLR to VLR prior to FTP

FTP files from mainframe to unix/linux

 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata/

 #1. rm -f d0ebc/*  <-- remove any old files if starting a new group
     =============      of files to be converted

 #2. cd d0ebc       <-- change into the data subdir
     ========

 #3. ftp xxx.xxx.xxx.xxx   <-- FTP from mainframe IP#
     ===================
 #3a. userid--> .....      <-- enter userid
 #3b. passwd--> .....      <-- enter password
 #3c. BINARY               <-- ensure BINARY transfer
 #3d. LITERAL SITE RDW     <-- required if any variable length files
 #3e. cd ...               <-- change to data files directory
 #3f. prompt               <-- turn prompt off for mget
 #3g. mget *.X             <-- get all the '.X' variable length files

 #4. chmod 664 *          <-- all files should be 664
     ===========

Re-Naming files as required

 #5. cd ..               <-- change back out of the d0ebc/ subdir
     =====

 #6. rename-X d0ebc .X   <-- remove the '.X' suffixes
     =================

 #7. renameL d0ebc       <-- rename (translate) all filenames to Lower case
     =============

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C6. Data File Conversion

Alternate FTP ON MAINFRAME to Unix/Linux

See utility 'genftpjcl1' at JCLcnv2real.htm#4D1 - 4D5 to generate mainframe JCL to FTP files from the mainframe to Unix/Linux. This was used at a site where we had a master control-file of all files to be converted with a group code for selecting the next group of files to be converted.

We wanted to FTP all files with the RDW option, but that worked only for variable length files, so we modified the genftpjcl1 utility to look in the control file to see if the file was variable or fixed, and generate the FTP JCL apropriately.

But, rather than using this 'genftpjcl1' job, we recommend using 'genF2Vjcl' on previous pages '5C1' - 5C4 to convert all files to variable length on the mainframe & then running FTP on unix/linux with option RDW.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_8. Data File Conversion - Comprehensive & Complete

Part 8 - listings of scripts for datafile conversion

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9X1. Listings of scripts for datafile conversion

mvsdatadirs - make subdirs for datafile conversions

 # mvsdatadirs - make data subdirs for MVS conversion, testing,& production
 #             - assumes you are in the $RUNDATA superdir (testdata or proddata)
 #             - might be /p1/apps/testdata, /p2/apps/prodata, etc
 #             - this is a subset of script 'mvstestdirs'
 #             - see www.uvsoftware.ca/jclcnv1demo.htm#3D1
 #Feb28/15 - changed subdirs data/ to data1/ & tape/ to tape1/
 #         - vital change in cjlunixop51 control file
 #
 echo "make subdirs for VU JCL,COBOL,& DATA conversion & testing"
 echo "MUST --> cd \$RUNDATA <-- before running this script"
 echo "        \$RUNDATA might be: \$HOME/testdata, /p1/apps/testdata, etc"
 echo "- see www.uvsoftware.ca/jclcnv1demo.htm#3D1"
 echo "--> enter to continue"
 read reply
 #
 mkdir ctl
 mkdir data1
 mkdir jobctl
 mkdir joblog
 mkdir jobmsgs
 mkdir jobtimes
 mkdir jobtmp
 mkdir sysout
 mkdir tape1
 mkdir tmp
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9X2. Listings of scripts for datafile conversion

renameL - rename all filenames in directory to lower case

 #!/bin/ksh
 # renameL - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # renameL - rename an entire directory of filenames to lower case
 #
 echo "rename all filenames in subdir to lower case"
 if [ -d "$1" ]; then :
    else echo "usage: renameL directory [y]"
         echo "       ====================="
         echo " - arg1 must be a directory"
         echo " - arg2 may be 'y' to bypass confirmation prompt"
         exit 1; fi
 #
 reply="$2"
 until [ "$reply" = "y" ]
 do echo "will rename all files in $1 to lower case OK ? y/n"
    read reply
 done
 #
 x=0; y=0
 for i in $1/*
 do let x=x+1
    f=${i##*/}
    typeset -l g=$f
    if [[ $g != $f ]]; then
       mv -i $1/$f $1/$g
       #================
       let y=y+1
       echo "file# $y (of $x) $1/$f - renamed to: $1/$g"
    fi
 done
 echo "total $y files in ${1}, $x renamed to lower case"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9X3. Listings of scripts for datafile conversion

catdata50 - extract control file from LISTCATs

 #!/bin/ksh
 # catdata50 - script to extract info from MVS LISTCAT reports
 #           - relevant info used to automate JCL & DATA file conversions
 #           - by Owen Townsend, UV Software, July 22/2014
 #           - runs uvcopy jobs catdata51, catdata52, & sort/load
 #             Indexed file of LISTCAT info for JCL & DATA conversions
 #           - see www.uvsoftware.ca/jclcnv2real.htm#2C1
 #
 # MVS LISTCAT reports FTP'd to the unix & stored in subdir ctl/cat0/...
 # - may be multiple reports, named for various mainframe discs
 # catdata51 extracts desired items & writes to subdir ctl/cat1/...
 # - AVGLRECL, MAXLRECL, RKP, KEYLEN, REC-TOTAL, GDG LIMIT
 # - code file info as keywords=... on right side of filename
 #
 if [[ "$1" == "all" && -d ctl/cat0 && -d ctl/cat1 && -d ctl/cat2 && -d ctl ]]; then :
   else echo "usage: catdata50 all"
        echo "       ============="
        echo "- arg1 must be 'all' & subdirs ctl,ctl/cat0,ctl/cat1,ctl/cat2 must be present"
        exit 91; fi
 #
 export UVCOPYROP=i30 # inhibit prompts for uvcopy jobs
 #
 uvcopyx catdata51 ctl/cat0 ctl/cat1 uop=q0i7,rop=r0
 #==================================================
 # - extract file info from MVS LISTCAT reports (recsize, key loc/len, etc)
 # - script uvcopyx repeats catdata51 for all files in directory
 #
 uvcopyx catdata52 ctl/cat1 ctl/cat2 uop=q0i7,rop=r0
 #==================================================
 # - translate filenames to lower case, append trailing '_' for GDG files
 # - convert any embedded '$ '#' to '_'s,& sort by filename
 # - script uvcopyx repeats catdata52 for all files in directory
 #
 uvcopy lrecl0cat2,fili1=ctl/lrecl0,filo1=ctl/cat2/lrecl2
 #=======================================================
 # - convert mainframe LRECL listing to LISTCAT UV format
 # - to be merged with other ctl/cat2/... files
 # - lrecl2cat1 job added to this script sf/IBM/catdata50 Dec01/2015
 # - copymvsctls creates empty ctl/lrecl1 in case LRECL listing unavailable
 #
 uvcopy catcat51,fild1=ctl/cat2,filo1=ctl/datacat51,uop=q0i7,rop=r0
 #=================================================================
 # sort/reduce any duplicate filenames to 1 entry
 # - combining significant keyword=... values on right side
 #
 uvcp "fili1=ctl/datacat51,rcs=191,typ=LST,filo1=ctl/datacat52I,typ=ISF,isk1=0(44)"
 #=================================================================================
 # - load indexed file, keyed by filename
 # - to lookup & transfer info to JCL & DATA conversion jobs
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9X4. Listings of scripts for datafile conversion

cpyrcs1A - create cross-ref copybooknames & record-sizes

 #!/bin/ksh
 # cpyrcs1A - script to create Indexed file of copybook-names & record-sizes
 #          - by Owen Townsend, UV Software, Jan2010, update Mar2017
 #
 # - runs cobmap1 to convert all copybooks to layouts
 # - runs 'cpyrcs1' (this job) to extract recsizes from copybook maps to ctl/cpyrcs1
 # - sorts & loads indexed file ctl/cpyrcs1I.dat & .idx
 #   for various following jobs to get recsizes from copybooknames
 #   (such as cobfil51, ???)
 #
 echo "extract rcsz from copybook maps, sort/load Indexed file ctl/cpyrcs1I"
 echo " - for cobfil51 (COBOL files report) to get recsize for copybooks"
 if [[ -d "$1" && -d "$2" && -d "$3" ]]; then :
    else echo "usage: cpyrcs1A cpys maps ctl"
         echo "       ======================"
         echo "- 3 subdirs must be present (as shown above)"
         exit; fi
 #
 echo "skip convert copybooks to maps y/n ? (save time if already done)"
 read reply;
 if [[ "$reply" == "y" ]]; then :
    else rm -f maps/*    # remove any old maps/*
         #===========
         uvcopyx cobmap1 cpys maps uop=q0i7p0
         #===================================
 fi
 uvcopy cpyrcs1,fild1=maps,filo1=ctl/cpyrcs1,rop=r0
 #=================================================
 #
 uvsort "fili1=ctl/cpyrcs1,rcs=256,typ=LST,filo1=ctl/cpyrcs1I,rcs=80,typ=ISF\
 ,isk1=0(20),key1=0(20)"
 #============================================================================
 exit

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9X5. Listings of scripts for datafile conversion

ctldata6A - run ctldata61 thru ctldata63 vs step by step

 # ctldata6A - script to create the Data-Conversion-Control-File
 #           - by Owen Townsend, UV Software, March 15/2017
 # - see documentation at www.uvsoftware.ca/jclcnv1demo.htm#Part_7
 # - intended for re-runs of uvcopy ctldata61 thru ctldata63
 #   after you have updated control files
 #
 # ctldata6A all  <-- run or re-run utilities ctldata61 thru ctldata63
 # =============
 #
 if [[ "$1" == "all" && -d d0ebc && -d ctl ]]; then :
   else echo "usage: ctldata6A all"
        echo "       ============="
        echo "- arg1 must be 'all' & subdirs d0ebc/ & ctl/ must be present"
        exit 91; fi
 #
 export UVCOPYROP=i14 # inhibit prompts for uvcopy jobs
 #
 uvcopy ctldata61,fild1=d0ebc,filo1=ctl/datactl61,filo2=tmp/datacopyxref
 #======================================================================
 # - create control-file from the datafilenames in d0ebc/...
 #
 #Note - ctldata61 (above) creates tmp/datacopyxref, but xrefcpy51 (below)
 #     - inputs ctl/datacopyxref - becuase this script is for Re-Runs
 #       assuming you have edited the copybooknames & copied to ctl/...
 #
 uvcopy xrefcpy51,fili1=ctl/datacopyxref,filo1=ctlcpy1I
 #=====================================================
 # - convert csv file from spreadsheet of datafilenames & copybooknames
 #   into Indexed file for lookup by datafilename to get copybookname
 #
 uvcopy ctldata62,fili1=ctl/datactl61,filo1=ctl/datactl62,filr2=ctl/ctlcpy1I,filr3=ctl/cpyrcs1I
 #=============================================================================================
 # - copy ctl/datactl61 to ctl/datactl62, inserting copybooknames from ctl/cpycpy1I
 #   & inserting record-sizes from ctl/cpyrcs1I
 #
 uvcopy ctldata63,fili1=ctl/datactl62,filo1=ctl/datactl63,filr2=ctl/datacat52I\
 ,filr3=ctl/datactl53I
 #=============================================================================
 # - copy ctl/datactl62 to ctl/datactl63, updating keyword info
 #   from Indexed file ctl/datacat52I
 #
 ##6. cp ctl/datactl63 ctl/datacpy52
 #    ==============================
 #    - copy/rename to agree with 'uvdata52' & earlier documentation.
 ## commented out here in the script, so you can check it before copying
 ## to ctl/datacpy52 for input to 'uvdata52' to complete conversion jobs
 ## with datafilenames (vs copybooknames) & update file info by keywords

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9Y0. Data File Conversion

summary of uvcopy jobs for datafile conversion

2. lrecl0cat2 - convert LISTCAT record-sizes for NONVSAM files to control file
   
   
3. cobmap1 - convert copybooks to record layouts (maps)
   
   
4. cpyrcs1 - converts copybook maps to ctl/cpyrcs1
   
   
   

   9Y5.	xrefcpy51 - convert spreadsheet csv file of datafilenames & copybooknames
   	to Indexed file for lookup by datafilename to get copybookname

5. ctldata61 - create control-file from the datafilenames in d0ebc/...
   
   
6. ctldata62 - insert copybooknames & record-sizes into control-file - copybooknames from ctl/cpycpy1I & record-sizes from ctl/cpyrcs1I
   
   
7. ctldata63 - update control file with LISTCAT info - copy updating keyword info from Indexed file ctl/datacat52I
   
   
8. cobmap1 - convert COBOL copybooks to cobmaps (record layouts)
   
   
9. uvdata51 - generate data conversion jobs from copybookmaps - codes fili1=... & filo1=... & jobnames sames as copybooknames
   
   
10. uvdata52 - copies pfx2/* to pfx3/... inserting datafilenames (vs copybooknames) on the fili1=... & filo1=... I/O filename declarations - also renames the jobs for the datafilenames (vs copybooknames) - also inserts/updates file characteristics from keywords in control file (record-size, file-type, Indexed-key-locations)
    
    
11. uvdata53 - copy pfx3/* to pfx4/... replicating conversion jobs for GDG files with multiple generations & inserting filenames depending on GDG members found in d1ebc/...
    
    
12. mkcopy2data1 - generate script copy files from $CNVDATA/d2asc to $RUNDATA/data1 creating final output file-types desired as coded in control-file.
    
    
13. keep1 - use the'keep1' utility to select desired entries
    
    
    

    9Y15.	xlsdata51 - create a keyword control-file from a user spreadsheet .csv file
    	- selecting datafilename,copybookname,record-size,file-type,gdgs,keys
    	- sample to be customized to the user's spreadsheet format

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9Y1. uvcopy jobs used for datafile conversion

mkcopyd0d1 - make script to change GDG filenames

 # mkcopyd0d1 - make script to copy datafiles from d0ebc to d1ebc
 #            - changing filenames to unix/linux & UV GDG standards
 #            - by Owen Townsend, Mar22/2017
 #
 # - assuming mainframe datafiles stored in $CNVDATA/d0ebc
 # - will create script to copy d0ebc/* to d1ebc
 # - converting filenames from mainframe to unix UV standards
 #
 #*1. uvcopy mkcopyd0d1,fild1=d0ebc,filo1=sf/copyd0d1
 #    ===============================================
 #    - create script to copy/rename files from d0ebc to d1ebc
 #    - changing filenames to unix/linux & UV GDG standards
 #
 # 2. vi sf/copyd0d1   <-- inspect created script
 #    ==============
 #
 # 3. sf/copyd0d1      <-- execute created script
 #    ===========
 #
 #                        ** filename changes **
 #
 # 1. translate to lower case
 # 2. convert embedded '$' to '_' (underscore)
 # 3. convert GDG filenames depending on option 'g'
 #    g0 - use existing gdg# ex: '.G1234V00' --> '_001234'
 #    g1 - resequence gdg#s  ex: '.G1234V00' --> '_000001'
 #                               '.G1235V00' --> '_000002', etc
 #    g2 - leave as is (conversions for JCL emulations)
 #
 #                    ** sample output script **
 #
 # # sf/copyd0d1 - script created by mkcopyd0d1 on 2017/03/22_16:20:25
 # #   - to copy all datafiles from d0ebc to d1ebc
 # #   - converting mainframe GDG filenames to VU GDG conventions
 # #   - 'filename.G1234V00' converted to 'filename_001234'
 # #   - default option g0, d1 to resequence from 000001, d2 leave as is
 # cp d0ebc/AR.CUSTOMER.ARMASTER                 d1ebc/ar.customer.armaster
 # cp d0ebc/AR.CUSTOMER.ARSALES                  d1ebc/ar.customer.arsales
 # cp d0ebc/AR.CUSTOMER.ARTRANS                  d1ebc/ar.customer.artrans
 # cp d0ebc/AR.CUSTOMER.PHONE.LIST               d1ebc/ar.customer.phone.list
 # cp d0ebc/CITY.TAX.ASSESS                      d1ebc/city.tax.assess
 # cp d0ebc/CITY.TAX.PAYMENTS                    d1ebc/city.tax.payments
 # cp d0ebc/GL.ACCOUNT.GLMASTER.G0123V00         d1ebc/gl.account.glmaster_000123
 # cp d0ebc/GL.ACCOUNT.GLMASTER.G0124V00         d1ebc/gl.account.glmaster_000124
 # cp d0ebc/GL.ACCOUNT.GLMASTER.G0125V00         d1ebc/gl.account.glmaster_000125
 # cp d0ebc/GL.ACCOUNT.GLTRAN.G0456V00           d1ebc/gl.account.gltran_000456
 # cp d0ebc/GL.ACCOUNT.GLTRAN.G0457V00           d1ebc/gl.account.gltran_000457
 # cp d0ebc/GL.ACCOUNT.GLTRAN.G0458V00           d1ebc/gl.account.gltran_000458
 # cp d0ebc/VENDOR.MASTER.NAMES                  d1ebc/vendor.master.names
 # cp d0ebc/VENDOR.MASTER.PAYMENTS               d1ebc/vendor.master.payments
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 opr='$jobname - make script to copy d0ebc to d1ebc changing filenames'
 opr='         - changing GDG filenames to UV GDG conventions'
 rop=r1    # prompt for output file disposition at EOJ
 fild1=?d0ebc,rcs=128,typ=DIR
 fild2=?d1ebc,rcs=128,typ=DIR
 fili1=tmp/$fild1.files,rcs=128,typ=LST
 filo1=?sf/copyd0d1,rcs=128,typ=LSTtp775
 #
 # load table of #comments for begining of output script
 lodv1=h0(100)
 # $filo1 - script created by $jobname on $datetime
 #   - to copy all datafiles from $fild1 to $fild2
 #   - converting mainframe GDG filenames to VU GDG conventions
 #   - 'filename.G1234V00' converted to 'filename_001234'
 #   - default option g0, d1 to resequence from 000001, d2 leave as is
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 @run
         opn    fild1                   open indir (prove existing)
         opn    fild2                   open outdir (prove existing)
         opn    filo1                   open output file for script
 # create file of filenames & open for input
         mvfv1  s0(80),'ls $fild1 >tmp/$fild1.files',x'00'
         sys    s0(80)                  create file of filenames
         opn    fili1                   open input file of filenames
         wtbe   filo1,h0(100),h0(100)   dump table of script hdr #cmts
 #
 # begin loop to get ls filenames & create script
 # - converting filenames (as described above)
 man20   get    fili1,a0                 get next filename
         skp>   man90
         mvc    b0(80),a0                copy to output area
         trl    b0(80)                   translate to lower case
 #
 # search for GDG file ID & convert depending on option 'g'
 man24   scnp   b0(80),'.g####v##'       GDG ID ?
         skp!   man60                    go output as is
 man26   mvn    $rg,$rx                  save ptr to '.g####v##'
 #
 # If current filename is 1st member of a GDG set
 #   - init generation# ctr (for resequence option)
 # If 2nd+ member - increment the GDG resequence ctr
 #   - compare current filename to prior saved on length up to '.g####v##'
 man30   cmc    a0($rg80),c0            same as prior up to '.g####v##' ?
         skp=   man34
         add    $ca3,1                  count GDG groups
         mvn    $ca4,0                  clear GDG member ctr
 #
 # common point to increment GDG# ctr
 man34   add    $ca4,1                  increment GDG# ctr
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # convert GDG file depending on option g
 #   g0 - use existing gdg# ex: '.G1234V00' --> '_001234'
 #   g1 - resequence gdg#s  ex: '.G1234V00' --> '_000001', '.G1235V00' --> '_000002', etc
 #   g2 - leave as is (conversions for JCL emulations)
 man40   tsb    o7(1),x'02'             leave as is ?
         skp=   man60                   goto format cp & write
         tsb    o7(1),x'01'             resequence GDG# ?
         skp=   man54
 #
 # default - use existing gdg#
 man50   clr    bg0(40),' '             clear from '.g####v##'
         mvc    bg0(1),'_'              append '_' gdg file ID for UV
         mvn    bg1(6),ag2(4)           convert mf 4 digits to 6 for UV
         skp    man60                   goto format cp & write
 #
 # resequence existing GDG#s from _000001, _000002, etc
 man54   clr    bg0(40),' '             clear from '.g####v##'
         mvc    bg0(1),'_'              append '_' gdg file ID for UV
         mvn    bg1(6),$ca4             insert gdg ctr for curreent group
         skp    man60                   goto format cp & write
 #
 # create 'cp' command, for example:
 # cp d0ebc/filename.g1234v00      d1ebc/filename_001234
 man60   mvf    d0(128),'cp d0ebc/'     setup cmd & input subdir
         mvc    d9(50),a0               append input filename
         mvc    d60(50),'d1ebc/'        setup output subdir
         mvc    d66(50),b0              append output filename
 #
 # shift output filename over if room
 man62   cmc    d45(15),' '             filename short enuf ?
         skp!   1
         mvc    d46(60),d60             yes - shift left
 #
 # write, save current for next compare GDG,& return to get next
 man70   put    filo1,d0                write output
         mvc    c0(80),b0               save current for next GDG compare
         skp    man20
 #
 # EOF - close files & eoj
 man90   cls    all
         eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9Y5. uvcopy jobs used for datafile conversion

xrefcpy51 - convert spreadsheet .csv to Indexed file

 # xrefcpy51 - convert spreadsheet csv file to Indexed file
 #          - to lookup copybooknames from datafilenames
 #          - by Owen Townsend, UV Software, Mar07/2017
 #
 # uvcopy xrefcpy51,fili1=ctl/datacopyxref,filo1=ctl/ctlcpy1I
 # ==========================================================
 #
 #                 ** sample input - ctl/datacopyxref **
 #
 # ar.customer.armaster,armaster.cpy
 # ar.customer.arsales,arsales.cpy
 # ar.customer.artrans,custran3
 # city.tax.assess,citytax1.cpy
 # city.tax.payments,citytax2
 # gl.account.glmaster_,glmaster
 # gl.account.gltran_,gltran
 # vendor.master.names,vendormas
 # vendor.master.payments,vendorpaymas
 #
 # Note - any '.cpy' suffixes in input above removed for output file as shown below
 #
 #                  ** sample output - ctl/ctlfile3 **
 #
 # ar.customer.armaster                         cpy=armaster
 # ar.customer.arsales                          cpy=arsales
 # ar.customer.artrans                          cpy=custran3
 # city.tax.assess                              cpy=citytax1
 # city.tax.payments                            cpy=citytax2
 # gl.account.glmaster_                         cpy=glmaster
 # gl.account.gltran_                           cpy=gltran
 # vendor.master.names                          cpy=vendormas
 # vendor.master.payments                       cpy=vendorpaymas
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 fili1=ctl/datacopyxref,rcs=128,typ=LST
 filo1=ctl/ctlcpy1I,rcs=80,typ=ISF,isk1=0(44)
 @run
        opn    all                      open file
 #
 # begin loop to read/reformat/write until EOF
 man20   get    fili1,a0                get next line of listcat2
         skp>   man90                   (cc set > at EOF)
 #
 # separate datafilename,copybookname from datafile/copybook crossref
 # - possible .csv file exported from spreadsheet
 man30   fix    b0(100),a0(80),3,','    fix csv fields 100 apart
 #
 # create desired output: datafilename(max 44)    cpy=__(max 20)__
 # - following jobs allow 12 max for copybookname
 man34   clr    c0(256),' '             clear output area
         mvu    c0(128),b0,' '          copy filename till ending blank
         mvf    c45(17),'cpy='
         mvu    c49(20),b100,'._ '      store until ending ./_/blank
 #
 man40   put    filo1,c0                write to output Indexed file
         skp    man20                   return to get next line
 #
 # EOF - close files & end job
 man90   cls    all                     close files
         eoj                            end job

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9Y15. uvcopy jobs used for datafile conversion

xlsdata51 - sample uvcopy job

'xlsdata51' is an example uvcopy job that extracts desired file info from a user spreadsheet into the keyword control-file format required for conversions. You can copy/modify it for your particular spreadshet format.

 # xlsdata51 - convert spreadsheet .csv to control-file for data conversion
 #           - by Owen Townsend, UV Software, March 10/2015
 #           - updated for data conversions vs JCL conversions March 2017
 #
 # uvcopy xlsdata51,fili1=ctl/xlsdata51.csv,filo1=ctl/cat2/dataxls51
 # =================================================================
 #
 #                       ** sample input **
 #
 # DSN,copybook,Organization,Reclen,Block,RecFormat,Allocated,USED,Extents,created,Referenced
 # ar.customer.armaster,armaster,PS,256,25600,FB,800,50,1,9/30/2014,3/14/2015,
 # ar.customer.arsales,arsales,PS,64,6400,FB,750,25,1,8/25/2014,2/17/2015,
 # gl.account.master,glmaster,PS,128,12800,FB,750,25,1,8/25/2014,2/17/2015,
 # gl.account.gltran,gltran,PS,80,8000,FB,750,25,1,8/25/2014,2/17/2015,
 # vendor.master.names,vendormas,PS,100,10000,VB,750,25,1,8/25/2014,2/17/2015,
 # vendor.master.payments,vendorpaymas,PS,100,10000,VB,750,25,1,8/25/2014,2/17/2015,
 #
 #                        ** sample outut **
 #
 # ar.customer.armaster    cpy=armaster     rca=00256 rcm=00256 rcs=00256 data=ps
 # ar.customer.arsales     cpy=arsales      rca=00064 rcm=00064 rcs=00064 data=ps
 # gl.account.master       cpy=glmaster     rca=00128 rcm=00128 rcs=00128 data=ps
 # gl.account.gltran       cpy=gltran       rca=00080 rcm=00080 rcs=00080 data=ps
 # vendor.master.names     cpy=vendormas    rca=00100 rcm=00100 rcs=00100 data=ps
 # vendor.master.payments  cpy=vendorpaymas rca=00100 rcm=00100 rcs=00100 data=ps
 #
 opr='$jobname - convert spreadsheet .csv to control-file for data conversion'
 rop=r1       # prompt for file disposition at EOF
 fili1=?ctl/dataxls51.csv,typ=LST,rcs=256
 filo1=?ctl/cat2/dataxls51,typ=LSTt,rcs=100
 # load keyword template
 #         1         2         3         4         5         6         7         8
 #12345678901234567890123456789012345678901234567890123456789012345678901234567890
 lod=h0(100)
 cpy=____________ rca=_____ rcm=_____ rcs=_____ data=_______
 cpy=____________ rca=_____ rcm=_____ rcs=_____ data=_______
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 @run
         opn    all
         get    fili1,a0                  bypass 1st record (header)
 #
 # begin loop to get/process/put records until EOF
 man20   get    fili1,a0                  get next record
         skp>   man90                     (cc set > at EOF)
         cmc    a0(1),'#'                 #cmt line to bypass ?
         skp=   man20
 # drop #cmt lines disabled so demo file #cmts retained for doc
         add    $ca1,1                    count records in
         trl    a0(100)                   translate to lower case
 #
 # convert csv to fixed fields 100 bytes apart
         fix    b100(100),a0(100),15,','  fix .csv fields 100 apart
 #
 # refresh keyword  template & insert desired data
 # store recsize in rca=... & rcs=..., store organization & format in typ=org_fm_
 man30   mvc    h0(80),h100               refresh template
         scn    h04(20),' '               find length of _________
         clr    h04($rx20),' '            clear __________
         mvu    h04(20),b200,' '          store copybook til ending blank
         mvn    $ca3,b400(5)              capture record size
         mvn    h21(5),$ca3               rca=...
         mvn    h31(5),$ca3               rcm=...
         mvn    h41(5),$ca3               rcs=...
         mvc    h52(4),b300               organization
         mvc    h56(3),b600               record format
 #
 # bypass records without rec-size, filenames ending in .data or .index,
 # & file organization 'pds'
 man50   cmn    $ca3,0                    any record size ?
         skp=   man20
         scn    b100(50),'.data '
         skp=   man20
         scn    b100(50),'.index '
         skp=   man20
         cmc    h52(3),'pds'
         skp=   man20
 #
 # store filename in output area followed by keywords
 man60   mvc    c0(50),b100               filename to output area
         mvc    c46(80),h0                append keywords
         put    filo1,c0(100)
         add    $ca2,1                    count records out
         skp    man20
 #
 # EOF - close files & exit
 man90   cls    all
         eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Visitor Counters for ThisYear and LastYear