| Part_0 | - Overview of data compare utilities |
| Part_1 | - Compare 'Fixed Length' Data files |
| - utility 'uvcmp1' will print pairs of records that are different | |
| - differences highlighted with '*'s under 64 byte record segments | |
| - uvcmp1 intended for files with packed/binary fields & no LineFeeds | |
| (use unix 'diff' for text files with LineFeeds) | |
| - options provided for up to 4 'exclude fields' (date/time stamps, etc) | |
| - scripts uvcmpFA1 (ASCII) & uvcmpFE1 (EBCDIC) make it easier | |
| to run uvcmp1 (shorter command lines) | |
| - uvcmp1all, script to compare all data files in 2 sub-directories |
| Part_2 | - Compare 'Variable Length RDW' Data files |
| - using uvcopy job 'uvcmp2' for RDW files (vs 'uvcmp1' for Fixed length) | |
| - RDW files have 'Record Descriptor Words' in 1st 4 bytes each record | |
| - binary record size in 1st 2 bytes (next 2 bytes null/unused) | |
| - script 'uvcmpRDW1' makes it easier to run uvcmp2 (shorter commands) |
| Part_3 | - Compare Micro Focus COBOL Variable Length Data files |
| - using uvcopy job 'uvcmp3' for IDX files (vs 'uvcmp1' or 'uvcmp2') | |
| - may be sequential or Indexed (IDXFORMAT3 or IDXFORMAT8) | |
| - file header record 128/1024 & binary record size in 1st 2 bytes | |
| - script 'uvcmpIDX81' makes the command line shorter & easier to key |
| Part_4 | - Compare Fixed length files that may have INSERTED records |
| - may use uvcmpI1 directly if files are already INDEXED | |
| - else use scripts uvcmpFAI1/uvcmpFEI1 to load sequential files to | |
| Indexed files before calling uvcmpI1 | |
| - need to rerun with files reversed to report inserts in file2 | |
| - use uvcmpFAI2/uvcmpFEI2 to show inserts in both files on 1 report | |
| - step1 unmatI2 writes unmatched records in file1 to tmp2/... | |
| by reading file2 sequentially & reading file1 randomly by key | |
| - uvcmpI2 then reads 3 files data1, data2,& unmatched in data1 |
| Part_5 | - Compare 'entire directories' of data files |
| - 'uvcmp5' uses a control file to supply the record sizes | |
| - script 'uvcmp5all' makes it easier to run | |
| - first run 'unixcmpsplit' to separate large non-matching files | |
| - uses unix 'cmp' to eliminate matching files before uvcmp5 is run | |
| (since 'cmp' is much faster than uvcmp5) |
| Part_6 | - Compare Data files 'Field by Field' |
| - COBOL copybooks used to generate data compare jobs | |
| - may compare files with different record layouts (as for Y2K) | |
| but can be used anytime record layouts are changed for any reason | |
| - unlimited field excludes (for date/time stamps or any reason) | |
| - script 'uvcopyxx' allows you to run all field by field compare jobs |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
These 'CMPjobs' (compare jobs) were developed for Year 2000 reasons, but have proven useful for many reasons long after Year 2000 work has ended.
The CMPjobs can be extremely valuable anytime you make major changes to soft- ware systems, involving parallel runs, to prove the validity of the changes.
This was proven the first time these jobs were used. An analyst had examined the printouts from the parallel runs & pronounced the test OK, but running CMPjobs on the data files highlighted several problems which would have been disastrous, had the system gone into production at that point.
These jobs to compare data files and create reports showing pairs of mismatched records, with differences highlighted by '*'s.
Records are printed in 64 byte segments. If packed or binary fields are present the segments are printed in vertical hexadecimal using 5 lines per segment (characters, zones, digits, '*' mismatches,& space between segments)
These jobs are useful for fixed & variable record length files without linefeeds that may have packed or binary fields. These files are commonly used by COBOL applications that may have been converted from a mainframe.
Note that the UNIX 'diff' utility is marvelous for text files, but cannot be used for files without linefeeds or that have packed or binary data.
The UNIX 'cmp' utility can be used for these files, but is very inadequate in showing you where the differences are. By default it quits on the first mismatch & the long option is not nice (1 byte mismatches per line in octal).
However UNIX 'cmp' is very QUICK to determine whether files are different. You can then run the uvcmp jobs only on the files which are in fact different. This plan will save time if you have very large files.
The uvcmp... jobs have a 'stop print' option, which prevents generating voluminous reports when you probably only need the first page to determine the problem, before fixing and retesting.
They also have a terminate option which allows the run to continue to count the mismatches for totals at end of job. If you have very large files, you can save time by setting the terminate count low (1000 for example) for the initial testing. For the final testing you would set it high, to ensure comparisons are performed on all records in the file.
Note that the above descriptions apply to all of the uvcmp... jobs The next page will describe the jobs in detail, highlighting the differences, and recommending which job you should use in various situations.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
There are 2 ways to run the data comparison jobs. You can run the 'uvcopy jobs' directly or you can run 'scripts' that call the uvcopy jobs, which is easier because it makes the command lines shorter.
First we will describe the most commonly used uvcopy jobs followed by several scripts that call them. Two pages ahead we will describe the less used jobs to compare entire directories & perform field by field comparisons for 2 files with different record layouts.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmp2 |
|
| uvcmp3 |
|
| uvcmpFA1 |
|
| uvcmpFE1 |
|
| uvcmpRDW1 |
|
| uvcmpIDX81 |
|
| uvcmpFAI1 |
|
| uvcmpFEI1 |
|
| uvcmpFAI2 |
|
| uvcmpFEI2 |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
For casual use (1 pair of files at a time), assuming same size records, & less than 5 exclude fields (date/time stamps that need to be ignored), I recommend using uvcmp1, uvcmpI1,& their various scripts, because there is no preparation time required.
For serious work (comparing dozens of files at a time), I recommend running 'uvcmp5all' on all files in the 2 directories & printing all mismatch reports. The stop count allows you to limit the reports to 1 or 2 pages each.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 1A1. | Overview of 'uvcmp1' for Fixed length record comparisons |
| - alternatives uvcmp2,3,5,6 (documented in later parts) | |
| - scripts to run uvcmp1 (shorter commands) |
| 1B1. | Setup Directories to demo uvcmp1 |
| - may setup 'cmpdata' directory in your homedir | |
| - cd cmpdata & run script 'cmpdatadirs' to setup 10 subdirs required |
| 1C1. | test file#1 supplied for uvcmp1 demo (dat1/warmas1) |
| test file#2 with differences (dat1/warmas1a) |
| 1D1. | uvcmp1 demo#1 - Compare 2 ASCII datafiles - Operating Instructions |
| 1D2. | uvcmp1 demo#1 - sample report |
| 1E1. | uvcmp1 demo#2 - Exclude options |
| 1E2. | uvcmp1 demo#2 - Exclude Op. Instrns. & sample report |
| 1F1. | scripts to make uvcmp1 easier to run |
| - will use script 'uvcmpFE1' for demo#3 | |
| - will use EBCDIC files for demo#3 (warmas1E & warmas1Ea) | |
| - uvcopy command vs script equivalent |
| 1F2. | script uvcmpFE1 demo#3 - Op. Instrns & sample report |
| 1G1. | demo#4 - creating test file for packed field comparison |
| 1G2. | demo#4 - Packed field Op. Instrns. & sample report |
| 1H1. | uvcmp1all - script to compare ALL files in 2 directories, for example: |
| - comparing re-converted EBCDIC fiels to original mainframe files | |
| to verify conversions |
| 1X1. | Summary of comparison jobs & scripts documented in this Part. |
| - listings of some scripts (uvcmpFA1, uvcmpFE1, uvcmp1all) |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'uvcmp_' jobs will print pairs of records that are different & highlight the differences with '*'s between the 64 byte record segments. This is usefull for comparing files with packed/binary fields, or that do not have LineFeeds.
UNIX 'diff' is good for text files, & UNIX 'cmp' (for binary files) is OK for indicating the files are the same or different, but very poor at illustrating where the differences are.
*uvcmp1 - compare Fixed length Sequential files (used here in Part 1)
- option y1 for EBCDIC (vs y0 ASCII default)
- option 'r' (for Record-size) must be specifeid
- allows up to 4 EXCLUDE fields defined by options a/b c/d e/f g/h
(example: a44b6 - exclude field#1, a44=displacement, b6=length)
- uvcmp1 does NOT allow for inserted records
| uvcmpI1 |
|
| uvcmpI2 |
|
| uvcmp2 |
|
| uvcmp3 |
|
| uvcmp5 |
|
| uvcmp6 |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmpFA1 |
|
| uvcmpFE1 |
|
| uvcmpRDW1 |
|
| uvcmpIDX81 |
|
| uvcmpFAI1 |
|
| uvcmpFEI1 |
|
| uvcmpFAI2 |
|
| uvcmpFEI2 |
|
| uvcmp1all |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
I recommend you setup a 'cmpdata' within your homedir & then make the subdirs required for comparison job uvcmp1 & its scripts uvcmpFA1 & uvcmpFE1.
#1. Login ---> /home/userxx <-- Login or change to your home directory
#2. mkdir cmpdata <-- make superdir for subdirs shown above
=============
#3. cd cmpdata <-- change into the comparison superdir
==========
#4. mkdir dat1 dat2 rptcmp tmp <-- make subdirs required
==========================
#5a. cp /home/uvadm/dat1/warmas1 dat1
=================================
- copy supplied test files to subdirs in your homedir
#5b. cp /home/uvadm/dat1/warmas1a dat1
==================================
- can store 2nd demo file in same subdir as 1st since filenames different
(store in dat2/... if filenames same)
#6a. vi dat1/warmas1 <-- view the demo files
================
#6b. vi dat1/warmas1a
================
#7a. vi dat1/warmas1 <-- print the demo files
================ see the files listed on the next page -->
#7b. vi dat1/warmas1a
================
/home/userxx <-- your home directory :-----... :-----cmpdata <-- working directory for datafile compares : :-----dat1 <-- subdir to store files to be compared : : :-----warmas1 <-- sample files for comparison : : :-----warmas1a
: :-----rptcmp <-- subdir to receive comparison reports : : :-----warmas1 <-- comparison report demo#1
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
We will demonstrate uvcmp1 on the following 2 pages using the test files shown below:
cust# prod# description purch-date expire-date price
1 2 3 4 5 6 7
01234567890123456789012345678901234567890123456789012345678901234567890
12345 12345 sony television 950101 1-year 960101 000002911 22222 00000 dish washer 960101 2-year 980101 000004922 33333 11111 refrigerator 961231 3-year 991231 000006733 33333 22222 air conditioner 970101 3-year 000101 000006744 44444 00000 freezer 970101 4-year 010101 000009955 55555 11111 clothes dryer 991231 5-year 041231 000055566 66666 00000 micro wave 001231 6-year 061231 000066677 88888 00000 HP computer 030101 3-year 070101 000129999
12345 12345 sony television 950101 1-year 960101 000002911 22222 00000 dish CRASHER 960101 2-year 980101 000004922 33333 11111 refrigerator 961215 4-year 001215 000005500 33333 22222 air conditioner 970101 3-year 000101 000006744 44444 00000 freezer 970101 4-year 010101 000009955 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 66666 00000 micro wave 001215 6-year 999999 000066677 88888 00000 HP computer 030101 3-year 070101 000129999
Note that we have created some intentional mismatches in the 2nd file.
The operating instructions are shown on the following page --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcopy uvcmp1,fili1=dat1/warmas1,fili2=dat1/warmas1a,uop=r64 ============================================================
uop=q1a0b0c0d0e0f0g0h0p30r256s8t500000u3x2y0 - option defaults
a# - dsplcmnt to exclude field #1
b# - length to exclude field #1
c# - dsplcmnt to exclude field #2
d# - length to exclude field #2
e# - dsplcmnt to exclude field #3
f# - length to exclude field #3
g# - dsplcmnt to exclude field #4
h# - length to exclude field #4
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r256 - record size
s8 - StopPrint count, after s# mismatched records
t500000 - StopRead count, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
090218:093751:uvcmp1: uvcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmp1
uvcopy DISAM ext=dat LNX L64 license=20090212V site=UV_Software
User OPtion (uop) defaults = q1p30r256s8t500000u3x2y0r64
-->null to accept or enter/override -->
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=8, File2Total=8, StopPrint=8, StopRead=500000
File1Reads=8, TotalMisMatches=4, MisMatchesPrinted=4
090218:093752:uvcmp1: EOF fili01 rds=8 size=512: dat1/warmas1 090218:093752:uvcmp1: EOF fili02 rds=8 size=512: dat1/warmas1a 090218:093752:uvcmp1: EOF fild03 size=4096: rptcmp 090218:093752:uvcmp1: EOF filo03 wrts=53 size=3058: rptcmp/warmas1 hits=4 EOJ, Output File written to: rptcmp/warmas1 default command = null, OR enter: more,print,uvlpr12,edit,null -->
enter command: vi,cat,more,lp,uvlp12,null --> vi <-- enter vi to view rpt
See differences report listed on the next page --->
Note the response to the options prompt was 'r64', specifying the record sizes as 64 bytes. We specified the options twice here to emphasize their importance, once on the command line & again at the prompt (which would be redundant).
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_09:37:51 uop=q1p30r256s8t500000u3x2y0r64
recsize reccount file-size typ Report=rptcmp/warmas1
1: 64 8 0 RSF File1=dat1/warmas1
2: 64 8 0 RSF File2=dat1/warmas1a
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish washer 960101 2-year 980101 000004922 .
3333323333326676276766722222223333332327667233333323333333332220
22222000000049380713852000000096010102D951209801010000004922000A
*******
2 2 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
3333323333326676245454452222223333332327667233333323333333332220
22222000000049380321385200000096010102D951209801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961231 3-year 991231 000006733 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096123103D951209912310000006733000A
** * ** ** ****
2 3 33333 11111 refrigerator 961215 4-year 001215 000005500 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096121504D951200012150000005500000A
** * ** ** ****
1 6 0 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
3333323333326667667267767222223333332327667233333323333333332220
5555501111103CF48530429520000099123105D951200412310000055566000A
******** **** * * * *****
2 6 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
3333323333326667667254544445223333332327667233333323333333332220
5555501111103CF485303829EB520099061506D951200512010000066699000A
******** **** * * * *****
1 7 0 66666 00000 micro wave 001231 6-year 061231 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000123106D951200612310000066677000A
** ******
2 7 66666 00000 micro wave 001215 6-year 999999 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000121506D951209999990000066677000A
** ******
==================== EOF or StopPrint/StopRead count reached ============== File1Total=8, File2Total=8, StopPrint=8, StopRead=500000 File1Reads=8, TotalMisMatches=4, MisMatchesPrinted=4
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The page headings give you the information you need to know about the file (record-size, no of records, filesize, filenames, options in effect, date).
The options specified at the prompt (or via uop= on the command line), are appended to & override the default options. Please see the options described on the previous page.
The EOF count statistics are vital, indicating whether the report shows you all the mismatches, or whether the stop count was reached. The stop count saves wasting a lot of paper; you probably want to fix the programs & rerun the program & the comparison jobs. You might set the stop count & termination count options higher as your testing gets cleaner to ensure you compare to end of file with no mismatches.
You could enter just 'uvcopy uvcmp1' & enter filenames & options at the prompts, but it is better to enter at least the filenames on the command line, so you can use the shell history if you need to repeat the command.
uvcopy uvcmp1 <-- will prompt for filenames & options if not on command =============
For uvcmp1 (intended for 1 pair of files at a time) you could wait for the prompt to specify options, but for uvcmp5/uvcmp6 it is usually advantageous to specify options on the command line & include the 'q0' option to disable the prompts for multiple files in the directory.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
There are options to exclude up to 4 fields from the mismatch report (for date/time stamps or any fields desired). When you run uvcmp1, it will display the option descriptions & prompt you for changes to the defaults. Please see the options displayed for demo#1 on page '1D1' above.
For demo#2, we will exclude just the 'day' of the 'purchase-date' & the entire 'expiry-date' (see the demo file listed on page '1C1', so the options will be:
uop=r64a34b2c44d6 <-- options for demo#2 (exclude fields)
=================
r64 - record size option mandatory
a34 - dsplcmnt to 1st exclude field
b2 - length of 1st exclude field
c44 - dsplcmnt to 2nd exclude field
d6 - length of 2nd exclude field
Here is the complete command line. You could enter just 'uvcopy uvcmp1' & wait for the prompts to enter filenames & options, BUT it is better to enter them on the command line, so you can use your command line history to easily repeat & modify prior commands.
uvcopy uvcmp1,fili1=dat1/warmas1,fili2=dat1/warmas1a,uop=r64a34b2c44d6 ======================================================================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_09:40:57 uop=q1p30r256s8t500000u3x2y0r64a34b2c44d6
recsize reccount file-size typ Report=rptcmp/warmas1
1: 64 8 0 RSF File1=dat1/warmas1
2: 64 8 0 RSF File2=dat1/warmas1a
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish washer 960101 2-year 980101 000004922 .
3333323333326676276766722222223333332327667233333323333333332220
22222000000049380713852000000096010102D951209801010000004922000A
*******
2 2 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
3333323333326676245454452222223333332327667233333323333333332220
22222000000049380321385200000096010102D951209801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961231 3-year 991231 000006733 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096123103D951209912310000006733000A
* ****
2 3 33333 11111 refrigerator 961215 4-year 001215 000005500 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096121504D951200012150000005500000A
* ****
1 6 0 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
3333323333326667667267767222223333332327667233333323333333332220
5555501111103CF48530429520000099123105D951200412310000055566000A
******** ** * *****
2 6 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
3333323333326667667254544445223333332327667233333323333333332220
5555501111103CF485303829EB520099061506D951200512010000066699000A
******** ** * *****
==================== EOF or StopPrint/StopRead count reached ============== File1Total=8, File2Total=8, StopPrint=8, StopRead=500000 File1Reads=8, TotalMisMatches=3, MisMatchesPrinted=3
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Compare demo#2 report above on page '1E2' (with exclude field options) to the prior demo#1 report on page '1D2' (without excludes) & note the following:
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmpFA1 |
|
| uvcmpFE1 |
|
| uvcmpRDW1 |
|
| uvcmpIDX81 |
|
| uvcmpFAI1 |
|
| uvcmpFEI1 |
|
We will demo script 'uvcmpFE1' which makes it easier to run the 'uvcmp1' uvcopy utility. As well as illustrating the script commands, we will also demonstrate comparing EBCDIC files (vs the prior ASCII files).
| dat1/warmas1 |
|
| dat1/warmas1a |
|
| dat1/warmas1E |
|
| dat1/warmas1Ea |
|
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcopy uvcmp1,fili1=dat1/warmas1E,fili2=dat1/warmas1Ea,uop=r64y1 ================================================================ - uvcopy command to execute uvcmp1 - using option 'y1' to translate character lines to ASCII
uvcmpFE1 dat1/warmas1E dat1/warmas1Ea r64y1 <-- script to execute uvcmp1 ===========================================
| Note |
|
uvcmpFE1 <-- calling script without filenames & options ======== - causes the 'usage' display shown below:
usage: uvcmpFE1 file1 file2 recsize
============================
example1: uvcmpFE1 dat1/warmas1E dat1/warmas1Ea r64
=========================================
ex2: uvcmpFE1 dat1/warmas1E dat1/warmas1Ea r64a34b2c44d6y1
=====================================================
uvcmpFE1 dat1/warmas1E dat1/warmas1Ea r64 <-- script to call uvcmp1 ========================================= - uvcmpFE1 for EBCDIC files
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_09:43:20 uop=q1p30r256s8t500000u3x2y0q1y1r64
recsize reccount file-size typ Report=rptcmp/warmas1E
1: 64 8 0 RSF File1=dat1/warmas1E
2: 64 8 0 RSF File2=dat1/warmas1Ea
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish washer 960101 2-year 980101 000004922 .
FFFFF4FFFFF488A84A8A8894444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
222220000000492806128590000000960101020851909801010000004922000A
*******
2 2 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
FFFFF4FFFFF488A84CDCECCD444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
222220000000492803912859000000960101020851909801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961231 3-year 991231 000006733 .
FFFFF4FFFFF4988988898A99444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
333330111110956997591369000000961231030851909912310000006733000A
** * ** ** ****
2 3 33333 11111 refrigerator 961215 4-year 001215 000005500 .
FFFFF4FFFFF4988988898A99444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
333330111110956997591369000000961215040851900012150000005500000A
** * ** ** ****
1 6 0 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
FFFFF4FFFFF4899A88A489A8944444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
555550111110336385204985900000991231050851900412310000055566000A
******** **** * * * *****
2 6 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
FFFFF4FFFFF4899A88A4ECDCDDCD44FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
555550111110336385202899525900990615060851900512010000066699000A
******** **** * * * *****
1 7 0 66666 00000 micro wave 001231 6-year 061231 000066677 .
FFFFF4FFFFF4988994A8A844444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
666660000000493960615500000000001231060851900612310000066677000A
** ******
2 7 66666 00000 micro wave 001215 6-year 999999 000066677 .
FFFFF4FFFFF4988994A8A844444444FFFFFF4F6A8894FFFFFF4FFFFFFFFF4440
666660000000493960615500000000001215060851909999990000066677000A
** ******
==================== EOF or StopPrint/StopRead count reached ============== File1Total=8, File2Total=8, StopPrint=8, StopRead=500000 File1Reads=8, TotalMisMatches=4, MisMatchesPrinted=4
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
For demo#4, we will use demo file dat1/custmas1 & ask you to copy & create a 2nd mismatched file by using the 'uvhd' utility (which you will find invaluable for investigating files with packed/binary date).
2. uvhd tmp/custmas1 r256u - enter uvhd to update 256 byte records
======================= - option 'u' to allow update
- uvhd displays 1st record & prompts for command
- we will then enter '6' to goto record #6
r# 1 1 2 3 4 5 6
b# 0 0123456789012345678901234567890123456789012345678901234567890123
0 130140 EVERGREEN MOTORS LTD. 1815 BOWEN ROAD NANA
3333332222454545444244545524542222233332445442544422222222224444
130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
64 IMO BC V9S1H1 604-754-5531 ........
4442222222222442535343222233323332333322222222222222222200000000
9DF00000000002306931810000604D754D55310000000000000000000000C000
128 .........W0....`........)X}..f3.....\.......................f...
0000000005300016000000002570063100095000000000000000000000016000
0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0000C0056C00
192 ...............................f.....<........f.C
0000008900000000880000000018000680001300000000694222222222222222
00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3000000000000000
--> 6 <-- enter '6' to goto record #6
r# 6 1 2 3 4 5 6
b# 1280 0123456789012345678901234567890123456789012345678901234567890123
0 147615 X O'CONNER R.V. CENTRE 44430 YALE ROAD WEST CHIL
3333332252424444452525224445542222233333254442544425455222224444
1476150080F73FEE5202E6E035E4250000044430091C502F140753400000389C
64 LIWACK BC V2P6J1 604-858-4611BONNIE o"conner ........
4454442222222442535343222233323332333344444426266666722200000000
C9713B00000002306206A10000604D858D46112FEE950F23FEE520000000C000
128 ...4Vx........BT|...W.............. .........GO.................
0013579000000045700059000000000000920000000004400000000000000000
0C2468C0000C0024C0017C0000C0000C0380C0000C0057F0000C0000C0000C00
192 ...........0.....\.........\........`...........
0000000000830000150000000995000000086000000000002222222222222222
00C0000C0220C0058C0000C0214C0000C0210C0000C0000C0000000000000000
Enter the following 'update' commands to make changes for uvcmp1 demo#4 (we have already shown the changes in the display above).
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcopy uvcmp1,fili1=dat1/custmas1,fili2=tmp/custmas1,uop=r256 =========================================================== - use the uvcopy job directly (longer command)
uvcmpFA1 dat1/custmas1 tmp/custmas1 r256 <-- OR use the script (easier) ========================================
uvcmpFA1 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
2008/05/24_16:50:28 uop=q1p30r256s8t500000v2y0q1r256
recsize reccount file-size typ Report=rptcmp/custmas1
1: 256 32 0 RSF File1=dat1/custmas1
2: 256 32 0 RSF File2=tmp/custmas1
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 6 0 147615 X O'CONNER R.V. CENTRE 44430 YALE ROAD WEST CHIL
3333332252424444452525224445542222233333254442544425455222224444
1476150080F73FEE5202E6E035E4250000044430091C502F140753400000389C
64 LIWACK BC V2P6J1 604-858-4161BONNIE ........
4454442222222442535343222233323332333344444422222222222200000000
C9713B00000002306206A10000604D858D41612FEE950000000000000000C000
** ********
128 ..............BT|...W.............. .........GL.................
0000000000000045700059000000000000920000000004400000000000000000
0C0000C0000C0024C0017C0000C0000C0380C0000C0057C0000C0000C0000C00
***** *
2 6 147615 X O'CONNER R.V. CENTRE 44430 YALE ROAD WEST CHIL
3333332252424444452525224445542222233333254442544425455222224444
1476150080F73FEE5202E6E035E4250000044430091C502F140753400000389C
64 LIWACK BC V2P6J1 604-858-4611BONNIE o"conner ........
4454442222222442535343222233323332333344444426266666722200000000
C9713B00000002306206A10000604D858D46112FEE950F23FEE520000000C000
** ********
128 ...4Vx........BT|...W.............. .........GO.................
0013579000000045700059000000000000920000000004400000000000000000
0C2468C0000C0024C0017C0000C0000C0380C0000C0057F0000C0000C0000C00
***** *
==================== EOF or StopPrint/StopRead count reached ============== File1Total=32, File2Total=32, StopPrint=8, StopRead=500000 File1Reads=32, TotalMisMatches=1, MisMatchesPrinted=1
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Here is a good illustration of the usefulness of the 'uvcmp1all' script:
When we convert mainframe EBCDIC files to ASCII, we can verify the conversions by converting back to EBCDIC & comparing the re-converted EBCDIC files to the original mainframe files.
You might also need the ASCII to EBCDIC conversions to return some files to the mainframe if you can not switch all applications over to unix/linux at the same time.
cpys -------> maps --------> pfx1 --------> pfx2 ---------> pfx3 -------> pfx4
cobmap1 uvdata51 uvdata52 copy/edit uvdata55
Main -----> d1ebc ----------> d2asc ----------> d3ebc --------> rptcmp frame FTP uvcopyxx 'pfx3/*' uvcopyxx 'pfx4/*' **uvcmp1all** mismatch rpts
#1. mkdir pfx4 <-- make subdir for ASCII-->EBCDIC jobs
========== (if not already existing)
#2. uvcopyx uvdata55 pfx3 pfx4 uop=q0i7 <-- generate ALL jobs
===================================
#3. uvcopyxx 'pfx4/*' <-- execute ALL jobs
=================
#4. mkdir rptcmp <-- make subdir for mismatch reprots
============
#5. uvcmp1all d1ebc d3ebc <-- compare all original files to reconverted files
=====================
#6a. vi rptcmp/* <-- inspect all mismatch reports
===========
#6b. uvlpd12 rptcmp <-- print all mismatch reports
==============
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmp1 |
|
| 1X1. | uvcmpFA1 - compare 2 Fixed length ASCII Sequential files |
| 1X2. | uvcmpFE1 - compare 2 Fixed length EBCDIC Sequential files |
| 1X3. | uvcmp1all - script to compare ALL files in 2 directories, for example: |
| - comparing re-converted EBCDIC fiels to original mainframe files | |
| to verify conversions |
The scripts will be listed below since they are only 1 or 2 pages each.
The uvcopy job (uvcmp1) will not be listed because it is much longer, but you can view it or print it as follows:
#1. cd /home/uvadm
==============
#2. vi pf/util/uvcmp1
=================
#3. uvlp12 pf/util/uvcmp1
=====================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmpFA1 - script to compare 2 Fixed length ASCII data files
# - may have packed & binary fields
# - by Owen Townsend, October 2007, May 2008
#Feb18/09 - rename uvcmp2->5, uvcmp3->6, RDW->uvcmp2, IDX->uvcmp3
#
# ** scripts not allowing for inserts/deletes **
#
#*uvcmpFA1 (this script) - compare 2 fixed length ASCII data files
# uvcmpFE1 - compare 2 fixed length EBCDIC data files
# uvcmpRDW1 - compare RDW variable length files - ASCII or EBCDIC (option y1)
# uvcmpIDX81 - compare Micro Focus COBOL IDXFORMAT3/8 Variable length files
#
# ** scripts allowing for INSERTS/DELETES **
#
# uvcmpFAI2 - FixLth ASCII w inserted recs by loading Index files typ=ISF
# uvcmpFEI2 - FixLth EBCDIC w inserted recs by loading Index files typ=ISF
#
# - the 2 datafiles may have same names if in different subdirs
# - run these scripts from a working directory with subdirs holding datafiles
# - will make subdir rptcmp for report (if not already existing)
# - output report will be created in subdir: rptcmp/filename1
#
export JOBID2=uvcmpFA1 # so uvcopy uvcmp1 can show script name on report
if [[ -f "$1" && -f "$2" && -n "$3" ]]; then :
else echo "usage: uvcmpFA1 file1 file2 options/recsize"
echo " ===================================="
echo "example1: uvcmpFA1 dat1/warmas1 dat1/warmas1a r64"
echo " ======================================="
echo "compare 2 Fixed length ASCII data files"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 option alphas+numerics, example r64a34b2c44d6"
echo " - Record-size optn 'r' must be specified"
echo " "
echo "ex2: uvcmpFE1 dat1/warmas1 dat1/warmas1a r64a34b2c44d6"
echo " ================================================="
echo " - ex#2 demos exclude field options (displacements & lengths)"
echo " - a34b2 excludes 34-35, c44d6 excludes 44-49"
echo " - see all options described in /home/uvadm/pf/util/uvcmp1"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) must be spcfd"
exit 99; fi
#
df1="$1"; df2="$2"; optns="$3";
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uvcopy uvcmp1,fili1=$df1,fili2=$df2,fild3=rptcmp,uop=q1$optns
#============================================================
# - uvcmp1 creates comparison report in rptcmp/filename1
# and prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmpFE1 - script to compare 2 Fixed length EBCDIC data files
# - files may have packed & binary fields
# - by Owen Townsend, October 2007, May 2008
#Feb18/09 - rename uvcmp2->5, uvcmp3->6, RDW->uvcmp2, IDX->uvcmp3
#
# ** scripts not allowing for inserts/deletes **
#
# uvcmpFA1 - compare 2 fixed length ASCII data files
#*uvcmpFE1 (this script) - compare 2 fixed length EBCDIC data files
# uvcmpRDW1 - compare RDW variable length files - ASCII or EBCDIC (option y1)
# uvcmpIDX81 - compare Micro Focus COBOL IDXFORMAT3/8 Variable length files
#
# ** scripts allowing for INSERTS/DELETES **
#
# uvcmpFAI2 - FixLth ASCII w inserted recs by loading Index files typ=ISF
# uvcmpFEI2 - FixLth EBCDIC w inserted recs by loading Index files typ=ISF
#
# - the 2 datafiles may have same names if in different subdirs
# - run these scripts from a working directory with subdirs holding datafiles
# - will make subdir rptcmp for report (if not already existing)
# - output report will be created in subdir: rptcmp/filename1
#
export JOBID2=uvcmpFE1 # so uvcopy uvcmp1 can show script name on report
echo "uvcmpFE1 - compare Fixed lth Ebcdic files (Nov28/07)"
if [[ -f "$1" && -f "$2" && -n "$3" ]]; then :
else echo "usage: uvcmpFE1 file1 file2 recsize"
echo " ============================"
echo "example1: uvcmpFE1 dat1/warmas1 dat1/warmas1a r64"
echo " ======================================="
echo "compare 2 Fixed length EBCDIC data files"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 option alphas+numerics, example r64a34b2c44d6y1"
echo " - Record-size optn 'r' must be specified"
echo " "
echo "ex2: uvcmpFE1 dat1/warmas1 dat1/warmas1a r64a34b2c44d6y1"
echo " ==================================================="
echo " - ex#2 demos exclude field options (displacements & lengths)"
echo " - a34b2 excludes 34-35, c44d6 excludes 44-49, y1 EBCDIC:ASCII"
echo " - see all options described in /home/uvadm/pf/util/uvcmp1"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) must be spcfd"
exit 99; fi
#
df1="$1"; df2="$2"; optns="$3";
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uvcopy uvcmp1,fili1=$df1,fili2=$df2,fild3=rptcmp,uop=q1y1$optns
#==============================================================
# - uvcmp1 creates comparison report in rptcmp/filename1
# and prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmp1all - script to compare all files in 2 directories
# - by Owen Towsnend, Feb 2010 at Everis
# calls 'uvcmp1' for fixed length files w/o LineFeeds
# - shows record pairs in vertical hexadecimal
# with mismatches flagged with '*'s
# uses Indexed control file to get record size (ctl/ctlfile4I)
# - same control file as used for EBCDIC to ASCII conversion
#
echo "uvcmp1all - compare all files in 2 directories, using uvcmp1"
echo "- to show record pairs in vertical hexadecimal with '*' mismatches"
echo "- for files with packed/binary & no LineFeeds (use diff for text files)"
d1="$1"; d2="$2";
if [[ -d "$1" && -d "$2" ]]; then :
else echo "usage: uvcmp1all dir1 dir2 [options]"
echo " ============================="
echo "ex1: uvcmp1all d2asc.old d2asc"
echo " ========================="
echo "ex2: uvcmp1all d1ebc d3ebc y1"
echo " ========================"
echo "- option y1 for EBCDIC not req'd if dirname contains *ebc*"
exit 99; fi
echo "uvcmp1 $d1/... $d2/..."
echo "- will write mismatch reports to subdir rptcmp/..."
reply=x;
until [[ "$reply" = "y" || "$reply" = "n" ]]
do echo "- clear all old mismatch reports from rptcmp/... y/n ?"
read reply; done
if [[ "$reply" == "y" ]]; then rm -f rptcmp/*; echo "rptcmp/* removed"; fi
x=0;y=0;
for d1f in $d1/*
{ let x=x+1
f=$(basename $d1f)
if [[ $f == *.idx ]]; then continue; fi
if [[ ! -f $d2/$f ]]; then
echo "bypassing nomatch in $d2 for $d1/$f"
continue; fi
#
# use uvcopy job to get recsize & key options from control file
# - will write to: /tmp/${LOGNAME}.uvcmp
# remove any '.dat' (not coded in control file)
fx=${f%.dat}
uvcopy uvcmpget2,filr1=${CNVDATA}/ctl/ctlfile4I\
,filo1=/tmp/${LOGNAME}.uvcmp,arg1=$fx
if (( $? != 0))
then echo "datafilename not in control file"; exit 2;fi
#
# open the file written by uvcopy & retrieve options into $variables
exec 3< /tmp/${LOGNAME}.uvcmp # open file written by uvcopy
read -u3 copyname ctlops other # read file contents into variables
exec 3<&- # close file
#
# if dirname contains 'ebc' add option y1 to trnslt chars to ASCII
if [[ $d1 == *ebc* ]]; then ctlops=${ctlops}y1; fi
#
echo "file# $x $d1/$f $d2/$f"
uvcopy uvcmp1,fili1=$d1/$f,fili2=$d2/$f,rop=r0,uop=q0i7$ctlops
#=============================================================
if (( $? != 0)); then ((y+=1)); fi
}
echo "$y mismatches of $x total files compared in $d1 & $d2"
echo "vi rptcmp/* <-- view all mismatch reports"
echo "==========="
echo "uvlpd12 rptcmp <-- print all mismatch reports"
echo "=============="
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 2A1. | Comparing Variable Length RDW files - Overview |
| 'uvcmp2' to compare Variable Length RDW files | |
| vs 'uvcmp1' for FIXED Length files (see Part_1) | |
| 'uvcmpRDW1' script easier way to run uvcmp2 (shorter commands) | |
| Variable Length file types (RDW, BDW/RDW, BDW/RDW unblocked, Micro Focus) |
| 2B1. | Setup Directories to demo uvcmp2 |
| - may setup 'cmpdata' directory in your homedir | |
| - make subdirs & copy test/demo files from /home/uvadm/dat1/... |
| 2C1. | testRDWe & testRDWea - test files supplied for VarLth comparison demo |
| - display with 'uvhd', since binary RDW conflicts with 'vi' |
| 2D1. | Op. Instrns to compare RDW files using uvcopy job 'uvcmp2' directly |
| 2D2. | console log |
| 2D3. | sample report |
| 2E1. | Op. Instrns to compare RDW files using script 'uvcmpRDW1' |
| 2E2. | console log |
| 2E3. | sample report |
| 2F1. | creating table summary stats of record sizes in variable length files |
| uvcopy job 'RDWstat1' Operating Instructions & sample report |
| 2X1. | Summary of comparison jobs & scripts documented in this Part. |
| - uvcopy jobs uvcmp2, RDWstat1 | |
| - scripts uvcmpRDW1 |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
We will document 2 ways to compare variable length RDW files
Before documenting the comparison procedures, we will illustrate the RDW variable length file format on the next few pages.
| 2B1. | testRDWe - sample VARIABLE length data file in RDW format. |
| 2B2. | testRDWea - Alternate test file for comparison |
| - with some intentional mismatches |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
/home/userxx <-- your home directory :-----... :-----cmpdata <-- working directory for datafile compares : : : :-----dat1 <-- subdir to store test files : : :-----testRDWe <-- sample RDW file for comparison : : :-----testRDWea <-- 'a'lternate file with some mismatches : : : :-----rptcmp <-- subdir to receive comparison reports : : :-----testRDWe <-- comparison report : : - name same as file#1 : : : :-----tmp <-- subdir required by uvhd for any Printing : : :-----testRDWe_090217_124800P <-- Print file created by uvhd
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#1. login to your homedir & mkdir cmpdata (if not already existing)
#2. cd cmpdata <-- change into comparison directory
==========
#3. mkdir dat1 rptcmp tmp <-- make subdirs required
=====================
#4a. cp /home/uvadm/dat1/testRDWe dat1
=================================
- copy 1st EBCDIC varlth file to be compared to dat1/...
#4b. cp /home/uvadm/dat1/testRDWea dat1
==================================
- copy 2nd EBCDIC varlth file to be compared to dat1/...
#5. uvhd dat1/testRDWe z4a
======================
Investigate 1st variable lth file with options:
z4 - use binary recsize in 1st 2 bytes of record
a - translate uvhd character line from EBCDIC to ASCII
#5a. --> p8 <-- Print 8 records (all)
- see printout below
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvhd dat1/testRDWe z4a1 <-- option z4 = variable lth binary 1st 2 bytes
======================= - option a1 = translate char line to Ascii
- zones & digits lines show EBCDIC codes
10 20 30 40 50 60
r# 1 0123456789012345678901234567890123456789012345678901234567890123
0 ....DELL10 - Dell Inc.
0100CCDDFF464C8994C98444
080045331000045330953B00
r# 2
24 ....HP0010 - Hewlett Packard
0100CDFFFF464C8A98AA4D889898
0C00870010000856353307132194
r# 3
52 ....IBM010 - International Business Machines
0200CCDFFF464C9A8998A899894CAA898AA4D888898A
0C009240100009535951396513024295522041389552
r# 4
96 ....MFC010 - Micro Focus COBOL
0200DCCFFF464D88994C98AA4CDCDD44
00004630100004939606634203626300
r# 5
128 ....MS0010 - Microsoft Corp.
0100DEFFFF464D8899A98A4C9994
0C0042001000049396266303697B
r# 6
156 ....REDH10 - Red Hat Linux
0100DCCCFF464D884C8A4D89AA44
0C00954810000954081303954700
r# 7
184 ....SUN010 - Sun Microsystems Ltd
0200EEDFFF464EA94D8899AAAA89A4DA8444
040024501000024504939628235420334000
r# 8
220 ....UVSI10 - UV Software Inc.
0200EEECFF464EE4E98AA8984C984444
0000452910000450266361950953B000
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvhd dat1/testRDWea z4a1 <-- option z4 = variable lth binary 1st 2 bytes
======================== - option a1 = translate char line to Ascii
- zones & digits lines show EBCDIC codes
10 20 30 40 50 60
r# 1 0123456789012345678901234567890123456789012345678901234567890123
0 ....DELL10 - Dell Inc.
0100CCDDFF464C8994C98444
080045331000045330953B00
r# 2
24 ....HP0010 - Hewlett Packard
0100CDFFFF464C8A98AA4D889898
0C00870010000856353307132194
r# 3
52 ....IBM010 - International Business Mothers
0200CCDFFF464C9A8998A899894CAA898AA4D9A889A4
0C009240100009535951396513024295522046385920
r# 4
96 ....MFC010 - Micro Focus COBOL
0200DCCFFF464D88994C98AA4CDCDD44
00004630100004939606634203626300
r# 5
128 ....MS0010 - Microsoft Corp.
0100DEFFFF464D8899A98A4C9994
0C0042001000049396266303697B
r# 6
156 ....REDH10 - Red Hat Linux
0100DCCCFF464D884C8A4D89AA44
0C00954810000954081303954700
r# 7
184 ....SUN010 - Sun Microsystems Corp
0200EEDFFF464EA94D8899AAAA89A4C99944
040024501000024504939628235420369700
r# 8
220 ....UVSI10 - UV Software Inc.
0200EEECFF464EE4E98AA8984C984444
0000452910000450266361950953B000
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Assuming the 2 EBCDIC RDW files have already been copied to subdir dat1/... in your homedir (as shown on page '2B1').
#1. uvcopy uvcmp2,fili1=dat2/testRDWe,fili2=dat2/testRDWea,uop=y1z4
===============================================================
- run comparison utility
option y1 - translate char line to ASCII (zone & digit lines EBCDIC)
option z4 - RDW file with 4 byte recsize prefixes (vs 2 bytes)
- see the console log displayed on the next page --->
#2. uvlp12 rptcmp/testRDWe
======================
- print output report
#1a. uvcmpRDW1 dat1/testRDWe dat1/testRDWea z4y1 <-- execute alternative
=========================================== (shorter command)
- option 'z4' (4 byte prefixes) is default & may be omitted
- option 'y1' translates character line to ASCII (for EBCDIC files)
| Note |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcopy uvcmp2,fili1=dat2/testRDWe,fili2=dat2/testRDWea,uop=y1z4 ===============================================================
uvcmp2 - compare files & print diffs in hex with "*" markers (Nov28/07)
uop=q1a0b0c0d0e0f0g0h0p30r8192s8t500000u3x2y0 - option defaults
a# - dsplcmnt to exclude field #1
b# - length to exclude field #1
c# - dsplcmnt to exclude field #2
d# - length to exclude field #2
e# - dsplcmnt to exclude field #3
f# - length to exclude field #3
g# - dsplcmnt to exclude field #4
h# - length to exclude field #4
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r8192- record size (largest RDW recsize, default 8192)
s8 - StopPrint count, after s# mismatched records
t500000 - StopRead count, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
z2 - RDWz2 2 byte RDW recsize prefixes (vs RDWz4 dflt)
090217:131431:uvcmp2: uvcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmp2
uvcopy DISAM ext=dat LNX L64 license=20090212V site=UV_Software
User OPtion (uop) defaults = q1p30r8192s8t500000u3x2y0y1
-->null to accept or enter/override -->
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=0, File2Total=0, StopPrint=8, StopRead=500000
File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2
090217:131431:uvcmp2: EOF fili01 rds=8 size=252: dat1/testRDWe 090217:131431:uvcmp2: EOF fili02 rds=8 size=252: dat1/testRDWea 090217:131431:uvcmp2: EOF fild03 size=4096: rptcmp 090217:131431:uvcmp2: EOF filo03 wrts=32 size=1496: rptcmp/testRDWe hits=2 EOJ, Output File written to: rptcmp/testRDWe default command = null, OR enter: more,print,uvlpr12,edit,null -->
See differences report listed on the next page --->
| Note |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
${JOBID2} uvcmp2 - compare RDW files, print mismatched records, '*' flag diffs
2009/02/17_13:14:31 uop=q1p30r8192s8t500000u3x2y0y1
recsize reccount file-size typ Report=rptcmp/testRDWe
1: 8192 0* 0 RDW File1=dat1/testRDWe
2: 8192 0* 0 RDW File2=dat1/testRDWea
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 3 0 IBM010 - International Business Machines
40 CCDFFF464C9A8998A899894CAA898AA4D888898A
9240100009535951396513024295522041389552
** ****
2 3 IBM010 - International Business Mothers
CCDFFF464C9A8998A899894CAA898AA4D9A889A4
9240100009535951396513024295522046385920
** ****
1 7 0 SUN010 - Sun Microsystems Ltd
32 EEDFFF464EA94D8899AAAA89A4DA8444
24501000024504939628235420334000
****
2 7 SUN010 - Sun Microsystems Corp
EEDFFF464EA94D8899AAAA89A4C99944
24501000024504939628235420369700
****
==================== EOF or StopPrint/StopRead count reached ============== File1Total=0, File2Total=0, StopPrint=8, StopRead=500000 File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Script 'uvcmpRDW1' makes it easier to compare variable length RDW files because:
#1. uvcmpRDW1 dat1/testRDWe dat1/testRDWea z4y1 <-- execute script
===========================================
- option 'z4' (4 byte prefixes) is default & may be omitted
- option 'y1' translates character line to ASCII (for EBCDIC files)
#2a. vi rptcmp/testRDWe <-- view report
==================
#2b. uvlp12 rptcmp/testRDWe <-- print report
======================
| Note |
|
Also see 'uvcmpRDWAI2/uvcmpRDWEI2' on pages '4H1' - 4H4, to compare variable length RDW files allowing for isnerts & deletes.
uvcmpRDWAI2 dat1/vendormas3 dat1/vendormas32 r64k0l6 ==================================================== - compare 2 RDW files, report created in rptcmp/vendormas3
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'RDWstat1' is a uvcopy job that you can run to create a summary table of record-sizes found in all records of a variable file typ=RDW or typ=RDWz2 (variable length records with 4 byte or 2 byte prefixes).
#1. cd $UV <-- change to /home/uvadm
#2. uvcopy RDWstat1,fili1=dat1/testRDW,filo2=tmp/testRDW_recsizes
=============================================================
#3. vi tmp/testRDW_recsizes <-- inspect report
=======================
RDWstat1 2007/12/19_18:10:29 record-sizes in dat1/testRDW
tbl#001 pg#001 -argument-
line# count % record-sizes
1 1 12 00020
2 3 37 00024
3 2 25 00028
4 1 12 00032
5 1 12 00040
8*100 *TOTAL*
#1. cd $CMPDATA <-- change to file compare superdir
#2. mkdir tmp1 <-- make tmp1 subdir if not existing #2a. rm -f tmp1/* <-- OR remove all files if tmp1 already exists
#3. uvcopyx RDWstat1 d0ebc tmp1 uop=q0i7,rop=r0
===========================================
- create table summary recsize counts for all files in d0ebc subdir
- output reports in tmp1 with same names as datafiles in d0ebc
#4. uvlpd12 tmp1 <-- print all reports in tmp subdir
============
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 2X1. | uvcmpRDW1 - compare 2 variable length RDW files |
| - option y1/y0 EBCDIC or ASCII data files | |
| - option z2/z4 for 2 byte or 4 byte RDW recsize prefixes | |
| - listed on the next page. |
| uvcmp2 |
|
| 2X2. | RDWstat1 - create table summary of record sizes in variable length files |
| - listed 2 pages below |
uvcopy job 'uvcmp2' is not listed because it is too long, but you can view or print it as follows:
#1. cd /home/uvadm
==============
#2. vi pf/util/uvcmp2
=================
#3. uvlp12 pf/util/uvcmp2
=====================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmpRDW1 - compare 2 Variable length EBCDIC files with 4 byte record headers
# - RDW big-end binary recsize in 1st 2 bytes, next 2 bytes x'0000'
# - by Owen Townsend, February 2009
# - 1 of several file comparison scripts for different file types
#Feb18/09 - rename uvcmp2->5, uvcmp3->6, RDW->uvcmp2, IDX->uvcmp3
#
#*uvcmpRDW1 (this script) - compare 2 RDW variable length files
# - either ASCII or EBCDIC (if option 'y1')
# uvcmpFA1 - compare 2 fixed length ASCII data files
# uvcmpFE1 - compare 2 fixed length EBCDIC data files
# uvcmpFAI2 - FixLth ASCII w inserted recs by loading Index files typ=ISF
# uvcmpFEI2 - FixLth EBCDIC w inserted recs by loading Index files typ=ISF
#
# This script (uvcmpRDW1) does not allow for inserted or deleted records
# - since RDW files are used mostly for file transfers from mainframe
# - not for COBOL production files (since COBOL does not process RDW files)
# - we could provide scripts to allow inserts/deletes in RDW files if required
#
# This script is an easier way to use the 'uvcmp2' comparison job for RDW files
# - shorter commands
#
export JOBID2=uvcmpRDW1 # so uvcopy uvcmp2 can show script name on report
if [[ -f "$1" && -f "$2" && -n "$3" ]]; then :
else echo "usage: uvcmpRDW1 file1 file2 [options]"
echo " =============================="
echo "example: uvcmpRDW1 dat1/testRDWe dat1/testRDWea y1z4"
echo " ==========================================="
echo "compare RDW files (variable length recsize in binary prefixes)"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 options (all options displayed by uvcmp2)"
echo " y1 - translate EBCDIC to ASCII (y0=no translate)"
echo " z4 - 4 byte recsize prefixes default (vs z2 2 bytes)"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) must be spcfd"
exit 99; fi
df1="$1"; df2="$2"; optns="$3";
if [[ -z "$optns" ]]; then optns=y0; fi
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uvcopy uvcmp2,fili1=$df1,fili2=$df2,fild3=rptcmp,uop=q1$optns
#============================================================
# - uvcmp2 creates comparison report in rptcmp/filename1
# and prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
# RDWstat1 - create summary table of record-sizes used in variable file # - for variable lth records with 4 byte hdrs (standard RDW) # - by Owen Townsend, Dec 12, 2007 #Dec19/07 - replace getvr2 subrtn with file typ=RDW & RDWz2 (optional) # # ** create report for 1 file (for testing) ** # # uvcopy RDWstat1,fili1=d0ebc/datafilename,filo2=tmp1/reportname # ============================================================== # # ** create reports for all files in directory ** # # 1. cd $CNVDATA <-- change to conversion superdir # - subdir d0ebc contains EBCDIC var lth files # # 2. mkdir tmp1 <-- make tmp1 subdir if not existing # 2a. rm -f tmp1/* <-- OR remove all files if tmp1 already exists # # 3. uvcopyx RDWstat1 d0ebc tmp1 uop=q0i7,rop=r0 # =========================================== # - create table summary recsize counts for all files in d0ebc subdir # - output reports in tmp1 with same names as datafiles in d0ebc # # 4. uvlpd12 tmp1 <-- print all reports in tmp subdir # ============ # # ** sample report ** # # RDWstat1 2006/12/17_18:15:17 record-sizes in d0ebc/E2121656 # tbl#001 pg#001 -argument- # line# count % record-sizes # 1 10,552 16 00046 # 2 4,451 7 00065 # 3 23,347 37 00066 # 4 367 0 00068 # 5 21,010 33 00083 # - - - etc - - - # 18 3 0 00218 # 19 441 0 00233 # 20 813 1 00239 # 62,115*100 *TOTAL* #
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
# This job designed for variable lth records typ=RDW & typ=RDWz2
# typ=RDW (standard RDW) - 4 byte record prefixes
# - binary recsize in 1st 2 bytes (3rd & 4th x'0000' unused)
# - binary recsize of data (includes the 4 byte prefix)
# - blk/rec lth binary BIG-END format, need 's'witch option for INTEL
# typ=RDWz2 (non-std RDW) - 2 byte record prefixes (2 nulls omitted)
# - binary recsize does not include the 2 byte prefix
#
opr='$jobname - create summary table of record-sizes used in variable file'
opr='uop=q1z0 - option defaults (message only, see uop=... in job)'
opr=' z0 - RDW 4 byte prefix, recsize 1st 2 BIG-END binary'
opr=' z2 - RDW 2 byte prefix, recsize 1st 2 BIG-END binary'
uop=q1z0 # option defaults
rop=r1 # option to prompt for report disposition
was=a16384
fili1=?d0ebc/input,rcs=8192,typ=RDW #Dec19/07 typ=RDW replaces getvr
filo1=?tmp1/$jobname.rpt,typ=LSTt,rcs=128
@run
#Dec19/07 - replace subrtn getvr with typ=RDW & RDWz2
# - for nonstd 2 byte prefix vs std 4 bytes
# if uop=z2, append file option z2 for typ=RDWz2
tsb o26(1),x'02' uop z2 for nonstd RDW ?
skp! 1
cata8 $fili1+180(15),'z2'
opn all
#
# begin loop to get records & build summary table of recsizes
# - to be dumped at end of file
man20 get fili1,a0 get next RDW record data
skp> eof (cc set > at EOF)
mvn c0(5),$rv cnvrt binary recsize to numeric
tblt1f1 c0(5),'record-sizes' build table in memory
skp man20 repeat get loop til EOF
#
# EOF - close files & end job
eof mvfv1 f0(80),'record-sizes in $fili1'
tbpt1s1 filo1,f0(50) dump table from memory to file
cls all
eoj
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 3A1. | Comparing Variable Length IDXFORMAT3/IDXFORMAT8 files - Overview |
| 'uvcmp3' to compare Variable Length IDXFORMAT files | |
| vs 'uvcmp1' for FIXED Length files (see Part_1) | |
| vs 'uvcmp2' for Variable Length RDW files (see Part_2) | |
| 'uvcmpIDX81' script easier way to run uvcmp3 (shorter commands) | |
| Variable Length file types (IDXFORMAT vs RDW) |
| 3B1. | Setup Directories to demo uvcmp3 |
| - may setup 'cmpdata' directory in your homedir | |
| - make subdirs & copy test/demo files from /home/uvadm/dat1/... |
| 3C1. | testIDX81 & testIDX81a - test files supplied for VarLth comparison demo |
| - display with 'uvhd', since you cannot use 'vi' for IDXFORMAT files |
| 3D1. | Op. Instrns to compare IDX files using uxcopy job 'uvcmp3' directly |
| 3D2. | console log |
| 3D3. | sample report |
| 3E1. | Op. Instrns to compare IDX files using script 'uvcmpIDX81' |
| 3E2. | console log |
| 3E3. | sample report |
| 3F1. | creating table summary stats of record sizes in variable length files |
| uxcopy job 'IDXstat1' Operating Instructions & sample report |
| 3X1. | Summary of comparison jobs & scripts documented in this Part. |
| - uxcopy jobs uvcmp3, IDXstat1 | |
| - scripts uvcmpIDX81 |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
We will document 2 ways to compare variable length IDX files
Before documenting the comparison procedures, we will illustrate the IDX variable length file format on the next few pages.
| 3B1. | testIDX81 - sample VARIABLE length data file in IDX format. |
| 3B2. | testIDX81a - Alternate test file for comparison |
| - with some intentional mismatches |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
/home/userxx <-- your home directory :-----... :-----cmpdata <-- working directory for datafile compares : : : :-----dat1 <-- subdir to store test files : : :-----testIDX81 <-- sample IDX file for comparison : : :-----testIDX81a <-- 'a'lternate file with some mismatches : : : :-----rptcmp <-- subdir to receive comparison reports : : :-----testIDX81 <-- comparison report : : - name same as file#1 : : : :-----tmp <-- subdir required by uvhd for any Printing : : :-----testIDX81_090217_124800P <-- Print file created by uvhd
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#1. login to your homedir & mkdir cmpdata (if not already existing)
#2. cd cmpdata <-- change into comparison directory
==========
#3. mkdir dat1 rptcmp tmp <-- make subdirs required
=====================
#4a. cp /home/uvadm/dat1/testIDX81 dat1
==================================
- copy 1st EBCDIC varlth file to be compared to dat1/...
#4b. cp /home/uvadm/dat1/testIDX81a dat1
===================================
- copy 2nd EBCDIC varlth file to be compared to dat1/...
#5. uvhd dat1/testIDX81 v
=====================
Investigate 1st variable lth file with options 'v'
- for IDXFORMAT3/IDXFORMAT8 files
#5a. --> p99 <-- Print All records (we know < 99)
- see printout below
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvhd dat1/testIDX81 v <-- option v = variable lth IDXFORMAT
===================== - option a1 = translate char line to Ascii
- zones & digits lines show EBCDIC codes
10 20 30 40 50 60
r# 1 0123456789012345678901234567890123456789012345678901234567890123
0 0~......0711180743423507111807434235.>...................@......
3700000033333333333333333333333333330300000000000000000004000100
0E00000007111807434235071118074342350E01000000001000000000000000
64 ............................................X"8*................
0000000000000000000000000000000000000000000052320000000000000000
00000000000000100000000000000000000000000000828A0000000000000000
10 20 30 40 50 60
r# 2 0123456789012345678901234567890123456789012345678901234567890123
128 @.DELL00 - Dell Inc.
41444433222466624662
0245CC000D045CC09E3E
r# 3
148 @.HP0000 - Hewlett Packard..
4145333322246766772566667600
088000000D0857C5440013B12400
r# 4
176 @(IBM000 - International Business Machines..
42444333222467676676666624776667724666666700
0892D0000D09E452E149FE1C02539E5330D1389E5300
r# 5
220 @.MFC000 - Micro Focus COBOL
4144433322246676246677244444
0AD630000D0D932F06F35303F2FC
r# 6
248 @.MS0000 - Microsoft Corp...
4145333322246676766724677200
08D300000D0D932F3F6403F20E00
r# 7
276 @.REDHAT - Red Hat Linux
415444452225662467246677
062548140D025408140C9E58
r# 8
300 @.SUN000 - Sun Microsystems Ltd.
41554333222576246676777766724760
0D35E0000D035E0D932F39345D30C440
r# 9
332 @.UVSI00 - UV Software Inc..
4155543322255256677676246620
095639000D05603F64712509E3E0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvhd dat1/testIDX81a v <-- option v = variable lth IDXFORMAT3/8 files ======================
10 20 30 40 50 60
r# 1 0123456789012345678901234567890123456789012345678901234567890123
0 0~......0711180747307707111807473077.>...................@......
3700000033333333333333333333333333330300000000000000000004000100
0E00000007111807473077071118074730770E01000000001000000000000000
64 ............................................X"8*................
0000000000000000000000000000000000000000000052320000000000000000
00000000000000100000000000000000000000000000828A0000000000000000
10 20 30 40 50 60
r# 2 0123456789012345678901234567890123456789012345678901234567890123
128 @.DELL00 - Dell Inc.
41444433222466624662
0245CC000D045CC09E3E
r# 3
148 @.HP0000 - Hewlett Packard..
4145333322246766772566667600
088000000D0857C5440013B12400
r# 4
176 @'IBM000 - International Business Mothers...
42444333222467676676666624776667724676677000
0792D0000D09E452E149FE1C02539E5330DF48523000
r# 5
220 @.MFC000 - Micro Focus COBOL
4144433322246676246677244444
0AD630000D0D932F06F35303F2FC
r# 6
248 @.MS0000 - Microsoft Corp...
4145333322246676766724677200
08D300000D0D932F3F6403F20E00
r# 7
276 @.REDHAT - Red Hat Linux
415444452225662467246677
062548140D025408140C9E58
r# 8
300 @.SUN000 - Sun Macrosystems Ltd.
41554333222576246676777766724760
0D35E0000D035E0D132F39345D30C440
r# 9
332 @.UVSI00 - UV Software Inc..
4155543322255256677676246620
095639000D05603F64712509E3E0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Assuming the 2 EBCDIC IDX files have already been copied to subdir dat1/... in your homedir (as shown on page '3B1').
#1. uxcopy uvcmp3,fili1=dat2/testIDX81,fili2=dat2/testIDX81a,uop=q1
===============================================================
- run comparison utility
option q1 = display prompts for option changes
- see the console log displayed on the next page --->
#2. uvlp12 rptcmp/testIDX81
======================
- print output report
#1a. uvcmpIDX81 dat1/testIDX81 dat1/testIDX81a q1 <-- execute alternative
============================================ (shorter command)
| Note |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uxcopy uvcmp3,fili1=dat2/testIDX81,fili2=dat2/testIDX81a,uop=q1 ===============================================================
uvcmp3 - compare files & print diffs in hex with "*" markers (Nov28/07)
uop=q1a0b0c0d0e0f0g0h0p30r8192s8t500000u3x2y0 - option defaults
a# - dsplcmnt to exclude field #1
b# - length to exclude field #1
c# - dsplcmnt to exclude field #2
d# - length to exclude field #2
e# - dsplcmnt to exclude field #3
f# - length to exclude field #3
g# - dsplcmnt to exclude field #4
h# - length to exclude field #4
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r8192- record size (largest RDW recsize, default 8192)
s8 - StopPrint count, after s# mismatched records
t500000 - StopRead count, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
090217:151629:uvcmp3: uxcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmp3
uxcopy DISAM ext=dat LNX LL64 license=20090212U site=UV_Software_Inc.
User OPtion (uop) defaults = q1p30r8192s8t500000u3x2y0
-->null to accept or enter/override -->
record pair size mismatch fili1=40, fili2=39, enter=continue
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=0, File2Total=0, StopPrint=8, StopRead=500000
File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2
090217:151630:uvcmp3: EOF fili01 rds=8 size=360: dat1/testIDX81 090217:151630:uvcmp3: EOF fili02 rds=8 size=360: dat1/testIDX81a 090217:151630:uvcmp3: EOF fild03 size=4096: rptcmp 090217:151630:uvcmp3: EOF filo03 wrts=32 size=1454: rptcmp/testIDX81hits=2 EOJ, Output File written to: rptcmp/testIDX81 default command = null, OR enter: more,print,uvlpr12,edit,null -->
See differences report listed on the next page --->
| Note |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpIDX81 uvcmp3 - compare RDW files, print mismatched records, '*' flag diffs
2009/02/17_15:36:51 uop=q1p30r8192s8t500000u3x2y0q1r8192
recsize reccount file-size typ Report=rptcmp/testIDX81
1: 8192 0 0 IDX File1=dat1/testIDX81
2: 8192 0 0 IDX File2=dat1/testIDX81a
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 3 0 IBM000 - International Business Machines
39 4443332224676766766666247766677246666667
92D0000D09E452E149FE1C02539E5330D1389E53
** ****
2 3 IBM000 - International Business Mothers
4443332224676766766666247766677246766772
92D0000D09E452E149FE1C02539E5330DF485230
** ****
1 7 0 SUN000 - Sun Microsystems Ltd
29 55433322257624667677776672476
35E0000D035E0D932F39345D30C44
*
2 7 SUN000 - Sun Macrosystems Ltd
55433322257624667677776672476
35E0000D035E0D132F39345D30C44
*
==================== EOF or StopPrint/StopRead count reached ============== File1Total=0, File2Total=0, StopPrint=8, StopRead=500000 File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Script uvcmpIDX81 makes it easier to compare variable length IDX files because:
#1. uvcmpIDX81 dat1/testIDX81 dat1/testIDX81a r64 <-- execute script
=========================================
#2a. vi rptcmp/testIDX81 <-- view report
==================
#2b. uvlp12 rptcmp/testIDX81 <-- print report
======================
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'IDXstat1' is a uxcopy job that you can run to create a summary table of record-sizes found in all records of a variable file typ=IDX or typ=IDXz2 (variable length records with 4 byte or 2 byte prefixes).
#1. cd $UV <-- change to /home/uvadm
#2. uxcopy IDXstat1,fili1=dat1/testIDX81,filo2=tmp/testIDX81_recsizes
=================================================================
#3. vi tmp/testIDX81_recsizes <-- inspect report
=========================
IDXstat1 2009/02/17_15:29:16 record-sizes in dat1/testIDX81 dat1/testIDX81
tbl#001 pg#001 -argument- -acum#1- %
line# count % record-sizes
1 1 12 00018 1 12
2 1 12 00022 1 12
3 2 25 00024 2 25
4 1 12 00025 1 12
5 1 12 00026 1 12
6 1 12 00029 1 12
7 1 12 00040 1 12
8*100 *TOTAL* 8 *100
#1. cd $CMPDATA <-- change to file compare superdir
#2. mkdir tmp1 <-- make tmp1 subdir if not existing #2a. rm -f tmp1/* <-- OR remove all files if tmp1 already exists
#3. uxcopyx IDXstat1 d0ebc tmp1 uop=q0i7,rop=r0
===========================================
- create table summary recsize counts for all files in d0ebc subdir
- output reports in tmp1 with same names as datafiles in d0ebc
#4. uvlpd12 tmp1 <-- print all reports in tmp subdir
============
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 3X1. | uvcmpIDX81 - compare 2 variable length IDX files |
| - listed on the next page. |
| uvcmp3 |
|
| 3X2. | IDXstat1 - create table summary of record sizes in variable length files |
| - listed 2 pages below |
uxcopy job 'uvcmp3' is not listed because it is too long, but you can view or print it as follows:
#1. cd /home/uvadm
==============
#2. vi pf/util/uvcmp3
=================
#3. uvlp12 pf/util/uvcmp3
=====================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmpIDX81 - compare 2 Variable length IDXFORMAT8 files
# - using 'uvcmp3' for IDXFORMAT files
# (vs 'uvcmp1' for Fiexd lth or 'uvcmp2' for RDW files)
# - by Owen Townsend, Feb 2009
#Feb18/09 - rename uvcmp2->5, uvcmp3->6, RDW->uvcmp2, IDX->uvcmp3
#
# See documentation at www.uvsoftware.ca/cmpjobs.htm#Part_3
# Part 1 documents compares for fixed length files (both ASCII & EBCDIC)
# Part 2 documents compares for RDW variable length files (ASCII & EBCDIC)
# *Part 3 (this script) documents compares for IDXFORMAT8 variable length files
# - no need to provide EBCDIC versions for IDXFORMAT8 (not mainframe files)
# - the 2 datafiles may have same names if in different subdirs
# - run this script from a working directory with subdirs holding datafiles
# - output report will be created in subdir: rptcmp/filename1
#
export JOBID2=uvcmpIDX81 # so uxcopy uvcmp3 can show script name on report
if [[ -f "$1" && -f "$2" ]]; then :
else echo "usage: uvcmpIDX81 file1 file2 [options]"
echo " ================================"
echo "example: uvcmpIDX81 dat1/testIDX81 dat1/testIDX82"
echo " ========================================"
echo "compare Variable length IDXFORMAT8 files"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 options (may also enter at uvcmp3 prompt)"
echo " - see options described in /home/uvadm/pf/util/uvcmp3"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) may be spcfd"
exit 99; fi
#
df1="$1"; df2="$2"; optns="$3";
if [[ -z "$optns" ]]; then optns=r8192; fi
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uxcopy uvcmp3,fili1=$df1,fili2=$df2,fild3=rptcmp,uop=q1$optns
#============================================================
# - uvcmp3 creates comparison report in rptcmp/filename1
# and prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
# IDXstat1 - create summary table of record-sizes used in variable file # - for variable lth records IDXFORMAT3 or IDXFORMAT8 # - Micro Focus COBOL compatible # - must execute with 'uxcopy' (not 'uvcopy') # (uxcopy has IDXFORMAT file handler, uvcopy does not) # - by Owen Townsend, Feb 2009 # # ** create report for 1 file (for testing) ** # # uxcopy IDXstat1,fili1=d0ebc/datafilename,filo2=tmp/reportname # ============================================================= # # ** create reports for all files in directory ** # # 1. cd $CNVDATA <-- change to conversion superdir # - subdir d0ebc contains EBCDIC var lth files # # 2. mkdir tmp <-- make tmp subdir if not existing # 2a. rm -f tmp/* <-- OR remove all files if tmp already exists # # 3. uxcopyx IDXstat1 d0ebc tmp uop=q0i7,rop=r0 # ========================================== # - create table summary recsize counts for all files in d0ebc subdir # - output reports in tmp with same names as datafiles in d0ebc # # 4. uvlpd12 tmp <-- print all reports in tmp subdir # =========== # # ** sample report ** # # IDXstat1 2006/12/17_18:15:17 record-sizes in d0ebc/E2121656 # tbl#001 pg#001 -argument- # line# count % record-sizes # 1 10,552 16 00046 # 2 4,451 7 00065 # 3 23,347 37 00066 # 4 367 0 00068 # 5 21,010 33 00083 # - - - etc - - - # 18 3 0 00218 # 19 441 0 00233 # 20 813 1 00239 # 62,115*100 *TOTAL* #
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
# This job designed for variable lth records typ=IDXf3 or typ=IDXf8
#
opr='$jobname - create summary table of record-sizes used in variable file'
#uop=q1 # option defaults
rop=r1 # option to prompt for report disposition
was=a16384
fili1=?d0ebc/input,rcs=8192,typ=IDXf8
filo1=?tmp/$jobname.rpt,typ=LSTt,rcs=128
@run
opn all
#
# begin loop to get records & build summary table of recsizes
# - to be dumped at end of file
man20 get fili1,a0 get next IDXf3/f8 record data
skp> eof (cc set > at EOF)
mvn c0(5),$rv cnvrt binary recsize to numeric
tblt1f1 c0(5),'record-sizes' build table in memory
skp man20 repeat get loop til EOF
#
# EOF - close files & end job
eof mvfv1 f0(80),'record-sizes in $fili1'
tbpt1s1 filo1,f0(50) dump table from memory to file
cls all
eoj
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 4A1. | 'uvcmpI1' to compare Indexed files with Inserted records |
| - alternatives uvcmp1, uvcmp2, uvcmp3, uvcmp5,& uvcmp6 | |
| 'uvcmpI2' allows for inserts in both files, on 1 report | |
| - scripts to run uvcmpI1 (shorter commands) |
| 4B1. | Setup Directories to demo uvcmpI1 |
| - may setup 'cmpdata' directory in your homedir | |
| with subdirs required for uvcmpI1 & scripts uvcmpFAI1 & uvcmpFEI1 | |
| (dat1, rptcmp, tmp, tmp1, tmp2) |
| 4C1. | test file#1 supplied for uvcmpI1 demo (dat1/warmas1I) |
| test file#2 with differences (dat1/warmas1Ia) |
| 4D1. | uvcopy jobs & scripts used for test/demos |
| uvcopy jobs - loadISF1, uvcmpI1, unmatI2, uvcmpI2 | |
| scripts uvcmpFAI1/uvcmpFEI1 show inserts, but need to rerun with files | |
| reversed to detect inserts in 1st file | |
| scripts uvcmpFAI2/uvcmpFEI2 show inserts in both files on 1 report |
| 4E1. | uvcmpFAI1/uvcmpI1 demo#1 - Compare ASCII files with Inserted records |
| - 1st run detects inserted record in file#1, but not file#2 | |
| 4E2. | - console log & sample report |
| 4F1. | uvcmpFAI1/uvcmpI1 demo#2 - Compare ASCII files with Inserted records |
| - 2nd run detects inserted record in file#2, but not file#1 | |
| - need to rerun with files reversed to detect inserts in 1st file | |
| 4E2. | - console log & sample report |
| 4G1. | uvcmpFAI2/uvcmpI2 demo#3 - Compare files with Inserts/Deletes (on 1 report) |
| - same as uvcmpI1, but allows for inserts in both files, on 1 report | |
| - prior job 'unmatI2' writes unmatched records in file1 to tmp2/... | |
| by reading file2 sequentially & reading file1 randomly by key | |
| - uvcmpI2 then reads 3 files data1, data2,& unmatched in data1 | |
| - options i1/i2 to ignore inserts/deletes OR mismatched pairs | |
| uvcmpFEI2 - for EBCDIC files (vs ASCII files) |
| 4G2. | - console log & sample report |
| 4H1. | uvcmpRDWAI2/uvcmpRDWEI2 - for RDW ASCII/EBCDIC files with inserts/deletes |
| - similar to above uvcmpFAI2/uvcmpFEI2 | |
| 4H2. | sample report from uvcmpRDWAI2 comparing vendormas3 & vendormas32 |
| 4H3. | options for max recsize & key displacment/length |
| 4H4. | unique key required to allow for inserts & deletes |
| 4X1. | Summary of comparison jobs & scripts documented in this Part. |
| uvcopy jobs - loadISF1 & uvcmpI1 - called by scripts | |
| scripts uvcmpFAI1 & uvcmpFEI1 | |
| - scripts listed since they are short |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Part 4 shows you how to compare Fixed length files that may have Inserted or Deleted records. See Part_0 for a summary of all file compare jobs.
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmpFAI1 |
|
| uvcmpFEI1 |
|
| uvcmpFAI2 |
|
| uvcmpFEI2 |
|
| uvcmpRDWAI2 |
|
| uvcmpRDWEI2 |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#1. Login ---> /home/userxx <-- Login or change to your home directory
#2. mkdir cmpdata <-- make superdir for subdirs shown above
=============
#3. cd cmpdata <-- change into the comparison superdir
==========
#4. mkdir dat1 dat2 rptcmp tmp <-- make subdirs required
==========================
#5a. cp /home/uvadm/dat1/warmas1I dat1
=================================
- copy supplied test files to subdirs in your homedir
#5b. cp /home/uvadm/dat1/warmas1Ia dat1
==================================
- can store 2nd demo file in same subdir as 1st since filenames different
(store in dat2/... if filenames same)
#6a. vi dat1/warmas1I <-- view the demo files
================ - see listed on page '4C1'
#6b. vi dat1/warmas1Ia
================
#7. cp /home/uvadm/dat1/vendormas3* dat1
====================================
- copies 4 RDW files for demos on pages '4H1' - 4H4
- ASCII files vendormas3/vendormas32
- EBCDIC files vendormas3e/vendormas3e2
#8a. uvhd /home/uvadm/dat1/vendormas3* dat1 z4
=========================================
- inspect RDW files (requires uvhd due to binary prefixes)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
/home/userxx <-- your home directory :-----... :-----cmpdata <-- working directory for datafile compares : :-----dat1 <-- subdir to store files to be compared : : :-----warmas1I <-- sample files for comparison : : :-----warmas1Ia
: :-----tmp1 <-- tmp1 subdir for Indexed version of file1 : : :-----warmas1I.dat : : :-----warmas1I.idx
: :-----tmp2 <-- tmp2 subdir for Indexed version of file2 : : :-----warmas1Ia.dat : : :-----warmas1Ia.idx
: :-----rptcmp <-- subdir to receive comparison reports
: : :-----warmas1I <-- comparison report demo#1
: : :-----warmas1Ia <-- comparison report demo#2
- with files reversed
- to detect inserted records in 2nd file
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
We will demonstrate uvcmpI1 on the following 2 pages using the test files shown below:
cust# prod# description purch-date expire-date price
1 2 3 4 5 6 7
01234567890123456789012345678901234567890123456789012345678901234567890
12345 12345 sony television 950101 1-year 960101 000002911 22222 00000 dish washer 960101 2-year 980101 000004922 33333 11111 refrigerator 961231 3-year 991231 000006733 33333 22222 air conditioner 970101 3-year 000101 000006744 44444 00000 freezer 970101 4-year 010101 000009955 55555 11111 clothes dryer 991231 5-year 041231 000055566 66666 00000 micro wave 001231 6-year 061231 000066677 77777 11111 panasonic HDTV 50 020601 5-year 070601 000219999 <-- Inserted 88888 00000 HP computer 030101 3-year 070101 000129999
12345 12345 sony television 950101 1-year 960101 000002911 22222 00000 dish CRASHER 960101 2-year 980101 000004922 33333 11111 refrigerator 961215 4-year 001215 000005500 33333 22222 air conditioner 970101 3-year 000101 000006744 44444 00000 freezer 970101 4-year 010101 000009955 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 55555 22222 miele ultra dryer 010301 5-year 050301 000099999 <-- Inserted 66666 00000 micro wave 001215 6-year 999999 000066677 88888 00000 HP computer 030101 3-year 070101 000129999
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmpFAI1 |
|
| uvcmpFEI1 |
|
| uvcmpFAI2 |
|
uvcmpFA1 - Fixed length SEQUENTIAL ASCII data files (see Part_1) uvcmpFE1 - Fixed length SEQUENTIAL EBCDIC data files (see Part_1) uvcmpRDW1 - compare variable length RDW files (Part_2) uvcmpIDX81 - MICRO FOCUS COBOL Variable length files (see Part_3)
| dat1/warmas1I |
|
| dat1/warmas1Ia |
|
| dat1/warmas1 |
|
| dat1/warmas1a |
|
| dat1/warmas1E |
|
| dat1/warmas1Ea |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| loadISF1 |
|
| uvcmpI1 |
|
| uvcmpI2 |
|
#1a. uvcopy loadISF1,fili1=dat1/warmas1I,filo1=tmp1/warmas1I,uop=q0i7r64k0l11
========================================================================
#1b. uvcopy loadISF1,fili1=dat1/warmas1Ia,filo1=tmp2/warmas1Ia,uop=q0i7r64k0l11
==========================================================================
#2. uvcopy uvcmpI1,fili1=tmp1/warmas1I,fili2=tmp2/warmas1Ia,uop=r64k0l11
====================================================================
uvcmpFAI1 dat1/warmas1I dat1/warmas1Ia r64k0l11 ===============================================
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI1 dat1/warmas1I dat1/warmas1Ia r64k0l11 ===============================================
uvcmpFAI1 - compare Fixed lth Ebcdic Indexed files (Dec11/07) 090218:112644:uvcmpFAI1: EOF fili01 rds=9 size=576: dat1/warmas1I 090218:112644:uvcmpFAI1: EOF filo01 wrts=9 size=0: tmp1/warmas1I 090218:112644:uvcmpFAI1: EOF fili01 rds=9 size=576: dat1/warmas1Ia 090218:112644:uvcmpFAI1: EOF filo01 wrts=9 size=0: tmp2/warmas1Ia
| Note |
|
uvcmpI1 - compare files & print diffs in hex with "*" markers (Nov28/07)
uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1 - option defaults
a# - dsplcmnt to exclude field #1
b# - length of exclude field #1
c# - dsplcmnt to exclude field #2
d# - length of exclude field #2
e# - dsplcmnt to exclude field #3
f# - length of exclude field #3
g# - dsplcmnt to exclude field #4
h# - length of exclude field #4
k# - dsplcmnt to Indexed key field#1
l# - length of Indexed key field#1
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r256 - record size
s8 - StopPrintCount, after s# mismatched records
t500000 - StopReadCount, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
090218:112644:uvcmpFAI1: uvcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmpI1
uvcopy DISAM ext=dat LNX L64 license=20090212V site=UV_Software
User OPtion (uop) defaults = q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1q1r64k0l11
-->null to accept or enter/override -->
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=9, File2Total=9, StopPrint=8, StopRead=500000
File1Reads=9, TotalErrs=5, MisCompares=4, File2NoRec=1
090218:112644:uvcmpFAI1: EOF fili01 rds=9 size=585: tmp1/warmas1I 090218:112644:uvcmpFAI1: EOF fili02 rds=9 size=585: tmp2/warmas1Ia 090218:112644:uvcmpFAI1: EOF fild03 size=4096: rptcmp 090218:112644:uvcmpFAI1: EOF filo03 wrts=60 size=3466: rptcmp/warmas1I hits=5 EOJ, Output File written to: rptcmp/warmas1I default command = null, OR enter: more,print,uvlpr12,edit,null -->
See differences report listed on the next page --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI1 uvcmpI1 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_11:19:25 uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1q1r64k0l11
recsize reccount file-size typ Report=rptcmp/warmas1I
1: 64 9 585 ISF File1=tmp1/warmas1I
2: 64 9 585 ISF File2=tmp2/warmas1Ia
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish washer 960101 2-year 980101 000004922 .
3333323333326676276766722222223333332327667233333323333333332220
22222000000049380713852000000096010102D951209801010000004922000A
*******
2 2 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
3333323333326676245454452222223333332327667233333323333333332220
22222000000049380321385200000096010102D951209801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961231 3-year 991231 000006733 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096123103D951209912310000006733000A
** * ** ** ****
2 3 33333 11111 refrigerator 961215 4-year 001215 000005500 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096121504D951200012150000005500000A
** * ** ** ****
1 6 0 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
3333323333326667667267767222223333332327667233333323333333332220
5555501111103CF48530429520000099123105D951200412310000055566000A
******** **** * * * *****
2 6 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
3333323333326667667254544445223333332327667233333323333333332220
5555501111103CF485303829EB520099061506D951200512010000066699000A
******** **** * * * *****
1 7 0 66666 00000 micro wave 001231 6-year 061231 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000123106D951200612310000066677000A
** ******
2 8 66666 00000 micro wave 001215 6-year 999999 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000121506D951209999990000066677000A
** ******
1 8 0 77777 11111 panasonic HDTV 50 020601 5-year 070601 000219999 .
3333323333327666766662445523323333332327667233333323333333332220
77777011111001E13FE9308446050002060105D951200706010000219999000A
****************************************************************
2 *** NoRecord in file#2 ***
==================== EOF or StopPrint/StopRead count reached ============== File1Total=9, File2Total=9, StopPrint=8, StopRead=500000 File1Reads=9, TotalErrs=5, MisCompares=4, File2NoRec=1
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI1 dat1/warmas1Ia dat1/warmas1I r64k0l11 ===============================================
uvcmpFAI1 - compare Fixed lth Ebcdic Indexed files (Dec11/07)
090218:112541:uvcmpFAI1: EOF fili01 rds=9 size=576: dat1/warmas1Ia
090218:112541:uvcmpFAI1: EOF filo01 wrts=9 size=0: tmp1/warmas1Ia
090218:112541:uvcmpFAI1: EOF fili01 rds=9 size=576: dat1/warmas1I
090218:112541:uvcmpFAI1: EOF filo01 wrts=9 size=0: tmp2/warmas1I
uvcmpI1 - compare files & print diffs in hex with "*" markers (Nov28/07)
uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1 - option defaults
a# - dsplcmnt to exclude field #1
b# - length of exclude field #1
c# - dsplcmnt to exclude field #2
d# - length of exclude field #2
e# - dsplcmnt to exclude field #3
f# - length of exclude field #3
g# - dsplcmnt to exclude field #4
h# - length of exclude field #4
k# - dsplcmnt to Indexed key field#1
l# - length of Indexed key field#1
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r256 - record size
s8 - StopPrintCount, after s# mismatched records
t500000 - StopReadCount, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
090218:112541:uvcmpFAI1: uvcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmpI1
uvcopy DISAM ext=dat LNX L64 license=20090212V site=UV_Software
User OPtion (uop) defaults = q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1q1r64k0l11
-->null to accept or enter/override -->
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=9, File2Total=9, StopPrint=8, StopRead=500000
File1Reads=9, TotalErrs=5, MisCompares=4, File2NoRec=1
090218:112541:uvcmpFAI1: EOF fili01 rds=9 size=585: tmp1/warmas1Ia 090218:112541:uvcmpFAI1: EOF fili02 rds=9 size=585: tmp2/warmas1I 090218:112541:uvcmpFAI1: EOF fild03 size=4096: rptcmp 090218:112541:uvcmpFAI1: EOF filo03 wrts=60 size=3467: rptcmp/warmas1Ia hits=5 EOJ, Output File written to: rptcmp/warmas1Ia default command = null, OR enter: more,print,uvlpr12,edit,null -->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI1 uvcmpI1 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_11:25:41 uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u1q1r64k0l11
recsize reccount file-size typ Report=rptcmp/warmas1Ia
1: 64 9 585 ISF File1=tmp1/warmas1Ia
2: 64 9 585 ISF File2=tmp2/warmas1I
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
3333323333326676245454452222223333332327667233333323333333332220
22222000000049380321385200000096010102D951209801010000004922000A
*******
2 2 22222 00000 dish washer 960101 2-year 980101 000004922 .
3333323333326676276766722222223333332327667233333323333333332220
22222000000049380713852000000096010102D951209801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961215 4-year 001215 000005500 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096121504D951200012150000005500000A
** * ** ** ****
2 3 33333 11111 refrigerator 961231 3-year 991231 000006733 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096123103D951209912310000006733000A
** * ** ** ****
1 6 0 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
3333323333326667667254544445223333332327667233333323333333332220
5555501111103CF485303829EB520099061506D951200512010000066699000A
******** **** * * * *****
2 6 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
3333323333326667667267767222223333332327667233333323333333332220
5555501111103CF48530429520000099123105D951200412310000055566000A
******** **** * * * *****
1 7 0 55555 22222 miele ultra dryer 010301 5-year 050301 000099999 .
3333323333326666627677626776723333332327667233333323333333332220
555550222220D95C505C421042952001030105D951200503010000099999000A
****************************************************************
2 *** NoRecord in file#2 ***
1 8 0 66666 00000 micro wave 001215 6-year 999999 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000121506D951209999990000066677000A
** ******
2 7 66666 00000 micro wave 001231 6-year 061231 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000123106D951200612310000066677000A
** ******
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=9, File2Total=9, StopPrint=8, StopRead=500000
File1Reads=9, TotalErrs=5, MisCompares=4, File2NoRec=1
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI2 dat1/warmas1Ia dat1/warmas1I r64k0l11 ===============================================
uvcmpFAI2 - compare Fixed lth ASCII Indexed files (May20/08) 110311:105326:uvcmpFAI2: EOF fili01 rds=9 size=576: dat1/warmas1Ia 110311:105326:uvcmpFAI2: EOF filo01 wrts=9 size=0: tmp1/warmas1Ia 110311:105326:uvcmpFAI2: EOF fili01 rds=9 size=576: dat1/warmas1I 110311:105326:uvcmpFAI2: EOF filo01 wrts=9 size=0: tmp2/warmas1I
unmatI2 - select unmatched records in file#2 for following uvcmpI2
uop=q1r256k0l0 - option defaults
r256 - record size
k0 - dsplcmnt to Indexed key
l0 - length of Indexed key (must specify)
110311:105326:uvcmpFAI2: uvcopy ver=20110307 pf=/home/uvadm/pf/util/unmatI2
uvcopy DISAM ext=.dat LNX L64 license=110307_00V_930601 site=UV_Software
110311:105326:uvcmpFAI2: EOF fili01 rds=9 size=585: tmp1/warmas1Ia
110311:105326:uvcmpFAI2: EOF fili02 rds=9 size=585: tmp2/warmas1I
110311:105326:uvcmpFAI2: EOF filo03 wrts=1 size=0: tmp2/warmas1I.unm
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpI2 - compare files allowing for inserts/deletes
uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y1u3 - option defaults
a# - dsplcmnt to exclude field #1
b# - length of exclude field #1
c# - dsplcmnt to exclude field #2
d# - length of exclude field #2
e# - dsplcmnt to exclude field #3
f# - length of exclude field #3
g# - dsplcmnt to exclude field #4
h# - length of exclude field #4
i1 - ignore inserts/deletes, show mismatched pairs
i2 - ignore mismatched pairs, show inserts/deletes
k# - dsplcmnt to Indexed key field#1
l# - length of Indexed key field#1
n0 - print column scale only at begin report
n1 - print column scale for each record mismatch pair
p30 - line# to test overflow at begin record only
r256 - record size
s8 - StopPrintCount, after s# mismatched records
t500000 - StopReadCount, after s# mismats, else read to EOF
u1 - stop print at last segment with mismatch
u2 - inhibit printing all matching segments after 1st
x# - hex or char print (x1 char, x2 hex, x0 auto)
y0 - do NOT translate, file already ASCII, default(y0)
y1 - translate char lines to ASCII (for EBCDIC files)
110311:105326:uvcmpFAI2: uvcopy ver=20110307 pf=/home/uvadm/pf/util/uvcmpI2
uvcopy DISAM ext=.dat LNX L64 license=110307_00V_930601 site=UV_Software
User OPtion (uop) defaults = q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u3q1r64k0l11
-->null to accept or enter/override -->
==================== EOF or StopPrint/StopRead count reached ==============
File1Total=9, File2Total=9, StopPrint=8, StopRead=500000
File1Reads=9, TotalErrs=6, MisCompares=4, File1NoRec=1, File2NoRec=1
110311:105328:uvcmpFAI2: EOF fili01 rds=9 size=585: tmp1/warmas1Ia 110311:105328:uvcmpFAI2: EOF fili02 rds=10 size=585: tmp2/warmas1I 110311:105328:uvcmpFAI2: EOF fili03 rds=1 size=65: tmp2/warmas1I.unm 110311:105328:uvcmpFAI2: EOF fild04 size=4096: rptcmp 110311:105328:uvcmpFAI2: EOF filo04 wrts=27 size=1561: rptcmp/warmas1Ia hits=6 EOJ, Output File written to: rptcmp/warmas1Ia default command = null, OR enter: more,print,uvlpr12,edit,null -->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpFAI2 uvcmpI2 - compare 2 files, print mismatched records, '*' flag diffs
2009/02/18_11:31:55 uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u3q1r64k0l11
recsize reccount file-size typ Report=rptcmp/warmas1Ia
1: 64 9 585 ISF File1=tmp1/warmas1Ia
2: 64 9 585 ISF File2=tmp2/warmas1I
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 00000 dish CRASHER 960101 2-year 980101 000004922 .
3333323333326676245454452222223333332327667233333323333333332220
22222000000049380321385200000096010102D951209801010000004922000A
*******
2 2 22222 00000 dish washer 960101 2-year 980101 000004922 .
3333323333326676276766722222223333332327667233333323333333332220
22222000000049380713852000000096010102D951209801010000004922000A
*******
1 3 0 33333 11111 refrigerator 961215 4-year 001215 000005500 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096121504D951200012150000005500000A
** * ** ** ****
2 3 33333 11111 refrigerator 961231 3-year 991231 000006733 .
3333323333327667666767672222223333332327667233333323333333332220
3333301111102562975214F200000096123103D951209912310000006733000A
** * ** ** ****
1 6 0 55555 11111 clothes SHRINKER 990615 6-year 051201 000066699 .
3333323333326667667254544445223333332327667233333323333333332220
5555501111103CF485303829EB520099061506D951200512010000066699000A
******** **** * * * *****
2 6 55555 11111 clothes dryer 991231 5-year 041231 000055566 .
3333323333326667667267767222223333332327667233333323333333332220
5555501111103CF48530429520000099123105D951200412310000055566000A
******** **** * * * *****
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
1 7 0 55555 22222 miele ultra dryer 010301 5-year 050301 000099999 .
3333323333326666627677626776723333332327667233333323333333332220
555550222220D95C505C421042952001030105D951200503010000099999000A
****************************************************************
2 *** NoRecord in file#2 ***
1 8 0 66666 00000 micro wave 001215 6-year 999999 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000121506D951209999990000066677000A
** ******
2 7 66666 00000 micro wave 001231 6-year 061231 000066677 .
3333323333326667627676222222223333332327667233333323333333332220
666660000000D932F071650000000000123106D951200612310000066677000A
** ******
1 *** NoRecord in file#1 ***
2 8+ 0 77777 11111 panasonic HDTV 50 020601 5-year 070601 000219999 .
3333323333327666766662445523323333332327667233333323333333332220
77777011111001E13FE9308446050002060105D951200706010000219999000A
****************************************************************
==================== EOF or StopPrint/StopRead count reached ============== File1Total=9, File2Total=9, StopPrint=8, StopRead=500000 File1Reads=9, TotalErrs=6, MisCompares=4, File1NoRec=1, File2NoRec=1
| option i1 |
|
| option i2 |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
In March 2011, 2 new scripts were added to make it easier to compare variable length RDW files allowing for inserts & deletes.
| uvcmpRDWAI2 |
|
| uvcmpRDWEI2 |
|
You can demo these scripts using the following supplied test files. These can be copied from $UV/dat1 to your homedir/subdir as previously shown on page '4B1'.
cp $UV/dat1/vendormas3* dat1/ <-- copy supplied test files to your homedir =============================
| vendormas3 |
|
| vendormas32 |
|
| vendormas3e |
|
| vendormas3e2 |
|
DELL10 M Dell Inc.
HP0010 M Hewlett Packard
IBM010 M International Business Machines
MFC010 M Micro Focus COBOL
MS0010 M Microsoft Corp.
REDH10 M Red Hat Linux
SUN010 M Sun Microsystems Ltd
UVSI10 M UV Software Inc.
DELL10 M Dell Inc.
HP0010 M Hewlett Packrat
IBM010 M International Business Machines
MFC010 M Micro Focus COBOL
MS0010 M Microhard Corp.
REDH10 M Red Hat Linux Incorporated
UVSI10 M UV Software Inc.
uvcp "fili1=dat1/vendormas2,typ=LST,rcs=64,filo1=dat1/vendormas3,typ=RDW" =========================================================================
uvcp "fili1=dat1/vendormas2,typ=LST,rcs=64,filo1=dat1/vendormas3e,typ=RDW,tre=0(64)" ================================================================================
UV Software has already created the test files as shown above. We list the 'uvcp' commands here in case you are interested in how to create RDW files from TEXT files.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpRDWAI2 dat1/vendormas3 dat1/vendormas32 r64k0l6 ==================================================== - compare 2 RDW files, report created in rptcmp/vendormas3 (listed below) - see options 'r64k0l6' explained on following page -->
uvcmpRDWAI2 uvcmpI2 - compare 2 files, print mismatched records, '*' flag diffs
2011/03/12_06:40:48 uop=q1a0b0c0d0e0f0g0h0k0l0p30r256s8t500000x2y0u3q1r64k0l6
recsize reccount file-size typ Report=rptcmp/vendormas3
1: 64 8 520 ISF File1=tmp1/vendormas3
2: 64 7 455 ISF File2=tmp2/vendormas32
1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 HP0010 M Hewlett Packard
4533332422467667725666676222222222222222222222222222222222222222
8000100D00857C5440013B124000000000000000000000000000000000000000
***
2 2 HP0010 M Hewlett Packrat
4533332422467667725666767222222222222222222222222222222222222222
8000100D00857C5440013B214000000000000000000000000000000000000000
***
1 5 0 MS0010 M Microsoft Corp.
4533332422466767667246772222222222222222222222222222222222222222
D300100D00D932F3F6403F20E000000000000000000000000000000000000000
****
2 5 MS0010 M Microhard Corp.
4533332422466766676246772222222222222222222222222222222222222222
D300100D00D932F812403F20E000000000000000000000000000000000000000
****
1 6 0 REDH10 M Red Hat Linux
5444332422566246724667722222222222222222222222222222222222222222
2548100D0025408140C9E5800000000000000000000000000000000000000000
************
2 6 REDH10 M Red Hat Linux Incorporated
5444332422566246724667724666776767662222222222222222222222222222
2548100D0025408140C9E5809E3F20F214540000000000000000000000000000
************
1 7 0 SUN010 M Sun Microsystems Ltd
5543332422576246676777766724762222222222222222222222222222222222
35E0100D0035E0D932F39345D30C440000000000000000000000000000000000
****************************************************************
2 *** NoRecord in file#2 ***
==================== EOF or StopPrint/StopRead count reached ============== File1Total=8, File2Total=7, StopPrint=8, StopRead=500000 File1Reads=8, TotalErrs=4, MisCompares=3, File1NoRec=0, File2NoRec=1
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmpRDWAI2 dat1/vendormas3 dat1/vendormas32 r64k0l6 ==================================================== - compare 2 RDW files, note options explained below:
| r64 |
|
| k0 |
|
| l6 |
|
uvcmpRDWEI2 dat1/vendormas3e dat1/vendormas3e2 r64k0l6 ====================================================== - compare 2 EBCDIC RDW files
vi rptcmp/vendormas3e <-- inspect report (created in subdir rptcmp/...) =====================
Since the report is similar to the ASCII version (on previous page), we will show only the 1st pair of mismatched records. Note the vertical hex now shows the EBCDIC zones & digits, but character line is translated to ASCII, so we can read it on the unix/linux systems.
1 2 0 HP0010 M Hewlett Packard
CDFFFF4D44C8A98AA4D889898422222222222222222222222222222222222222
8700100400856353307132194000000000000000000000000000000000000000
****
2 2 HP0010 M Hewlett Packrat
CDFFFF4D44C8A98AA4D88998A222222222222222222222222222222222222222
8700100400856353307132913000000000000000000000000000000000000000
****
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Records must have some unique field to be used as a key to load the Indexed files required to allow for inserts & deletes. If the files did not have any unique key, we could modify the script to create a sequential# key as files are converted to Indexed files.
vi $UV/sf/util/uvcmpRDWAI2 <-- inspect script to compare 2 RDW files ==========================
This script is similar to 'uvcmpFAI2' listed on page '4X2'. Note line 70 loads the 1st file into an Indexed files as follows (line 75 loads the 2nd file).
uvcopy loadISF1,fili1=$df1,typ=RDWz8,filo1=tmp1/$f1,uop=q0i7$optns #================================================================= # - load 1st seqntl file to Indexed file for uvcmpI2 to match by key
We could copy/modify this job to insert a 10 byte key (sequential#) as the file is being load to the Indexed file. We might name the new job 'loadISF2'.
#1. cd $UV <-- change to /home/uvadm
#2. cp pf/util/loadISF1 pf/util/loadISF2 - copy/rename for alternate version
====================================
#3. vi pf/util/loadISF2 - edit alternate job to insert sequence# key
===================
We would also copy/rename uvcmpRDWAI2 & modify to call the alternate job loadISF2 on lines 70 & 75.
#4. cp sf/util/uvcmpRDWAI2 sf/util/uvcmpRDWAI3 - copy/rename for alt version
==========================================
#5. vi pf/util/uvcmpRDWAI2 - edit alt job to call loadISF3 (vs loadISF2)
======================
The alternate jobs/scripts have not yet been created as of March 2011, but let us know if you need them.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| uvcmpI1 |
|
| uvcmpI2 |
|
| 4X1. | uvcmpFAI1 - compare Fixed ASCII files that may have INSERTs & DELETEs |
| ____ uvcmpFEI1 |
|
| 4X2. | uvcmpFAI2 - compare Fixed ASCII files that may have INSERTED records |
| - creates 1 report with inserts/deletes in both files |
| ____ uvcmpFEI2 |
|
| ____ uvcmpRDWAI2 |
|
| ____ uvcmpRDWEI2 |
|
Two scripts will be listed below since they are short. The uvcopy jobs will not be listed because they are much longer, but you can view or print as follows:
#1. cd /home/uvadm
==============
#2. vi pf/util/uvcmpI1
==================
#3. uvlp12 pf/util/uvcmpI1
======================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh # uvcmpFAI1 - script to compare 2 Fixed length ASCII Indexed files # - by Owen Townsend, December 2007, May 2008 #Feb18/09 - rename uvcmp2->5, uvcmp3->6, RDW->uvcmp2, IDX->uvcmp3 # # ** scripts allowing INSERTs/DELETEs BUT on 2 reports ** # #*uvcmpFAI1 - FixLth ASCII w inserted recs by loading Index typ=ISF # uvcmpFEI1 - FixLth EBCDIC w inserted recs by loading Index typ=ISF # uvcmpFAX1 - FixLth ASCII w inserted recs by loading Index typ=IDXf1 # uvcmpFEX1 - FixLth EBCDIC w inserted recs by loading Index typ=IDXf1 # - sort/load Indexed files, read 1st seqntl by key, read 2nd randomly by key # - need to repeat with files reversed to ensure all mismatches detected # - above scritps now obsoleted by following scripts # # ** scripts allowing for INSERTs/DELETEs on 1 report ** # # uvcmpFAI2 - FixLth ASCII w inserted recs by loading Index typ=ISF # uvcmpFEI2 - FixLth EBCDIC w inserted recs by loading Index typ=ISF # - pre-compare step unmatI2 writes unmatched records in file1 to tmp2/... # by reading file2 sequentially & reading file1 randomly by key # - uvcmpI2 then reads 3 files data1, data2,& data2 recs w/o matches in data1 # # ** scripts NOT allowing Inserts/deletes ** # # uvcmpFA1 - compare Fixed length ASCII data files # uvcmpFE1 - compare Fixed length EBCDIC data files # uvcmpRDW1 - compare RDW variable length files - ASCII or EBCDIC (option y1) # uvcmpIDX81 - compare Micro Focus COBOL IDXFORMAT3/8 Variable length files # - 1 pass only, no need to load Indexed files, or rerun files reversed # (since no inserts/deletes are allowed for these scripts) # - the 2 datafiles may have same names if in different subdirs # - run this script from a working directory with subdirs holding datafiles # - output report will be created in subdir: rptcmp/filename1 # (will make subdir ./rptcmp if not already existing) #
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
export JOBID2=uvcmpFAI1 # so uvcopy uvcmp1 can show script name on report
echo "uvcmpFAI1 - compare Fixed lth Ebcdic Indexed files (Dec11/07)"
if [[ -f "$1" && -f "$2" && -n "$3" ]]; then :
else echo "usage: uvcmpFAI1 file1 file2 Recsize/Keydsp/Keylth/Excludes"
echo " ===================================================="
echo "example1: uvcmpFAI1 dat1/warmas1I dat1/warmas1Ia r64k0l6"
echo " =============================================="
echo "compare 2 Fixed length ASCII Indexed data files"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 options alphas+numerics, example r64k0l6"
echo " - r64=Recsize, k0=Keydsplcmnt, l6=keyLength"
echo " "
echo "ex2: uvcmpFAI1 dat1/warmas1I dat1/warmas1Ia r64k0l6a34b2c44d6"
echo " ========================================================"
echo " - ex#2 demos exclude field excludes (displacements & lengths)"
echo " - a34b2 excludes 34-35, c44d6 excludes 44-49"
echo " - see all options described in /home/uvadm/pf/util/uvcmp1"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) must be spcfd"
exit 99; fi
#
df1="$1"; df2="$2"; optns="$3";
f1=$(basename $df1)
f2=$(basename $df2)
#
if [[ ! -d tmp1 ]]; then mkdir tmp1; fi
if [[ ! -d tmp2 ]]; then mkdir tmp2; fi
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uvcopy loadISF1,fili1=$df1,filo1=tmp1/$f1,uop=q0i7$optns
#=======================================================
if (($?)); then exit 91; fi # exit if load fails
# - load 1st seqntl file to Indexed file for uvcmpI1 to match by key
#
uvcopy loadISF1,fili1=$df2,filo1=tmp2/$f2,uop=q0i7$optns
#=======================================================
if (($?)); then exit 92; fi # exit if load fails
# - load 2nd seqntl file to Indexed file for uvcmpI1 to match by key
#
uvcopy uvcmpI1,fili1=tmp1/$f1,fili2=tmp2/$f2,fild3=rptcmp,uop=q1$optns
#=====================================================================
# - uvcmpI1 creates comparison report in rptcmp/filename1
# and prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh # uvcmpFAI2 - script to compare 2 Fixed length ASCII Indexed files # - by Owen Townsend, May 2008 #Mar11/11 - option i1 ignore inserts/deletes, show only mismatched pairs # option i2 ignore mismatched pairs, show only inserts/deletes # # ** scripts allowing for INSERTs/DELETEs on 1 report ** # #*uvcmpFAI2 - FixLth ASCII w inserted recs by loading Index typ=ISF # uvcmpFEI2 - FixLth EBCDIC w inserted recs by loading Index typ=ISF # - pre-compare step unmatI2 writes unmatched records in file1 to tmp2/... # by reading file2 sequentially & reading file1 randomly by key # - uvcmpI2 then reads 3 files data1, data2,& data2 recs w/o matches in data1 # # ** scripts allowing INSERTs/DELETEs BUT on 2 reports ** # # uvcmpFAI1 - FixLth ASCII w inserted recs by loading Index typ=ISF # uvcmpFEI1 - FixLth EBCDIC w inserted recs by loading Index typ=ISF # uvcmpFAX1 - FixLth ASCII w inserted recs by loading Index typ=IDXf1 # uvcmpFEX1 - FixLth EBCDIC w inserted recs by loading Index typ=IDXf1 # - sort/load Indexed files, read 1st seqntl by key, read 2nd randomly by key # - need to repeat with files reversed to ensure all mismatches detected # - above scritps now obsoleted by uvcmpFAI2 & uvcmpFEI2 # # ** scripts NOT allowing Inserts/deletes ** # # uvcmpFA1 - compare Fixed length ASCII data files # uvcmpFE1 - compare Fixed length EBCDIC data files # uvcmpRDW1 - compare RDW variable length files - ASCII or EBCDIC (option y1) # uvcmpIDX81 - compare Micro Focus COBOL IDXFORMAT3/8 Variable length files # - 1 pass only, no need to load Indexed files, or rerun files reversed # - use these scripts for sequential files without inserts & deletes # - simpler, faster, 1 step process (no need for Indexed files) # # - the 2 datafiles may have same names if in different subdirs # - run this script from a working directory with subdirs holding datafiles # - output report will be created in subdir: rptcmp/filename1 # (will make subdir ./rptcmp if not already existing) #
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
export JOBID2=uvcmpFAI2 # so uvcopy uvcmpI2 can show script name on report
echo "uvcmpFAI2 - compare Fixed lth ASCII Indexed files (May20/08)"
if [[ -f "$1" && -f "$2" && -n "$3" ]]; then :
else echo "usage: uvcmpFAI2 file1 file2 Recsize/Keydsp/Keylth/Excludes"
echo " ===================================================="
echo "example1: uvcmpFAI2 dat1/warmas1I dat1/warmas1Ia r64k0l6"
echo " =============================================="
echo "compare 2 Fixed length ASCII Indexed data files"
echo " - arg1 & arg2 are filenames that must be present"
echo " - arg3 options alphas+numerics, example r64k0l6"
echo " - r64=Recsize, k0=Keydsplcmnt, l6=keyLength"
echo " "
echo "ex2: uvcmpFAI2 dat1/warmas1I dat1/warmas1Ia r64k0l6a34b2c44d6"
echo " =========================================================="
echo " - ex#2 demos exclude field excludes (displacements & lengths)"
echo " - a34b2 excludes 34-35, c44d6 excludes 44-49"
echo " - see all options described in /home/uvadm/pf/util/uvcmpI2"
echo " "
echo "ERROR - arg1 & arg2 must be files, arg3(options) must be spcfd"
exit 99; fi
#
df1="$1"; df2="$2"; optns="$3";
f1=$(basename $df1)
f2=$(basename $df2)
#
if [[ ! -d tmp1 ]]; then mkdir tmp1; fi
if [[ ! -d tmp2 ]]; then mkdir tmp2; fi
if [[ ! -d rptcmp ]]; then mkdir rptcmp; fi
#
uvcopy loadISF1,fili1=$df1,filo1=tmp1/$f1,uop=q0i7$optns
#=======================================================
if (($?)); then exit 91; fi # exit if load fails
# - load 1st seqntl file to Indexed file for uvcmpI2 to match by key
#
uvcopy loadISF1,fili1=$df2,filo1=tmp2/$f2,uop=q0i7$optns
#=======================================================
if (($?)); then exit 92; fi # exit if load fails
# - load 2nd seqntl file to Indexed file for uvcmpI2 to match by key
#
uvcopy unmatI2,fili1=tmp1/$f1,fili2=tmp2/$f2,filo3=tmp2/$f2.unm,uop=q0$optns
#===========================================================================
# pass #1 - detect unmatched records in file2 & write to tmp2/filename2.unm
# - by reading file2 seqntly & file1 randomly by key
#
uvcopy uvcmpI2,fili1=tmp1/$f1,fili2=tmp2/$f2,fili3=tmp2/$f2.unm\
,fild4=rptcmp,uop=q1$optns
#===============================================================
# pass #2 - create the report, reads file1 seqntly, random read file2 by key,
# - detects matched keys with mismatched data & unmatched recs in file2
# - unmatched recs in file1 are merged in from unmatI2 output above
#
# report created: rptcmp/filename1
# - prompts for disposition, may enter vi, more, uvlp12, etc
exit 0
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 5A1. | Comparing 'entire directories' of data files - Overview |
| - 'uvcmp5' uses a control file to supply the record sizes | |
| (vs manual entry for uvcmp1 used in Parts 1,2,3,4) | |
| - script 'uvcmp5all' makes it easier to run | |
| - first run 'unixcmpsplit' to isolate non-matching files | |
| - uses unix 'cmp' to eliminate matching files before uvcmp5 is run | |
| (since 'cmp' is much faster than uvcmp5) |
| 5B1. | unixcmpall - script to run unix 'cmp' for all pairs of files in 2 subdirs |
| - creates a report used by unixcmpsplit to remove matches |
| 5C1. | unixcmpsplit - move non-matching files out to 2 compare directories |
| - saves running uvcmp5 on matching files (unix 'cmp' much faster). |
| 5D1. | uvcmp5 - compare all files in 2 directories & create comparison reports |
| - uses a control file to supply record sizes | |
| (recsize was entered manually for uvcmp1) |
| 5E1. | preparation to demo uvcmp5 - compare all files in 2 subdirs |
| 5F1. | Execute test/demo uvcmp5 - compare all files in 2 subdirs |
| 5G1. | vtocr1 - VTOC reports may assist your directory comparisons |
| - could run before & after batch updates |
| 5X0. | Summary of scripts & uvcopy jobs used in this Part |
| - listings of scripts (& uvcopy jobs only if short) | |
| - unixcmpsplit, unixcmpall, uvcmp5all, |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'uvcmp5' can be used to compare all Fixed Length Sequential files in 2 directories & generate difference reports in subdir rptcmp.
uvcmp5 is driven by the 'uvcmp5all' script & a pre-edited control file to supply the record sizes, so we can perform the comparisons with no interventions. (vs manual entry for uvcmp1 used in Parts 1,2,3,4)
The control file may be prepared with the vi editor & loaded into an ISAM file with the supplied utility job (uvcopy loadctlI).
You might 1st run script 'unixcmpsplit' if you have a lot of large files. unixcmpsplit uses unix 'cmp' to eliminate matching files before uvcmp5 is run (since 'cmp' is much faster than uvcmp5).
The UNIX 'cmp' utility is inadequate for reporting the differences between the files, BUT is very QUICK to determine whether the files are different.
We will also demonstrate the 'unixcmpall' script (also based on UNIX cmp), which simply creates a brief report indicating matching/mismatching files.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
I recommend you setup a 'cmpdata' within your homedir & then use the the 'cmpdatadirs' script to setup subdirs required for uvcmp1, uvcmp5,& uvcmp6 (see procedures following the directory illustrations).
/home/userxx
:-----cmpdata <-- superdir for DATA comparisons
: :-----cpy1 - copybooks
: :-----cpy2 - copybooks for modified layouts (as in Y2K)
: :-----dat0 - EBCDIC files transferred from mainframe
- may need to convert variable to fixed in dat1
- might need to copy/rename/reformat dat0/... to dat1
: :-----dat1 - subdir to hold 1st file(s) to be compared
- can put demo files here since filenames are different
: :-----dat2 - subdir to hold 2nd file(s) to be compared
(need separate subdir if 2nd filenames same as 1st)
: :-----map1 - copybook layouts
: :-----map1I - copybook layouts loaded to Indexed files
: :-----map2 - modified layouts (as in Y2K) OR same as map1
: :-----pfc1 - uvcmp6 comparison jobs generated from layouts
: :-----pfc2 - comparison jobs copied here for any mods & execution
: :-----rptcmp - reports generated here
: :-----tmp - for misc outputs (keep working dir free of files)
The cpy1,2,map1,1I,map2,pfc1,2 subdirs are needed to generate 'uvcmp6' jobs that allow different file layouts for the files to be compared (as in Y2K).
#1. Login ---> /home/userxx <-- Login or change to your home directory
#2. mkdir cmpdata <-- make superdir for subdirs shown above
=============
#3. cd cmpdata <-- change into the comparison superdir
==========
#4. cmpdatadirs <-- run script to make subdirs shown above
===========
#5a. cp /home/uvadm/dat1/warmas1 dat1
================================
- copy supplied test files to subdirs in your homedir
#5b. cp /home/uvadm/dat1/warmas1a dat1
=================================
- can store 2nd demo file in same subdir as 1st since filenames different
(store in dat2/... if filenames different)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
1a. cd $HOME - change to your home directory 1b. mkdir cmpdata - make a sub-directory for the test/demo 1c. cd cmpdata - change into it
2a. mkdir dat1 - make subsubdir for original data files 2b. mkdir dat2 - make subsubdir for modified data files 2c. mkdir rptcmp - make subsubdir for mismatch reports
3a. cp /home/uvadm/dat1/warmas* dat1 - copy some test files to dat1 3b. cp /home/uvadm/dat1/warmas* dat2 - & dat2, then modify some in dat2
4a. vi dat2/* - create some differences to illustrate mismatch reports
- for the demo report below, we modified as follows:
warmas3 rec#5: change 'dish washer' to 'disk CRASHER'
warmas6 rec#6: change 'clothes dryer' to 'clothes SHRINKER'
Note - warmas9 has packed fields & we will change packed date field in
record #2, byte 60-63 from x'0990101C' to x'0990909C'
- You can't do this with 'vi', so use 'uvhd' as follows:
4b. uvhd dat2/warmas9 r64u - call uvhd to update warmas9 (rcsz 64)
- goto rec#2 & enter following command:
u 60(4),x'0990909C' - change bytes 60-63 to x'0990909C'
5a. unixcmpall dat1 dat2 <-- run unixcmpall script (using UNIX cmp)
==================== - report will be displayed on screen (stdout)
5b. unixcmpall dat1 dat2 >rptcmp/cmpall.rpt <-- to capture report in a file
=======================================
file# 1 cmp dat1/warmas1 dat2/warmas1 file# 2 cmp dat1/warmas1x dat2/warmas1x file# 3 cmp dat1/warmas3 dat2/warmas3 dat1/warmas3 dat2/warmas3 differ: char 274, line 5 file# 4 cmp dat1/warmas6 dat2/warmas6 dat1/warmas6 dat2/warmas6 differ: char 341, line 6 file# 5 cmp dat1/warmas6x dat2/warmas6x file# 6 cmp dat1/warmas9 dat2/warmas9 dat1/warmas9 dat2/warmas9 differ: char 127, line 1 >> 3 mismatches of 6 total files compared in dat1 & dat2
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
We will now demo the 'unixcmpsplit' script (based on UNIX cmp) to move the non-matching files from 2 directories out to 2 other directories, prior to running the 'uvcmp5all' script (based on uvcopy uvcmp5, see next section).
The demo preparation is the similar to unixcmpall (shown on the previous page). We only need to make 2 more subdirs to receive the non-matching files & run unixcmpsplit vs unixcmpall. Here are the additions & replacement OP. Instrns:
2e. mkdir dat1X - make subdirs to receive non-matching files 2f. mkdir dat2X (selected out by unixcmpsplit)
5a. unixcmpsplit dat1 dat2 dat1X dat2X <-- run unixcmpsplit script
================================== - report displayed on screen
Note - To rerun the demo, we must move the non-matching files back
5b. mv dat1X/* dat1 5c. mv dat2X/* dat2
5d. unixcmpsplit dat1 dat2 dat1X dat2X >rptcmp/unixcmpsplit.rpt
===========================================================
- capture report in a file
5e. uvlp12 rptcmp/unixcmpsplit.rpt - print the unixcmpsplit report
==============================
file# 1 cmp dat1/warmas1 dat2/warmas1 file# 2 cmp dat1/warmas1x dat2/warmas1x file# 3 cmp dat1/warmas3 dat2/warmas3 dat1/warmas3 dat2/warmas3 differ: char 274, line 5
file# 4 cmp dat1/warmas6 dat2/warmas6 dat1/warmas6 dat2/warmas6 differ: char 341, line 6
file# 5 cmp dat1/warmas6x dat2/warmas6x file# 6 cmp dat1/warmas9 dat2/warmas9 dat1/warmas9 dat2/warmas9 differ: char 127, line 1
>> 3 matching of 6 total files compared in dat1 & dat2 >> 3 non-matching files have been moved to directories dat1X & dat2X
See the 'unixcmpsplit' script listed at the end of this Part --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'uvcmp5' is similar to uvcmp1, but includes 2 significant enhancements.
Uvcmp2 does require the 1-time preparation of the control file, but this is well worth the time if you want to compare dozens of files multiple times.
The control file may be prepared with the editor & loaded into an ISAM file with the supplied utility job (uvcopy loadctlI). You would hard-code it's pathname within the uvcmp5 job.
For casual 1-shot use, use the uvcmp1 job which requires no preparation (& you can enter all required parameters on the command line).
Note that the reports generated are the same as for uvcmp1. Several demo reports were illustrated for uvcmp1 in Part_1 and they will not be repeated here. Please see the following pages for:
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
# ctlcmp1 - control file to generate complete jobs from skeletons # - used by uvdata52 to encode data-file-names & ISAM keys # # uvcmp5 - this ctlcmp1 must be converted to an ISAM file so uvcmp5 can # determine the record sizes of sequential files, use loadctlI: # # uvcopy loadctlI,fili1=ctl/ctlcmp1,filo1=ctl/ctlcmp1I # ==================================================== # # uvcmp6 - the procedure of generating field by field comparison jobs for # each file uses the uvdata52 utility which converts the uvcmp5 # skeletons to complete uvcopy jobs with data filenames & keys. # - uvdata52 uses this file directly as a sequential text file # to relate the data files to the copybooks & code ISAM key info. # custmas1 cpy=custmas1 rcs=00256 custmast cpy=custmas1 rcs=00256 keys=(0,6,nn,10,25,nn,60,16,nn,etc) warmas1 cpy=warmas1 rcs=00064 warmas3 cpy=warmas3 rcs=00064 warmas6 cpy=warmas6 rcs=00064 warmas8 cpy=warmas8 rcs=00064 keys=(0,6,nn)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
You can run the following demo, using the test files provided in uvadm/dat1 We will setup a test sub-directory (within your home directory), change into it and setup several sub-sub-directories. This will keep our home directory uncluttered & make it easier to clean up after the test/demo.
Sub-directories are made for the different types of files, so that we can keep the same filenames for different versions of the files. These data files will also be used for the demos in the next section.
Multiple sub-directories work well if we have many files to be compared We can compare 2 entire directories & output the corresponding mismatch reports to a 3rd directory.
1a. cd $HOME - change to your home directory 1b. mkdir cmpdata - make a sub-directory for the test/demo 1c. cd cmpdata - change into it
2a. mkdir dat1 - make subsubdir for original data files 2b. mkdir dat2 - make subsubdir for modified data files 2c. mkdir rptcmp - make subsubdir for mismatch reports 2d. mkdir ctl - make directory for control files 2e. mkdir tmp - subdir for misc outputs (keep working dir clean)
3a. cp /home/uvadm/dat1/custmas1 dat1 3b. cp /home/uvadm/dat1/warmas1 dat1 3c. cp /home/uvadm/dat1/warmas6 dat1
3d. uvcp "fili1=/home/uvadm/dat1/warmas8,rcs=64,typ=RSF,filo1=dat1/warmas8\
=======================================================================
,typ=ISF,isk1=0(6)" - convert warmas8 to ISAM (see uvcp.doc)
===================
6a. cp /home/uvadm/ctl/ctlcmp1 ctl/ctlcmp1
======================================
6b. uvcopy loadctlI,fili1=ctl/ctlcmp1,filo1=ctl/ctlcmp1I
====================================================
- copy ctlcmp1 & create indexed version for uvcmp5
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
uvcmp5all dat1 dat2 rptcmp filr4=ctl/ctlcmp1I,uop=q0i7s5 ========================================================
compare files in dir dat1 to dir dat2, write mismatch reports to rptcmp
EOF fili1 32 rds, 0 wrts, 8192 size, fname=dat1/custmas1 EOF fili2 32 rds, 0 wrts, 8192 size, fname=dat2/custmas1 EOF filo3 0 rds, 45 wrts, 2689 size, fname=rptcmp/custmas1 1 hits EOF filr4 1 rds, 0 wrts, 0 upds, 0 dels, 906 size, fname=ctl/ctlcmp1I ctl/ctlcmp1I DISAM STATS: recs=6,rcsz=150,keys=1
EOF fili1 7 rds, 0 wrts, 448 size, fname=dat1/warmas1 EOF fili2 7 rds, 0 wrts, 448 size, fname=dat2/warmas1 EOF filo3 0 rds, 24 wrts, 1161 size, fname=rptcmp/warmas1 2 hits EOF filr4 1 rds, 0 wrts, 0 upds, 0 dels, 906 size, fname=ctl/ctlcmp1I ctl/ctlcmp1I DISAM STATS: recs=6,rcsz=150,keys=1
EOF fili1 7 rds, 0 wrts, 448 size, fname=dat1/warmas6 EOF fili2 7 rds, 0 wrts, 448 size, fname=dat2/warmas6 EOF filo3 0 rds, 45 wrts, 2151 size, fname=rptcmp/warmas6 5 hits EOF filr4 1 rds, 0 wrts, 0 upds, 0 dels, 906 size, fname=ctl/ctlcmp1I ctl/ctlcmp1I DISAM STATS: recs=6,rcsz=150,keys=1
EOF fili1 6 rds, 0 wrts, 0 upds, 0 dels, 390 size, fname=dat1/warmas8 dat1/warmas8 DISAM STATS: recs=6,rcsz=64,keys=1 EOF fili2 6 rds, 0 wrts, 0 upds, 0 dels, 426 size, fname=dat2/warmas8 dat2/warmas8 DISAM STATS: recs=6,rcsz=70,keys=1 EOF filo3 0 rds, 45 wrts, 2183 size, fname=rptcmp/warmas8 5 hits EOF filr4 1 rds, 0 wrts, 0 upds, 0 dels, 906 size, fname=ctl/ctlcmp1I ctl/ctlcmp1I DISAM STATS: recs=6,rcsz=150,keys=1
5 files compared in dat1 & dat2, mismatch reports in rptcmp
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
You will find the 'VTOC' reports a valuable asset in comparing the resulting directories of files from your parallel tests. Run these first & review any differences in files present & record counts within the files.
Note that 'ls -l' does not give you any record counts, because UNIX does not carry a record size within directory entries (as did mainframes). The 'vtocr1' utility uses the same control file that we created on page '5D2' for the 'uvcmp5' job to know the record sizes for file compares.
uvcopy vtocr1,fild1=dat1,filr1=ctl/ctlcmp1I =========================================== vtocr1 prompts for report disposition report=tmp/vtocr1.rpt - more/vi/uvlp12 --------> uvlp12 <-- print at 12 cpi
uvcopy vtocr1,fild1=dat2,filr1=ctl/ctlcmp1I --> uvlp12 <- print dat2 report ===========================================
vtocr1 Directories: dat1
1998/11/15_08:08:47 options: q1a1s2k2s1
file# file-size rcsz records kloc klen last-modify filename
1 8,192 256 32 19981114 16:19 custmas1
2 448 64 7 19981114 09:23 warmas1
3 448 64 7 19981102 18:13 warmas6
4 390 64+ 6 0 6 19981104 09:35 warmas8
Total Records 59
vtocr1 Directories: dat2
1998/11/15_08:11:09 options: q1a1s2k2s1
file# file-size rcsz records kloc klen last-modify filename
1 8,192 256 32 19981114 16:23 custmas1
2 448 64 7 19981114 09:32 warmas1
3 561 70 8* 19981105 09:39 warmas6
4 426 70+ 6 0 6 19981104 09:35 warmas8
Total Records 60
The '+' beside the rcsz of warmas8 (64+ in dat1 & 70+ in dat2) indicates that the physical record size is 1 greater (ISAM record delete flag x'0A' at EOR).
The '*' beside record count (8*) of warmas6 in dat2 indicates the file size is not evenly divisible by the record size. See explanation next page --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
vtocr1 Directories: dat2
1998/11/15_08:11:09 options: q1a1s2k2s1
file# file-size rcsz records kloc klen last-modify filename
1 8,192 256 32 19981114 16:23 custmas1
2 448 64 7 19981114 09:32 warmas1
3 561 70 8* 19981105 09:39 warmas6
4 426 70+ 6 0 6 19981104 09:35 warmas8
Total Records 60
The '*' beside record count (8*) of warmas6 in dat2 indicates the file size is not evenly divisible by the record size.
In this case it is because our control file has the correct record size for warmas6 in dat1, but not for warmas6 in dat2.
The records in warmas6 & warmas8 of dat2 have been expanded to 70 bytes to test 'uvcmp6' (see next section) which can do field by field comparisons of different record sizes (due to date field expansion).
The solution is to create a different control file for the dat2 directory of expanded record sizes. We could simply copy our ctl/ctlcmp1 to say ctl/ctlcmp2,& correct the record sizes with the editor.
Note that uvcmp5all requires the Indexed version of the control file, and we don't want to attempt to edit the Indexed file directly. Is is much safer to copy the non-indexed version of the control file, modify that,& use it to create the indexed version, for example:
1. cp ctl/ctlcmp1 ctl/ctlcmp2 - copy sequential version of ctlcmp1
==========================
2. vi ctl/ctlcmp2 - update ctlcmp2 for rcsz's in dat2
==============
3. uvcopy loadctlI,fili1=ctl/ctlcmp2,filo1=ctl/ctlcmp2I
====================================================
- create Indexed version of ctlcmp2
for use by vtocr1 report utility
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 5X1. | unixcmpall - script using UNIX 'cmp' to compare all files in 2 subdirs |
| & create a report showing which files are different. |
| 5X2. | unixcmpsplit - script using UNIX 'cmp' to compare all files in 2 subdirs |
| & move non-matching files out to 2 other subdirs |
| 5X3. | uvcmp5all - script to execute uvcmp5 for every pair of files in the |
| 2 input directories |
| uvcmp5 |
|
The uvcopy jobs in this section are not listed here to save space. They are of course readily available in the /home/uvadm/pf/... directories and you can inspect them with the editor or list them as follows. Using uvcmp1 as an example:
cd /home/uvadm - change to uvadm home directory ============== vi pf/util/uvcmp5 - inspect uvcmp5 (or any uvcopy job desired) =================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# unixcmpall - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
# unixcmpall - use UNIX 'cmp' to compare corresponding files in 2 directories
# - very fast since it stops at the 1st miscompare (no -l option)
# - for large files run this 1st to determine which files are diff
# - then run uvcmp5 on the diff files to get the diff reports
#
# unixcmpall-l - alternate script with '-l' (long) option
# - do NOT use this, creates very long reports (1 byte per line)
# - use 'uvcmp5all' which shows 64 byte diffs per line
#
if [[ -d "$1" && -d "$2" ]]; then :
else echo "usage: unixcmpall dir1 dir2"
echo " ===================="
echo " - arg1 & arg2 must be directories"
exit 9; fi
#
x=0;y=0;
for i in $1/*
{ let x=x+1
f=${i##*/}
echo "file# $x cmp $1/$f $2/$f"
cmp $1/$f $2/$f >tmp/$f
#=============================
if (($? != 0))
then echo " mis-compares--->tmp/$f"
let y=y+1
fi
}
echo ">> $y mismatches of $x total files compared in $1 & $2"
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# unixcmpsplit - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
# unixcmpsplit - use UNIX 'cmp' to compare all files in 2 directories
# & move Non-Matching files out to 2 other directories
#
# - UNIX 'cmp' will quickly determine if files are same or different,
# but very inadequate for showing where & what the differences are.
#
# - We recommend running unixcmpsplit to select out Non-Matching files prior to
# running 'unixcmpall' which is very good at reporting the differences.
# - see complete documentation in CMPjobs.doc
#
if [[ -d "$1" && -d "$2" && -d "$3" && -d "$4" ]]; then :
else echo "usage: unixcmpsplit dir1 dir2 dir1X dir2X"
echo " =================================="
echo " - arg1,2,3,4 must be subdirs"
exit 9; fi
#
x=0;y=0;z=0;
for i in $1/*
{ let x=x+1
f=${i##*/}
echo "file# $x cmp $1/$f $2/$f"
cmp $1/$f $2/$f # compare current pair of files in dir1 & dir2
if (($? == 0))
let y=y+1
else
mv -i $1/$f $3 # move non-matching files to dir3 & dir4
mv -i $2/$f $4
let z=z+1
fi
}
echo ">> $y matching files in $x total compared in $1 & $2"
echo ">> $z Mis-Matching files moved to subdirs $3 & $4"
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# uvcmp5all - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
# uvcmp5all - compare all files in 2 directories with output to a 3rd
# - script calls 'uvcmp5' for each pair of files in 2 dirs
# - writes the mismatch reports to a 3rd directory
# - could run uvcmp5 manually for any pair of files, but this script saves
# a lot of manual entry when you have multiple files to compare.
#
#usage: uvcmp5all indir1 indir2 reportdir [ctlfile,options]
# ===================================================
#example:
# 1. cd /home/uvadm/tstcmp
# 2. mkdir dat1,dat2,rptcmp,ctl,etc, copy test data from uvadm/dat1, etc
# - see demo setup in COMPjobs.doc
#
# 3. uvcmp5all dat1 dat2 rptcmp uop=q0i7s3,filr4=ctlI/ctlcmp1
# ========================================================
#
# - uvcmp5 requires a 1 time preparation of a control file of information
# about the files at your site (filename,recsize,filetype,& ISAM keys)
# - Hard-code full pathname of the control file in the uvcmp5 job since it
# does not change often & saves you entering it on the command line
# - arg4 then used only for options & you can set your desired defaults
# in the script or in the uvcmp5 uvcopy job
# - you will probably default option q0 to inhibit prompts on each file
# & option i7 to inhibit extraneous console messages (leaving EOF stats)
# - you might want to modify some options from run to run, such as the
# print stop count (default s3) & the terminate count (default t5000)
# - be sure to use terminate count t99999999 for your final comparison
# tests (when most problems resolved) to ensure no mismatches ignored.
#
# The script command above would generate 1 uvcopy command for each file
# in the input#1 directory, for example:
#
# uvcopy uvcmp5,fili1=dat1/file1,fili2=dat2/file1,filo3=rptcmp/file1\
# ===================================================================
# ,filr4=ctlI/ctlcmp1,uop=q0i7s5
# ==============================
#
echo "compare files in dir $1 to dir $2, write mismatch reports to $3"
if [[ (-d "$1") && (-d "$2") && (-d "$3") ]]; then :
else echo "ERR usage: uvcmp5all indir1 indir2 outdir uop=q0i7"; exit 1; fi
x=0
for i in $1/*
do
f=${i##*/}
if [[ $f != *.idx ]]; then
uvcopy uvcmp5,fili1=$1/$f,fili2=$2/$f,filo3=$3/$f,uop=q0i7,$4
let x=x+1
fi
done
echo "$x files compared in $1 & $2, mismatch reports in $3 "
exit 0
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 6A1. | uvcmp6 - field by field comparison - Overview |
| - allows date/time stamps to be removed | |
| - allows comparing different size records (after Y2K expansion) | |
| - separate job per file, automatically generated from copybooks |
| 6B1. | Illustrate 'field by field' data file comparisons |
| - comparing files with same record sizes, but with date/time stamps. | |
| - sample/demo: data-file, copybook, generated compare job, mismatch- | |
| report, demo operating instructions, options. | |
| - Was used for Year2000 testing after DATE WINDOWING changes | |
| but could be used anytime you need to exclude some fields |
| 6C1. | Preparation to generate field by field compare jobs |
| - from 1 or 2 copybooks (2nd could have different layout) |
| 6D1. | use 'uvcmp6' to Generate field by field compare jobs 1 at a time |
| 6D2. | Execute generated field by field compare job |
| 6E1. | script 'genuvcmp6' to generate field by field compare job |
| 6F1. | File comparison demo for Year2000 19xx testing for DATE EXPANSION |
| - Compare files that have different record sizes due to Y2K expansions. | |
| or whenever record layouts are changed for any reason. | |
| - sample/demo: data-file, copybook, generated compare job, mismatch- | |
| report, demo operating-instructions. |
| 6G1. | Operating Instructions to generate comparison jobs for all copybooks |
| in a directory & all corresponding data files. | |
| 6G1. | Preparation for generating for all copybooks & data files. |
| 6G3. | Generating comparison jobs for all copybooks & all data files. |
| 6G4. | Alternative generation for duplicate fieldnames due to qualification |
| 6G5. | Sample comparison job fully completed with filenames & indexed keys. |
| 6G6. | Executing the fully generated jobs (1 at a time or all or by prefix). |
| 6G7. | Inspecting, printing,& using the mismatch reports. |
| 6I1. | Operating Summary & Recommendations |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 6J0. | Modifying generated jobs. |
| - summary of functions performed by called subroutines. |
| 6J1. | summary uvcmp6 - generates uvcopy comparison jobs from copybooks |
| - generated uvcopy jobs | |
| - uvcmp6.subs - supplied subrtns called by generated jobs |
| 6J2. | Alternate subrtn 'cmpp' vs 'cmp', to compare data files from the |
| - mainframe to data files created on UNIX. |
| 6J3. | Splitting combined fields to eliminate subfields |
| 6J4. | Generating 1 instruction per field (vs combining like fields) |
| 6K1. | Inserting record-type tests for files with redefined records |
| 6L1. | Corrections for occurs clauses with mixed data types |
| 6M1. | Exceptions, Problems,& Solutions |
1. options h,j,k to allow comparison where 1 file has an expanded date
within the 1st key (used to chain to the matching record in 2nd file)
2. Complement dates cause problems if present within the 1st key, which
is used to chain to the 2nd file.
Alternate versions of uvcmp1, uvcmp5,& uvcmp6.subs are provided
(renamed as uvcmp1X, uvcmp5X,& uvcmp6X.subs) for this problem.
| 6N1. | Alternative method to compare data files field by field, showing |
| corresponding COBOL copy-book field-names (uvhdcob & uvhddif2). | |
| Sample report illustrated here, complete doc in uvhdcob.doc of vol 1. |
| 6O1. | Applications |
| 1. Year2000 19xx testing (windows, expansion, or combinations) | |
| 2. Year2000 20xx testing (windows, expansion, or combinations) | |
| 3. Mainframe to UNIX conversions |
| 6X1. | Listings of the comparison scripts documented in this Part. |
| - uvcmp1, uvcmp5, uvcmp6 & uvcmp6.subs genuvcmp6 |
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Part_5 shows you how to automatically generate uvcopy jobs to compare data file records (field by field) based on the COBOL copybooks, tables of date-field-names,& a control file of data file names & related copybooks.
'uvcmp6' is the most important part of this process, but there are several other jobs involved which are documented in other sections of the Vancouver Utilities documentation.
cobmap1 - documented in COBaids.doc
- creates 'cobmaps' from COBOL copybooks, encoding the field
length, type, start,& end bytes on the right hand side.
- copies the entire copybook directory to an alternate directory
while adding the 'map' information.
reform1 - documented in DATAcnvt1.doc
- loads the old cobmap (copybook record layout) into an indexed
file, so uvcmp6 will have access to the old field locations
as well as the new copybook locations (primary input file).
*uvcmp6 - the HEART of the process, but the complete procedure depends on
several other pre-programmed jobs in the Vancouver Utilities.
- uvcmp6 generates a skeleton uvcopy for each pair of copybook/maps.
- the skeleton job will have 'bal' instructions to compare the
records field by field. The job contains tables of fieldnames
that should not be compared (date/time stamps, etc).
uvdata52 - documented in DATAcnvt1.doc
- uvdata52 will complete the skeletons created by uvcmp6, supplying
the actual data-file names & any indexed keys, from a control
file which relates the data-file names to the copybook names.
- A sample control file (uvadm/ctl/ctlcmp1) is supplied & you can
use this as a guide to create your version.
uvcopyxx - script supplied to execute all the generated uvcopy jobs
to increment the dates in multiple files. Or you can, of course,
execute them individually to convert one file at a time.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Assume we have modified our programs for Year2000 using 'windows' solutions, and we want to test our modified programs using 19xx data, to verify that we have not broken the programs while modifying for Year 2000. We might proceed as follows:
| Note |
|
12345 12345 sony television 970101 1-year 980101 981201091501 22222 11111 refrigerator 970101 2-year 990101 981201091502 33333 22222 dish washer 961231 3-year 991231 981201103003 33333 11111 refrigerator 970101 3-year 000101 981201103004 44444 33333 freezer 970101 4-year 010101 981201124505 55555 55555 clothes dryer 971231 5-year 021231 981201124506 66666 66666 blank/zero dates 6-year 000000 981201125507
cobmap1 start-end bytes for cobol record fields 199906061246 pg# 0001
cpy1/warmas6 RCSZ=0064 bgn-end lth typ
*warmas6 - warranty master record (warmas1 with date-time vs
05 wm-cust pic x(6). 000-0005 006
05 wm-prod pic x(6). 006-0011 006
05 wm-descrip pic x(18). 012-0029 018
05 wm-purch-date pic x(6). 030-0035 006
05 wm-policy pic x(8). 036-0043 008
05 wm-expiry.
10 wm-exp-year pic 9(2). 044-0045 002 n 02
10 wm-exp-month pic 9(2). 046-0047 002 n 02
10 wm-exp-day pic 9(2). 048-0049 002 n 02
05 filler001 pic x(1). 050-0050 001
05 wm-date-time pic x(12). 051-0062 012
05 filler002 pic x(1). 063-0063 001
*RCSZ=0064 0064
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The previous page showed a demo data file & corresponding copybook, that we are using to illustrate the 'uvcmp6' utility, which automatically generates a uvcopy job (interpretive code) to compare data files, field by field. This allows disabling fields that should not be compared (date/time stamps).
This also allows comparing records of different layouts, which will be illustrated in the next section (after date-field expansions).
opr='JOBNAME warmas6 - generated by cobmap1,uvcmp6,uvdata52'
rop=j64000k16000,uop=q1p30s3t1000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=FILEIN1,rcs=0064,typ=RSF
fili2=FILEIN2,rcs=0070,typ=RSF
filo3=rptcmp/FILEOUT,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0032d0006','wm-purch-date c0030d0008'
bal cmpc,'a0036b0008c0038d0008','wm-policy'
bal cmpc,'a0044b0002c0048d0002','wm-exp-year c0046d0004'
bal cmpc,'a0046b0004c0050d0004','wm-exp-month : wm-exp-day'
bal cmpc,'a0050b0001c0054d0001','filler001'
#2 bal cmpc,'a0051b0012c0057d0012','wm-date-time c0055d0014'
bal cmpc,'a0063b0001c0069d0001','filler002'
bal cmpd output records if any mismatches
skp loop repeat loop until EOF
# EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
Please see the sample report (data mismatches) on the next page --->
Note that this utility compares multiple contiguous fields of the same type with 1 'bal' instruction. The first & last fieldnames are shown in operand 3 on the right side (eg - 'wm-cust : wm-descript'). The copybook on the previous page shows you this includes wm-prod. The total length is coded as option 'b' (b0030) 30 bytes combined.
Uvcmp3 contains a table of fieldnames to force breaking contiguous compares. This table includes names such as 'date', 'year', etc & you can add to it.
A 2nd table is used to inhibit compares on fields such as date/time stamps. This table might include names such as 'time', 'stamp', etc.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Here is a sample report created by the uvcopy job shown on the previous page (which was automatically generated from the copybook).
Note that, we created some intentional mismatches by using the vi editor to change some data in one of the files (cmpdata/dat2/warmas6).
uvcmp6 - compare 2 files & "*" differences (uops=q0p30s3t1000v0) record-size count file-size typ filename file1: 64 7 448 RSF dat1/warmas6 file2: 64 7 448 RSF dat2/warmas6 1998/11/08_15:32:57 1 2 3 4 5 6 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123 ============================================================================
1 3 0 33333 22222 dish washer 961231 3-year 991231 981201103003.
******* * *
2 3 33333 22222 dish CRASHER 961231 2-year 981231 981202133333.
******* * *
1 6 0 55555 55555 clothes dryer 971231 5-year 021231 981201124506.
******** * *
2 6 55555 55555 clothes SHRINKER 971231 6-year 031231 981201162666.
******** * *
======================= EOF or stop count reached ========================== Total=000007, read=000007, mismatch=000002, stop=000003, term=000100
Please see the operating instructions for this demo on the next page --->
Note the '*'s marking the differences between the record pairs, and also note that there are no '*'s under bytes 51-62 (on the right hand side), even though these fields are clearly different.
This field is a date/time stamp which is excluded from the comparison by table #2 in the 'uvcmp6' generation job. It's fieldname is 'wm-date-time' & is excluded because it has a match on 'time' in table #2. Actually the 'bal' comparison instruction is generated, but #commented out, so you can reinstate, in case of table #2 unintended matches.
Note that the page headings give you the information you need to know about the file (rcsz, no of records, filesize, filenames, options in effect, date.
The EOF count statistics are vital, indicating whether the report shows you all the mismatches, or whether the stop count was reached. The stop count saves wasting a lot of paper; you probably want to fix the programs & rerun the program & the comparison jobs. You might set the stop count higher as your testing gets cleaner to ensure you compare to end of file.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The report below was created from the same pair of data files as was the report on the previous page, but we specified the *'v2'* option to force vertical hexadecimal record displays.
The default option is 'v0' which automatically determines whether to print char only or both char+hex, depending on the presence of packed/binary data.
You are prompted to change the default options when you run the compare job but if desired you can specify the option on the command line as follows: (Please see the detailed operating instructions on page '6D1' ahead)
uvcopy pfc2/warmas6,fili1=dat1/warmas6,fili2=dat2/warmas6
=========================================================
,filo3=rptcmp/warmas6x,uop=v2
=============================
uvcmp6 - compare 2 files & "*" differences (uops=q0p30s3t1000v0v2) record-size count file-size typ filename file1: 64 7 448 RSF dat1/warmas6 file2: 64 7 448 RSF dat2/warmas6 1998/11/08_15:30:09 1 2 3 4 5 6 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123 ============================================================================
1 3 0 33333 22222 dish washer 961231 3-year 991231 981201103003.
3333323333326676276766722222223333332327667233333323333333333330
33333022222049380713852000000096123103D951209912310981201103003A
******* * *
2 3 33333 22222 dish CRASHER 961231 2-year 981231 981202133333.
3333323333326676267676672222223333332327667233333323333333333330
33333022222049380321385200000096123102D951209812310981202133333A
******* * *
1 6 0 55555 55555 clothes dryer 971231 5-year 021231 981201124506.
3333323333326667667267767222223333332327667233333323333333333330
5555505555503CF48530429520000097123105D951200212310981201124506A
******** * *
2 6 55555 55555 clothes SHRINKER 971231 6-year 031231 981201162666.
3333323333326667667276766667223333332327667233333323333333333330
5555505555503CF485303829EB520097123106D951200312310981201162666A
******** * *
======================= EOF or stop count reached ========================== Total=000007, read=000007, mismatch=000002, stop=000003, term=000100
| NOTE |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Now you have data 'difference' reports, how do you find the problems ? You may want to investigate the records just before the differences occurred.
'uvhd' is the perfect tool for displaying the desired records, either by record# (shown on the difference reports, or by searching for patterns.
uvhd dat2/warmas6 r64 <-- initiate uvhd on dat/warmas6 with record size 64 =====================
dat2/warmas6 bytes=448 recs=7 mdate=1998110309 today=199811050647
r# 1 1 2 3 4 5 6
b# 0 0123456789012345678901234567890123456789012345678901234567890123
0 12345 12345 sony television 970101 1-year 980101 981201091111.
3333323333327667276667676662223333332327667233333323333333333330
1234501234503FE9045C56939FE00097010101D951209801010981201091111A
r# 2 1 2 3 4 5 6
b# 64 0123456789012345678901234567890123456789012345678901234567890123
0 22222 11111 refrigerator 970101 2-year 990101 981201091222.
3333323333327667666767672222223333332327667233333323333333333330
2222201111102562975214F200000097010102D951209901010981201091222A
r# 3 1 2 3 4 5 6
b# 128 0123456789012345678901234567890123456789012345678901234567890123
0 33333 22222 dish CRASHER 961231 2-year 981231 981202133333.
3333323333326676267676672222223333332327667233333323333333333330
33333022222049380321385200000096123102D951209812310981202133333A
fname=dat2/warmas6 fsz=448 rsz=64 recs=7 current#=3 fptr=128 null=next,r#=rec,b#=byte,+/-recs,s=search,u=update,p=print,w=write ,q=quit,?=help -->
When you start uvhd it displays the first record & prompts for your command.
s '961231' <-- search for 961231 anywhere in records s 30(6),'961231' <-- search for 961231 in bytes 30-35 of records
See detailed documentation for uvhd in uvhd.doc .
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
These demo Op. Instrns. generate compare jobs '1 FILE AT A TIME', BUT for production, I recommend the 'ALL FILES in DIRECTORY' procedures. See '6G1' + ahead --->
We will setup a test sub-directory (within your home directory), change into it and setup several sub-sub-directories. This will keep our home directory uncluttered & make it easier to clean up after the test/demo.
Sub-directories are made for the different types of files, so that we can keep the same name as the original copybook.
Multiple sub-directories works well if we have many files to be compared We can generate these jobs for an entire directory of COBOL copybooks, and their corresponding data files (see page '6G1' +).
This procedure provides for comparing data files before & after expansion (or any kind of conversion) by allowing you to specify 2 different copybooks (1 for the original file & 1 for the expanded or converted file).
In this case (19xx windows testing) the data files are the same, and we will specify the same copybook for the old file as for the new file. The next demo (expansion testing, page '6E1' +) will need 2 copybooks.
/home/userxx
:-----cmpdata <-- superdir for DATA comparisons
: :-----cpy1 - copybooks
: :-----cpy2 - copybooks for modified layouts (as in Y2K)
: :-----dat0 - EBCDIC files transferred from mainframe
- may need to convert variable to fixed in dat1
: :-----dat1 - data files for comparison to dat2/...
: :-----dat2 - data files 2nd set (same names as in dat1)
: :-----map1 - copybook layouts
: :-----map1I - copybook layouts loaded to Indexed files
: :-----map2 - modified layouts (as in Y2K) OR same as map1
: :-----pfc1 - uvcmp6 comparison jobs generated from layouts
: :-----pfc2 - comparison jobs copied here for any mods & execution
: :-----rptcmp - reports generated here
: :-----tmp - for misc outputs (keep working dir free of files)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
You may perform uvcmp6 data file comparisons within a subdirectory of your home directory (as illustrated above). We will refer to your userid as 'userxx' & will use 'cmpdata' as the superdir containing the several subdirs.
You must have setup your profile as documented in ADMjobs.htm#Part_1.
The profiles now include 'export CMPDATA=$HOME/cmpdata' & an alias 'cdk' to take you there.
#1. Login ---> /home/userxx <-- Login or change to your home directory
#2. mkdir cmpdata <-- make superdir for subdirs shown above
=============
#3. cd cmpdata <-- change into data comparison superdir
==========
#4. cmpdatadirs <-- run script to make subdirs shown above
===========
#5a. cp /home/uvadm/mf/cpys/warmas6 cpy1 - copy demo copybook to subdir
===================================
#5b. cp /home/uvadm/mf/cpys/warmas6 cpy2 - same copybook for new layouts
=================================== (diff for Y2K or any reformats)
#6a. cp /home/uvadm/dat1/warmas6 dat1 - copy test-data to original subdir
================================
#6a. cp /home/uvadm/dat1/warmas6 dat2 - copy test-data to modified subdir
================================
#7. vi dat2/warmas6 - create some differences to illustrate uvcmp6
=============== - don't modify record sizes since these are fixed length
The following demos generate 1 compare job at a time. For production we will generate jobs for all copybooks & all data files. The Instructions are shorter since you specify directories not filenames. See page '6G1' ahead --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
<----- see preparations on the previous page.
1. uvcopy cobmap1,fili1=cpy1/warmas6,filo1=map1/warmas6
====================================================
- run cobmap1 for the original copybook
2. uvcopy cobmap1,fili1=cpy2/warmas6,filo1=map2/warmas6
====================================================
- run cobmap1 for the new copybook
3. uvcopy reform1,fili1=map1/warmas6,filo1=map1I/warmas6
=====================================================
- load old map into an Indexed file
4. uvcopy uvcmp6,fili1=map2/warmas6,filr1=map1I/warmas6,filo1=pfc1/warmas6\
========================================================================
,arg2=e9999p30s3t9999v0 (arg2 defaults as shown OK)
=======================
- generate uvcopy job to compare files
6. cp pfc1/warmas6 pfc2 <-- copy job to alt subdir before modify/execute
==================== - to protect edits from inadvertent regen
7. vi pfc2/warmas6 - modify generated job if required
=============== - add record-type tests for redefined fields
- might change data-file-names
8. uvcopy pfc2/warmas6,fili1=dat1/warmas6,fili2=dat2/warmas6,filo3=rptcmp/warmas6
=============================================================================
- execute the job to compare the data file
8a. uvcopy pfc2/warmas6 <-- same as above (files default as shown)
=================== - OK if you changed datafilenames to match copybook
9a. vi rptcmp/warmas6 - inspect report for differences (* markers)
=================
9b. uvlp12 rptcmp/warmas6 - print report
=====================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Uvcmp3 has 'options' that affect behavior at generation time & 'arguments' that affect behavior of the generated jobs at execution time.
The arguments specified to the uvcmp6 generator are passed thru to the generated jobs as the 'default options' at execution time.
uop=q1g0c14 - option defaults
q1 - prompt operator for option changes (q0 inhibits)
g0 - combine contiguous same type fields into 1 compare
(breaks on fieldname matches to table date/time)
g1 - generate separate bal/cmpc instrns for each field
c__ - control action when field lth expanded
c14 - default compare leftmost bytes if new field > 14 bytes
default compare rightmost bytes if new field <= 14 bytes
c08 - change compare left/right decision to 8 bytes
c00 - would disable any change to generated dsplcmnts(lengths)
User OPtion (uop) defaults = q1g0c14
null to accept or re-specify (1 or more) -->
enter options for generated jobs (defaults=q1p30s3t1000v0) null to accept defaults, or respecify all options
| q1 |
|
| q0 |
|
uvcopy uvcmp6,fili1=map2/warmas6,filr1=map1I/warmas6,filo1=pfc1/warmas6\
========================================================================
,arg2=q1e9999p30s3t9999v0,uop=q1g0c14 <-- defaults no need to enter
=====================================
| Note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Page '6D1' presented the 'step by step' method of generating a uvcmp6 job to compare data files 'field by field'.
Alternatively, you can use the 'genuvcmp6' script to generate the uvcmp6 job with 1 command assuming the following:
3a. The copybook describing the 1st data file has been stored in cpy1/...
3b. The copybook describing the 2nd data file has been stored in cpy2/...
- will be same as the 1st if no layout differences exist
4a. The 1st data file is stored in dat1/...
4b. The 2nd data file is stored in dat2/...
6. genuvcmp6 copybookname <-- call script to generate uvcmp6 job
====================== - generated job stored in pfc1/...
6a. genuvcmp6 warmas6 <-- demo with test file used on page '6D1'
=================
7. cp pfc1/warmas6 pfc2 <-- copy to alt subdir before edit/execute
==================== - protects edits in case inadvertent regen
8. vi pfc2/warmas6 <-- modify generated job if required
=============== - might change data-file-names ?
9. uvcopy pfc2/warmas6 <-- execute the file comparison job
===================
10a. vi rptcmp/warmas6 <-- inspect the comparison report
=================
10b. vi rptcmp/warmas6 <-- print the comparison report
=================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Assume we have modified our programs for Year2000 using DATE EXPANSION, and we want to test our modified programs using 19xx data, to verify that we have not broken the programs while modifying for Year 2000. We might proceed as follows:
| Note |
|
12345 12345 sony television 970101 1-year 980101 981201091501 22222 11111 refrigerator 970101 2-year 990101 981201091502 33333 22222 dish washer 961231 3-year 991231 981201103003 33333 11111 refrigerator 970101 3-year 000101 981201103004 44444 33333 freezer 970101 4-year 010101 981201124505 55555 55555 clothes dryer 971231 5-year 021231 981201124506 66666 66666 blank/zero dates 6-year 000000 981201125507
12345 12345 sony television 19970101 1-year 19980101 19981201091501 22222 11111 refrigerator 19970101 2-year 19990101 19981201091502 33333 22222 dish CRASHER 19961231 3-year 19991231 19981201103003 33333 11111 refrigerator 19970101 3-year 20000101 19981201103004 44444 33333 freezer 19970101 4-year 20010101 19981201124505 55555 55555 clothes SHRINKER 19971231 5-year 20021231 19981201124506 66666 66666 blank/zero dates 6-year 20000000 19981201125507 77777 77777 extra rec file2 19980101 7-year 20050101 19980101164559XX
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
cobmap1 start-end bytes for cobol record fields 199906061246 pg# 0001
cpy1/warmas6 RCSZ=0064 bgn-end lth typ
*warmas6 - warranty master record (warmas1 with date-time vs
05 wm-cust pic x(6). 000-0005 006
05 wm-prod pic x(6). 006-0011 006
05 wm-descrip pic x(18). 012-0029 018
05 wm-purch-date pic x(6). 030-0035 006
05 wm-policy pic x(8). 036-0043 008
05 wm-expiry.
10 wm-exp-year pic 9(2). 044-0045 002 n 02
10 wm-exp-month pic 9(2). 046-0047 002 n 02
10 wm-exp-day pic 9(2). 048-0049 002 n 02
05 filler001 pic x(1). 050-0050 001
05 wm-date-time pic x(12). 051-0062 012
05 filler002 pic x(1). 063-0063 001
*RCSZ=0064 0064
cobmap1 start-end bytes for cobol record fields 199906061246 pg# 0001
cpy2/warmas6 RCSZ=0070 bgn-end lth typ
*warmas6x - warmas6 expanded date,year,date-time stamp
05 wm-cust pic x(6). 000-0005 006
05 wm-prod pic x(6). 006-0011 006
05 wm-descrip pic x(18). 012-0029 018
05 wm-purch-date pic x(8). <-- 8 bytes vs 6 030-0037 008
05 wm-policy pic x(8). 038-0045 008
05 wm-expiry.
10 wm-exp-year pic 9(4). <-- 4 bytes vs 2 046-0049 004 n 04
10 wm-exp-month pic 9(2). 050-0051 002 n 02
10 wm-exp-day pic 9(2). 052-0053 002 n 02
05 filler001 pic x(1). 054-0054 001
05 wm-date-time pic x(14). <-- 14 bytes vs 12 055-0068 014
05 filler002 pic x(1). 069-0069 001
*RCSZ=0070 0070
The records have been expanded from 64 bytes to 70 bytes, since there were 3 date fields expanded by 2 bytes each.
This forces the record display onto 2 lines, since records are displayed in 64 byte segments.
See the sample report 2 pages ahead --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The previous pages showed demo data files & corresponding copybooks, that we are using to illustrate the 'uvcmp6' utility, which automatically generates a uvcopy job (interpretive code) to compare data files, field by field. This allows disabling fields that should not be compared (date/time stamps).
This also allows comparing records of different layouts, which is required due to date field expansion. This is illustrated on the next page --->
opr='JOBNAME warmas6 - generated by cobmap1,uvcmp6,uvdata52'
rop=j64000k16000,uop=q1p30s3t1000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=FILEIN1,rcs=0064,typ=RSF
fili2=FILEIN2,rcs=0070,typ=RSF
filo3=rptcmp/FILEOUT,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0032d0006','wm-purch-date c0030d0008'
bal cmpc,'a0036b0008c0038d0008','wm-policy'
bal cmpc,'a0044b0002c0048d0002','wm-exp-year c0046d0004'
bal cmpc,'a0046b0004c0050d0004','wm-exp-month : wm-exp-day'
bal cmpc,'a0050b0001c0054d0001','filler001'
#2 bal cmpc,'a0051b0012c0057d0012','wm-date-time c0055d0014'
bal cmpc,'a0063b0001c0069d0001','filler002'
bal cmpd output records if any mismatches
skp loop repeat loop until EOF
# EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
Please see the sample report (data mismatches) on the next page --->
Note that this utility compares multiple contiguous fields of the same type with 1 'bal' instruction. The first & last fieldnames are shown in operand 3 on the right side (eg - 'wm-cust : wm-descript'). The copybook on the previous page shows you this includes wm-prod. The total length is coded as option 'b' (b0030) 30 bytes combined.
Uvcmp3 contains a table of fieldnames to force breaking contiguous compares. This table includes names such as 'date', 'year', etc & you can add to it.
A 2nd table is used to inhibit compares on fields such as date/time stamps. This table might include names such as 'time', 'stamp', etc.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Here is a sample report created by the uvcopy job shown on the previous page (which was automatically generated from the copybook). Note that, we created some intentional mismatches by using the vi editor to change some data in one of the files (cmpdata/dat2/warmas6).
uvcmp6 - compare 2 files & "*" differences (uops=q0p30s3t1000v0) record-size count file-size typ filename file1: 64 7 448 RSF dat1/warmas6 file2: 70 8* 561 RSF dat2/warmas6 1998/11/08_15:32:57 1 2 3 4 5 6 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123 ============================================================================
1 3 0 33333 22222 dish washer 961231 3-year 991231 981201103003.
******* * *
2 3 33333 22222 dish CRASHER 19961231 2-year 19981231 199812333
******* * *
64 33333.
1 6 0 55555 55555 clothes dryer 971231 5-year 021231 981201124506.
******** * *
2 6 55555 55555 clothes SHRINKER 19971231 6-year 20031231 199812666
******** * *
64 66666.
1 8 0 ~EOF~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*************************************************** *
2 8 77777 77777 extra rec file2 19981105 7-year 20000000 BADRECORD
****************************** ************** *******
64 LENGTH
*
======================= EOF or stop count reached ==========================
Total=000007, read=000009, mismatch=000004, stop=000003, term=000100
Please see the operating instructions for this demo on the next page --->
Since the records have been expanded to 70 bytes, it now requires 2 lines to display records from file #2 (records are displayed in 64 byte segments). Note that file2 has an extra record after EOF already reached in file2. If the extra record occurs any sooner we would have mismatches from there on.
The extra record also has wrong length (not 70 bytes like others in file2). This causes the '*' flag beside the file2 count - see '8*' in page headings. The count is flagged when filesize/recsize is not evenly divisible. For ISAM files the extra byte (delete control byte) is allowed for by adding 1 to the recsize before division.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The following files are supplied to demo uvcmp6 on different record sizes:
uvadm/dat1/warmas6 - demo data-file supplied in 'dat1' directory
uvadm/dat1/warmas6x - same data file after dates expanded
uvadm/mf/cpys/warmas6 - demo copy-book for the warmas6 data file above uvadm/mf/cpys/warmas6x - demo copy-book for the expanded warmas6x above
3a. cp /home/uvadm/mf/cpys/warmas6 cpy1
===================================
- copy demo copybook for original file to cpy1 subdir
3b. cp /home/uvadm/mf/cpys/warmas6x cpy2/warmas6
============================================
- change name of expanded copy-book in cpy2 to match cpy1
4a. cp /home/uvadm/dat1/warmas6 dat1
================================
- copy original test-data to dat1 subdir
4b. cp /home/uvadm/dat1/warmas6x dat2/warmas6
=========================================
- change name of expanded data-file in dat2 to match dat1
See test/demo job executions on the next page --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| NOTE |
|
<----- see preparations on the previous page.
1. uvcopy cobmap1,fili1=cpy1/warmas6,filo1=map1/warmas6
====================================================
- run cobmap1 for the original copybook
2. uvcopy cobmap1,fili1=cpy2/warmas6,filo1=map2/warmas6
====================================================
- run cobmap1 for the new copybook
3. uvcopy reform1,fili1=map1/warmas6,filo1=map1I/warmas6
=====================================================
- load old map into an Indexed file
4. uvcopy uvcmp6,fili1=map2/warmas6,filr1=map1I/warmas6,filo1=pfc1/warmas6\
========================================================================
,arg2=e9999p30s3t9999v0 <-- defaults, no need to enter
=======================
- generate uvcopy job to compare files
5a. uvcopy uvdata52 - optional for demo (see pages '6G1'+)
- definitely use for production (serious work)
- copies jobs from pfc1 to pfc2 inserting datafilenames
using a control file to relate to copybooknames
- - - - OR - - - - <-- use 5b. if you do not use 5a.
5b. cp pfc1/warmas6 pfc2 <-- copy job to alt subdir before modify/execute
==================== - to protect edits from inadvertent regen
6. vi pfc2/warmas6 - modify generated job if required
=============== - add record-type tests for redefined fields
- might change data-file-names
7. uvcopy pfc2/warmas6,fili1=dat1/warmas6,fili2=dat2/warmas6,filo3=rptcmp/warmas6
============================================================================
- execute the job to compare the data file
7a. uvcopy pfc2/warmas6 <-- same as above (files default as shown)
=================== - OK if data-file-names match copy-book-names
8a. vi rptcmp/warmas6 - inspect report for differences (* markers)
=================
8b. uvlp12 rptcmp/warmas6 - print report
=====================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The previous page suggested executing the generated compare job as follows:
uvcopy pfc2/warmas6,fili1=dat1/warmas6,fili2=dat2/warmas6,filo3=rptcmp/warmas6 ==============================================================================
Be patient, we will eventually show you a better way begining on page '6G1'.
This long command is necessary at this point because the jobs in 'pfc2' have been generated using only the information in the copybooks, which does not include the actual data-file-names or Indexed key info.
If desired, you could edit the correct data filenames into the pfc2/jobs (replacing the generic FILEIN & FILEOUT), then you could run as follows:
uvcopy pfc2/warmas6 ===================
The next section '6G1' + provides a better solution, using a control file to relate the actual data-file-names & indexed key info to the copy-book info. The following illustrates 2 lines from the control file used in section '6G1'+.
# ctlcmp1 - control file to generate complete jobs from skeletons # - used by uvdata52 to encode data-file-names & ISAM keys warmas6 cpy=warmas6 rcs=00064 warmas8 cpy=warmas8 rcs=00064 keys=(0,6,nn)
An additional job 'uvdata52' is provided to read the control file & for each data-file, randomly retrieve the corresponding skeleton uvcopy (previously generated from the copybook & now in the pfc1 subdir).
The actual filenames & any ISAM key info will be inserted into the skeleton jobs from pfc1 to create complete uvcopy jobs in the pfc2 subdir.
It is also important to realize that 1 copy-book could represent the data file record layouts for several data-files. The control file will have 1 line for each data-file, but the total number could be more than the total number of copy-books. In fact it is usually less because there are many other copy-books that do not represent any data-file.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Previous operating instructions were demos for comparing 1 file at a time, but for industrial strength serious work, we want to generate the compare jobs for all copy-books & all data-files.
If you are ready real work, you will use your own directories of copybooks & data-files, but if you are not yet confident in using these techniques, I suggest you use first try these procedures using the supplied test files.
1a. cd $HOME - change to your home directory 1b. mkdir cmpdata - make a sub-directory for the test/demo 1c. cd cmpdata - change into it
2a. mkdir cpy1 - for original copybooks 2b. mkdir cpy2 - for expanded copybooks 2c. mkdir map1 - for original copybook 'cobmaps' (record layouts) 2d. mkdir map2 - for new copybook cobmaps 2e. mkdir map1I - for original cobmaps loaded into indexed files 2f. mkdir pfc1 - for generated uvcopy jobs (to compare files) 2g. mkdir pfc2 - for uvcopy jobs (with filenames & indexed keys) 2h. mkdir rptcmp - for mismatch reports 2i. mkdir ctl - for control file (relate copybook to datafiles) 2j. mkdir dat1 - for test data file inputs 2k. mkdir dat2 - for test data file outputs 2i. mkdir tmp - subdir for misc outputs (keep working dir clean)
3a. cp /home/uvadm/mf/cpys/warmas* cpy1
====================================
- copy all warmas* copybooks to cmpdata/cpy1
(warmas1,warmas1x,warmas3,warmas6,warmas6x,warmas8,warmas8x)
3b. move the 'x' (expanded versions) to cpy2, renaming (removing the 'x')
to match the names in cpy1.
3b1. mv cpy1/warmas1x cpy2/warmas1
3b2. mv cpy1/warmas6x cpy2/warmas6
3b3. mv cpy1/warmas8x cpy2/warmas8
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
4a. cp /home/uvadm/dat1/warmas* dat1 - copy warmas* datafiles to cmpdata/dat1
================================
4b. move the 'x' (expanded versions) to dat2, renaming (removing the 'x')
to match the names in dat1.
4b1. mv dat1/warmas1x dat2/warmas1
4b2. mv dat1/warmas6x dat2/warmas6
4b3. mv dat1/warmas8x dat2/warmas8
4b4. mv dat1/warmas8x.idx dat2/warmas8.idx
| Note |
|
uvcp "fili1=dat2/warmas6,rcs=70,typ=RSF,filo1=dat2/warmas8,typ=ISF,isk1=0(6)" =============================================================================
5a. cp /home/uvadm/ctl/ctlcmp1 ctl - copy control file to cmpdata subdir
5b. vi ctl/ctlcmp1 - modify (remove extraneous files)
==============
# ctlcmp1 - control file to generate complete jobs from skeletons # - used by uvdata52 to encode data-file-names & ISAM keys # warmas1 cpy=warmas1 rcs=00064 warmas3 cpy=warmas3 rcs=00064 warmas6 cpy=warmas6 rcs=00064 warmas8 cpy=warmas8 rcs=00064 keys=(0,6,nn)
The control file is input to the 'uvdata52' step (see next page), which completes the skeleton compare jobs created by 'uvcmp6', by inserting the actual data filenames & any indexed file keys.
Preparing the control file is the only non-automatic part of this project. An accurate control file is vital to your file comparison success.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
<--- this assumes you have completed the preparations on the previous page.
1. uvcopyx cobmap1 cpy1 map1 uop=q0p0i7
====================================
- convert original copybooks into cobmap1 layouts
2. uvcopyx cobmap1 cpy2 map2 uop=q0p0i7
====================================
- convert modified copybooks into cobmap1 layouts
3. uvcopyx reform1 map1 map1I uop=q0i7
===================================
- convert original cobmap1 into ISAM files (for reform2)
4. uvcopyxr uvcmp6 map2 pfc1 map1I uop=q0i7g0c14,arg2=q1e9999p30s3t9999v0
======================================================================
- generate uvcopy jobs to compare data-files
5. uvcopy uvdata52,fili1=ctl/ctlcmp1,fild2=pfc1,fild3=pfc2\
========================================================
5a. ,arg1=/u2/prod/dat1,arg2=/u2/prod/dat2 <- paths to data-files
======================================
5b. ,arg1=${UVDATA1},arg2=${UVDATA2} <- alternative (see note)
================================
The fully generated jobs in subdir 'pfc2' are renamed for the datafiles, rather than the copybooks as were the skeleton jobs in 'pfc1'. This makes sense because there could be multiple data-files for 1 copybook.
See a "Sample Fully Generated Job" on the next page ---> See the "Execution Operating Instructions" 2 pages ahead --->
uvcopy pfc2/warmas6 - execute job any 1 job to compare 2 files uvcopyxx 'pfc2/*' - execute all compare jobs in pfc2 directory
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
These procedures use 'reform1a' (vs 'reform1') & include an extra step 'reform1b'. An extra directory is required 'map2a'.
1. uvcopyx cobmap1 cpy1 map1 uop=q0p0i7
====================================
- convert original copybooks into cobmap1 layouts
2. uvcopyx cobmap1 cpy2 map2 uop=q0p0i7
====================================
- convert modified copybooks into cobmap1 layouts
3. uvcopyx reform1a map1 map1I uop=q0i7 <-- reform1a vs reform1
====================================
- convert original cobmap1 into ISAM files (for reform2)
3a. uvcopyxr reform1b map2 map2a map1I uop=q0i7 <-- reform1b creates map2a
===========================================
4. uvcopyxr uvcmp6 map2a pfc1 map1I uop=q0i7g0c14,arg2=q1e9999p30s3t9999v0
=======================================================================
- generate uvcopy jobs to compare data-files
| Note |
|
5. uvcopy uvdata52,fili1=ctl/ctlcmp1,fild2=pfc1,fild3=pfc2
======================================================
| reform1a |
|
| reform1b |
|
| note |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The demo jobs generated on pages '6D1' + were incomplete, and filenames were supplied on the command line at execution time. This is OK for a demo, but not good if you have 100 file compare jobs to run.
The comparison jobs created in this section '6G1' + (see prior page) are complete, including actual data filenames & indexed keys if any. This is achieved by editing the 'control file' (see page '6G2'), and running an extra step 'uvdata52' which is driven by the control file to relate the copybook names to the actual datafile names.
This makes the jobs easier to run, just enter: 'uvcopy + jobname'.
uvcopy pfc2/warmas8 <--- command to execute fully completed comparison job ===================
opr='warmas8 warmas8 - generated by cobmap1,uvcmp6,uvdata52'
rop=j64000k16000,uop=q1p30s3t1000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=${UVDATA1}/warmas8,rcs=0064,typ=ISFl1,isks=(0,6n)
fili2=${UVDATA2}/warmas8,rcs=0070,typ=ISFl1,isks=(0,6n)
filo3=rptcmp/warmas8,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0032d0006','wm-purch-date c0030d0008'
bal cmpc,'a0036b0008c0038d0008','wm-policy'
bal cmpc,'a0000b0000c0048d0000','wm-exp-cent c0046d0002'
bal cmpc,'a0044b0002c0048d0002','wm-exp-year'
bal cmpc,'a0046b0004c0050d0004','wm-exp-month : wm-exp-day'
bal cmpc,'a0050b0001c0054d0001','filler001'
#2 bal cmpc,'a0051b0012c0057d0012','wm-date-time c0055d0014'
bal cmpc,'a0063b0001c0069d0001','filler002'
bal cmpd output records if any mismatches
skp loop repeat loop until EOF
# EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
Compare this fully completed job to the demo job on page '6E3' and note:
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
To be completely flexible we should be able to execute our comparison jobs from anywhere. This can be achieved by environmental variables.
When you ran 'uvdata52' you were prompted for pathnames to the 2 sets of data files to be compared,& we suggested that you specify ${UVDATA1} & ${UVDATA2}. You would then add exports to your .profile, for example:
export UVDATA1=$HOME/cmpdata/dat1; export UVDATA2=$HOME/cmpdata/dat2 ====================================================================
Then during testing, you could modify the pathnames simply by exporting different values at your shell prompt (in effect until you logoff).
The other problem to be resolved is 'How does the uvcopy interpreter know where to find our generated comparison jobs ?'
Uvcopy provides environmental variable PFPATH to be searched. The provided profiles (see ADMjobs.htm#Part_1) includes the following:
export UV=/home/uvadm
export PFPATH=$UV/pf/adm:$UV/pf/demo:$UV/pf/IBM:$UV/pf/util ===========================================================
If you have generated the comparison jobs in $HOME/cmpdata/pfc2, you could include that subdir in the PFPATH as follows:
export PFPATH=$PFPATH:$HOME/cmpdata/pfc2 ========================================
Simply enter 'uvcopy' (the interpreter) followed by the name of the parameter file (interpretive code / uvcopy job), which matches the name of the data file, for example:
uvcopy custmast - execute job to compare the pair of custmast files =============== (cmpdata/dat1/custmast & cmpdata/dat2/custmast)
'uvcopyxx' is a script which will repeatedly execute the uvcopy interpreter for all parameter files found in the specified directory. uvcopyxx allows for pattern matching for, but you must specify the full pathname:
uvcopyxx 'pfc2/*' - execute all compare jobs in pfc2 directory =============== (assuming you are in $HOME/cmpdata)
uvcopyxx '$HOME/cmpdata/pfc2/gl*' - execute all jobs for the GL system ================================= (regardless of current directory)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The mismatch reports will be created in subdir 'rptcmp' within whatever directory you are in when you run the comparison jobs. I recommend you clean it out before you begin each new series of comparisons.
vi rptcmp/gl.master -inspect mismatch report for the gl.master file ===================
You can inspect the reports with vi, but I recommend printing them out, so you have your terminal free to inspect the data files (with uvhd) using the hard-copy mismatch reports as your guide.
uvlpd12 rptcmp - print all mismatch reports ==============
You will find uvhd an invaluable tool for inspecting your data files especially if you have packed/binary files without linefeeds. See details in uvhd.htm.
uvhd dat1/warmas6 r64 - investigate problem file original =====================
uvhd dat2/warmas6 r70 - investigate problem file expanded =====================
With uvhd you can inspect any record in the file, for example you might want to see the data records just prior to a mismatch report difference. You can search for any desired pattern (& restrict the search to specified columns).
You can also print out any desired records in hex similar to mismatch reports. For example, when you records you want to print, use the immediate print:
i3 <-- print the next 3 records ==
See the sample uvhd printout on page '6B5'. The printout will go to the laser printer configured in the 'uvlp12' script. See the install.doc guide in volume 1 or see /home/uvadm/sf/util/uvlp12.
uvcopy vtocr1,fild1=dat1,filr1=ctl/ctlcmp1I --> uvlp12 <- print at 12 cpi ===========================================
uvcopy vtocr1,fild1=dat2,filr1=ctl/ctlcmp2I --> uvlp12 <- print at 12 cpi ===========================================
See sample VTOC reports on page '5G1'.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
I recommend you initially work your way thru the 1 file at a time demos begining on page '6B1', to get familiar with these procedures.
First run the demos with the demo copybooks & data files supplied in ~uvadm, then try them with some of your own copybooks & data files. For these you, can use the 'gencmp3a' script (see page '6X3') to automatically perform the multiple steps required to generate the comparison job.
If you only have a few data files to compare you can stick with gencmp3a.
If you have more than a few data files to compare, I recommend using the operating instructions begining on page '6G1' to generate complete jobs for all copybooks & all data files.
The instructions for ALL are in fact shorter than for 1 at a time & it is very fast to generate the uvcopy comparison jobs for hundreds of files.
The only manual editing required is to prepare the control file which relates the copybook names to the actual data file names, record sizes,& indexed file keys (if applicable).
Generating the comparison jobs for ALL files is very convenient , because you can then run comparisons whenever required during your testing.
For your first mass generation, I recommend you follow the step by step instructions begining on page '6G1'. The 'gencmp3b' script might be used if you have to subsequently regenerate all jobs due to changes in copybooks. If you are only changing the control file, you could simply run the 'uvdata52' job (which is the last step of generation).
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 6J1. | summary of functions performed by called subroutines. |
| 6J2. | alternate subrtn 'cmpp' vs 'cmp. This allows you to compare data files |
| from the mainframe to data files created on UNIX. |
6J3. To split combined fields to eliminate subfields, not to be compared
- The generation job contains a table of fieldnames that force separate
'bal' compare instructions. This table contains date,year,yy, etc.
- A 2nd table contains patterns that cause the 'bal' instruction to be
#commented out (date/time stamps, that will not compare).
- You can modify these tables (within ~uvadm/pf/util/uvcmp6), but it will
never be perfect & you may want to split combined fields, or modify code
that compares different length fields due to date expansion.
| 6J4. | Option 'g1' could have been used to generate separate instructions for |
| each copybook/data field, but you might not want to use it because the | |
| generated jobs would be much longer, especially for big copybooks with | |
| hundreds of fields. |
6K1. Inserting record-type tests for files with redefined records.
- Some files have multiple record layouts depending on record type codes.
The code to compare the multiple sets of record fields will have been
generated, but you need to insert code to test the record type & skip
to the proper set of generated bal instructions.
6L1. Corrections for occurs clauses with mixed data types.
- Under these circumstances uvcmp6 will generate code only for the 1st
set of occurs fields. You may have to insert a code loop to test
all fields in the occurs set.
Please see these modifications illustrated on the following pages --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
'uvcmp6' generates a worker uvcopy job for each pair of copybook/maps. The generated jobs have 'bal' instructions to compare the records field by field.
The 'bal's (branch & link) to subroutines to perform the various functions required in the comparison job. These subrtns are loaded at execution time from ~uvadm/pf/util/uvcmp6.subs (stored in Vancouver Utility home directory).
This is advantageous since it keeps the comparison worker jobs to the absolute minimum no of lines & allows any changes to the common subrtns to be made once to the master copy.
There are 5 subrtns called by the comparison jobs:
3a. cmpp - alternate subrtn 'cmpp' vs 'cmp', to compare data files from
the mainframe to data files created on UNIX.
These subrtns are stored in /home/uvadm/pf/util/uvcmp6.subs. You probably wont have to change them, since you can usually achieve the desired effect by modifying the worker jobs that call these subrtns.
If you do require changes, please call UV Software for assistance.
The uvcmp6 master generation job is not listed here, but is available in /home/uvadm/pf/util/uvcmp6. This of course is more complex and call UV Software if you think you need to change something.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
This allows you to compare data files from the mainframe to data files created on UNIX. Mainframes generated x'_C' signs in arithmetic operations, but Micro Focus COBOL generates C signs for comp-3 signed, and F signs for comp-3 unsigned.
Subrtn cmpp (alternate to cmpc) considers packed field signs C or F to match equal. Bytes with either x'_C' or x'_F' are assumed to be part of packed fields & match equal if the units digit matches. Uvcmp3 always generates 'bal cmpc' for all fields, and you can simply change these to 'bal cmpp' if you wish packed signs C/F to be considered a match.
For example, demo file custmas1 has 12 * 5 = 60 bytes packed fields for this year & last year sales, and generates the following bal's:
bal cmpc,'a0000b0006c0000d0006','cm-num'
bal cmpc,'a0006b0114c0006d0114','cm-delete : cm-contact'
bal cmpc,'a0120b0060c0120d0060','cm-thisyr-sales occur 12'
bal cmpc,'a0180b0060c0180d0060','cm-lastyr-sales occur 12'
bal cmpc,'a0250b0006c0250d0006','filler003'
You can simply change the bal's for the packed fields as follows:
bal cmpp,'a0120b0060c0120d0060','cm-thisyr-sales occur 12'
bal cmpp,'a0180b0060c0180d0060','cm-lastyr-sales occur 12'
Actually, you can change all the 'cmpc's to 'cmpp's. It will run a little slower, but you will save a lot of time not having to determine which 'bal's have packed fields.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Here is just the 'bal' compare instructions extracted from the complete job, that was listed on page '6E3'.
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0032d0006','wm-purch-date c0030d0008'
bal cmpc,'a0036b0008c0038d0008','wm-policy'
bal cmpc,'a0044b0002c0048d0002','wm-exp-year c0046d0004'
bal cmpc,'a0046b0004c0050d0004','wm-exp-month : wm-exp-day'
bal cmpc,'a0050b0001c0054d0001','filler001'
#2 bal cmpc,'a0051b0012c0057d0012','wm-date-time c0055d0014'
bal cmpc,'a0063b0001c0069d0001','filler002'
Referring to the copybook (listed on page '6E2') you can see that the first instruction compares 3 contiguous fields for a total of 30 bytes.
05 wm-cust pic x(6). 006 000-0005
05 wm-prod pic x(6). 006 006-0011
05 wm-descrip pic x(18). 018 012-0029
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a____b____c____d____','1stfield : lastfield'
---file1--|--file2--
'-dspl-lth-|-dspl-lth'
The above should clarify the coding which uses option letters a,b,c,d. (note that displacements are zero relative)
option 'a' = displacement to first byte of current field in file #1 option 'b' = length of current field in file #1 option 'c' = displacement to first byte of current field in file #2 option 'b' = length of current field in file #2
| EXAMPLE |
|
| Solution |
|
bal cmpc,'a0000b0006c0000d0006','wm-cust'
### bal cmpc,'a0006b0006c0006d0006','wm-prod'
bal cmpc,'a0012b0018c0012d0018','wm-descrip'
Wm-prod has been commented out (using ### for emphasis). Of course you could simply omit it, but #commenting out allows easy future changes.
OR you could have generated 1 instruction per field (via option g1). See illustration on the next page --->
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
g0 - combine contiguous same type fields into 1 compare (default)
(breaks on fieldname matches to table date/time)
g1 - generate separate bal/cmpc instrns for each field
Option 'g0' is the default & you can see the resulting separate instructions on page '6B2'. If you specified *'g1'* the results would be as follows:
opr='JOBNAME warmas6 - generated by cobmap1,uvcmp6,uvdata52'
rop=j64000k16000,uop=q1p30s3t1000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=FILEIN1,rcs=0064,typ=RSF
fili2=FILEIN2,rcs=0070,typ=RSF
filo3=rptcmp/FILEOUT,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
bal cmpc,'a0000b0006c0000d0006','wm-cust'
bal cmpc,'a0006b0006c0006d0006','wm-prod'
bal cmpc,'a0012b0018c0012d0018','wm-descrip'
bal cmpc,'a0030b0006c0032d0006','wm-purch-date c0030d0008'
bal cmpc,'a0036b0008c0038d0008','wm-policy'
bal cmpc,'a0044b0002c0048d0002','wm-exp-year c0046d0004'
bal cmpc,'a0046b0002c0050d0002','wm-exp-month'
bal cmpc,'a0048b0002c0052d0002','wm-exp-day'
bal cmpc,'a0050b0001c0054d0001','filler001'
#2 bal cmpc,'a0051b0012c0057d0012','wm-date-time c0055d0014'
bal cmpc,'a0063b0001c0069d0001','filler002'
bal cmpd output records if any mismatches
skp loop repeat loop until EOF
# EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
In many cases, you will not have to modify the compare jobs, for files with multiple record types. The compare jobs are less likely to need modifications than are the other date processing jobs (REFORMjobs, AGEjobs, & VALjobs). This is because we can usually compare the entire record and we do not have to process the dates individually as for the other jobs. Even if there are multiple record types, we can usually compare the entire record & not have to care that some have dates & some do not.
There are 2 situations where we would need to test record types:
We will illustrate the 2nd situation on the next few pages using the warmas3 file, which has 2 record types (dtl & note). Let's pretend that the expiry date is a time stamp & cannot be compared whereas the entire note record should be compared.
12345 12345 sony television 970101 1-year 980101 000002500 12345 12345*warmas3 alternate warmas file, demo multi rec types 22222 11111 refrigerator 970101 2-year 990101 000004500 22222 22222 dish washer 971231 2-year 991231 000004500 22222 22222*rectype col 12, blank for detail, '*' for note recs 33333 11111 refrigerator 970101 3-year 000101 000006500 33333 11111*special 3 yr warranty to John Smith GE refrigerator
cobmap1 start-end bytes for cobol record fields 199909151419 pg# 0001
cpy1/warmas3 wm-rec RCSZ=0064 bgn-end lth typ
01 wm-rec.
05 wm-cust pic x(6). 000-0005 006
05 wm-prod pic x(5). 006-0010 005
05 wm-type pic x(1). <-- Record Type 011-0011 001
05 wm-dtl.
10 wm-descrip pic x(18). 012-0029 018
10 wm-purch-date pic x(6). 030-0035 006
10 wm-policy pic x(8). 036-0043 008
10 wm-expiry_date pic x(6). <-- time-stamp ? 044-0049 006
10 filler001 pic x(1). 050-0050 001
10 wm-paid-amt pic 9(9). 051-0059 009 n 09
10 filler002 pic x(4). 060-0063 004
05 wm-note redefines wm-dtl.
10 wm-comment pic x(52). 012-0063 052
*RCSZ=0064 0064
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
opr='warmas3 warmas3 - generated by cobmap,uvcmp6,uvdata52'
arg1=warmas3,uop=q1p30s3t10000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=${UVDATA1}/warmas3,rcs=0064,typ=RSF
fili2=${UVDATA2}/warmas3,rcs=0064,typ=RSF
filo3=${CMPRPTS}/warmas3,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
# --- <-- R/T tests required ?
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0030d0006','wm-purch-date'
bal cmpc,'a0036b0008c0036d0008','wm-policy'
bal cmpc,'a0044b0006c0044d0006','wm-expiry_date'
bal cmpc,'a0050b0014c0050d0014','filler001 : filler002'
skp put1
# --- <-- redefined record, R/T code reqd?
bal cmpc,'a0012b0052c0012d0052','wm-comment'
skp put1
#----------------------------------------------------------------------
cmpall bal cmpc,'a0b0064c0d0064','allrecord' (use if no dates)
put1 bal cmpd report if any mismatches
skp loop repeat loop until EOF
#EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
Note that different instructions are generated for the different record definitions and it is simply a matter of inserting some code to test the record type & conditionally execute the proper set of instructions.
See the modified job listed on the next page --->
Note that only bytes 12-63 (52 bytes) of the comment record will be compared, since only bytes 12-63 were redefined in the copybook.
Also note that uvcmp6 has generated the 'unixcmpall' instruction for just such situations. On the next page, we could change 'skp= note' to 'skp= cmpall' & eliminate or just ignore the 'bal' generated for the comment record.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
opr='warmas3 warmas3 - generated by cobmap,uvcmp6,uvdata52'
arg1=warmas3,uop=q1p30s3t10000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=${UVDATA1}/warmas3,rcs=0064,typ=RSF
fili2=${UVDATA2}/warmas3,rcs=0064,typ=RSF
filo3=${CMPRPTS}/warmas3,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
#-------------------------------------------------------------------------
cmc a11(1),' ' detail record ? <-INSERT
skp= dtl <-INSERT
cmc a11(1),'*' note record ? <-INSERT
skp= note <-INSERT
msgw 'Invalid record type - enter to copy unchanged ?' <-INSERT
skp put1 <-INSERT
#-------------------------------------------------------------------------
dtl bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0030d0006','wm-purch-date'
bal cmpc,'a0036b0008c0036d0008','wm-policy'
### bal cmpc,'a0044b0006c0044d0006','wm-expiry_date'<-disable TIME-STAMP
bal cmpc,'a0050b0014c0050d0014','filler001-002'
skp put1
#
note bal cmpc,'a0012b0052c0012d0052','wm-comment'
skp put1
#----------------------------------------------------------------------
cmpall bal cmpc,'a0b0064c0d0064','allrecord' (use if no dates)
put1 bal cmpd report if any mismatches
skp loop repeat loop until EOF
#EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
We inserted 6 instructions & added 2 tags to already generated instructions.
We ### commented out the compare for wm-expiry-date, which we are pretending is a time-stamp. In fact this would be automatically #commented out if the fieldname included 'time'.
In more complex files, there might be 3 or more record types. However it is still a matter of inserting the record type tests - all the code to compare the different record type fields would already be generated.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Dates within occurs is not usually a problem for these 'CMPjobs', as they were for other date processing jobs (REFORMjobs, AGEjobs, VALjobs). This is because we can simply compare the entire occurs area, without having to process dates individually as we do for REFORMjobs, AGEjobs,& VALjobs.
Dates in occurs would be a problem in the following 2 situations:
1 - Comparing unexpanded files with 6 digit dates to expanded files with 8 digit dates. 2 - When the occurs fields also contains other fields which must not be compared (usually time-stamps).
The solution for both of these is add a register loop to avoid having to code a 'bal' instruction for each occurrence. We will illustrate the solution using the warmat1 test file, which has an occurs 4 containing a date & policy type. Let's assume that the policy type is a time stamp.
Here is the copybook/cobmap for our test file. We will compare 2 files using the same file layout (with unexpanded 6 digit dates).
cobmap1 start-end bytes for cobol record fields 199905130550 pg# 0001
cpy1/warmat1 RCSZ=0100 bgn-end lth typ
*warmas1 - warranty master record
05 wm-cust pic x(6). 000-0005 006
05 wm-prod pic x(6). 006-0011 006
05 wm-descrip pic x(18). 012-0029 018
05 wm-policy occurs 4.
07 wm-policy-date pic x(6). 030-0035 006
07 wm-policy-type pic x(8). 036-0043 008
05 wm-expiry.
07 wm-exp-year pic 9(2). 086-0087 002 n 02
07 wm-exp-month pic 9(2). 088-0089 002 n 02
07 wm-exp-day pic 9(2). 090-0091 002 n 02
05 wm-paid-amt pic 9(7). 092-0098 007 n 07
05 filler001 pic x(1). 099-0099 001
*RCSZ=0100 0100
Note that only the 1st set of occurs fields are shown in the cobmap. The 2nd,3rd,4th sets would be in bytes 44-57, 58-71,& 72-85.
When you add your code loop, I suggest you insert #comment lines as shown below to check your calculations.
# occurs 4 * (6 date + 8 type) = 56 + 30 start = 86 end
Remember that for this demo, we are pretending that the policy type is a time-stamp field which must not be compared.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Here is the comparison report BEFORE & AFTER the modifications required.
The BEFORE report detects mismatches only within the 1st occurs group, because the cobmap1 utility maps only the 1st occurrence, to avoid long repetitive code in the cobmap reports.
Since the compare jobs are generated from the copybooks/cobamps, they will contain the code only to compare the 1st field of each occurs group. The next 2 pages illustrate the BEFORE & AFTER compare job for this file.
uvcmp6 file comparison, Copybook=warmat1, uops=q1p30s3t10000v0q0i3
recsize reccount dels file-size typ filename
1: 100 8 0 800 RSF dat1/warmat1
2: 100 8 0 800 RSF dat2/warmat1
1999/09/15_09:40:56 1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 11111 refrigerator 970101 21year 970201 22year 970301
** *
64 23year 970401 24year 9901010004925.
2 2 22222 11111 refrigerator 970199 21Xear 970288 22XXar 970377
** *
64 23XXXr 970466 24XXXX 9901010004925.
======================= EOF or terminate count rea==========================
Total=000008, read=000008, mismatch=000001, stop=000003, term=010000
uvcmp6 file comparison, Copybook=warmat1, uops=q1p30s3t10000v0
recsize reccount dels file-size typ filename
1: 100 8 0 800 RSF dat1/warmat1
2: 100 8 0 800 RSF dat2/warmat1
1999/09/15_11:29:25 1 2 3 4 5 6
f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
============================================================================
1 2 0 22222 11111 refrigerator 970101 21year 970201 22year 970301
** ** **
64 23year 970401 24year 9901010004925.
**
2 2 22222 11111 refrigerator 970199 21Xear 970288 22XXar 970377
** ** **
64 23XXXr 970466 24XXXX 9901010004925.
**
======================= EOF or terminate count rea==========================
Total=000008, read=000008, mismatch=000001, stop=000003, term=010000
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
opr='warmat1 warmat1 - generated by cobmap,uvcmp6,uvdata52'
arg1=warmat1,uop=q1p30s3t10000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=${UVDATA1}/warmat1,rcs=0100,typ=RSF
fili2=${UVDATA2}/warmat1,rcs=0100,typ=RSF
filo3=${CMPRPTS}/warmat1,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
# --- <-- R/T tests required ?
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
bal cmpc,'a0030b0006c0030d0006','wm-policy-date occur 04'
bal cmpc,'a0036b0008c0036d0008','wm-policy-type occur 04'
bal cmpc,'a0086b0002c0086d0002','wm-exp-year'
bal cmpc,'a0088b0012c0088d0012','wm-exp-month : filler001'
skp put1
#----------------------------------------------------------------------
cmpall bal cmpc,'a0b0100c0d0100','allrecord' (use if no dates)
put1 bal cmpd report if any mismatches
skp loop repeat loop until EOF
#EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
| WARNING |
|
Setup 'pfc3' for all jobs that require manual modifications.
| pfc1 |
|
| pfc2 |
|
| pfc3 |
|
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
opr='warmat1 warmat1 - generated by cobmap,uvcmp6,uvdata52'
arg1=warmat1,uop=q1p30s3t10000v0
was=a8000b8000c8000d8000e8000f8000g8000h8000i8000j8000
fili1=${UVDATA1}/warmat1,rcs=0100,typ=RSF
fili2=${UVDATA2}/warmat1,rcs=0100,typ=RSF
filo3=${CMPRPTS}/warmat1,rcs=128,typ=LSTt
@run
bal cmpa open files, write report headers
# begin loop to get each record pair, compare, report(if mismatch), until EOF
loop bal cmpb get record pair & init possible output
skp> eof
# --- <-- R/T tests required ?
bal cmpc,'a0000b0030c0000d0030','wm-cust : wm-descrip'
#----------------------------------------------------------------------
## bal cmpc,'a0030b0006c0030d0006' <-- original date, see new below
## bal cmpc,'a0036b0008c0036d0008' <-- original time, removed
##occurs with dates & time-stamps to be bypassed, original above, new below
# - use rgstrs j & k with code loop to compare date only & bypass time
# - occurs 4 * (6 byte date + 8 byte time) = 56 + 30 start = 86 end
mvn $rj,0 init rgstr j (file 1)
mvn $rk,0 init rgstr k (file 2)
lup1 bal cmpc,'aj30b6ck30d6','date occur 04' <-- NOTE rgstrs J & k
add $rj,14 up file1 rgstr to next occurs
add $rk,14 up file2 rgstr to next occurs
cmn $rj,52 end 4 loops 4 * (6+8) = 52 ?
skp< lup1
#----------------------------------------------------------------------
bal cmpc,'a0086b0002c0086d0002','wm-exp-year'
bal cmpc,'a0088b0012c0088d0012','wm-exp-month : filler001'
skp put1
#----------------------------------------------------------------------
cmpall bal cmpc,'a0b0100c0d0100','allrecord' (use if no dates)
put1 bal cmpd report if any mismatches
skp loop repeat loop until EOF
#EOF - write reports stats, close files,& end job
eof bal cmpe write rpt stats, close files, end job
@pf2=uvcmp6.subs #load(at run-time) common subrtns (cmpa,cmpb,cmpc,cmpd,cmpe)
Note that you must use registers 'j' & 'k' as shown on the following line:
lup1 bal cmpc,'aj30b6ck30d6','date occur 04' <-- NOTE rgstrs J & k
I recommend you insert #comment lines as shown above to check your math.
# occurs input 4 * (6 date + 8 type) = 56 + 30 start = 86 end
Please relate these calcs to the copybook shown on page '6L1'. These #comments are a big help to you or other people who may later need to verify that the job was done correctly.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
For sequential (non-keyed) files, the uvcmp... jobs read both files sequentially & assume the matching record pairs can be compared.
For Indexed files, the uvcmp jobs read the 1st file sequentially, and chain to the 2nd file ('chain' means random read by key of 1st file).
Chaining (random read by key) allows for inserted records, which would be a problem if we were reading sequentially & assuming matching pairs.
However, if we have expanded dates in the first key of the 2nd file, we will have a problem using the unexpanded key of the first file to chain to the second file.
Options h,j,k are provided to solve this problem, by allowing for expanded dates prior to the key, and for inserting '19's for the century of any expanded dates within the key, before using the key to chain to the 2nd file.
| option 'h' |
|
| option 'j' |
|
| option 'k' |
|
To demo all options in use, assume the old record had 3 * 6 byte dates in the 1st 18 bytes 0(18) & that the key is date2+date3 6(12). Also assume that all dates have been expanded in the 2nd file to 24 bytes, so the key is now 8(16).
date1,date2,date3.
<----key--->
1st file key: 990122990222990322
--date1,--date2,--date3.
<------key----->
2nd file key: 199901221999022219990322
For this example, you would enter 'h0j8k16' at the options prompt when you run the 'pfa2' job generated by uvcmp6. This will shift the key up 2 bytes to compensate for the expansion of the 1st date (not part of the key), and insert '19' at bytes 8 & 16 before chaining to the 2nd file.
uvcopy pfc2/filename,uop=h2j8k16 ================================
You can enter the options on the command line as shown above, or even better (in case of reruns) edit the generated job to append these options on line 2.
rop=j64000k16000,uop=q1p30s3t1000v0h2j8k16
^^^^^^^
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Some sites use complement dates, to make transactions appear in date sequence.
complement date = 1000000 - yymmdd yymmdd = 1000000 - complement date
For Year2000 (with a window of 68) this was changed to:
complement date = 20680000 - yymmdd yymmdd = 20680000 - complement date
Complement dates are excluded from comparison by using the uvcmp6 generated jobs & inserting their fieldnames into the exclude table within uvcmp6. In the example above, the customer always coded the word 'pseudo' in the complement dates, so this word was added to the exclude table.
However, there is still a problem if the complement date is part of the first key of an Indexed file, which is normally used to chain to the matching record in the second file.
For these cases, the old original uvcmp1,5,6 jobs have been preserved with their names changed to uvcmp1X, uvcmp5X, & uvcmp6X.subs.
For this situation, we would be using the jobs generated by uvcmp6, which normally call uvcmp6.subs at the end of the generated code. All you have to do is change:
@pf2=uvcmp6.subs
-to-
@pf2=uvcmp6X.subs
Of course this is a trade-off, these jobs will not now handle inserted records in the file, which is why the original jobs were changed to read the 1st file sequentially & use its key to chain to the 2nd file.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
The 'uvhdcob' utility & 'uvhddif2' script can be used to compare 2 datafiles & show the differences with the corresponding COBOL copybook names.
Please see complete details in uvhdcob.doc of UVdoc.
** sample output report **
1c1
< dat1/warmas1 fsz=576 rsz=64 totrecs=9 current=1
---
> dat2/warmas1 fsz=576 rsz=64 totrecs=9 current=1
34c34
< wm-descrip 12 29 dish washer
---
> wm-descrip 12 29 dish CRASHER
76c76
< wm-descrip 12 29 clothes dryer
---
> wm-descrip 12 29 clothes SHRINKER
Note that this method of comparing data-files is not as exact as the uvcmp... jobs documented in CMPjobs.doc because:
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
2a. Assuming a test set of data is available (19xx data input).
2b. Create a new set of test data with EXPANDED dates (19xx data input).
- see the reform2 job in DATAcnvt1.htm.
2a. Create test data by aging dates into Year2000 (20xx unexpanded data input).
- see the dateage1 job in AGEjobs.doc
2b. Create 2nd set of test data with EXPANDED dates (20xx expanded data input).
- see the reform2 job in DATAcnvt1.htm.
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
4a. Transfer mainframe input test data to UNIX & convert
- EBCDIC to ASCII, except for packed/binary, correct numeric signs
- see DATAcnv1.htm
4b. Transfer mainframe output data to UNIX & convert (as for inputs)
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
| 6X1. | genuvcmp6 - script to generate uvcmp6 job to compare 2 files field by field |
| - saves having to key 4 uvcopy jobs with long command lines | |
| (cobmap1, cobmap1, reform1, uvcmp6) |
| uvcmp6 |
|
| uvcmp6.subs |
|
The uvcopy jobs in this section are not listed here to save space. They are of course readily available in the /home/uvadm/pf/... directories and you can inspect them with the editor or list them as follows. Using uvcmp1 as an example:
cd /home/uvadm - change to uvadm home directory ============== vi pf/util/uvcmp6 - inspect uvcmp6 (or any uvcopy job desired) =================
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page
#!/bin/ksh
# genuvcmp6 - Generate uvcopy job to compare 2 data files field by field
# - based on 2 COBOL copybooks
# (same or different if datafile#2 has different layout than #1)
# - see /home/uvadm/doc/CMPjobs.doc or www.uvsoftware.ca/cmpjobs.htm
#
# genuvcmp6 must run from a superdir containing subdirs
# cpy1,cpy2,dat1,dat2,map1,map2,map1I,pfc1,pfc2,rptcmp,tmp)
# cpy_=copybooks, map_=layouts from copybooks, pfc_=uvcopy jobs from cobmaps
# dat1=datafile#1, dat2=datafile#2
#
# Multiple subdirs allow us to use the same filename for copybooks, layouts,
# datafiles,& reports. This simplifies the generation & execution.
#
fn="$1" #capture copybook filename (must be no .cpy extension)
echo "genuvcmp6 using: cpy1,cpy2,dat1,dat2,map1,map2,map1I,pfc1,pfc2,rptcmp"
if [[ (-f "cpy1/$fn") && (-f "cpy2/$fn") && ("$fn" != *.*) ]]; then :
else echo "usage: genuvcmp6 CopyBookName"
echo " ======================"
echo " copybook (arg1) must be stored in cpy1 & cpy2 (with no .extnsn)"
exit 91; fi
if [[ -d cpy1 && -d cpy2 && -d map1 && -d map2 && -d map1I && -d pfc1 ]]; then :
else echo "subdirs missing (cpy1,cpy2,map1,map2,map1I,pfc1)"
exit 92; fi
#
uvcopy cobmap1,fili1=cpy1/$fn,filo1=map1/$fn,uop=q0i7p0
#======================================================
uvcopy cobmap1,fili1=cpy2/$fn,filo1=map2/$fn,uop=q0i7p0
#======================================================
uvcopy reform1,fili1=map1/$fn,filo1=map1I/$fn,uop=q0i7
#=====================================================
uvcopy uvcmp6,fili1=map2/$fn,filr1=map1I/$fn,filo1=pfc1/$fn
#==========================================================
exit 0
#
Goto: Begin this doc , End this doc , Index this doc , Contents this library , UVSI Home-Page