Parm File Nulls - detect, delete, or blank

Nulls (x'00's) in parm files is a common problem when migrating mainframe files to Unix/Linux/Windows. This newsletter will provide both unix & Vancouver Utilities solutions to detect, delete,or remove nulls from parm files. These solutions might also apply to other text files such as COBOL & JCL.

'dat1/parmfilenulls' is a sample file with only 3 records illustrating: nulls at the end of a record, nulls within a record followed by blanks, & nulls followed by a sequence# (common in mainframe parm files). Here is how the file would appear in 'vi'. For Windows, you can download 'vim' from "vim.org".


 vi dat1/parmfilenulls   <-- investigate sample file with 'vi' (or 'vim')
 =====================
 * PARM example#1 - nulls in columns 51-80 -------*^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 * PARM example#2 - nulls 51-72, blanks 73-80 ----*^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 * PARM example#3 - nulls 51-72, seq# 73-80 ------*^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@00001234

The nulls appear as '^@' in vi, and for this documentation, I have replaced each x'00' with 2 characters '^@' (a carat & at-sign). To search for a real null x'00' you would enter control-V & control-@ in vi or vim --> /^V^@ <--. You could use grep to search a directory for files with nulls as follows:


 grep -Pa '\x00' dat1/*   <-- search all files in dat1/... for nulls
 ======================

Goto:   Begin this document End this document UVSI Home-Page

using 'uvhd' to display/search nulls in files

 Note that 'uvhd' is a free download from http://uvsoftware.ca/libuvhd.htm.
 It displays any file in "vertical hexadecimal" - data segments shsown on 3 lines
 - characters, hex zones,& hex digits. Segments default to 64 bytes, but below
 option 'l81' sets line length to 81 bytes (parm files 80 bytes + LineFeed x'0A').
 Option 't' (of 'tl81') specifies a Text file (lines terminated by linefeeds vs
 the default of fixed chunks of 256 bytes).

 uvhd dat1/parmfilenulls tl81  <-- display sample file with 'uvhd'
 ============================    - data in vertical hexadecimal as follows:
 rec#=1 rsize=81 fptr=0 fsize=243 rcount=3
                10        20        30        40        50        60        70        80
 r#  1 012345678901234567890123456789012345678901234567890123456789012345678901234567890
     0 * PARM example#1 - nulls in columns 51-80 -------*...............................
       225454267667662322267667266266676672332332222222220000000000000000000000000000000
       A0012D0581D0C5310D0E5CC309E03FC5DE3051D800DDDDDDDA000000000000000000000000000000A
 rec#=2 rsize=81 fptr=81 fsize=243 rcount=3
                10        20        30        40        50        60        70        80
 r#  2 012345678901234567890123456789012345678901234567890123456789012345678901234567890
    81 * PARM example#2 - nulls 51-72, blanks 73-80 ----*......................        .
       225454267667662322267667233233226666672332332222220000000000000000000000222222220
       A0012D0581D0C5320D0E5CC3051D72C02C1EB3073D800DDDDA000000000000000000000000000000A
 rec#=3 rsize=81 fptr=162 fsize=243 rcount=3
                10        20        30        40        50        60        70        80
 r#  3 012345678901234567890123456789012345678901234567890123456789012345678901234567890
   162 * PARM example#3 - nulls 51-72, seq# 73-80 ------*......................00001234.
       225454267667662322267667233233227672233233222222220000000000000000000000333333330
       A0012D0581D0C5330D0E5CC3051D72C03513073D800DDDDDDA000000000000000000000000001234A

Notes re uvhd display above

  1. Record #1 above contains nulls x'00's in zero relative bytes 50-79, or columns 51-80 with LineFeed in column 81. The scale above each record starts with 0 by default, but option 'g1' would start with 1.

  2. Record #2 shows has nulls in columns 51-72, and BLANKS x'20's in columns 73-80.

  3. Record #3 shows has nulls in columns 51-72, and a SEQUENCE# in columns 73-80. Sequence#s in columns 73-80 is a throwback to "punched cards" and I recommend blanking them out because the nulls can cause shifting over into the data area.

Goto:   Begin this document End this document UVSI Home-Page

convert nulls to Blanks or Delete nulls

You can use 'sed' to convert nulls to Blanks x'20's or Delete nulls. Here are the sed commands & the uvhd results only for record record #3 (with the Sequence#)


 sed 's/\x00/ /g'  dat1/parmfilenulls >tmp/parmfilenulls
 =======================================================
 - Substitute nulls x'00's with Blanks x'20's
                10        20        30        40        50        60        70        80
 r#  3 012345678901234567890123456789012345678901234567890123456789012345678901234567890
   162 * PARM example#3 - nulls 51-72, seq# 73-80 ------*......................00001234.
       225454267667662322267667233233227672233233222222222222222222222222222222333333330
       A0012D0581D0C5330D0E5CC3051D72C03513073D800DDDDDDA000000000000000000000000001234A

 sed 's/\x00//g'  dat1/parmfilenulls >tmp/parmfilenulls
 ======================================================
 - Delete nulls (Substitute with nothing)
                10        20        30        40        50        60        70        80
 r#  3 012345678901234567890123456789012345678901234567890123456789012345678901234567890
   162 * PARM example#3 - nulls 51-72, seq# 73-80 ------*00001234.
       22545426766766232226766723323322767223323322222222333333330
       A0012D0581D0C5330D0E5CC3051D72C03513073D800DDDDDDA00001234A

Deleting nulls is a problem if any data follows embedded nulls - the data (sequence# in this case) is shifted over into the data area.

It would be best if we could clear the sequence# AND then Delete the nulls. We can do this with 'uvhd' or 'uvcp' as shown below. uvhd is a free download, but uvcp is part of the Vancouver Utilities.

Goto:   Begin this document End this document UVSI Home-Page

using 'uvhd' to clear seq#s & Delete nulls


 uvhd dat1/parmfilenulls tl81u  <-- start uvhd adding option 'u' (Update mode)
 =============================    - displays 1st record & prompts for command

 --> u99  72(8),'        '      <-- Update all records ('99' signals all records)
     =====================        - with blanks in columns 73-80
 --> v99  x'00',''              <-- scan/replace all nulls with nothing
     =============
 --> q                          <-- Quit
     ===
                10        20        30        40        50        60        70        80
 r#  3 012345678901234567890123456789012345678901234567890123456789012345678901234567890
   162 * PARM example#3 - nulls 51-72, seq# 73-80 ------*        .
       22545426766766232226766723323322767223323322222222222222220
       A0012D0581D0C5330D0E5CC3051D72C03513073D800DDDDDDA00000000A

using 'uvcp' to clear seq#s & Delete nulls

The 'uvhd' solution was presented 1st because uvhd is a free download, but 'uvcp' can do better job (by removing trailing blanks & shortening records), and uvcp could be used in a script to process all files in a directory.


 uvcp "fili1=dat1/parmfilenulls,rcs=256,typ=LSTt,filo1=tmp/pfn,clr=72(28)"
 =========================================================================
 - copy sample file to tmp/pfn, clearing bytes 72-99 (columns 73-100)
 - clearing 28 bytes (vs 8) in case some records have extra unwanted data
 - uvcp file typ=LSTt automatically clears all trailing blanks & nulls
                10        20        30        40        50        60        70        80
 r#  3 012345678901234567890123456789012345678901234567890123456789012345678901234567890
   162 * PARM example#3 - nulls 51-72, seq# 73-80 ------*.
       225454267667662322267667233233227672233233222222220
       A0012D0581D0C5330D0E5CC3051D72C03513073D800DDDDDDAA

uvcp instructions - clr & rep

You might think uvcp should have an instruction to 'rep'lace nulls with blanks as shown below (omitting some parameters ... to focus on most relevant).


 uvcp "fili1=...,typ=LSTt,filo1=...,clr=72(28),rep=0(256):0x00:0x20"
 ===================================================================

The 'rep'lace instruction would replace all nulls within 1st 256 bytes with 0x20 which is a hexadecimal blank.

BUT in this case, the 'rep'lace instruction is not required, because file 'typ=LSTt' will scan back to the last nonblank byte > x'20' & insert the LineFeed x'0A' after that, thus removing trailing nulls as well as blanks.

Goto:   Begin this document End this document UVSI Home-Page

processing All files in Directory

The above solutions fixed 1 file at a time, but it is much more valuable to have solutions to process entire directories with hundreds or thousands of files. Here are 2 scripts that will call 'sed' or 'uvcp' for each file found in the input directory & copy the files to the output directory blanking nulls (and removing trailing blanks for uvcp). The 1st script (parmnullfix1) is listed at http://uvsoftware.ca/scripts1.htm#J1 & also here for your convenience.

 #!/bin/ksh
 # parmnullfix1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # parmnullfix1 - copy all files from 1 directory to a 2nd directory
 #              - converting any nulls to blanks, using sed
 # parmnullfix2 - alternate script calling 'uvcp' (vs sed)
 #              - also clears any sequence#s in cols 73-80
 d1="$1"; d2="$2";
 if [[ -d "$d1" && -d "$d2" ]]; then :
    else echo "usage: parmnullfix1 dir1 dir2"
         echo "       ======================"
         echo "- arg1 & arg2 must be input & output directories"
    exit 9; fi
 if [[ ! -d tmp ]]; then mkdir tmp; fi
 ls $d2 >tmp/$d2.emptytest
 if [[ -s tmp/$d2.emptytest ]]; then
    echo "the output directory must be empty $d2";exit 93; fi
 integer fcount=0 lcount=0 tlcount=0 nlcount=0 nfcount=0
 for d1f in $d1/*
 do
    f=$(basename $d1f)
    ((fcount+=1)); nlcount=0;
    nlcount=$(grep -Pac '\x00' $d1/$f)
    if ((nlcount)); then ((nfcount+=1)); fi
    wcount=$(wc -l $d1/$f 2>/dev/null); lcount=${wcount% *}; ((tlcount+=lcount));
    echo "file#=$fcount lines=$lcount nulls=$nlcount file=$d1/$f"
    sed 's/\x00/ /g' $d1/$f >$d2/$f
    #==============================
 done
 echo "total $fcount files, $nfcount with nulls converted to blanks"
 echo "total $fcount files, $tlcount lines, copied from $d1 to $d2"
 echo "- also see parmnullfix2 also clears 73-80 (using uvcp vs sed)"
 exit 0

Notes re script parmnullfix1


    sed 's/\x00/ /g' $d1/$f >$d2/$f    <-- most vital line
    #==============================

We could have made the script much shorter if we had omitted the error checking to ensure the 2 directories are specified & that the output directory is empty, and if we had omitted collecting statistics. See the demo on the next page.

Note that Windows 10 can now execute Bash shell scripts. Most Vancouver Utility scripts are coded as Korn shell on the 1st line, but this is easily changed.

Goto:   Begin this document End this document UVSI Home-Page

demo parmnullfix1 - script calling 'sed'

You do not need to purchase Vancouver Utilities to use the parmnullfix1 solution since it is a unix script using the unix 'sed' utility. You could test in your home directory as follows:


 #1. mkdir scripts tmp tmp1 tmp2  <-- make subdirs for test
     ===========================

 #2. Download parmnullfix1 from http://uvsoftware.ca/scripts1.htm#J1 or cut out
     of this documentation into your script file ($HOME/scripts) & make executable.

 #3. Copy some parm files with nulls into $HOME/tmp1/...
     Could create files like parmfilenulls shown at begining of this newsletter
     - using vi editor to create some nulls with control-V & control-@.
     I will assume 3 files created (parmfilenulls1,parmfilenulls2,parmfilenulls3)

 #4. scripts/parmnullfix1 tmp1 tmp2  <-- copy tmp1/* to tmp2/ blanking nulls
     ==============================    - script statistics as follows:
      file#=1 lines=4 nulls=3 file=tmp1/parmfilenulls1
      file#=2 lines=5 nulls=2 file=tmp1/parmfilenulls2
      file#=3 lines=2 nulls=0 file=tmp1/parmfilenulls3
      total 3 files, 2 with nulls converted to blanks
      total 3 files, 11 lines, copied from tmp1 to tmp2

Goto:   Begin this document End this document UVSI Home-Page

demo parmnullfix2 - script calling 'uvcp'

parmnullfix2 has the advantage of clearing 73-80 as well as blanking nulls. See the parmnullfix2 script listed at http://uvsoftware.ca/scripts1.htm#J2. We will not list it here since it is similar to parmnullfix1 with the 'sed' instruction on line# 26 replaced by a 'uvcp' instruction as follows:


    uvcp "fili1=$d1/$f,rcs=256,typ=LSTt,filo1=$d2/$f,clr=72(28)"
    #===========================================================

You would need Vancouver Utilities to use the parmnullfix2 solution since it calls 'uvcp' (vs unix 'sed'). uvcp can address column positions which most unix utilities can not do. The test setup is the same as for parmnullfix1 so we will show only the statistics messages:


 #4. sf/util/parmnullfix2 tmp1 tmp2  <-- copy tmp1/* to tmp2/...
     ==============================    - blanking nulls & clearing cols 73-80
     - for Vancouver Utilities, parmnullfix2 would be in /home/uvadm/sf/util/...
     - script & uvcp statistics as follows:
 file#=1 nulls=3 file=tmp1/parmfilenulls1
 160907:193818:uvcp: uvcp fili1=tmp1/parmfilenulls1,filo1=tmp2/parmfilenulls1
 160907:193818:uvcp: EOF fili01 4 rds, 267 size; tmp1/parmfilenulls1
 160907:193818:uvcp: EOF filo01 4 wrts, 177 size; tmp2/parmfilenulls1
 file#=2 nulls=2 file=tmp1/parmfilenulls2
 160907:193818:uvcp: uvcp fili1=tmp1/parmfilenulls2,filo1=tmp2/parmfilenulls2
 160907:193818:uvcp: EOF fili01 5 rds, 234 size; tmp1/parmfilenulls2
 160907:193818:uvcp: EOF filo01 5 wrts, 174 size; tmp2/parmfilenulls2
 file#=3 nulls=0 file=tmp1/parmfilenulls3
 160907:193818:uvcp: uvcp fili1=tmp1/parmfilenulls3,filo1=tmp2/parmfilenulls3
 160907:193818:uvcp: EOF fili01 2 rds, 48 size; tmp1/parmfilenulls3
 160907:193818:uvcp: EOF filo01 2 wrts, 48 size; tmp2/parmfilenulls3
 total 3 files, 2 with nulls converted to blanks & 73-80 cleared
 total 3 files, 11 lines, copied from tmp1 to tmp2

Goto:   Begin this document End this document UVSI Home-Page

uvcopy solution

'uvcopy' is an 'interpretive assembler' (power of assembler with no compile). It can do whatever kind of complex data manipulations required to solve your problems. It is ideal for processing files migrated from mainframes because it understands mainframe concepts such as column addressing, packed fields, Indexed files, etc. It has the ability to read all files in a directory without requiring a script. There are hundreds of the uvcopy instruction files supplied by UV Software to assist mainframe migrations & file maintenance. We will demo 2 uvcopy jobs to blank nulls & clear cols 73-80.

parmnullfix3
  • copy 1 file at a time blanking nulls & clearing 73-80
  • presented in contrast to the more practical parmnullfix4
  • listed below
parmnullfix4

parmnullfix3 - One file at a time

 # parmnullfix3 - convert nulls to blanks & clear cols 73-80
 #              - ONE file at a time (vs parmnullfix4 All files in directory)
 #
 # uvcopy parmnullfix3,fili1=dat1/parmfilenulls,filo1=tmp/pfn  <-- demo
 # ==========================================================
 #              - also see other scripts to perform similar functions
 # parmnullfix1 - script copy all files, blank nulls, using 'sed'
 # parmnullfix2 - script copy all files, blank nulls, clear 73-80, using 'uvcp'
 #*parmnullfix3 - uvcopy job blank nulls & clear cols 73-80 - 1 file at a time
 # parmnullfix4 - uvcopy job blank nulls & clear cols 73-80 - All files in directory
 #              - listed at http://uvsoftware.ca/uvcopy5.htm#L2
 #
 opr='$jobname - copy 1 file blanking nulls & clearing cols 73-80'
 fili1=?infile,typ=LST,rcs=4096       # '?' prompts for filename
 filo1=?outfile,typ=LSTt,rcs=4096     # - if not specified on command line
 @run
        opn    all                      open files
 #
 # begin loop to get/process/put records from/to current file until EOF
 man20  get    fili1,a0(4096)           get each record into area 'a'
        skp>   man90
        mvc    b0(4096),a0              copy input area to output area
 # -------- instructions to clear cols 73-80 & convert any nulls to blanks
        clr    b72(28),' '              clear cols 73-80
        rep    b0(256),x'00',x'20'      convert any nulls to blanks
 # -------------------------------
        put    filo1,b0(4096)           output record
        skp    man20                    return to get next record
 #
 # EOF - close files & end job
 man90  cls    all                      close files
        eoj                             end job

Goto:   Begin this document End this document UVSI Home-Page

demo parmnullfix4 - All files in directory

See the uvcopy job listed at http://uvsoftware.ca/uvcopy5.htm#L2 - not listed here since longer (40 instructions) than parmnullfix3 (14 instructions) due to processing All files in directory vs 1 file at a time.


 #4. uvcopy parmnullfix4,fild1=tmp1,fild2=tmp2
     =========================================
     - copy tmp1/* to tmp2/... blanking nulls & clearing cols 73-80
 parmnullfix3 - copy all files blanking nulls & clearing cols 73-80
 160907:195527:parmnullfix4: uvcopy ver=20160901 pf=/home/uvadm/pf/util/parmnullfix3
 uvcopy LNX H64 license=20160901_99V_930630 site=UV_Software
 did you create empty outdir (or remove all files) y/n ?
 160907:195528:parmnullfix3: EOF fili01 rds=4 size=267: tmp1/parmfilenulls1
 160907:195528:parmnullfix3: EOF filo02 wrts=4 size=177: tmp2/parmfilenulls1
 160907:195528:parmnullfix3: EOF fili01 rds=5 size=234: tmp1/parmfilenulls2
 160907:195528:parmnullfix3: EOF filo02 wrts=5 size=174: tmp2/parmfilenulls2
 160907:195528:parmnullfix3: EOF fili01 rds=2 size=48: tmp1/parmfilenulls3
 160907:195528:parmnullfix3: EOF filo02 wrts=2 size=48: tmp2/parmfilenulls3
 160907:195528:parmnullfix3: EOF fili03 rds=3 size=45: tmp/infiles
 160907:195528:parmnullfix3: EOF fild01 size=69: tmp1
 160907:195528:parmnullfix3: EOF fild02 size=69: tmp2

applications for Vancouver Utilities

Blanking nulls & clearing sequence#s is just 1 example of the usefulness of the Vancouver Utilities. You can see many more examples thruout the documentation - such as at http://uvsoftware.ca/cnvaids.htm

What DATA problems do you have ?

Please tell me about any data problems that you have and I will show you how the problem could be fixed using unix scripts or Vancouver Utilities.

Goto:   Begin this document End this document UVSI Home-Page

links to more information

 http://www.uvsoftware.ca                 - UV Software Home Page
 http://www.uvsoftware.ca/news/           - all newsletters past & present
 http://www.uvsoftware.ca/scripts1.htm    - most useful script listings
 http://www.uvsoftware.ca/jclcnv1demo.htm - JCL Conversion to scripts
 http://www.uvsoftware.ca/uvsort.htm      - uvsort documentation
 http://www.uvsoftware.ca/uvprices.htm    - prices & partner agreements
 http://www.uvsoftware.ca/uvintro.htm#G1  - customer list & reference letters
 http://www.uvsoftware.ca/libuvhd.htm     - download uvhd FREE utility

I invite your feedback. Let me know if these news letters are useful to you, and any suggestions you may have for improvements.

Thank You, Owen Townsend, UV Software Inc.

Goto:   Begin this document End this document UVSI Home-Page