uvsort - sort utility for Unix/Linux systems - CONTENTS

A1. uvsort Overview
 'uvsort' is a powerful utility that provides mainframe sorting capability to
 UNIX systems. It is about 20% faster than Micro Focus 'mfsort'. It is not as
 fast as the unix system sort, but the unix system sort is totally unsatisfactory
 for mainframe file types because: it can not sort files without Line-Feeds,
 can not sort Packed fields, can not sort Indexed files, & does not have the
 select/delete/reformat capabilities to match those of DFSORT & SYNCSORT.
A2. Preliminary examples, before presenting details of many commands & options.
A3. uvsort handles many types of files (sequential, fixed length, RDW variable
 length, variable length & Indexed files compatible with Micro Focus COBOL.
A4. uvsort provides for 20 input files & 60 output files. You can achieve
 unlimited number of input files by specifying ALL files in a directory
 (or all files matching a pattern).
A5. The JCL converter automatically connerts mainframe SORT steps to uvsort.
 DFSORT/SYNCSORT syntax is converted to the more compact uvsort equivalents.
 INCLUDE/OMIT is converted to sel/del, INREC/OUTREC FIELDS are converted to
 uvsort equivalents. uvsort provides up to 600 selects/deletes on input
 and another 600 on output. Up to 600 FIELD reformats are also allowed at
 both input & output time.
B1. uvsort FUNCTION Summary
B2.  uvsort INSTRUCTION Summary (same as uvcp)
C1. Sort Key field data types, sequence codes
C2.  Sort Key formats & options
C3.  Sort collation sequence for field type 'g'
C4.  Sort collation sequence for field type 's'
C5.  sort key option 'u' (unique) to drop records with duplicate keys
C6.  sort key option 's' (select) to select records with duplicate keys
C7.  sort key option 'f1' to write separate files for each key1 dup set
C8.  Year 2000 field type & options to modify window
C9.  Illustrated demo of sliding windows
D1. FILE DEFINITION parameters
 (fili1/filo1, typ, rcs, isk)
E1. File 'TYP' codes
 (typ=RSF/RSV/ISF/IDX/LST/RDW/STL)
E4.  Indexed Sequential Variable record length files (typ=IDX)
 compatible with Micro Focus COBOL IDXFORMAT3
E5.  File type options (a thru z)
E8.  File type options for typ=IDX files (f,v,x,y,z)
E9.  output record size defaults for variable length files

Goto:   Begin this document End this document UVSI Home-Page

uvsort - sort utility for Unix/Linux - CONTENTS (continued)

F1. Run options a-z
F3.  Run option examples (assigning more memory, stop & bypass counts)
F4.  Run option 'f2' to input all files from a directory
- optionally matching any specified prefix &/or suffix
F5.  SORTDIR environmental variable for the sort work subdir.
 Interrupt processing can be enabled via run option e1.
G1. Select & Delete records
G2.  Sample select/delete problem & uvsort solution
G3.  Instruction option 'o#' to select/delete records to different output files
G4.  option u1 to code blanks as underscores & u2 to code commas as at-signs
- for sel/del, also mvc & rep (documented in uvcp.htm)
H1. SUM function - accumulates fields into 1 record for duplicate key sets.
I1. uvsort Instructions - same as uvcp (see documentation in uvcp.htm)
I2.  Instruction option 'o' to execute instructions
 at sort output time vs sort input time (the default).
I3.  Instruction option 'o#' to execute instructions on different output files
I4. 'seq' - sequence# instruction most applicable to uvsort
I5.  test/demo seq instruction using /home/uvadm/tf/test100
J1. FILE DECLARATION - 4 METHODS
K1. MULTI-file declaration - 3 METHODS
L0. uvsort examples - using demo files provided
 demo file /home/uvadm/dat1/sales1
- 64 byte records, mainframe type, fixed length records, no LineFeeds
L1. ex#1 - sort sales1 demo file on 1 field (product#)
L2. ex#2 - sort sales1 demo file on 2 fields (salesman# & product#)
- also change output file to typ=LSTt to insert LineFeeds
L3.  Sort sales file by product within salesman
 select HAMmers for salesmen 21,44,& 65
L4.  Sort & Load an Indexed file with 2 keys (customer# & city-name)
L5.  Read, Sort,& re-Write an Indexed file back into the same file
- a good way to reorganize indexed files & drop deleted records
L6.  Create customer telephone list - sorted by telephone#
- reformat 256 byte record to 50 bytes (cust#, name,& telephone#)
L7.  Convert text file to fixed length Indexed file
- create with 'vi' editor & use 'uvsort' to load Indexed file with
  required record-size & Indexed key.
M1. Using uvsort to rebuild indexed files from the data partition of
 ISAM files with corrupted Indices. (both Microfocus & MBP COBOL)
N1. Record Sequential Variable (typ=RSV) files
- compatible with Micro Focus COBOL IDXFORMAT3 files

Goto:   Begin this document End this document UVSI Home-Page

uvsort - sort utility for Unix/Linux - CONTENTS (continued)

O1. Test/demo uxsort creating Indexed Sequential Variable files (typ=IDXf3/f8)
O2.  uvhd hexdump of a sample IDXFORMAT3 file
O3.  uvhd hexdump of a sample IDXFORMAT8 file
O4.  IDXf3/IDXf8 file header & record header layout
O5.  uxsort scripts used at UVSI to write various IDX files & read back
O6.  uxsortIDXf3w test/demo script - writes 6 IDXf3 files with various options
O7.  uxsortIDXf8w test/demo script - writes 6 IDXf8 files with various options
O8.  directory listings after writing IDXf3/IDXf8 files
P1. typ=RDW (default typ=RDWz4) Record Descriptor Word variable length files
- 4 byte record prefix with 2 byte binary record size & 2 unused nulls
- using 'uvhd' to display & convert EBCDIC RDW files to ASCII text
- using 'uvcp' to convert EBCDIC RDW files to ASCII fixed length files
 typ=RDWz2 - option 'z2' for 2 byte record prefix (vs 4 above)
 typ=RDWz8 - option 'z8' same as RDWz4, but bypasses any BDW prefixes
Q1. typ=STL STandard Language file system (used by AIX CBOL)
STLs - STL Sequential files (input & output)
STLi - STL Indexed files (input only)
STLr - STL Relative record files (input only)
Q2.  Converting STL to other file types (RSF,LST,RDW,etc)
- may convert STLi & STLr to STLs (only output for typ STL)
S1. CoSort to uvsort converter
 'cosort2uvsort1' converts unix/linux CoSort parameters to uvsort
 sample CoSort parameters & uvsort equivalent
- 3 output files with selections by salesman & amounts > $200.00
T1. uvsort vs MFSORT (Micro Focus COBOL standalone sort utility)
- samples of uvsort & equivalent mfsort
- sort time tests on big files (4 gig & 8 gig)
U1. JCL conversion of Mainframe SORT/SYNCSORT to uvsort
- INCLUDE/OMIT's converted to uvsort sel/del's
- complex INCLUDE/OMIT's with multiple mixed AND/OR's in ((parens))
  may require manual correction
V0. Examples of SORT/INCLUDEs uvsort/selects that require corections

uvhd - free sample of Vancouver utilities

'uvhd' is a free sample of Vancouver utilities, that you can download from http://uvsoftware.ca/libuvhd.htm. uvhd is a file investigation utility, especially relevant for files with packed/binary fields. It displays any file in vertical hexadecimal & prompts for commands to browse, search, select, update, translate, etc. uvhd has 25 help screens and is all my customers favourite tool.

Goto:   Begin this document End this document UVSI Home-Page

A1. uvsort Overview

'uvsort' is a powerful utility that provides mainframe sorting capability to UNIX systems. It is about 20% faster than Micro Focus 'mfsort'. It is not as fast as the unix system sort, but the unix system sort is totally unsatisfactory for mainframe file types (migrated to unix systems) because:

  1. unix sort cannot sort fixed length record files without LineFeeds
  2. unix sort cannot sort on packed decimal fields
  3. unix sort cannot sort Indexed files (Micro Focus compatible)
  4. unix sort does not have the select/delete/reformat capabilities of uvsort or mfsort (both uvsort & mfsort have capabilties similar to mainframe DFSORT or SYNCSORT)

uvsort provides for 600 record selects/deletes for input time & a further 600 record selects/deletes for output time. These select/deletes may be based on multiple and/or conditions comparing record fields to constants. uvsort also provides for 600 instructions for reformatting records at both input time & output time (move,add,subtract,multiply,compare,skip,etc).

uvsort is very similar to the uvcp utility with the addition of sort keys. Keys may be specified as separate parameters or as a string, for example:

        key1=0(8),key2=20(15),key3=30(5pd)
                      -or-
        keys=(0,8,c,a,20,15,c,a,30,5,p,d)

Up to 9 keys are allowed in format 1 & up to 15 keys in format 2.

This uvsort section will repeat some documentation for file declarations which are similar to uvcp but not identical (re multifile input).

This uvsort section will not repeat any documentation for the processing instructions (add,sub,mpy,mvc,etc) since they are identical to the uvcp utility & have already been documented in the uvcp section.

uvsort pricing

uvsort is sold as part of the Vancouver Utiltities package class B, which includes all utilities except tof the JCL converter. The class B price is $4800 US, but see http://uvsoftware.ca/uvprices.htm#B3 for all details including support options.

Goto:   Begin this document End this document UVSI Home-Page

A2. uvsort Overview

uvsort prelimnary examples

Here are some preliminary examples, before presenting all the details for the many parameters & options documented on the following pages.


 #1. uvsort "fili1=dat1/sales2,rcs=64,typ=RST,filo1=tmp/s2,rcs=64,key1=30(6),sel1=10(2):21"
     ======================================================================================
     - sort sales2 demo file by product#, selecting only salesman #21
     - output 'typ=RSF' Record Sequential Fixed, recsize=64, no linefeed
     - could specify 'typ=RST' to force Linefeed into the last 64th byte
     - 'key1=30(6)' is product# in bytes 30-35 (zero relative) columns 31-36 (one relative)
     - see http://uvsoftware.ca/uvsort.htm#L3 for sales file layout & results expected
     - this example has only 1 select, but page 'L3' has 6 selects (600 maximum)

 #2. uvsort "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/cm1,typ=ISF,key1=0(6),isk1=0(6)"
     ======================================================================================
     - sort & load an Indexed sequential file (C-ISAM & Micro Focus COBOL compatible)
     - 'key1=0(6)' is the sort key (cust# 1st 6 bytes)
     - 'isk1=0(6)' Indexed Sequential Key to load output file (1st 6 bytes)
     - 'rcs=256' input recsize, output will be 257 (Indexed file status byte added)
     - see http://uvsoftware.ca/uvsort.htm#L4 for customer demo file layout

 #3. uvsort "fili1=dat1/custmas1,rcs=256,typ=ISF,filo1=tmp/custphones,rcs=80,typ=LSTt\
            ,key1=10(25),clr=35(45),mvc=35(12)90"
     =================================================================================
     - sort customer master by customer-name & reformat into a telephone list
     - converts Indexed file 'typ=ISF' to a text file 'typ=LSTt'
     - input fixed 256 byte records, output max 80 LineFeed after last non-blank
     - '\' continues script to a 2nd line
     - 'key1=10(25)' sorts on customer-name cols 11-35 (zero relative 10-34)
     - 'clr=35(45)' blanks the address, before telephone# moved here
     - 'mvc=35(12)90' moves the telephone# from cols 91-112 to cols 36-47
     - see http://uvsoftware.ca/uvsort.htm#L6 for demo file results expected

 #4. uvsort "fili1=dat1/sales1,rcs=64,typ=RST,filo1=tmp/salecustomers,key1u1=0(6)"
     =============================================================================
     - sort sales file with data reduction option 'u1' of 'key1u1=0(6)'
     - unique key option retains only 1st record of duplicate key set (cust#s)
     - see http://uvsoftware.ca/uvsort.htm#C5 for duplicate drop & select options

 #5. uvsort "fili1=dat1/sales1,rcs=64,typ=RST,filo1=tmp/salesman_,key1f1=10(2)"
     ==========================================================================
     - sort sales file with option 'f1' of 'key1f1=10(2)' (salesman cols 11-12)
     - option 'f1' creates a separate output file for each salesman
     - appending the sort key (salesman#) on to output filename
     - see http://uvsoftware.ca/uvsort.htm#C7 for demo file results expected

 #6. uvcp "fili1=dat1/saleshist,typ=RSF,rcs=100,filo1=tmp/saleshist,rop=x20000000"
     =============================================================================
      This illustrates Run OPtion 'x' (rop=x____) to split output into multiple
      files of a Max number of records (except for last output file).
      'rop=x20000000' 20,000,000 records * 100 = 2,000,000,000 bytes
      Assuming a 5 gig input file, the output files would be:
      saleshist_001 (2 gig), saleshist_002 (2 gig), saleshist_003 (1 gig)

Goto:   Begin this document End this document UVSI Home-Page

A3. uvsort: similar sort facility is also provided in uvcopy

alternate version 'uxsort' for IDXFORMAT3/8 files

'uxsort' is an alternate version of uvsort that includes support for Micro Focus COBOL IDXFORMAT3/8 files (Indexed Sequential Variable length records). You must execute uxsort to read or write 'typ=IDX' files. Note that Fixed length Indexed files (compatible with Micro Focus COBOL IDXFORMAT1) are supported by both uvsort & uxsort.

'uxsort' must be compiled with the Micro Focus 'cob' command. Please see install.doc which describes the use of the 'cobccuv' script which makes it easy to compile Vancouver Utility programs using the 'cob' command.

similar sort facility also provided in uvcopy

Uvcopy also includes the uvsort facility, implemented in a more powerful manner, but not as convenient for simple sorts, because uvsort can specify all parameters on the command line.

In uvcopy, the sort is implemented as several instructions for the various sort phases, which provides a lot of flexibility.

For example, you can reformat the records as you are giving them to the sort, then execute the sort,& then load an indexed file as you are retrieving the records from the sort (possibly appending sequence#s to make unique keys).

See the sort instructions documented at the end of uvcopy3.doc (sxo,sxp,sxs,sxg,sxc).

For some particularly good examples of using the sort within uvcopy please see the following jobs in other sections of the documentation.

   index  - in section TEXTjobs.doc
          - create an index from a document containing keywords & page ejects
   permute - also in the TEXTjobs section.
           - create a permuted index from a table of contents

Goto:   Begin this document End this document UVSI Home-Page

A4. Enhancements to uvsort

Jan2004 Enhancements to uvsort

uvsort now allows you to specify up to 20 input files & up to 60 output files. On input you may specify record selects/deletes/reformats that are applied to all records. On output you may specify record selects/deletes/reformats that are different for different output files.

A maximum of 20 input & 60 output files apply when you specifically name the desired files (fili1=...,fili2=...,fili20=...,filo1=...,filo2=...,filo60=...).

An alternative to specifying multiple input filenames is to use the new run option 'f2', to input all files from a specified directory, OR all filenames matching a specified prefix &/or suffix.

For output files, you may specify option 'f1' on key1 (key1f1) to output multiple separate files for duplicate record key sets based on key1. This is a unique & powerful feature that can be used to write out separate files for each customer#, salesman#, state/province, etc.

The output filenames are automatically created in the specified output directory, with an optional prefix, followed by the current key1 data value. These enhancements make uvsort even more powerful than SYNCSORT.

links to enhancements

C5. sort key option 'f1' to write separate files for each key1 duplicate set
F4.  Run option 'f2' to input all files from a directory
- optionally matching any specified prefix &/or suffix
G3.  Instruction option 'o#' to select/delete records to different output files
I2.  Instruction option 'o#' to execute instructions on different output files
H1.  SUM function - accumulates fields into 1 record for duplicate key sets.

Nov 2004 Enhancements to uvsort

uvsort now maintains the original file record sequence. This is effective when multiple records exist for the specified sort keys. This feature is implemented by automatically adding an extra low order sort field with an input record sequence number. For best performance, uvsort uses the 'qsort' algorithm which does not guarantee original record sequence.

sort field type 'g'

Sort field type 'g' changes the character collation sequence. The exact sequence is shown on page C3 of http://www.uvsoftware.ca/uvsort.htm#C3. This was requested by customers converting DB2 UDB applications from the mainframe to unix/linux. Other sites might use this if they want UPPER & lower case letters to sort together (vs the separate ranges of ASCII).

Goto:   Begin this document End this document UVSI Home-Page

A5. Enhancements to uvsort

DFSORT/SYNCSORT to uvsort by JCL converter

The JCL converter automatically connerts mainframe SORT steps to uvsort. DFSORT/SYNCSORT syntax is converted to the more compact uvsort equivalents. INCLUDE/OMIT is converted to sel/del, INREC/OUTREC FIELDS are converted to uvsort equivalents. uvsort provides up to 600 selects/deletes on input and another 600 on output. Up to 600 FIELD reformats are also allowed at both input & output time. See example of converting JCL with SORT OMIT to Korn shell script uvsort delete at http://www.uvsoftware.ca/JCLcnv1demo.htm#2B1.

uvsort Max Memory Increased

You can now specify up to 2 gigabytes of memory for uvsort. The same sort is also built in to some other utilities (uvcopy,uvqrpg,uxcopy,uxqrpg). The default Max Memory is now 512 megabytes (vs 64 megabytes). Sort times may not be much different for small files, but could be dramatically improved for large files (especially if over 12 gigabytes).

The number of workfiles has also been increased from 50 to 100. These work files are created in the 'tmp'subdir while reading the input file(s). These sorted workfiles are then merged during the sort output phase. If (WorkFiles * MaxMemory) < filesize, the sort will be efficient, there will be no need for intermediate sorts & merges.

You may increase the Max Memory via run option 'm'. For example 'rop=640' to specify 640 megabytes (vs the 512 meg default):


 uvsort "fili1=bigfile,typ=RSF,rcs=351,filo1=sorted,key1=71(20),rop=m640"
 ========================================================================

max filename lengths increased

The maximum filename lengths for uvcopy,uvqrpg,uxcopy,& uxqrpg have been increased from 80 bytes to 180 bytes. Note that uvsort,uvcp,uxsort,& uxcp already allowed 200 bytes for maximum filename length.

Goto:   Begin this document End this document UVSI Home-Page

B1. uvsort FUNCTION SUMMARY

 fili1=infilename           - declare input filename
 fili20=...                 - up to 20 input files may be declared
 fili1=dir/*                - input files from directory option (rop=f2)
 typ=LST,RSF,RST,RSV,ISF,IDX,STL - declare file type
 typ=LSTt                          & possible options appended to file type
 rcs=9999                   - declare record size
 filo1=outfilename          - declare output filename
 filo60=...                 - up to 60 output files may be declared
 filo1=dir/*                - multi-file output by option 'f1' on key1 breaks
 iskx=start(length)         - declare indexed file keys
 keys=(0,8,c,a,30,5,p,d)    - declare sort keys (format 1)
         -or-
 key1=0(8),key2=30(5pd)     - declare sort keys (alternate format)
                            - options to delete or select duplicate keys
 rop=abc...xyz              - declare misc run options
 sel=start(lth):constant     - select records with spcfd value in spcfd cols
 del=start(lth):constant     - delete records with spcfd value in spcfd cols
prm=paramfile
  • declare name of parameter file containing uvsort parameters
  • paramfile searched in $PFPATH, defined in profile, example:
    export PFPATH=$HOME/pf:$UV/pf/util
  • use for complex sorts with many sels/dels &/or output files
  • not required for simple sorts since all parameters
    may be specified on the command line
  • see example on page 'J1'
  • could use the sort facility of uvcopy vs complex uvsorts

prm updates Oct 2013

uvsort now allows any combination of prm=... & command line args, examples:

 uvsort prm=pfile1      <-- all commands in pfile1 (searches $PFPATH)
 uvsort pfile1          <-- same as above
 uvsort prm=pfile1,cmd1=aaa,cmd2=bbb,cmd3=ccc,etc,prm=pfile2
 uvsort      pfile1,cmd1=aaa,cmd2=bbb,cmd3=ccc,etc,prm=pfile2
 - could have all 3 combinations, more practical examples follow:
 uvsort "fili1=dat1/sales2,rcs=64,typ=RST,filo1=tmp/s2,key1=30(6),del=13(2)<12"
        - "del=13(2)<12" drops records older than 2012
        - might put in prm=file if used on multiple sorts thruout scripts
 uvsort "fili1=dat1/sales2,rcs=64,typ=RST,filo1=tmp/s2,key1=30(6),prm=sortdropold"

Goto:   Begin this document End this document UVSI Home-Page

B2. uvsort INSTRUCTION SUMMARY

      logical:  mvc, mvn, clr
   arithmetic:  add, sub, mpy, div, pac, unp, seq
 compare/skip:  cmc, cmn, skp
 scan/replace:  rep, scn, sqz
   translates:  tra, tre, trl, tru, trp
        other:  anc, orc, chx, hxc, tim
          I/O:  put
Note
  • the instructions are the same as for the 'uvcp' utility
  • please refer to the uvcp documentation

Goto:   Begin this document End this document UVSI Home-Page

C1. uvsort: sort KEY specification

sort key format #1 - up to 28 keys allowed

keys=(displacement,length,data-type,sequence, ... etc )

 keys=(dsp1,lth1,typ1,seq1,dsp2,lth2,typ2,seq2,----etc----)
 keys=(0,8,c,a)            - sortkey = cols 1-8, character, ascending
 keys=(0,8)                - same (character,ascending is the default)
 keys=(0,8,c,a,228,3,p,d)  - sortkey #1 = cols 1-8 character, ascending
                           - sortkey #2 = cols 229-231 packed, descending

sort key format #2 - up to 9 keys allowed

key#=displacement(length)

key1=dsp1(lth1),key2=dsp2(lth2),key3=dsp3(lth3),etc

 key1=0(8ca)            - sortkey #1 is cols 1-8
                        - character ascending
 key1=0(8)              - sortkey #1 is cols 1-8
                        - character ascending is the default
 key2=228(3pd)          - sortkey #2 is cols 229-231
                        - packed decimal, descending

Sort key format#2 allows fewer sort fields, but more options, which will be described on the following pages.

Goto:   Begin this document End this document UVSI Home-Page

C2. uvsort: sort key specification

sort field data types

a
  • alphabetic (case insensetive, lower & upper treated equally)
b
  • binary (bit for bit) - same as character (see below)
c
  • character ASCII (the default)
e
  • EBCDIC collating sequence (for ASCII data sort fields)
f
  • first blank high, same as type 'c' (character ASCII), except that
    if the 1st byte of the field is a blank, it will be sorted as though
    it were a tilde x'7E'
g
  • alternate collating sequence for DB2 UDB with some German characters
  • sorts UPPER & lower case alphas together (vs separate ranges)
  • see exact sequence on the next page
h
  • sort all blanks high regardless of field location (re type f)
  • otherwise the same as type 'c' (character ASCII)
i
  • integer (2 or 4 bytes) Intel reversed byte architecture
p
  • packed decimal
  • considers sign in right nibble of right hand byte
  • x'0D' is negative, all else is positive
s
  • sorts lower-case & UPPER-case letters together with lower-case
    preceding each pair
  • see sort sequence for field type 's' documented on page 'C4'
u
  • unsigned integer (length must be 2 or 4)
y
  • special field type for the Year2000 problem (2 digit years)
  • default window is '50' meaning years 50-99 = 1950-1999
    & years 00-49 = 2000-2049
  • specify run option 'y__' to modify window & 'z1' for sliding windows
    rop=y70z1 in 1998 means 68-99=1968-1999,& 00-67=2000-2067
    see 'rop' run options in sub-section 6 ahead --->
z
  • zoned decimal
  • considers sign in left nibble of right hand byte
  • negative if x'40' bit present, else positive

sort field sequence codes

a
  • ascending sequence (the default)
d
  • descending sequence

Goto:   Begin this document End this document UVSI Home-Page

C3. uvsort: Sort Key Specifications

Sort Sequence for field type 'g'


 uvsort "fili1=infile,rcs=80,typ=RSF,filo1=outfile,key2=20(30g)"
 ============================================================*==

Field type 'g' can be used to sort to the sequence shown in the table below:

 00-1F - control characters, reassigned to x'FF' to sort high
 20-2F - B _ - , ; : ! ? / . ` ^ ~ ' " (  ('B' is Blank)
 30-3F - ) [ ] { } @ $ * \ & # % + < = >
 40-4F - | 0 1 2 3 4 5 6 7 8 9 a A a A b  (2nd 'a A' with diaeresis)
 50-5F - B c C d D e E f F g G h H i I j
 60-6F - J k K l L m M n N o O o O p P q  (2nd o O with diaeresis)
 70-7F - Q r R s S s t T u U u U v V w W  (2nd 's' sharp, 2nd 'u U' diaeresis)
 80-85 - x X y Y z Z
 86-FE - not used
       - some German characters reasigned to lower sort sequences
 FF    - null & control characters reassigned to sort high

ASCII ISO 8859-1 sort sequence

For comparison the normal (default) character sort sequence is shown below:

 00-1F - control characters (null,soh,stx,etc,...)
 20-2F - B !  " # $ % & ' () * + , - .  /   ('B' is Blank)
 30-3F - 0 1 2 3 4 5 6 7 8 9 : ; < = > ?
 40-4F - @ A B C D E F G H I J K L M N O
 50-5F - P Q R S T U V W X Y Z [ \ ] ^ _
 60-6F - ` a b c d e f g h i j k l m n o
 70-7F - p q r s t u v w x y z { | } ~ D    ('D' is DEL char)
 80-9F - extended control characters
 AF-FF - extended characters for European languages
       - not used in US ASCII ISO 646 (subset of ISO 8859)
       - a few characters German language characters are shown
       - field type 'g' reassigns these to lower values (see above)
 0xC4 - 'a' with diaeresis
 0xD6 - 'O' with diaeresis
 0xDC - 'U' with diaeresis
 0xDF - 'S' sharp
 0xE4 - 'A' with diaeresis
 0xF6 - 'o' with diaeresis
 0xFC - 'u' with diaeresis

Goto:   Begin this document End this document UVSI Home-Page

C4. uvsort: Sort Key Specifications

Sort Sequence field type 's'

Field type 'g' can be used to sort to the sequence shown in the table below:

 00-1F - control characters (null,soh,stx,etc,...)
 20-2F - B !  " # $ % & ' () * + , - .  /   ('B' is Blank)
 30-3F - 0 1 2 3 4 5 6 7 8 9 : ; < = > ?
       - above same as shown for ASCII ISO 8859-1 on previous page
 40-4F - @ a A b B c C d D e E f F g G h
 50-5F - H i I j J k K l L m M n N o O p
 60-6F - P q Q r R s S t T u U v V w W x
 70-7F - X y Y z Z [ \ ] ^ _ ` { | } ~ D    ('D' is DEL char)
 80-FF - extended control characters
       - sequence unchanged

test sort sequence type 's'

You can test type 's' sort sequence as follows:


 uvsort "fili1=tf/testsortseq,rcs=8,typ=LSTt,filo1=tmp/s2,key1=0(8s)"
 =================================================================*==
 The input file is actually 94 records of 1 byte each
 (defined as 8 byte records & 8 byte sort field above).
 Here are the 94 characters in the input file sequence
 - shown with 26 characters per line to save space
 Z Y X W V U T S R Q P O N M L K J I H G F E D C B A
 z y x w v u t s r q p o n m l k j i h g f e d c b a
 9 8 7 6 5 4 3 2 1 0 ~ } | { ` _ ^ ] \ [ ? > = < ; :
 / .  - , + *) ( ' & % $ # " !
 The output file should be as follows:
 - but 1 character per record

! " # $ % & ' () * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? a A b B c C d D e E f F g G h H i I j J k K l L m M n N o O p P q Q r R s S t T u U v V w W x X y Y z Z [ \ ] ^ _ ` { | } ~

Goto:   Begin this document End this document UVSI Home-Page

C5. Sort Key Options - Unique Keys (key#u1/2) - Drop Duplicate Keys

option 'u' (key#u1/2/4) - drop duplicate records

Option 'u' in the 5th byte of 'key#u' specifies 'unique keys' (data reduction) which outputs only 1 record (or none) for the keys at the specified level '#'.

key#u1
  • will drop duplicates EXCEPT for the FIRST record of a duplicate set
    at the specified key level#. ('u1' is the default if u1/2/4 omitted)
key#u2
  • will drop duplicates EXCEPT for the LAST record of a duplicate set
    at the specified key level#.
key#u4
  • will drop ALL records of DUPLICATE KEY SETS, retaining only those
    records that had no duplicate keys at the specified key level.

run option 's' - insert dup count at specified column#

rop=s##
  • run option 's' may specify a displacement where the dup count
    is to be inserted in the last record of a duplicate set.
  • run option s (rop=s99) may be used with user option 'u'
    (this page) as well as with user option 's' (next page)
  • please see the example shown on the next page --->

ex#1 - option 'u1' keep only 1st of dup sets

given
  • the sales detail file listed on page 8 of uvtrain.doc
    with salesman# cols 11-12, cust# 1-6, $amount 54-62.
problem
  • create a list of customers for each salesman, listing only 1 record
    of multiple sales to any 1 customer for that salesman.
key1=10(2)
  • sortkey #1 (salesman# major)
key2u1=0(6)
  • sortkey #2 (customer# minor)
  • option 'u1' for unique keys on customer within slsmn
key3=53(9)
  • optionally specify key3, so that the one record in
    each duplicate key set will be the highest $amount.

ex#2 - option 'u4' drop all dup sets

given
  • 1 directory of COBOL source programs (gl101.bat,gl102.bat,etc)
  • 2nd directory of compiled programs (gl101.int,gl102.int,etc)
required
  • create a list of any missing source programs or compiled programs
    (ie - a list of filenames without matches in the other directory)

 #1. ls cobsdir >tmp/cobslist
 #2. ls cobxdir >tmp/cobxlist

 #3. uvsort "fili1=tmp/cobslist,fili2=tmp/cobxlist,rcs=80,typ=LSTt\
             filo1=tmp/misslist,key1u4=0(20)"
                               ============

Goto:   Begin this document End this document UVSI Home-Page

C6. Sort Key Options (key#s1/2) - SELECT records for DUPLICATE Keys

option 's' (key#s1/2) - select duplicate records

Option 's' in the 5th byte of 'key#s' specifies selection of records for duplicate key sets at the specified key level#. IE - drop all records that had no duplicate keys at the specified key level#.

key#s1
  • select only the FIRST record of a duplicate set
    (at the specified key# level)
key#s2
  • select only the LAST record of a duplicate set
    (at the specified key# level)
key#s4
  • select all records of a duplicate set
    (at the specified key# level)
key#s4b
  • select all records of a duplicate set, with option 'b'
    to insert a blank line between duplicate sets

run option 's' - insert dup count at specified column#

rop=s##
  • run option 's' may specify a displacement where the dup count
    is to be inserted in the last record of a duplicate set.
  • displacement is zero relative, length is always 6 bytes

ex#3 - option 's2' - select last rec of dup set

given
  • the sales detail file listed on page 8 of uvtrain.doc
required
  • determine how many sales invoices for each customer

 uvsort "fili1=dat1/sales1,rcs=64,typ=RSF,filo1=sales1dups,\
 ==========================================================
 rcs=80,typ=LSTt,key1s2=0(6),key2=13(6),rop=s70"
 ===============================================
                     ^^                 ^^^^^^^
key1s2=0(6)
  • select the last record of each duplicate set
  • 0(6) defines the customer#
key2=13(6)
  • also sort on the date, to ensure that the last customer
    record in a dup set is the latest date
rop=s70
  • optional, to insert the duplicate record count into
    columns 71-76 of the last record in each duplicate set.

Goto:   Begin this document End this document UVSI Home-Page

C7. uvsort: key1f1 option to write multiple files

key1f1 option to write multiple files

This is a unique & powerful feature that can be used to write out separate files for each customer#, salesman#, state/province, etc. Whenever a control break occurs in key1, a new file is opened whose filename is taken from the declared filo1=... with the new key1 data appended.

For example, we can sort the dat1/sales3 demo file by salesman# (bytes 10-11) & write separate files to the tmp/ subdir for each salesman# encountered.


 uvsort "fili1=dat1/sales3,typ=RSF,rcs=64,filo1=tmp/sm*,key1f1=10(2)"
 ====================================================================

1st 6 of 20 records in dat1/sales3 demo file

 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+
 130140    21 940802 IN111001  SCR012  00021+ 0001001 00021021+
 130140    21 940802 IN111001  CHR001  00022+ 0002001 00044022+
 139923    35 950802 IN111002  TAB013  00023+ 0003001 00069023+
 139923    35 950807 IN111002  TAB013  00024+ 0004001 00096024+
 150825    44 960804 IN1122    HAM001  00025+ 0005001 00125025+

Option f1 on key1f1 will write a separate output file for each salesman#, in the directory specified by filo1 & with any specified prefix:

tmp/sm21
  • output file for salesman 21 (6 records for demo)
tmp/sm35
  • output file for salesman 35 (4 records for demo)
tmp/sm44
  • output file for salesman 44 (6 records for demo)

Rules for option f1 (key1f1)

  1. Option f1 can only be coded on key1 (key1f1=...)

  2. Only 1 output file may be declared, for example: filo1=tmp/sm*

  3. The output filename must include a directory (at least 1 '/') & optionally may include a filename prefix (on which key1 data will be appended).

  4. You may append an '*' on the directory/filenameprefix*, but this is just a visual reminder. Only the option 'f1' on key1f1 is required.

  5. There is no limit on the number of output files that could be created but you would use this option where you knew the outputs are manageable.

  6. You might use this option when you need to send out separate files to each salesman, or each customer, etc.

Note
  • Also see option 'rop=f2' which is the reverse of 'key1f1'.
  • inputs all files from a directory (see page 'F4').
Note
  • Also see demo parameter files uvsortsplit1/2 on pages 'J1' & 'J2'

Goto:   Begin this document End this document UVSI Home-Page

C8. Year 2000 field type (y) - Illustrated Demonstration


    key1=30(6y)     - field type 'y' specifies Year2000 windowed sort
    =========^=
                    - 1st 2 digits assumed to be the year (no century)
                    - the default window is '50', meaning:
                      years 50-99 are considered 1950-1999,
                      years 00-49 are considered 2000-2049.

Year2000 sort demo

Note: You can run the following Y2K sort demo using the supplied test file.


 uvsort "fili1=tf/testY2K,rcs=80,typ=LSTt,filo1=tmp1,key1=30(6y)
 ===============================================================

INPUT file (uvadm/tf/testY2K)

          1         2         3         4         5
 12345678901234567890123456789012345678901234567890
                             purchase        expiry  <-- sort field 31-36
 cust#   product description   yymmdd policy yymmdd
 00000 test Year2000 date sort 000101 0-year 000101
 11111 test Year2000 date sort 100101 1-year 110101
 22222 test Year2000 date sort 200101 2-year 220101
 33333 test Year2000 date sort 301231 3-year 331231
 44444 test Year2000 date sort 400101 4-year 440101
 55555 test Year2000 date sort 500101 5-year 550101
 66666 test Year2000 date sort 601231 6-year 661231
 77777 test Year2000 date sort 700101 7-year 770101
 88888 test Year2000 date sort 800101 8-year 880101
 99999 test Year2000 date sort 901231 9-year 991231

Y2Ksort OUTPUT results

 55555 test Year2000 date sort 500101 5-year 550101  <-- Y2K windowed sort
 66666 test Year2000 date sort 601231 6-year 661231      (default window 50)
 77777 test Year2000 date sort 700101 7-year 770101
 88888 test Year2000 date sort 800101 8-year 880101
 99999 test Year2000 date sort 901231 9-year 991231
 00000 test Year2000 date sort 000101 0-year 000101
 11111 test Year2000 date sort 100101 1-year 110101
 22222 test Year2000 date sort 200101 2-year 220101
 33333 test Year2000 date sort 301231 3-year 331231
 44444 test Year2000 date sort 400101 4-year 440101

---> Rerun with run options to modify window value & with sliding option


 uvsort "fili1=tf/testY2K,rcs=80,typ=LSTt,filo1=tmp1,key1=30(6y),rop=y70z1"
 ==========================================================================
                                                              ^  ^^^^^^^^^
 ---> Try 'key1=30(6yd)' for descending sort on type y fields.

Goto:   Begin this document End this document UVSI Home-Page

C9. Year 2000 sort demo - Options to change window & specify sliding

Options to modify window & specify sliding

The default window is '50', you can change with run options 'y__' & 'z1'. NOTE - run options are specified on the 'rop' command.

rop=y35z1
  • set Year2000 window for field type 'y' on sort keys
    (& also for any sel1y/del1y selects/deletes)
  • option 'y__' assumes 1st 2 bytes of field is year (no century)
  • 'y35' means that years 35-99 = 1935-1999,
    00-34 = 2000-2034

Y2K demo sort with run options y35z1

<--- Please see the input file listed on the previous page

---> Rerun with run options to modify window value & with sliding option


 uvsort "fili1=tf/testY2K,rcs=80,typ=LSTt,filo1=tmp1,key1=44(6y),rop=y35z1"
 ==========================================================================
                                                              ^  ^^^^^^^^^
 cat tmp1       - display sort output file
 ========
                  ** output results **
 33333 test Year2000 date sort 301231 3-year 331231
 44444 test Year2000 date sort 400101 4-year 440101
 55555 test Year2000 date sort 500101 5-year 550101
 66666 test Year2000 date sort 601231 6-year 661231
 77777 test Year2000 date sort 700101 7-year 770101
 88888 test Year2000 date sort 800101 8-year 880101
 99999 test Year2000 date sort 901231 9-year 991231
 00000 test Year2000 date sort 000101 0-year 000101
 11111 test Year2000 date sort 100101 1-year 110101
 22222 test Year2000 date sort 200101 2-year 220101
Note
  • we have changed the sort field to the 2nd date in cols 45-50
    to better demonstrate the sliding window option.
  • with option 'z1' the window option y35 is adjusted to 33
    & the record with date '331231' in cols 45-50 sorts low.
  • assuming it is now 1998 (35 + 98 - 100 = 33)
    so 33-99 are 1933-1999 & 00-32 are 2000-2032
Now
  • Rerun the sort without option 'z1' & note that the record with
    date '331231' in cols 45-50 will sort high (vs low)
    since the window is then 35 (1935-1999 & 2000-2034)

Goto:   Begin this document End this document UVSI Home-Page

D1. uvsort: FILE DEFINITION PARAMETERS

      fili1=, typ=, rcs=, [isk#=,]  filo1=, [typ=,] [rcs=,] [isk#=]
fili1
  • specifies input filename (fili1=pathname)
  • up to 20 input files
  • also see run option rop=f2 to input multi-files from directory
filo1
  • specifies output filename (filo1=pathname)
  • up to 60 output files
  • also see option 'f1' on key1f1 to output multi-files based on
    control breaks on key1.
Note
  • following keywords apply to the preceding file (fili1 or filo1)
  • keywords specified for fili1 also default for filo1 unless respecified
typ
  • file type codes - 3 upper case alpha + misc options a-z
    RSF/RSV/STL/ISF/IDX/LST - see details ahead
    file options - misc options may be appended to the 3 alpha codes
    - see details on next page
rcs
  • record size for the preceding file (fili1/filo1)
  • may omit declaring rcs for output file (will default same as input)
  • input ISAM files know their rcs, but you have to specify something
    to get past the error-check
isk?
  • declare isk1-isk9 for the preceding file (fili1/filo1)
  • required only for filo1 to build/load a new Indexed file
        isk1=0(8)         - declare 1st key (bytes 1-8)
        isk2=59(15)       - declare 2nd key (bytes 61-75)
        isk1p1=0(8)       - declare key#1 part#1
        isk1p2=20(6)      - declare key#1 part#2
        isk1p3=30(4)      - declare key#1 part#3

fili1/filo1 example


 uvsort "fili1=xx,typ=RSF,rcs=80,filo1=yy,typ=ISF,isk1=0(8),key1=0(8)"
 =====================================================================
Note
  • the 'isk1=0(8)' could be omitted in this case, since it would be
    defaulted from 'key1=0(8)' when output typ=ISF.
  • you would have to code any additional ISAM keys (isk2,isk3,etc).

Goto:   Begin this document End this document UVSI Home-Page

D2. Indexed file definition - additional options

alternate format for ISAM keys

 uvsort "fili1=x,typ=RSF,rcs=80,filo1=y,typ=ISF,isk1=0(8),isk2=59(5),key1=0(8)"
 uvsort "fili1=x,typ=RSF,rcs=80,filo1=y,typ=ISF,isks=(0,8,59,5),key1=0(8)"

ISAM option for DUPS/NODUPS

 uvsort "fili1=x,typ=RSF,rcs=80,filo1=y,typ=ISF,isks=(0,8,40,20),key1=0(8)"
 uvsort "fili1=x,typ=RSF,rcs=80,filo1=y,typ=ISF,isks=(0,8n,40,20d),key1=0(8)"
 uvsort "fili1=x,typ=RSF,rcs=80,filo1=y,typ=ISF,isks=(0,8,40,20),key1=0(8)"
 uvsort "fili1=x,typ=RSF,rcs=80,\
       filo1=y,typ=ISF,isk1=0(8),isk2=40(20n),key1=0(8)"

do not need to specify ISAM keys for input files

 uvsort "fili1=x,rcs=80,typ=ISF,isks=(0,8,40,20),filo1=y,typ=RST,key1=0(8)"
 uvsort "fili1=x,rcs=80,typ=ISF,filo1=y,typ=RST,key1=0(8)"

option 'k' for key of reference (read sequence)

 uvsort "fili1=x,rcs=80,typ=ISFk2,filo1=y,typ=RST,key1=0(8)"

Goto:   Begin this document End this document UVSI Home-Page

E1. uvsort file TYP codes

File type codes are 3 upper case alpha + misc options (a-z), for example:


       uvsort "fili1=test1,typ=RSF,rcs=80,filo1=test2,typ=LST,key1=0(8)"
       ====================*******====================*******===========
RSF
  • Record Sequential Fixed (record size)
RST
  • Record Sequential Terminated (fixed rcsz)
    Terminated by LF *in* last byte of stated record size
    (whether input or output)
RSR
  • Record Sequential Relative (Micro Focus COBOL compatible)
  • actual record size is 1 greater than declared
  • extra byte x'0A' if valid, x'00' if deleted
    on output - the x'0A' is always written
    on input - 1 extra byte is read but not tested by default
    - option 'r1' required to drop record if last byte x'00'
RUF
  • UNIBOL file type
  • if input: will bypass the 8 byte header
  • if output: will create the 8 byte header as follows:
    bytes 00-05 - x'07ff00430000'
    bytes 06-07 - record size in binary
LST
  • Line Sequential Terminated
  • by LF *after* last byte of rcs
  • LST input files are usually variable rcsz
  • LST output files would have the LF after the stated rcs
  • the 't' option would truncate any trailing traling blanks
    to create variable output rcsz's if the data varies.
  • use LST only when required (for variable length text records)
  • LST is 4 times slower than RSF or RST because it reads or writes
    1 character at a time (vs 1 record at a time).
  • record size determined by position of the LineFeed, not by rcs=xxx
  • you should specify record size generously to avoid splitting records
RSV
  • Record Sequential Variable files compatible with Micro Focus COBOL
    IDXFORMAT3 files, see more explanation on the next page --->
STL
  • STandard Language file system, used by AIX COBOL
    - STLs=Sequential, STLi=Indexed, STLr=Relative
    - see STL sort examples on page 'Q1'

Goto:   Begin this document End this document UVSI Home-Page

E2. uvsort file TYP codes

ISF
  • Indexed Seqential Fixed record length D-ISAM/C-ISAM/IDXFORMAT1
  • see more explanation on the next page --->
IDX
  • Indexed Sequential Variable files compatible with Micro Focus COBOL
    IDXFORMAT3/8 files, see more explanation 2 pages ahead --->
RDWz4
  • 'Record Descriptor Word' variable length files (RDW defaults to RDWz4)
  • 4 byte record prefix with 2 byte binary record size & 2 unused nulls
  • record size in prefix is the total record size including the prefix
  • 2 byte record size is 'BIG-END' binary (like mainframe or SUN,AIX,DEC)
  • 'little-end' machines (Intel,AMD) need to switch ends to perform math
RDWz2
  • option 'z2' for 2 byte prefixes (vs 4 default above)
RDWz8
  • option 'z8' same as RDWz4, but bypasses any BDW prefixes

Goto:   Begin this document End this document UVSI Home-Page

E3. TYP codes for Indexed files

typ=ISF Indexed Sequential Fixed files

typ=ISF
  • Indexed Sequential Fixed records (D-ISAM for Micro Focus COBOL)
    (MBP-ISAM & SCO-ISAM now obsolete)

 uvsort "fili1=dat1/sales2,typ=LST,rcs=64,filo1=xx,typ=ISF,isk1=0(6),key1=0(6)"
 ==============================***=====================***=====================

The example above will convert a text input file to an Indexed Sequential Fixed record size file with key in 1st 6 bytes. This is an easy way to create indexed files - just use the editor to create the keys & data fields, with no need to worry about record sizes or key sequences, then use uvsort to create the Indexed file of fixed data records with .idx partition.

export ISDATEXT=.dat

This applies only to D-ISAM typ=ISF Indexed files. You may export ISDATEXT=".dat" if you wish to use '.dat' extensions on D-ISAM files read or written by uvcp (+ uvsort + uvcopy). The default is no extension on data partition, which is compatible with Micro Focus COBOL. Add to /etc/profile if your site always needs '.dat'. Add the export to /etc/profile if your site always needs '.dat'.

typ=RSV MF COBOL file type IDXFORMAT3

typ=RSV
  • Record Sequential Variable files compatible with Micro Focus
    COBOL IDXFORMAT3 files (data partition only not indexed).
  • As of Nov2002 uvcp & uvsort support OUTPUT as well as INPUT RSV files
  • For input record size declared does not matter, make sure it is
    big enough, for example:

 uvsort "fili1=indir/paymast,typ=RSV,rcs=4000,\
 ==============================================
         filo1=outdir/paymast,typ=ISF,rcs=200,isk1=0(6),key1=0(6)"
         =========================================================

The above would convert a Record Sequential Variable (Micro Focus IDXFORMAT3) data file to an Indexed Sequential Fixed record length file. The 'ISF' file is compatible with MF COBOL fixed length Indexed files & C-ISAM files.

Please see pages 'N1' thru 'N3' for more details & examples --->

Goto:   Begin this document End this document UVSI Home-Page

E4. file TYP codes for fili1/filo1 (continued)

typ=IDX Indexed Sequential Variable files

'typ=IDX' declares Indexed Sequential Variable length record files compatible with Micro Focus COBOL IDXFORMAT1,3,8 files. See the 'File Handling' documentation for Micro Focus Object or Server Express COBOL.

The Micro Focus COBOL 'EXTFH' file handler is compiled into an alternate version of uvsort called 'uxsort' using the 'cobccuv' script as explained in install.doc.

'uxsort' supports the same file types as uvsort with typ=IDX in addition. Two versions of the program are required since only Micro Focus COBOL users (server express 2.2+) could compile 'uxsort' whereas any user can compile 'uvsort' which uses D-ISAM to support Indexed Sequential FIXED record length files.

Note that 'typ=RSV' (Record Sequential Variable with no indexed keys) files are supported by both uvsort & uxsort. 'uxsort' must be used when either input or output or both files are typ=IDX.

example - sort text file & load Indexed Variable


 uxsort "fili1=dat1/idxfL,typ=LST,rcs=256\
 =========================================
        ,filo1=tmp/idxf3v,typ=IDXf3v60y4,rcs=252,isk1=0(6d),key1=0(6)"
        ==============================================================

This would read text file dat1/idxfL & build an Indexed Sequential Variable file tmp/idxf3v whose key is the 1st 6 bytes of each record. The output records could vary from 60 to 252 bytes depending on the last nonblank character of input record. 'rcs=252' sets the maximum size & option 'v60' sets the minimum. The records will actually vary from 64 to 256 allowing for the record header (see pages 'N1' to 'N3' which explain the record format).

You can run this example using a test file provided (uvadm/dat1/idxfL).

Goto:   Begin this document End this document UVSI Home-Page

E5. uvsort: file TYP OPTIONS

File options may be appended to the 3 byte file typ code (lower case letters a-z with possible numeric suffixes).

file 'typ' option examples


   uvsort "fili1=text1,typ=LSTb8c1,rcs=80,filo1=text2,typ=LSTtd3,key1=0(20)"
   =========================================================================
                              ^^^^                           ^^^
 a   - append to the file's existing contents
       (vs default of erasing existing contents)
b1
  • reduce multi consecutive blank lines to 1 blank line
b2
  • drop blank records
c1
  • convert unprintable characters (< x'20' or > x'7E') to blanks
c2
  • convert unprintable characters (< x'20' or > x'7E') to periods
  • OK for output file typ=LST since LF inserted after record converted
c4
  • convert nulls to blanks
c8
  • convert tabs to blanks
c16
  • inhibit EOR (end-of-record) on FormFeed x'0C' (as well as LineFeed x'0A')
d1
  • McIntosh file, record terminator is CR x'0D' for typ=LST & typ=RST files
    (a single CR vs LF for unix vs CRLF for DOS option d1)
  • CR is appended for output records & removed from input records
d2
  • Unix files (the default), inserts LF x'0A' on output, removes on input
    for typ=LST & typ=RST files
d3
  • MSDOS file, inserts CR (and LF) when writing typ=LST/RST files
  • removes when reading into the input area
d4
  • insert null as terminator on output typ=LST files
d8
  • insert tilde as terminator on output typ=LST files
e2
  • insert EBCDIC Line-Feed x'25' in last byte of record (typ=RSTe2)
e1
  • insert Carriage-Return x'0D' in 2nd last byte (use with e2+e1=e3)
e3
  • insert CR+LF in last 2 bytes of record (applies only to 'typ=RSTe3'
f#
  • format for typ=IDX files
f1
  • fixed length Indexed records, compatible with C-ISAM, same as typ=RSF
f3
  • variable length Indexed files (file size < 2 gig)
f8
  • variable length Indexed files (allowing file size > 2 gig)
  • IDXf3 & IDXf8 could be fixed records in variable format
g_
  • for typ=IDX/RSV MF COBOL ISAM variable length Indexed/Sequential files
g4
  • get active data records only (default)
g2
  • get deleted data records only
g6
  • get both active & deleted data records
  • option 'g' in this uvcp is similar to record display option 'v'
    in the uvhd display utility (v1=filehdr,v2=deleted,v4=data,v8=all)

Goto:   Begin this document End this document UVSI Home-Page

E6. file TYP OPTIONS (continued)

h1
  • wait on locked file, retries every 10 seconds until lock released
    (when other process closes the file)
  • retry time is gradually increased to every 5 minutes after 30 mins
  • displays waiting message showing total wait time
  • you would use option 'h1' with EXCLUSIVE lock ('l0' or not coded)
  • typ=ISFh1l1 would not make sense, since it will never get a lock
  • See file lock test results in separate section uvlocks.htm
j1
  • translate filename to lower case (before opening)
j2
  • translate filename to UPPER case (before opening)
j4
  • translate any extension to the opposite case

Option 'j' useful when you are processing all files in directory (with uvcopyx) And you want the output files in a different case than input files.

k#
  • key of reference for an ISAM input file
    typ=ISFk2 would cause reading in sequence by key #2
l_
  • file locking options for DISAM indexed files
    & for UNIX Sequential files as of January 1999
  • see more discussion & test/demo's in uvlocks.doc (follows uvcopy5)
l0
  • EXCLUSIVE (default) RDlock+WRlock
  • other jobs cannot access, unless they specify READONLY for ISAM
    or NO-locking for non-indexed sequential files.
  • always use for output files (building new ISAM files).
l1
  • READONLY
  • multiple jobs could read same file if all jobs specify RDlock
  • open rejected if another job has opened with WRlock (or Exclusive)
    OR use option 'h1' to wait until other jobs with WRlock/Exclock close
Note
  • see option 'h1' above (options in alpha sequence) to WAIT on file LOCK
l2
  • AUTOLOCK (applies only to ISAM files in uvcopy utility, not uvsort)
  • only 1 record locked at 1 time in this file
  • records are locked by the 'red' instruction
    & unlocked by whatever instruction follows
    (might be 'upd' to update the record)
l4
  • MANULOCK (applies only to ISAM files in uvcopy utility, not uvssort)
  • assumes the programmer will handle locking manually
    using the 'lck' uvcopy instruction ('islock' DISAM function)
  • Use this option if you want to open 2 paths for the same file
    (fili1 & filr1 for the same filename with typ=ISFl4)
Note
  • you probably wont use ISAM 'RECORD locking' options l2 & l4
  • it is easier to use the 'FILE locking' options
    (l0,l1 for ISAM files; l0,l1,l8 for Sequential files)
l8
  • NO locking, used only for Sequential files (not Indexed)
  • For non-indexed files, only lock options l0, l1, l8 apply.

Goto:   Begin this document End this document UVSI Home-Page

E7. file TYP OPTIONS (continued)

m#
  • file progress messages every m# records
m100000
  • default is every 100,000 records
n1
  • inhibit errmsg if data exceeds record length specified for typ=LST
  • ie, if LineFeed x'0A' not found before specified rcs=...
  • causes over-length records to be split into 2 records.
  • use this when you don't want errmsgs & it is OK to split to 2 recs
n4
  • typ=RDW: specifies 4 byte record-size prefix Numeric digits
  • vs default binary 2 bytes + 2 bytes null
  • also see typ=RDW & options z2/z4/z8
p#
  • parts in multi-part keyed files
q8
  • round up typ=RSV slot size to a nultiple of 8 (vs default 4)
vq8
  • use with option v (typ=RSVvq8), may be required if recsize >4095
  • see option v in alpha sequence below
Note
  • options q1 NO LONGER REQUIRED as of Feb 2016
  • if record-size at 54(4) of file hdr > 4095, we can assume that
    the file hdr size is 1024 & data records rounded up to multiple 8
    (vs multiples of 4), only if option q8 specified
r1
  • for typ=RSRr1 Relative files (MicroFocus COBOL compatible)
  • drop record if extra byte = x'00'
s4
  • detailed ISAM file key statistics (in addition to record counts)
    typ=ISFs4 will display key start byte & length for each key
t
  • truncate Line Variable records typ=LSTt
  • will scan back from the end of each record & insert the LF
    after the last non-blank data character
t4
  • write output recsize same as input recsize
  • LSTt4 inhibits the usual LSTt effect of shortening to last non-blank
    (allows trailing blanks to remain as on input record)

option 't' on typ=RDWt_ controls record padding & truncation

typ=RDWt0
  • default blank fills
typ=RDWt1
  • null fills to end of record
typ=RDWt2
  • scans back to last non-blank or non-null
  • to shorten output record
typ=RDWt3
  • same as RDWt2
  • since either nulls or blanks are truncated

Goto:   Begin this document End this document UVSI Home-Page

E8. file TYP OPTIONS (continued)
  ** file typ option 'v' for typ=ISF **
v#
  • min recsize for typ=ISF C-ISAM variable length Indexed files
  • max recsize specified by rcs=___
note
  • above variable length C-ISAM indexed files NOT compatible with MF COBOL
  • UVSI recommends following IDXf3 for variable length Indexed files
  • Micro Focus COBOL compatible
w2
  • Warning prompt if output file exists
  • may overwrite, change name, or cancel

file typ options for typ=IDX & typ=RSV

Note
  • you must use 'uxsort' (vs uvsort) to process IDXFORMAT3 & IDXFORMAT8
  • options v,x,y apply mostly to uxsort
  • but option v might be used with uvsort for file typ=RSV
IDXf1
  • fixed length Indexed records, compatible with C-ISAM, same as typ=RSF
IDXf3
  • variable length Indexed files (file size < 2 gig)
IDXf8
  • variable length Indexed files (allowing file size > 2 gig)
  • IDXF3 & IDXf8 could be fixed records in variable format
 v9999  - minimum record size option for typ=IDX & typ=RSV files
          For example: 'typ=IDXv400,rcs=5000' v400 declares minimum size 400
          while rcs=5000 declares the maximum size 5000.
        - if option v not specified or specified with no value on typ=IDX or typ=RSV,
          all data records will be fixed size rcs=____, but with variable length
          record headers.
 v#y4   - also see type 'y4' below to determine record size by scanning back
          from max size to the last nonblank/nonnull beyond min size
 v#q8  - q8 roundup output record size (pad x'20's) to multiple of 8 (vs 4)
       - may be required if record size > 4095
x
  • file organization for typ=IDX files
x1
  • Sequential, no Index created
x2
  • Indexed, assumed if 'isk' (Indexed Seqntl Key) is declared
y
  • Recording Mode
y1
  • Fixed Length records, all records forced to size spcfd by rcs=____
  • default for if option 'v' NOT specified (w or w/o min recsize)
y2
  • Variable Length records
  • default if option 'v' IS specified (w or w/o min recsize value)
  • min recsize specified by option 'v', max recsize by rcs=____
y4
  • determine the record size for each record by scanning back from
    max size to last non-blank or non-null beyond the min size
y8
  • fixed length records in variable length format
  • in case you want to copy a variable length file & output all records
    as the max size specified on rcs=___ (but in variable length format

Goto:   Begin this document End this document UVSI Home-Page

E9. file TYP OPTIONS (continued)

z4
  • RDW variable length files with 4 byte record prefixes (default)
    (typ=RDW defaults to typ=RDWz4)
  • 4 byte record prefix with 2 byte binary record size & 2 unused nulls
  • record size in prefix is the total record size including the prefix
  • 2 byte record size is 'BIG-END' binary (like mainframe or SUN,AIX,DEC)
  • 'little-end' machines (Intel,AMD) need to switch ends to perform math
z2
  • option 'z2' for 2 byte prefixes (vs 4 byte default above)
z8
  • option 'z8' same as RDWz4, but bypasses any BDW prefixes
n4
  • note option n4 = 4 byte recsize Numeric digits (vs Binary)

output record size defaults for variable length files

The output record size for IDXf3/IDXf8 variable length records defaults to the corresponding input record size. The record size is passed thru the sort by appending it on the end of the record & using it on the output phase.

This works well if both input & output are variable length (typ=IDXf3v/IDXf8v). If input was fixed (typ=RSF/ISF/IDXf1) & output was variable (typ=IDXf3/IDXf8v), then output records would be in the variable length format, but all same size.

See pages 'N1' thru 'O3' for examples of RSV/IDX & options f,v,x,y --->

Goto:   Begin this document End this document UVSI Home-Page

F1. uvsort: 'rop' - RUN OPTIONS

'rop' is used to declare various options for the entire run. Do not confuse with file options which apply only to a file (fili1/filo1) & which are appended to the file typ code (example: typ=LSTtd). Do not confuse with instruction options which are appended to instruction opcodes (see the uvcp documentation).


 uvsort "fili1=...,typ=...,rcs=...,filo1=xxx,key1=...,rop=m1024v0"
 =================================================================
 - example, option 'm' 1024 meg MAXMEMORY vs default 512 meg
   and option 'v' 0 to inhibit sort progress messages

 export UVSORTROP=m1024v0
 ========================
 - alternate way to specify run options via Environmental Variable 'UVSORTROP'
 - could define in your profile to specify options desired for all uvsort's
b99999
  • input file record bypass count
i99999
  • input file record stop count
o99999
  • put file record stop count
j#
  • input every j#th record (create smaller test files)
j100
  • input every 100th record (drop 99 of each 100 records)
Note
  • options 'i' 'o' 'j' are duplicated here out of alpha sequence
    since they are related to option 'b'
e1
  • enable interupt (usually '^C' or the DELETE key)
  • will report current I/O record counts & allow close or continue
  • disabled by default since 'uvsort's within scripts might be zombied
  • enable when you run uvcp from command line by appending ',rop=e1'
f2
  • input all files (unlimited number) from the fili1 directory
  • or all files matching a specified prefix &/or suffix (if any).
  • see the demo on page 'F4'.
h1
  • Header record pass-thru to output (without sorting)
  • 1st record of input saved & written as 1st record to output file
i99999
  • input file record stop count
m512
  • sort work memory in megabytes (default m512 = 512 meg)
  • max is m2000 (2 Gig), but m640 appears to be max effective
  • WorkFiles * MaxMemory = max filesize for efficient sort
    (see workfiles/mergefiles specified by option 'w' below)
  • 100 workfiles * 512 meg work memory = 51.2 gig (max filesize efficient)
  • workfiles * maxmemory >= filesize means no intermediate sort passes required
    (can create merge files on input phase & merge on output phase)
    (sort time will be mostly input time + output time)
  • No workfiles/mergefiles reqquired if input filesize < sort work memory

Goto:   Begin this document End this document UVSI Home-Page

F2. uvsort: rop options - continued

Note
  • sort work memory default increased from 64 meg to 512 meg in July 2015
  • to take advantage of newer machines with large memories
  • older sites wwith small memories may need to reduce the default
  • if necessary, append ",rop=m64" on the end of uvsort commands
    OR add "export UVSORTROP=m64" to the common_profile or all .profiles
  • the new default m512 may not increase sort speed much, but will greatly
    reduce the number of tmp/sortwork files & reduce chance of matching old ones
    (see bottom page 'F5' re clearing old sortwork files)
n1
  • inhibit wait for reply on all prompts soliciting operator input
  • intended for unattended night shifts
  • could append on each uvsort "fili1=....,rop=n1"
  • could code 'export UVSORTROP=n1' at begining of a night shift script
  • or put in profile of night shift to inhibit all prompts all scripts
o99999
  • input file record stop count
Note
  • both input & output stop counts are provided since the records
    selected could be very different (due to sorting).
r99999
  • declare maximum record size to be 99999 (or whatever)
    vs the default max 32768
  • this also changes the max size allowed for the 'w' work area
s999
  • column# (zero relative displacement) for record count of
    duplicate record sets (see 'key' specifications).
t1
  • (default) puts any SUM totals into FIRST record of the dup key set
t2
  • puts any SUM totals into the LAST record of the dup key set
  • see the 'SUM' feature discussed on page 'H1'
v0
  • inhibit the sort progress messages (% of merge pass completion)
  • Actually you will seldom see these messages because it takes
    a very large sort before they are displayed.
w100
  • workfiles maximum, default is 100 & this is max recommended.
x99999
  • set Max records per output file, causes multi-file output
  • appends _001,_002,etc on declared output filename
  • this option added in October 2016
y70
  • set Year2000 window for field type 'y' on sort keys
    & also for any sel1y/del1y selects/deletes
  • option 'y__' assumes 1st 2 bytes of field is the year (no century)
  • 'y70' means that years 70-99 = 1970-1999, 00-69 = 2000-2069
z
  • option 'z1' specifies sliding windows, for example in 1998
y70z1
  • window value '68' meaning 68-99 = 1968-1999, 00-67 = 2000-2067

Goto:   Begin this document End this document UVSI Home-Page

F3. uvsort: rop options - continued

uvsort options for performance

Most sort times are determined by input & output times. Intermediate sort merges are required only for files larger than 6.4 gigabytes, assuming the default sort memory of 64 megs & 100 sort work files. As the input file is read, each 64 meg portion is sorted & written to up to 100 work files in the tmp subdir. These are then merged during the output phase.

See options 'm' & 'w' (above) for sort memory & number of work files (default 64 megs & 100 work files max). There is no advantage increasing the memory unless the file is > 6.4 gig.

rop options - complete examples


 #1. uvsort "fili1=bigfile,typ=RSF,rcs=351,filo1=bigsorted,key1=71(20),rop=m640"
     ==================================================================********=

rop=m640 - assign 640 megs for internal sort memory (vs 512 megs default)


 #3. uvsort "fili1=xx,typ=RSF,rcs=256,filo1=yy,key1=0(8),rop=b8000,i4000"
     ====================================================***************
     rop=b8000 - bypass the 1st 8000 records in the input file
         i4000 - stop input after 4000 records (8001-12000)

 #3. uvcp "fili1=dat1/saleshist,typ=RSF,rcs=100,filo1=tmp/saleshist,rop=x20000000"
     =============================================================================
      This illustrates Run OPtion 'x' (rop=x____) to split output into multiple
      files of a Max number of records (except for last output file).
      For example, given a large file (5 gigabytes) of 100 byte records,
      You might need to split into multiple files of 2 gig each, for transfer
      to an older system with the 2 gig file limit.
       Use 'rop=x20000000' 20,000,000 records * 100 = 2,000,000,000 bytes
       output: saleshist_001 (2 gig), saleshist_002 (2 gig), saleshist_003 (1 gig)

 #4. uvsort "fili1=warmas1a,typ=RSF,rcs=64,filo1=wm2,key1=44(6y),rop=y70z1"
     ============================================================*********=
     rop=y70z1 - set Year2000 sliding window '70' for sort field type 'y'
                 & also for any sel1y/del1y selects/deletes

Goto:   Begin this document End this document UVSI Home-Page

F4. uvsort: rop options - continued

rop=f2 input all files from directory

Run option f2 allow you to input all files from a directory, or all files matching a specified prefix &/or suffix.

The fili1 pathname must include at least 1 '/' & exactly 1 '*' (a directory & optionally a prefix &/or suffix)

This is the reverse of option 'f1' on key1f1 (see page 'C7'), which writes multiple separate files to a directory based on key1 control breaks.

For example, we could recombine the multiple salesman files that were created on the key1f1 demo on page 'C7'.

ex#1
  • 'fili1=tmp/*' will read all files from the tmp subdir.
ex#2
  • 'fili1=tmp/*.txt' reads all files matching suffix '.txt'
ex#3
  • 'fili1=tmp/sm*.txt' reads files matching prefix 'sm' & suffix '.txt'

example rop=f2 input all files in subdir


 uvsort "fili1=tmp/sm*,typ=RSF,rcs=64,filo1=tmp/allsm,key1=10(2),rop=f2"
 =======================================================================
 - input all files with matching pprefix 'sm'

UVSORTROP - env-var to specify uvsort Run OPtions

Environmental Variable 'UVSORTROP' can be defined in your profile to specify options for all uvsort executions (vs appending ,rop=... on the uvsort command). There are only a few options that you might want to specify this way, for example:


 export UVSORTROP=m640n1v0
 #========================
 # m640 - allocate 640 megabytes for uvsort memory space
 # n1 - inhibit waiting for any reply to uvsort prompts
 # v0 - inhibit sort progress messages (large sorts > 12 MB)

Goto:   Begin this document End this document UVSI Home-Page

F5. environmental variable for sortwork directory


 export SORTDIR=/u2/sortwork   #<-- change sortworkdir from default ./tmp/
 ===========================

'SORTDIR' is an optional environmental variable that may be used to assign the sortwork directory. You could code this in your .profile.

The default sortwork directory is 'tmp' within the current directory (wherever uvsort is executed - not the system '/tmp' subdir). The sortwork directory will be created if it does not already exist.

Note that sort work (merge) files are not created if the data file size is less than the memory assigned to uvsort via run option 'm' (see page 'F2'). The default is 'rop=m512' which assigns 512 megabytes to internal sort work You could increase up to 2 gigabytes, but you wont see much improvement unless your file is larger than about 6 gigabytes, which is the point when uvsort would need to do multi-pass merges. m640 seems to be max effective.

uvsort creates sort work files using the process ID as part of the filename. The process ID is converted to hexadecimal, for example:

      tmp/SORT008466.001   <-- porcess ID 8466
         ---- etc ---      <-- 10 merge files reqquired to sort 4 gig file
      tmp/SORT008466.010

If uvsort fails for whatever reason (hardware, software, or no space left in filesystem where tmp subdir located), then the SORT work files will be left in the tmp subdir & you might want to remove them before rerunning the sort. uvsort automatically removes SORT work files only if more than 2 days old (in case multiple uvsorts are running at the same time).


 find tmp/SORT* -mtime +2 -exec rm -f {} \;
 ==========================================

clear SORTDIR by nightly cron

As noted above, uvsort failures can leave sort work/merge files in the SORTDIR (default ./tmp/... of current dir). There is a chance that new sortwork files might match old ones if the process-id wraps, since the process-id is part of sortwork filenames (as illustrated above).

So, I recommend you clear the SORTDIR by cron at some point when you know no sorts will be running. This will be easier if you assign SORTDIR to a fixed location (vs ./tmp/... in current dir of current user). For example:


 export SORTDIR=/var/sortwork   #<-- in common_profile
 ============================   # sortwork perms 775 & group matches users

 00 3 * * * rm -f /var/sortwork/*   #<-- remove sortwork files at 3 AM nightly
 ================================

Above in crontab file for root, see sample crontab file at http://uvsoftware.ca/admjobs.htm#5F2

Goto:   Begin this document End this document UVSI Home-Page

G1. uvsort: sel/del - SELECT/DELETE RECORDS

      sel1=start(length):dataconstant
      sel1=10(2):21          - select records with '21' in cols 11-12 (slsmn)
      sel2=30(3):HAM           AND 'HAM' in cols 31-33 (product code)
      sel3=38(6)>000025        AND >'000025' in cols 49-54 (quantity)
                               ('AND' condition if 4th byte increases)

      sel1=10(2):21,sel2=30(3):HAM,sel3=38(6)>000025  <-- code contigously
      ==============================================      with ',' separators
      sel1=10(2):21          - select records with '21' in cols 11-12 (slsmn)
      sel1=10(2):44            OR with '44' in cols 11-12
      sel1=10(2):85            OR with '85' in cols 11-12
                               (OR condition if 4th byte reverts to '1')
      del1=90(2)>55          - delete if cols 91-92 > '55'
      del2=99(1)!0xff          AND if column 100 not = x'ff'

'sel' & 'del' are mutually exclusive within Inputs & within Outputs. You can have all dels for Input files & all sels for Output files, but do not mix sels/dels within Inputs or Outputs. You can have a maximum of 500 sels/dels.

The AND/OR condition is indicated by the 4th byte, sel1, sel2, etc to sel8 max. The 'AND' condition is indicated as long as 4th byte increases (sel1,sel2,etc). The 'OR' condition is indicated whenever 4th byte reverts to '1'.

select to multiple output files

You can select to multiple output files using option 'o#' where '#' is the number of the output file declared (filo1,filo2,... max 60). You can use the 'SAVE' option to select all non-selected records to the last file#, for example:

      sel1o1=0(1):A         select records with 'A' in column 1 to filo1
      sel1o2=0(1):B         select records with 'B' in column 1 to filo2
      sel1o3=0(1):C         select records with 'C' in column 1 to filo3
      sel1o4=0(4):SAVE      select all non-selected records to filo4

Goto:   Begin this document End this document UVSI Home-Page

G2. uvsort: sel/del - SELECT/DELETE records

sel/del example

Given
  • /home/uvadm/dat1/sales2 file (3 records listed below)
  • see all 20 records listed on page 'L0'
          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 139923    35 950802 IN111002  TAB013  000023 0003001 000069023
 400002    85 010812 CR245     HAX129  00001v 0006001 00009601v
Required
  • sort by product within salesman & select following records:

 #1. Salesman 21,44,& 65 (columns 10-11)
 #2. Product hammers ('HAM' in cols 31-33)
 #3. sales QTY 10 or more (columns 39-44)

Solution as follows:

 uvsort "fili1=dat1/sales2,rcs=64,typ=RSF,filo1=tmp/s2,key1=10(2),key2=30(6)\
 ,sel1=10(2):21,sel2=30(3):HAM,sel3=38(6z)>9\
 ,sel1=10(2):44,sel2=30(3):HAM,sel3=38(6z)>9\
 ,sel1=10(2):65,sel2=30(3):HAM,sel3=38(6z)>9"

SORT INCLUDE/OMIT conversions to uvsort sel/del

The JCL converter converts mainframe SORT steps to the uvsort equivalent. Any 'INCLUDE/OMIT's are converted to 'sel/del's. Complex SORTs with multiple mixed INCLUDEs & OMITs may require manual changes and a warning is given. Please see several examples at the end of this uvsort documentation.

U1. JCL conversion of Mainframe SORT/SYNCSORT to uvsort
- INCLUDE/OMIT's converted to uvsort sel/del's
- complex INCLUDE/OMIT's with multiple mixed AND/OR's in ((parens))
V0. Examples of SORT/INCLUDEs uvsort/selects that require corections

Goto:   Begin this document End this document UVSI Home-Page

G3. uvsort: sel/del (continued) - option 'o'

option 'o' & 'o#' on sel/del

Option 'o' on sel/del functions cause the record selection to be performed at output time (vs input time). Without any number attached these selects/ deletes will be applied to all output files.

Option 'o#' may specify the output file# to which the selects/deletes are to be applied. For example, to select different salesmen to separate files:


 uvsort "fili1=dat1/sales3,rcs=64,typ=RSF,key1=10(2)\
 ===================================================
 ,filo1=tmp/sm21,filo2=tmp/sm35,filo3=tmp/sm44\
 =============================================
 ,sel1o1=10(2):21,sel1o2=10(2):35,sel1o3=10(2):44"
 =================================================

This is somewhat similar to option 'f1' on key1f1 already demonstrated on page 'C7', but this can be much more selective. For example, we can add additional conditions to select by product & quantity as well as salesman#:


 ,sel1o1=10(2):21,sel2o1=30(3):HAM,sel3o1=38(6)>000009\
 =====================================================
 ,sel1o2=10(2):35,sel2o2=30(3):HAM,sel3o2=38(6)>000009\
 =====================================================
 ,sel1o3=10(2):44,sel2=30(3):HAM,sel3=38(6)>000009\
 =================================================

Rules for option 'o' on sel/del

  1. Output files (filo1=...,filo2=...,etc) must have been declared to match the option 'o#'s specified (o1,o2,...,o60).

  2. The option 'o#'s must appear in sequence, but in fact any out of sequence o#s will be included with the current highest o# yet encountered.

  3. This means if you inadvertently omit the o# option as shown above for the sel2 & sel3 on the 3rd line, the conditions are still properly applied to filo3 as declared on the sel1o3.

Goto:   Begin this document End this document UVSI Home-Page

G4. uvsort: sel/del - SELECT/DELETE RECORDS

option 'u1' - allow coding ' 's as '_'s

Option 'u1' allows you to code blanks as underscores, which is convenient because blanks are not allowed in the uvcp command string. The alternative would be to code blanks in hexadecimal (' ' = 0x20). Here are some examples:

option 'u2' - allow coding ','s as '@'s

Option 'u2' allows you to code commas as at-signs, which is convenient because commas are used to separate commands in the command string and therefore can not be coded as constants.

examples - coding ' ' as '_' & ',' ar '@'


 sel1u1=0(1):_,sel1u2=0(1):@,sel...
 ==================================
 - select record if 1st byte blank ' ' OR 1st byte comma ','

 sel1u1=0(1):_,sel2u2=1(1):@,sel...
 ==================================
 - select record if 1st byte blank ' ' AND 2nd byte comma ','

 sel1u3=0(2):_@,sel...
 ==================================
 - select record if 1st byte blank ' ' AND 2nd byte comma ','
 - coding a 2 byte constant (vs separate 1 byte constants)
 - note option u3 = (u1 + u2)

option 'y' for Year2000 windowed date field selections

     sel1y=20(6)>980630   - select records with dates in cols 21-26
                            from 19980701 up to 20491231

Option 'y' assumes 1st 2 bytes of specified field is the 2 digit year regardless of field length (yymmdd, yymm, yy).

The default window is '50', may be changed by run option 'y' to specify the window value,& option 'z1' for sliding windows.

Note
  • run options are specified on the 'rop' command (ex: rop=y50z1)
     rop=y50  -------- means 50-99 = 1950-1999,& 00-49 = 2000-2049
     rop=y50z1 in 1998 means 48-99 = 1948-1999,& 00-47 = 2000-2047
     sel1y=20(2)>98,rop=y70z1  - select records with years in cols 21-22
                                 from 1998 to 2067 (rop=y70z1) if now 1998

Goto:   Begin this document End this document UVSI Home-Page

H1. uvsort: SUM Function

 sum1=38(6z),sum2=53(9z)        <-- sum example

The 'sum' function accumulates fields from multiple records into 1 record for each duplicate key set. Consider the following complete uvsort example:


 uvsort="fili1=dat1/sales2,rcs=64,typ=RSF,filo1=tmp/sales2,key1=0(6)\
 ===================================================================
 ,sum1=38(6z),sum2=53(9z)"
 =========================

input file

          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 130140    21 940802 IN111001  SCR012  000021 0001001 000021021
 130140    21 940802 IN111001  CHR001  000022 0002001 000044022
 139923    35 950802 IN111002  TAB013  000023 0003001 000069023
 139923    35 950807 IN111002  TAB013  000024 0004001 000096024
 150825    44 960804 IN1122    HAM001  000025 0005001 000125025
 150825    44 960804 IN1122    HAX129  000026 0006001 000156026
                     ** output file **
                                       (sum1)           (sum2)
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 940802 IN111001  HAM001  000063 0000001 000065063
 139923    35 950802 IN111002  TAB013  000047 0003001 000165047
 150825    44 960804 IN1122    HAM001  000051 0005001 000281051

Note that qty & amount have been accumulated into 1 record for each customer#. By default the SUM totals go into the FIRST record of the dup key record set. If you wish to use the LAST record of the dup key set, specify run option 't2'. For example:


 uvsort="fili1=dat1/sales2,rcs=64,typ=RSF,filo1=tmp/sales2,key1=0(6)\
 ===================================================================
 ,sum1=38(6z),sum2=53(9z),rop=t2"
 =========================******

Goto:   Begin this document End this document UVSI Home-Page

I1. uvsort Instructions

This uvsort.doc documents only functions (fili1,filo1,keys,sel/del,etc), but most instructions (mvc,mvn,add,rep,etc) are documented in uvcp.doc.

      logical:  mvc, mvn, clr
   arithmetic:  add, sub, mpy, div, pac, unp, seq
 compare/skip:  cmc, cmn, skp
 scan/replace:  rep, scn, sqz
   translates:  tra, tre, trl, tru, trp
        other:  anc, orc, chx, hxc, tim

Some instructions support 'options' (appended to the instruction code), and these were also documented in uvcp.doc. Instruction option 'o' is unique to uvsort & may be coded on most instructions. Option 'o' identifies instructions to be executed at sort 'output time' vs sort 'input time' (the default).

I2. uvsort Instructions

option 'o' example


 mvno=70(9)53,divo=70(9)38(6)
 ============================

The complete uvsort coding might be:


 uvsort="fili1=dat1/sales2,rcs=64,typ=LSTt,filo1=tmp/sales2,rcs=80,key1=0(6)\
 ===========================================================================
 ,sum1=38(6z),sum2=53(9z),mvno=64(9)53,divo=64(9)38(6)"
 =========================****************************

The 'sum' functions accumulate the qty & amount into 1 record for each customer. The 'mvno' moves the accumulated amount to the end of the record and the 'divo' calculates the average sale (total sales/total qty). The 'o' option is essential to force these instructions to be executed at sort output time (vs the default sort input time).

example input file

          1         2         3         4         5         6         7         8
 12345678901234567890123456789012345678901234567890123456789012345678901234567890
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================================
 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 130140    21 940802 IN111001  SCR012  000021 0001001 000021021
 130140    21 940802 IN111001  CHR001  000022 0002001 000044022
 139923    35 950802 IN111002  TAB013  000023 0003001 000069023
 139923    35 950807 IN111002  TAB013  000024 0004001 000096024
                     ** example output file **
                                       (sum1)           (sum2)
 cust#   slmn YYMMDD invoice  product    qty   price    amount   avg amount
 ===============================================================================
 130140    21 940802 IN111001  CHR001  000063 0002001 000065063  000001032
 139923    35 950807 IN111002  TAB013  000047 0004001 000165047  000003511

Goto:   Begin this document End this document UVSI Home-Page

I3. uvsort Instructions

option 'o#' on instructions for filo1,filo2,etc

Instructions coded with option 'o', but without a number (as on prior page) are applied to all output files.

Instructions coded with o1,o2,o3,...o60 are applied to the corresponding output file (filo1,filo2,filo3,...filo60).

Option 'o' on instructions could be combined with option 'o' on selects/deletes. For example, we will append the salesman's name on the output records as we select the different salesmen to separate files.


 uvsort "fili1=dat1/sales3,rcs=64,typ=RSF,key1=10(2)\
 ===================================================
 ,filo1=tmp/sm21,filo2=tmp/sm35,filo3=tmp/sm44\
 =============================================
 ,sel1o1=10(2):21,sel1o2=10(2):35,sel1o3=10(2):44\
 ================================================
 ,mvco1=64(4):John,mvco2=64(5):Henry,mvco3=64(6):George"
 =======================================================

Goto:   Begin this document End this document UVSI Home-Page

I4. uvsort Instructions

seq - sequence number

Most instrucitons are documented in uvcp.htm, but 'seq' is documented here in uvsort.doc, because it is most applicable to uvsort.

      seq=displacement(length)    <-- instruction format
      seq=75(5)     - sequence# in 75-79 (cols 76-80)
                    - 5 bytes zoned numeric (unpacked)
      seq=75(5p)    - 5 bytes packed
      seq=76(5b)    - 4 bytes binary
      seqo=75(5)    - sequence# at Output time (vs input time)
                    - more applicable to sorts
      seqo1=75(5)   - seq# Output file #1
      seqo2=75(5)   - seq# Output file #2
      seqo3=75(5)   - seq# Output file #3
      seqo1c1=75(5)   - seq# Outfile#1 using counter#1 (default)
      seqo2c2=75(5)   - seq# Outfile#2 using counter#2 (default)
      seqo1c1=75(5)   - seq# Outfile#1 using counter#1 (default)
      seqo1c2=80(5)   - 2nd seq# in Outfile#1 using counter#2
      seqo1c2=85(5)   - can have multiple seq#s in 1 file
      seqo1i10=75(5)  - Increment by 10 (vs 1 default)
      seqo1b1000i10=75(5) - Begin at 1000 & Increment by 10
option 'o#'
  • specifies Output file# 1-40
  • common to most instructions (not just seq#)
option 'c#'
  • Counter#
  • defaults to c0 for input files
  • defaults same as 'o#' for output files
  • specify explicitly for multiple ctrs in 1 file
option 'i#'
  • Increment, defaults to 1
option 'b#'
  • Begin value (before 1st increment)
  • dfaults to zero

 uvsort "fili1=tf/test100,rcs=80,typ=LSTt,filo1=tmp/t1,key1=0(4cd),seqo=35(5)"
 =============================================================================
 - sequence# in 35-39 (cols 36-40)

Goto:   Begin this document End this document UVSI Home-Page

I5. uvsort Instructions

seq - demo using Test File tf/test100

Here is a test/demo of the 'seq' (sequence number instruction), using test file tf/test100. You can run this test/demo as follows:


 #1. Login uvadm --> /home/uvadm

 #2. more tf/test100     <-- display input file
     ===============
      0001 tf/test100 one
      0002 tf/test100 two
      0003 tf/test100 three
        --- 94 records omitted ---
      0098 tf/test100 ninety eight
      0099 tf/test100 ninety nine
      0100 tf/test100 one hundred

 #3. uvsort "fili1=tf/test100,rcs=80,typ=LSTt,filo1=tmp/t1,key1=0(4cd),seqo=35(5)"
     =============================================================================
     - sort on 1st 4 bytes Char Descending
     - sequence# in 35-39 (columns 36-40)

 #4. more tmp/t1        <-- display output file
     ===========
      0100 tf/test100 one hundred        00001
      0099 tf/test100 ninety nine        00002
      0098 tf/test100 ninety eight       00003
             --- 94 records omitted ---
      0003 tf/test100 three              00098
      0002 tf/test100 two                00099
      0001 tf/test100 one                00100

test other seq# instructions


 #5. seq=35(4),seqo=40(4)   <-- seq# 35-39 at Input time
     ====================     - and  40-44 at Output time
                              - 1st 3 records shown below
      0100 tf/test100 one hundred        0100 0001
      0099 tf/test100 ninety nine        0099 0002
      0098 tf/test100 ninety eight       0098 0003

 #6. seqob1000i10=40(6)     <-- seq# 40-45 at Output time
     ==================       - Begin from 1000, Incrementing by 10
                              - 1st 3 records shown below
      0100 tf/test100 one hundred             001010
      0099 tf/test100 ninety nine             001020
      0098 tf/test100 ninety eight            001030

Goto:   Begin this document End this document UVSI Home-Page

J1. uvsort: I/O file declaration - 3 methods

  1. declaring filenames on the COMMAND LINE


    uvsort "fili1=dat1/sales2,rcs=64,typ=RSF,filo1=tmp/sales2,key1=10(8)"
    ========***************==================***************============
  1. declaring input filenames via the QUERY option


    uvsort "fili1=?,rcs=64,typ=RSF,filo1=tmp/allsales,key1=10(8)"
    ========*******=============================================
  1. declaring Filenames in a PARAMETER FILE


    uvsort pf/demo/uvsortsplit1  <-- execute instructions in parameter file
    ===========================
 # uvsortsplit1 - split sales to separate files for specified salesman
 #              - output separate file for salesmen 21,35,44,65,85
 #              - select sales only if extended amount > $ 500.00
 #              - see doc at http://www.uvsoftware.ca/uvsort.htm#J1
 #
 # Compare to 'uvsortsplit2' using 'f1' option on 'key1f1=...'
 # - allows unlimited number of output files with 1 line of code
 # But this 'uvsortsplit1' more flexible if you do not want all salesmen
 # - or if sort key not salesman#
 #
 # export PFD=/home/uvadm/pf/demo  <-- Vancouver Utility Parameter File Demos
 # uvsort $PFD/uvsortsplit2        <-- execute uvsort on this parameter file
 #
 fili1=dat1/sales2,rcs=64,typ=RST
 key1=10(8),key2=53(9z)
 filo1=tmp1/sales2_21,rcs=64,typ=RST,sel1o1=10(2):21,sel2o1=53(9)>000050000
 filo2=tmp1/sales2_35,sel1o2=10(2):35,sel2o2=53(9)>000050000
 filo3=tmp1/sales2_44,sel1o3=10(2):44,sel2o3=53(9)>000050000
 filo4=tmp1/sales2_65,sel1o4=10(2):65,sel2o4=53(9)>000050000
 filo5=tmp1/sales2_85,sel1o5=10(2):85,sel2o5=53(9)>000050000

Goto:   Begin this document End this document UVSI Home-Page

J2. uvsort: I/O file declaration - 3 methods

notes re Multiple Files in uvsort Parameter File

A parameter file would be used when the command line would be too long to be comfortable with, especially if the command line is to be entered from the UNIX prompt, and to a lesser extent when the command is within a script file. Use a parameter file for complex sorts with many sels/dels &/or output files.

Alternatively, you could use the sort facility within the 'uvcopy' utility (vs lengthy complex uvsorts). The uvcopy instructions are more powerful & easier to code but not as compact as uvsort fucntions such as sel/del.

Alternative for Multiple Output Files

 # uvsortsplit2 - demo uvsort split sales to separate files for each salesman
 #              - method #2 using 'f1' option on 'key1f1=...'
 #              - allows unlimited number of output files with 1 line of code
 #              - see doc at http://www.uvsoftware.ca/uvsort.htm#C6 & J2
 #
 # Compare to 'uvsortsplit1' which specifies salesman# & multiple output files
 # (but more flexible if you do not want all slsmn or if sort key not slsmn#)
 #
 # export PFD=/home/uvadm/pf/demo  <-- Vancouver Utility Parameter File Demos
 # uvsort $PFD/uvsortsplit2        <-- execute uvsort on this parameter file
 #
 fili1=dat1/sales2,typ=RST,rcs=64,filo1=tmp1/sales_*,key1f1=10(2)
 #===============================================================
 #
 # Alternative to executing uvsort on this parameter file
 # - can enter uvsort command at shell prompt as follows:
 #
 # uvsort "fili1=dat1/sales2,typ=RST,rcs=64,filo1=tmp1/sales_*,key1f1=10(2)"
 #==========================================================================

Goto:   Begin this document End this document UVSI Home-Page

K1. uvsort: MULTIPLE INPUT FILES - 3 methods

  1. declaring multiple input files on the COMMAND LINE


    uvsort "fili1=sales1,fili2=sales2,rcs=128,typ=RSF,filo1=sales,key1=10(8)"
    ========*************************========================================
  1. run option rop=f2 to input all files from directory - or all files matching a specified prefix &/or suffix


    uvsort "fili1=dat1/sales*,rcs=64,typ=RSF,filo1=allsales,key1=10(8),rop=f2"
    ==============***********==========================================******=
  1. declaring multiple input files via the '??' QUERY OPTION


    uvsort "fili1=??,rcs=64,typ=RSF,filo1=allsales,key1=10(8)"
    ========********==========================================
Note
  • The '??' was removed in Jan2004 when major enhancements made to
    allow multiple inputs from a directory. May be revived sometime
    in future if anybody requests it.

Goto:   Begin this document End this document UVSI Home-Page

L0. uvsort examples

Demo file '/home/uvadm/dat1/sales1' for examples 1-3

          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 990802 IN111001  HAM001  000020 0000101 000002020
 130140    21 000802 IN111001  SCR012  000021 0001001 000021021
 130140    21 010802 IN111001  CHR001  000022 0010001 000220022
 139923    35 980802 IN111002  TAB013  000023 0000201 000004623
 139923    35 990807 IN111002  VAR200  000024 0002001 000048024
 150825    44 010804 IN1122    HAM001  000025 0020001 000500025
 150825    44 020804 IN1122    HAX129  000026 0000301 000007826
 201120    44 030807 CR5234    BBQ001  000027 0003001 000081027
 223240    65 010816 CR955     HAM001  000028 0030001 000840028
 223240    65 020816 IN441     BBQ001  000029 0000101 000002929
 308685    21 020812 IN6605    SAW051  00001p 0000101 00000101p
 308685    21 030812 IN6605    BIT075  00001q 0001001 00001101q
 308685    21 040812 CR8835    TAB013  00001r 0010001 00012001r
 315512    44 040805 IN2251    HAM001  00001s 0000201 00000261s
 315512    44 050805 IN2251    SAW051  00001t 0002001 00002801t
 315512    44 060805 IN2255    BIT075  00001u 0020001 00030001u
 400002    85 990812 CR245     HAX129  00001v 0000101 00000161v
 406082    35 010815 IN33001   BBQ001  00001w 0001001 00001701w
 406082    35 020815 IN33001   TAB013  00001x 0010001 00018001x
 406082    65 030816 IN441     VAR200  00001y 0000101 00000191y

Notes re demo files

'dat1/sales1' is a typical mainframe file - fixed record length & no LineFeeds. This means you cannot view/print using the unix vi, cat, more, lp, etc.


 vi dat1/sales1   <-- try to view with 'vi'
 ==============

'vi' will report the entire file as 1 line & the records we know to be 64 bytes will be all run together on the 80 byte screen lines.


 uvhd dat1/sales1 r256  <-- use 'uvhd' to display files w/o LineFeeds
 =====================
                  10        20        30        40        50        60
 r#    1 0123456789012345678901234567890123456789012345678901234567890123
       0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
      64 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
     128 139923    35 950802 IN111002  CHR001  00002B 0002001 00004402B
     192 139923    35 950802 IN111002  TAB013  00002C 0003001 00006902C

The record size is actually 64, but was specified as 256, which explains why you see the 1st 4 records in alignment (256/4 = 64).

Goto:   Begin this document End this document UVSI Home-Page

L1. uvsort examples

uvsort ex#1 - simple sort

Given
  • the dat1/sales1 file illustrated above.
Required
  • sort by product#

uvsort ex#1 - solution


 uvsort "fili1=dat1/sales1,rcs=64,typ=RSF,filo1=tmp/s1,typ=RST,key1=30(6)"
 =========================================================================

output file

 201120    44 970807 CR5234    BBQ001  00002G 0007001 00018902G
 406082    35 020815 IN33001   BBQ001  00001P 0007001 00011901P
 223240    65 980816 IN441     BIT001  00002I 0009001 00026102I
 139923    35 950802 IN111002  CHR001  00002B 0002001 00004402B
 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 150825    44 960804 IN1122    HAM001  00002E 0005001 00012502E
 223240    65 980816 CR955     HAM001  00002H 0008001 00022402H
 315512    44 000805 IN2251    HAM001  00001L 0003001 00003901L
 400002    85 010812 CR245     HAX129  00001O 0006001 00009601O
 150825    44 960804 IN1122    PLN012  00002F 0006001 00015602F
 308685    21 990812 IN6605    SAW001  00001} 0000001 00000001}
 315512    44 000805 IN2251    SAW051  00001M 0004001 00005601M
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
 139923    35 950802 IN111002  TAB013  00002C 0003001 00006902C
 308685    21 990812 CR8835    TAB013  00001K 0002001 00002401K
 406089    35 020815 IN33001   TAB013  00001Q 0008001 00014401Q
 139923    35 950807 CR5421    VAR200  00002D 0004001 00009602D
 406082    65 020816 IN441     VAR200  00001R 0009001 00017101R
 308685    21 990812 IN6605    WID115  00001J 0001001 00001101J
 315512    44 000805 IN2255    WID115  00001N 0005001 00007501N
                         ** Notes **
 1. Note that the file is now in sequence by product code bytes 30-35
    (or columns 31-36).
  1. Note that we specified 'typ=RST' for the output filo1=tmp/s1, to insert LineFeeds in the last byte, so we can inspect the file with vi,cat,more,etc & have each record on a separate line, (vs run together across width of your screen).

Goto:   Begin this document End this document UVSI Home-Page

L2. uvsort examples

uvsort ex#2 - simple sort

Given
  • the dat1/sales1 file illustrated on page 'L1'.
Required
  • sort by product# within salesman.
  • also create a text output file
    (with LineFeeds, so we can inspect with vi,cat,more,etc)

uvsort ex#2 - solution


 uvsort "fili1=dat1/sales1,rcs=64,typ=RSF,filo1=tmp/s1,typ=LSTt\
 ================================================================
 ,key1=10(2),key2=30(6)"
 =======================

The command is shown on 2 lines for this documentation & but you can enter it all on 1 line or you can use the '\' to complete the command on a 2nd line.

output file

 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 308685    21 990812 IN6605    SAW001  00001} 0000001 00000001}
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
              - - - - - some records omitted - - - - -
 223240    65 980816 CR955     HAM001  00002H 0008001 00022402H
 406082    65 020816 IN441     VAR200  00001R 0009001 00017101R
 400002    85 010812 CR245     HAX129  00001O 0006001 00009601O

Notes

  1. Note that the records are in sequence 1st by key1=10(2) (salesman), 2nd by key2=30(6) (product#).

  2. The input typ=RSF means 'Record Sequential Fixed' (no LineFeeds).

  3. The output file type is now changed to 'typ=LSTt' which means 'Line Sequential Terminated (with LineFeeds)'.

  4. The 't' option on 'LSTt' truncates trailing blanks. The LineFeed is inserted after the right hand non-blank.

Goto:   Begin this document End this document UVSI Home-Page

L3. uvsort examples

uvsort ex#3 - selecting records

Given
  • the dat1/sales1 file illustrated on page 'L1'.
Required
  • sort by product within salesman & select following records:

 #1. Salesman 21,44,& 65 (columns 10-11)

 #2. Product hammers ('HAM' in cols 31-33)

uvsort solution


 uvsort "fili1=dat1/sales2,rcs=64,typ=RST,filo1=tmp/sales2,key1=10(2),key2=30(6)\
 ================================================================================
 ,sel1=10(2):21,sel2=30(3):HAM,sel1=10(2):44,sel2=30(3):HAM\
 ===========================================================
 ,sel1=10(2):65,sel2=30(3):HAM"
 ==============================

output file


 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 990802 IN111001  HAM001  000020 0000101 000002020
 150825    44 010804 IN1122    HAM001  000025 0020001 000500025
 315512    44 040805 IN2251    HAM001  00001s 0000201 00000261s
 223240    65 010816 CR955     HAM001  000028 0030001 000840028

Notes

  1. We have used dat1/sales2 as the input file for this example, so you will find it easier to check the outputs with 'vi', because sales2 has LineFeeds in the last byte.

  2. 'typ=RST' indicates LF in last byte of rcs=64. The LF will be removed when read in but reinserted on output since 'typ=RST' is also in effect for output (unless typ=... re-specified).

Goto:   Begin this document End this document UVSI Home-Page

L4. uvsort examples

uvsort ex#4 - loading indexed (ISAM) files

given
  • input test file uvadm/dat1/custmas1,
  • record layout & 1st record (of 32) is listed below.
  • since the record contains many packed fields, we used the 'uvhd'
    (hexdump utility) to list the 1st record.
           001-006 - customer#
           009-009 - Delete flag 'D'
           011-035 - customer name
           036-060 - address
           061-076 - city
           078-079 - province
           081-090 - postal code
           091-102 - phone#
           103-120 - contact name
           121-180 - this year sales 12 x 5 byte packed fields
           181-240 - last year sales 12 x 5 byte packed fields
           241-256 - unused (no LineFeed at end record)
Note
  • above layout is ONE relative
  • you must specify ZERO relative displacements for uvsort (subtract 1)

    uvhd dat1/custmas1 r256  - execute uvhd to display 1st record
    =======================
                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    604-754-5531                  ..4V|...
             4442222222222442535343222233323332333322222222222222222201357000
             9DF00000000002306931810000604D754D55310000000000000000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ...............................f.....<........f.C
             0000008900000000880000000018000680001300000000694222222222222222
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3000000000000000
REQUIRED
  • sort & load an ISAM file with 2 keys
  • customer# (cols 1-6) & city name (cols 61-76)

uvsort solution


 uvsort "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/custmas1,typ=ISF,\
 =======================================================================
 isk1=0(6),isk2=60(16),key1=0(6),key2=60(16)"
 ===========================================

The sort keys are coded as key1,key2,etc. 'isk' is the code for 'Indexed Sequential Key', which could be different than the sort keys, but usually the same.

Goto:   Begin this document End this document UVSI Home-Page

L5. uvsort examples

uvsort ex#5 - Re-Organizing Indexed files

Example #2 (on the previous page) sorted a 'sequential' file & loaded an 'indexed' file. This example will input an indexed file, sort,& write back to the same file.

Writing back to the same file could be a problem if your machine just happened to crash at while writing the output, so you should ensure that a backup has preceded a job such as this.

Sorting & reloading an indexed file should be done periodically to reorganize the indexed file & drop any deleted records. Deleted records could be both physical & logical.

'Reorganizing' means to put the data partition records back into sequence to improve sequential access times. Randomly added records will be at the end of the data partition & reading, sorting,& reloading will put them back into physical sequence depending on their key fields.

Physical deletes are usually performed by COBOL program 'delete' commands. The micro Focus COBOL file handler changes the delete flag (an extra byte at end of each record) from x'0A' to x'00'. The uvsort program wont see these records because the D-ISAM file handler (compatible with COBOL file handler) drops them automatically.

Logical deletes are probably also performed by a COBOL program, but rather than use the 'delete' command, the program sets a data byte to some agreed upon value that other programs will check. In our uvsort example below we will assume that a 'D' in byte 8 (column 9) identifies a logically deleted record. uvsort will drop it due to the 'del=8(1):D'.

Indexed file sort/reload demo


 #1. cp dat1/custmast* tmp      - copy custmast & custmast.idx to tmp subdir
     =====================

 #2. uvsort "fili1=tmp/custmast,rcs=256,typ=ISF,filo1=tmp/custmast\
     =============================================================
     ,key1=0(6),isk1=0(6d),del=8(1):D"
     =================================

Console Log

 uvsort:050330:115707: uvsort fili1=tmp/custmast,filo1=tmp/custmast
 uvsort:050330:115707: EOF fili01 30 rds, 8224 size; tmp/custmast
 uvsort:050330:115707: DISAM: recs=30,rcsz=256,keys=7; tmp/custmast
 uvsort:050330:115707: sort phase complete, used 64 megabytes, sort output begin
 uvsort:050330:115707: EOF filo01 27 wrts, 0 size; tmp/custmast
 uvsort:050330:115707: DISAM: recs=27,rcsz=256,keys=1; tmp/custmast

Goto:   Begin this document End this document UVSI Home-Page

L6. uvsort examples

uvsort ex#6 - reformatting records

given
  • input test file uvadm/dat1/custmas1
  • same file as on previous example
  • input record layout is same as previous example
    (see 1st record listed on prior page)
  • output record layout as follows:
           001-006 - customer#
           009-009 - delete flag (D or X)
           011-035 - customer name
           036-047 - phone# (from 091-102 of input)
REQUIRED
  • Create a customer Telephone Number list
  • sort by telephone#
  • reformat the record to include only cust#, cust name,& telephone#

uvsort solution


 uvsort "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/phones,typ=LSTt,rcs=80\
 ============================================================================
 clr=35(45):0x20,mvc=35(12)90,key1=35(12)"
 =========================================

sample output file

 313720    MONITREX ENGINEERING LTD
 306959  X 356582 ALBERTA LTD.          -   -
 208060    E&L TRUCKING              403-231-7312
 201120    ALLTYPE RENTAL LTD.       403-246-5274
 308685    FOOTHILLS ELECTRIC        403-296-4856
 211140  D FORD NEW HOLLAND          403-312-2760
 318833    TOP NOTCH CONSTRUCTION    403-385-2965
 224700    OTTO MOBILES WESTERN LTD. 403-467-3197
 223240  X NICHOLSON CHEVROLET LTD.  403-467-4257
 234300    TERRITORIAL REWIND        403-512-3948
 204700    CASE POWER EQUIPMENT      403-538-1743
 231550    SPARROW ELECTRIC CO. LTD. 403-742-6139
 301120  D ALBERTA GAS CHEMICALS LTD 403-952-2475
 401210    COAST RANGE CONSTRUCTION  604 681 2626

We might have dropped the deleted records (X or D in column 9), by adding the following to the uvsort command string:


 ,del1=8(1):D,del1=8(1):X
 ========================

The 'mvc=35(12)90' is performed during input, so the sort key is specified as 'key1=35(12)'. The 'key=...' is specified following the 'mvc=...', but in fact the command sequence does not matter. You can perform 'mvc's at output time using the 'o' option (mvco=... see page 'I3').

Goto:   Begin this document End this document UVSI Home-Page

L7. uvsort examples

uvsort ex#7 - convert text file to fixed length Indexed file

COBOL programs (originally from a mainframe) often require Fixed length records & Indexed files. It is difficult to create such a file with the usual unix utilities. here is an example of creating such a file with the 'vi' editor & then using 'uvsort' to load into an Indexed file with the reqqquired record-size & Indexed key.

GIVEN
  • input text file uvadm/dat1/products
  • variable length text records (terminated by LineFeeds)
  • could prepare with vi editor for loading with uvsort
  • sample records as follows
          BBQ010  BAR-B-Q
          CHR015  LAWN CHAIR
          HAM010  CLAW HAMMER
          --- column layout 1 relative ---
          001-006 - product code
          008-008 - reserve for delete flag (D)
          011-030 - product description
          031-039 - sales (to be initialized to zeros)
REQUIRED
  • convert variable length text records to fixed length Indexed file
  • record size 40, Indexed key product code (1-6)
  • initialize columns 31-39 with zeros for sales acumulation

uvsort solution

 uvsort "fili1=dat1/products,rcs=80,typ=LST,filo1=tmp/prodmaster,rcs=40,typ=ISF\
        ,isk1=0(6),key1=0(6),mvn=30(9z):0"
 ===============================================================================

sample output file

          BBQ010   BAR-B-Q              000000000
          CHR015   LAWN CHAIR           000000000
          HAM010   CLAW HAMMER          000000000
  1. Of course we might have several more data fields in real situation.

  2. 'mvn=30(9):0' initializes a zoned (unpacked) field of 9 bytes could use 'mvn=30(5p)' to initialize a packed 9 digit field

Goto:   Begin this document End this document UVSI Home-Page

M1. rebuilding Indexed files from the data partition of corrupted file s

Microfocus COBOL Indexed files (IDXFORMAT1)

 Data    partition filename: 'filename'     <-- NO .dat extension
                             'filename.dat'   - OR with .dat if ISDATEXT=".dat"
 Indexed partition filename: 'filename.idx' <-- Indexed part always ".idx"

C-ISAM data partition filename: 'filename.dat' <-- C-ISAM convention

No control area at begining of data records (vs MBP ISAM files)

Data records are 1 byte larger than record size loaded because a LineFeed is appended to each data record. The LineFeed is converted to a x'00' if the record is deleted

recovering Microfocus data records

The following will demonstrate the process by loading an Indexed file, pretending that the Index is corrupted,& then rebuilding the Indexed file from the data partition only.


 #0. Create an Indexed file for recovery test/demo

     uvcp "fili1=dat1/custmas2,rcs=128,typ=RSF,filo1=tmp/cm2,typ=ISF,isk1=0(6)"
     ==========================================================================

 #1. Copy the data partition to a save file

     cp tmp/cm2.dat tmp/cm2.sav
     ==========================

 #2. erase the original filename.dat & filename.idx

     rm tmp/cm2.dat tmp/cm2.idx
     ==========================

 #3. sort/reload the Indexed file from the data partition

     uvsort "fili1=tmp/cm2.sav,rcs=129,typ=RSF,filo1=tmp/cm2,rcs=128,typ=ISF\
     =======================================================================
            ,isk1=0(6),key1=0(6),del=128:0x00"
            =================================
Note
  • Also see alternative methods using uvcopy in section uvcopy4.doc
    recovISMF - recover Microfocus COBOL files with corrupted indices
    recovISMB - recover MBP COBOL ISAM files with corrupted indices

Goto:   Begin this document End this document UVSI Home-Page

M2. rebuilding Indexed files from the data partition of corrupted files

10b - MBP COBOL Indexed files

      Data    partition filename is: 'filename.DAT'
      Indexed partition filename is: 'filename.KEY'

588 byte binary control area at begining of data records (Microfocus ISAM files do not have).

Data records are 2 bytes larger than record size loaded because a 2 byte binary record size is prepended to each data record. The record size is converted to a x'0000' if the record is deleted.

Note
  • See tip# 11b in UNIXtips.doc for an illustration of the control
    area at the begining of an MBP COBOL ISAM file.

recovering MBP data records from an Indexed file

The following will demonstrate the process by loading an Indexed file, pretending that the Index is corrupted,& then rebuilding the Indexed file from the data partition only.


 #0. Create an Indexed file for recovery test/demo

     uvcp "fili1=dat1/custmas2,rcs=128,typ=RSF,filo1=tmp/cm2,typ=ISF,isk1=0(6)"
     ==========================================================================

 #1. Copy the data partition to a save file

     cp tmp/cm2.DAT tmp/cm2.sav1
     ===========================

 #2. erase the original filename.DAT & filename.KEY

     rm tmp/cm2.DAT tmp/cm2.KEY
     ==========================

 #3. drop the control area from the begining of the data partition
    (using the UNIX 'dd' utility)

     dd if=tmp/cm2.sav1 of=tmp/cm2.sav2 bs=588 skip=1
     ================================================

 #4. sort/reload the Indexed file from the data partition

     uvsort "fili1=tmp/cm2.sav2,rcs=130,typ=RSF,filo1=tmp/cm2,rcs=128,typ=ISF\
     ========================================================================
             ,isk1=0(6),key1=0(6),del=0(2):0x0000,mvc=0(128)2"
              ================================================

Goto:   Begin this document End this document UVSI Home-Page

N1. Record Sequential Variable length files (typ=RSV)

As of Nov 2002 'typ=RSV' supports OUTPUT files as well as INPUT files. 'typ=RSV' provides compatibility with Micro Focus COBOL IDXFORMAT3 Variable length files, but only for Sequential. Indexed variable length support is provided by an alternate version of uvsort (uxsort linked with EXTFH).

For complete documentation on these files, please see the 'FILE STRUCTURES' chapter in Micro Focus Object COBOL or Server Express COBOL, or the web site http://supportline.microfocus.com/documentation/books/ocds4140/fhfile.htm.

'typ=RSV' files have 128 byte file header. Each record has 2 or 4 byte record header/prefix which has x'40' in the first nibble to identify a data record, followed by the record size in the next 12 bits or 28 bits, depending on max recsize <= 4095 or > 4095 bytes.

uvcp/uvsort RSV Output Implementation

  1. The maximum rcsz is specified by 'rcs=____' & the minimum rcsz is specified by option 'v' of 'typ=RSVv____'. Record size determined by scanning backward to last non-blank & round up to a multiple of 4 (includes 2/4 byte rec hdr).

  2. If minimum rcsz (option v) is not specified, all output records will be written with maximum record size & blank padded as necessary.

  3. 'uvsort' allows any combination of INPUT/OUTPUT file types (LST,RSF,ISF,RSV) & allows you to convert files from any type to any other type. For example you can create test files by using 'vi' to create text records & then use uvcp/uvsort to convert the text file (typ=LST) to a typ=RSV output file.

    uvcp/uvsort typ=RSV examples


 #1. uvcp "fili1=dat1/testRSV1,rcs=256,typ=LST,filo1=tmp/a,rcs=254,typ=RSV"
     ======================================================================

 #2. uvcp "fili1=dat1/testRSV1,rcs=256,typ=LST,filo1=tmp/b,rcs=254,typ=RSVv62"
     =========================================================================
    Same as ex#1 but writes typ=RSV records varying from 128 to 256
    - 064 bytes (2 hdr + 062 data) if bytes 062+ all blank
      ??? bytes (2 hdr + ??? data) depending on last nonblank (multiple of 4)
    - 256 bytes (2 hdr + 254 data) is maximum

 #3. uvcp "fili1=dat1/testRSV1,rcs=256,typ=LST,filo1=tmp/c,rcs=5000,typ=RSVv400"
     ===========================================================================
    This example writes 4 byte record headers (vs 2 bytes in above examples)
    because the maximum record size is > 4095 bytes (rcs=5000).
    Record sizes vary from 404 to 5004 depending on last nonblank
    -  404 bytes (4 hdr +  400 data) if bytes 400+ all blank
      ???? bytes (4 hdr + ???? data) depending on last nonblank (multiple of 4)
    - 5004 bytes (4 hdr + 5000 data) is maximum

Goto:   Begin this document End this document UVSI Home-Page

N2. Record Sequential Variable length files (typ=RSV)

using uvcp to create a typ=RSV file


 #2. uvcp "fili1=dat1/testRSV3,rcs=256,typ=LST,filo1=tmp/test3,rcs=254,typ=RSVv80"
    =============================================================================

input text file (3 records)


 #1 testRSV3 - text to create Rec Seqntl Variable(RSV) file to test uvcp & uvsort
 #2 uvcp "fili1=dat1/testRSV3,rcs=256,typ=LST,filo1=tmp/test3,rcs=254,typ=RSVv80"
 #3 output records vary from min(84) to max(256) depending on last nonblank-----|
 -----this record will be 164 (data 160 + 2 byte rechdr + 2 round to mult 4)---->

output RSV file (displayed with uvhd)


 uvhd tmp/test3 v      <-- display file using uvhd with option 'v'
 ================
                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 0~......030114102019.................>.......................P..
             370000003333333333330000000000000000030000000000000000000F000500
             0E00000003011410201900000000000000000E0100000000100000000E000000
          64 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
                       1         2         3         4         5         6
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 @P#1 testRSV3 - text to create Rec Seqntl Variable (RSV) file to
             4523276775553222767727626766762566256767625676666622555226666276
             00310453423630D0458404F032514502530351E4C0612912C5082369069C504F
          64  test uvcp/uvsort
             27677277672777677222
             0453405630F563F24000
                       1         2         3         4         5         6
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         212 @P#2 uvcp "fili1=dat1/testRSV3,rcs=256,typ=LST,filo1=tmp/test3,r
             4523277672266663366732767755532767333327773455266663376727677327
             0032056300269C91D4141F45342363C233D256C490DC34C69CF1D4D0F45343C2
          64 cs=254,typ=RSVv80"
             67333327773555733222
             33D254C490D236680200
                       1         2         3         4         5         6
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
         296 @.#3 output records vary from min(84) to max(256) depending on l
             4A23267777727666767276772676626662332276266723332266766666626626
             00330F540540253F24306129062FD0D9E884904F0D188256904505E49E70FE0C
          64 ast nonblank-----|-----this record will be 164 (data 160 + 2 byt
             6772666666662222272222276672766676276662662333226676233322232677
             1340EFE2C1EBDDDDDCDDDDD48930253F24079CC025016408414101600B020294
         128 e rechdr + 2 round to mult 4)---->
             627666672223276766276267672322222322
             502538420B0202F5E404F0D5C4049DDDDE00

Goto:   Begin this document End this document UVSI Home-Page

N3. Record Sequential Variable length files (typ=RSV)

File Header record for typ=RSV (& typ=IDX) files

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 0~......021125110120.................>.......................~..
             370000003333333333330000000000000000030000000000000000000F000700
             0E00000002112511012000000000000000000E0100000000100000000E000E00
          64 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000

00-00 - x'30' identifies the file header record (vs x'40' for data records)

 01-01 - data size of this record x'7E' = (7*16+14*1) = 112+14 = 126
       - total size is 128 including the 2 byte record header
 00-03 - If max rcsz > 4095, the record header would be 4 bytes x'3000007C'
       - 7*16 + 12*1) = 112+12 = 124 + 4 byte header = 128 total record size

08-19 - file creation date/time stamp (yymmddHHMMSS)

36-36 - reserved value x'3E' required for Micro Focus COBOL IDXFORMAT3 files

39-39 - file ORGanization x'01' = Sequential

48-48 - Recording Mode x'01' = Variable format

 54-57 - Maximum record length
       - In this example x'000000FE' = (15*16+14*1) = 240+14 = 254
 58-61 - Minimum record length
       - In this example x'0000007E' = (07*16+14*1) = 112+14 = 126

typ=IDX files same as typ=RSV with index added

The file structure described above for typ=RSV files also applies to typ=IDX files which include an indexed partition & some extra system records in the data partition. You must use the alternate program 'uxsort' (vs 'uvsort) if either input or output or both files are typ=IDX.

Goto:   Begin this document End this document UVSI Home-Page

O1. 'uxsort' test/demo Indexed Seqntl Variable length files

test/demo uxsort for Indexed Seqntl Variable length files

'uxsort' supports Indexed Sequential Variable length record files & must be compiled with the Micro Focus COBOL 'cob' command as shown on page D1 of install.htm. You can demo uxsort using test files provided as shown below:

dat1/idxtest - test file to demo uxsort

 0010.0060 idxtest text file to create Var Lth Indexed files.
 0020.0064 - input to COBOL,uxcp,uxcopy to create IDXf1,3,8 files
 0030.0065 - 01-04=ISAM key, 05=separator, 06-09=size, 10-400 text
 0040.0067 - output record varies from 64 to 256, depending on 06-09
 0050.0080 - here is an 80 byte record------------------------------------------>
 0060.0160 - here is a 160 byte record...........................................
 0070.0240 - here is a 240 byte record (last record in file).....................
 ===============================================================================>

uxsort - read text (typ=LST), write Indexed file (typ=IDXf3)


 #1. uxsort "fili1=dat1/idxtest,rcs=256,typ=LST\
     ===========================================
             ,filo1=tmp/idxf3v60,rcs=254,typ=IDXf3v60,isk1=0(4),key1=0(4)"
             ============================================================

 #2. uvhd tmp/idxf3v60.dat v    <-- examine output data partition with 'uvhd'
     =======================      - need option 'v' for IDXf3 files
Note
  • See uvhd output (in vertical hexadecimal) on the next page --->

uxsort - read Indexed (typ=IDXf3), write text (typ=LST)


 #3. uxsort "fili1=tmp/idxf3v60,rcs=256,typ=IDXf3,filo1=tmp/idxf3v60vL,typ=LSTt,key1=0(4)"
     ============================================================================

 #4. vi tmp/idxf3v60L           <-- examine output file with 'vi'
     ================

This converts the typ=IDXf3 file back to a text file (typ=LSTt) which should be the same as the original file used in step #1. There is no need to specify the indexed keys or the 'v' option for input files.

Goto:   Begin this document End this document UVSI Home-Page

O2. 'uxsort' test/demo Indexed Seqntl Variable length files

uvhd hexdump of IDXf3 varlth indexed file


 #1a. uvhd tmp/IDXf3v60.dat v   <-- examine output data partition with 'uvhd'
      =======================     - uvhd option 'v' for IDXf3 or IDXf8 files
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 0~......0705141326133507051413261335.>.......................<..
             370000003333333333333333333333333333030000000000000000000F000300
             0E00000007051413261335070514132613350E0200000000100000000E000C00
          64 ............................................X"8*................
             0000000000000000000000000000000000000000000052320000000000000000
             00000000000020100000000000000000000000000000828A0000000000000000
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         384 @<0010.0060 idxtest text file to create Var Lth Indexed files...
             4333332333326677677276772666627626766762567247624666766266667200
             0C0010E00600948453404584069C504F032514506120C4809E45854069C53E00
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         448 @@0020.0064 - input to COBOL,uxcp,uxcopy to create IDXf1,3,8 fil
             4433332333322266777276244444277672776677276267667624456323232666
             000020E00640D09E05404F03F2FCC5830C583F0904F0325145094861C3C8069C
          64 es..
             6700
             5300
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
         516 @A0030.0065 - 01-04=ISAM key, 05=separator, 06-09=size, 10-400 t
             4433332333322233233345442667223337676767672233233376762233233327
             010030E00650D001D04D931D0B59C005D3501214F2C006D09D39A5C010D40004
          64 ext.
             6770
             5840
                      10        20        30        40        50        60
 r#        5 0123456789012345678901234567890123456789012345678901234567890123
         584 @C0040.0067 - output record varies from 64 to 256, depending on
             4433332333322267777727666762767667267662332762333226676666662662
             030040E00670D0F540540253F240612953062FD06404F0256C04505E49E70FE0
          64 06-09...
             33233000
             06D09000
                      10        20        30        40        50        60
 r#        6 0123456789012345678901234567890123456789012345678901234567890123
         656 @P0050.0080 - here is an 80 byte record-------------------------
             4533332333322266762672662332677627666762222222222222222222222222
             000050E00800D0852509301E080029450253F24DDDDDDDDDDDDDDDDDDDDDDDDD
          64 ----------------->..
             22222222222222222300
             DDDDDDDDDDDDDDDDDE00
                ----- last 3 records omitted to keep this on 1 page -----

Goto:   Begin this document End this document UVSI Home-Page

O3. 'uxsort' test/demo Indexed Seqntl Variable length files

create IDXf8 varlth indexed file & examine with uvhd


 #1. uxsort "fili1=dat1/idxtest,rcs=256,typ=LST\
     ==========================================
             ,filo1=tmp/IDXf8v60,rcs=254,typ=IDXf8v60,isk1=0(4),key1=0(4)"
             =============================================================

 #2. uvhd tmp/IDXf8v60.dat v  <-- examine output data partition with 'uvhd'
     =======================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 3.......0705141522382607051415223827.>.......................<..
             3F0000003333333333333333333333333333030000000000000000000F000300
             3E00000007051415223826070514152238270E0200180000106000000E000C00
          64 ............................................X"8*X"8*............
             0000000000000000000000000000000000000000000052325232000000000011
             00000007000022100000000000000000000000000008828A828A000000000000
         128 ................................................................
             00000011000000000000000000000000000000000000000F0000000000000000
             000000004244010400000040000000C0000000000000003E0000000000000000
                     ----- bytes 192-960 removed (all x'00's) -----
         960 ................................................................
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
        3328 @<0010.0060 idxtest text file to create Var Lth Indexed files...
             4333332333326677677276772666627626766762567247624666766266667200
             0C0010E00600948453404584069C504F032514506120C4809E45854069C53E00
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
        3392 @@0020.0064 - input to COBOL,uxcp,uxcopy to create IDXf1,3,8 fil
             4433332333322266777276244444277672776677276267667624456323232666
             000020E00640D09E05404F03F2FCC5830C583F0904F0325145094861C3C8069C
          64 es..
             6700
             5300
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
        3464 @A0030.0065 - 01-04=ISAM key, 05=separator, 06-09=size, 10-400 t
             4433332333322233233345442667223337676767672233233376762233233327
             010030E00650D001D04D931D0B59C005D3501214F2C006D09D39A5C010D40004
          64 ext.
             6770
             5840
                      10        20        30        40        50        60
 r#        5 0123456789012345678901234567890123456789012345678901234567890123
        3536 @C0040.0067 - output record varies from 64 to 256, depending on
             4433332333322267777727666762767667267662332762333226676666662662
             030040E00670D0F540540253F240612953062FD06404F0256C04505E49E70FE0
          64 06-09...
             33233000
             06D09000
                    ----- records 6,7,8 not shown to fit 1 page -----

Goto:   Begin this document End this document UVSI Home-Page

O4. 'uxsort' test/demo Indexed Seqntl Variable length files

IDXORMAT3/IDXFORMAT8 file header layout

The Vancouver Utilities (uxcopy,uxsort,uxqrpg,uxcp,uvhd) are compatible with Micro Focus COBOL variable length sequential & Indexed files (IDXFORMAT3 & 8).

We will document the most important fields in the file header for IDXf3/IDXf8. If desired, you can find all fields defined on the Micro Focus website.

Note that you can investigate the file headers using 'uvhd' with option 'v' for IDXFORMAT3 or IDXFORMAT8. See sample on the previous page.

 000-003 - file header record length
         - 1st 4 bits x'30' identifies a system record
         - next 12 bits is header length if max record length is < 4095
         - next 28 bits is header length if max record length is => 4095

008-014 - creation date/time yymmddhhmmsscc (indexed files only)

 039-039 - organization (1=seqntl, 2=indexed, 3=relative)
 043-043 - file type (3=IDXFORMAT3, 8=IDXFORMAT8)
 048-048 - recording mode (0=fixed, 1=variable)
 054-057 - maximum record length (big end format)
         - example recsize 80 = x'00000050'
 058-061 - minimum record length
 068-071 - record count (4 bytes Big-End Binary)
         - Indexed files only (not in variable length non-indexed file hdrs)

128-135 - offset to End-of-File (8 bytes binary big end format)

 140-141  - number of keys in Indexed file

record header layout

 000-000 - 1st 4 bits identifies the record type
 x'1_'   - system record - duplicate occurrence record
 x'2_'   - deleted record
 x'3_'   - system record
 x'4_'   - user data record
 x'5_'   - reduced user data record
 x'6_'   - pointer record
 x'7_'   - user data record referenced by a pointer record
 x'8_'   - reduced user data record referenced by a pointer record
 000-001 - next 12 bits is the record length if max recsize < 4095
 000-003 - next 28 bits is the record length if max recsize =? 4095
 002-... - record data if max recsize < 4095
 004-... - record data if max recsize => 4095
Note
  • all records are padded with x'00's to a multiple of 4 bytes

Goto:   Begin this document End this document UVSI Home-Page

O5. 'uxsort' test/demo Indexed Seqntl Variable length files

Test typ=IDX file options

See page 'E6' which documents file options 'v' 'x' 'y' 'z' to modify the default behavior of the Micro Focus COBOL EXTFH file handler. Note that these tests are the same as #1 on the previous page with the addition of options x,y,or z on the output file typ=IDXv60__?__

scripts to test typ=IDX file options

The next few pages will list some scripts used at UV Software to test/demo using uxsort to write IDXFORMAT3/IDXFORMAT8 files & read them back to verify.

These scripts are supplied in /home/uvadm/sf/demo/..., which is in the PATH of the supplied profile. You can run these at your site. See instructions & explanations embedded within the scripts.

O6. uxsortIDXf3w - write various IDXf3 file formats
  - sorts a text file & writes out 6 files with various options
O7. uxsortIDXf8w - write various IDXf8 file formats
  - reads a text file & writes out 6 files with various options

can use 'uxcp' scripts to read back

uxcpIDXf3r
  • read back IDXf3 files & convert back to text files
    to verify result same as original text file input
uxcpIDXf8r
  • read back various IDXf8 files written by uxcpIDXf8w
    to verify result same as original text file input
Note
  • we will list the scripts only for uxsortIDXf3 & uxsortIDXf8
  • you can all scripts in /home/uvadm/sf/demo/...
  • you can see some uxcp scripts listed at uvcp.htm#J4
O8. tmp1 subdir listings after uxsortIDXf3w & after uxsortIDXf8w
Note
  • see VSECOBOL.htm for samples of COBOL with IDXf3/IDXf8 files
  • also includes scripts to prove uxsort compatibility with COBOL

Goto:   Begin this document End this document UVSI Home-Page

O6. 'uxsort' test/demo Indexed Seqntl Variable length files

 # uxsortIDXf3w - write various IDXFORMAT3 files to test uxsort
 #              - by Owen Townsend, Jan28/06, updated May14/07
 #              - IDXFORMAT8 files created by Micro Focus COBOL EXTFH
 #              - file handler used by uxcp, uxsort, uxcopy, uxqrpg
 #
 #*uxsortIDXf3w - write various IDXf8 files for read back by uxcpIDXf3r
 #                   ** can use 'uxcp' scripts to read back **
 # uxcpIDXf3r   - read back various IDXf8 files written by uxsortIDXf3w
 #              - converts output files back to text records
 # uxcpIDXf3c   - copy various IDXf8 files written by uxsortIDXf3w
 #              - writes output files with same types & record-sizes
 #
 # OPTIONS: f1=IDXFORMAT1, f3=IDXFORMAT3, f8=IDXFORMAT8 (see www.microfocus.com)
 # x1=noindex, x2=indexed; y1=fixed, y2=variable, y4=scanback to last non-blank
 # v=min recsize, (maxsize declared by rcs=___)
 #
 # 1. typ=LST-->IDXf8 indexed, varfmt but all 252+4 since no option v minsize
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8,typ=IDXf8,isk1=0(4)"
 #===============================================================================
 #
 # 2. typ=LST-->IDXf8v60 indexed, variable recsize pass thru from input text recs
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60,typ=IDXf8v60,isk1=0(4)"
 #===============================================================================
 #
 # 3. typ=LST-->IDXf8v60y4 indexed variable option y4 recsize at last nonblank
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60y4,typ=IDXf8v60y4,isk1=0(4)"
 #===============================================================================
 #
 # 4. typ=LST-->IDXf8x1 noindex(x1), fixed 252, no IDX filehdr, same as typ=RSF
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8x1,typ=IDXf8x1,isk1=0(4)"
 #===============================================================================
 #note - must read back using typ=RSF, since no index & fixed lth (no v)
 #
 # 5. typ=LST-->IDXf8v60x1 noindex(x1), varfmt, but all 252+4
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60x1,typ=IDXf8v60x1,isk1=0(4)"
 #===============================================================================
 #
 # 6. typ=LST-->IDXf8v60x1y4 noindex(x1), variable 60-252, depending on LNB (y4)
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60x1y4,typ=IDXf8v60x1y4,isk1=0(4)"
 #===============================================================================
 #
 echo "You can examine output files with 'uvhd', example for output#1 above"
 echo " "
 echo "uvhd tmp1/idxf8.dat v   <-- use 'v' option for IDXf8 files"
 echo "====================="
 #
 #Note - script may bomb on uxcp #6 if you only have 1 Micro Focus COBOL license
 #     - But you can add following to your profile to wait 1 minute (vs bomb)
 #
 # export ASLMFRETRY="5,15"
 # ========================

Goto:   Begin this document End this document UVSI Home-Page

O7. 'uxsort' test/demo Indexed Seqntl Variable length files

 # uxsortIDXf8w - write various IDXFORMAT files to test uxsort
 #            - by Owen Townsend, Jan28/06, updated May14/07
 #            - IDXFORMAT8 files created by Micro Focus COBOL EXTFH
 #            - file handler used by uxcp, uxsort, uxcopy, uxqrpg
 #
 #*uxsortIDXf8w - write various IDXf8 files for read back by uxcpIDXf8r
 #                   ** can use 'uxcp' scripts to read back **
 # uxcpIDXf8r   - read back various IDXf8 files written by uxsortIDXf8w
 #              - converts output files back to text records
 # uxcpIDXf8c   - copy various IDXf8 files written by uxsortIDXf8w
 #              - writes output files with same types & record-sizes
 #
 # OPTIONS: f1=IDXFORMAT1, f3=IDXFORMAT3, f8=IDXFORMAT8 (see www.microfocus.com)
 # x1=noindex, x2=indexed; y1=fixed, y2=variable, y4=scanback to last non-blank
 # v=min recsize, (maxsize declared by rcs=___)
 #
 # 1. typ=LST-->IDXf8 indexed, varfmt but all 252+4 since no option v minsize
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8,typ=IDXf8,isk1=0(4)"
 #===========================================================================
 #
 # 2. typ=LST-->IDXf8v60 indexed, variable recsize pass thru from input text recs
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60,typ=IDXf8v60,isk1=0(4)"
 #===============================================================================
 #
 # 3. typ=LST-->IDXf8v60y4 indexed variable option y4 recsize at last nonblank
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60y4,typ=IDXf8v60y4,isk1=0(4)"
 #===============================================================================
 #
 # 4. typ=LST-->IDXf8x1 noindex(x1), fixed 252, no IDX filehdr, same as typ=RSF
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8x1,typ=IDXf8x1,isk1=0(4)"
 #===============================================================================
 #note - must read back using typ=RSF, since no index & fixed lth (no v)
 #
 # 5. typ=LST-->IDXf8v60x1 noindex(x1), varfmt, but all 252+4
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60x1,typ=IDXf8v60x1,isk1=0(4)"
 #===============================================================================
 #
 # 6. typ=LST-->IDXf8v60x1y4 noindex(x1), variable 60-252, depending on LNB (y4)
 uxsort "fili1=dat1/idxtest,typ=LST,rcs=252,filo1=tmp1/idxf8v60x1y4,typ=IDXf8v60x1y4,isk1=0(4)"
 #===============================================================================
 #
 echo "You can examine output files with 'uvhd', example for output#1 above"
 echo " "
 echo "uvhd tmp1/idxf8.dat v   <-- use 'v' option for IDXf8 files"
 echo "====================="
 #
 #Note - script may bomb on uxcp #6 if you only have 1 Micro Focus COBOL license
 #     - But you can add following to your profile to wait 1 minute (vs bomb)
 #
 # export ASLMFRETRY="5,15"
 # ========================

Goto:   Begin this document End this document UVSI Home-Page

O8. 'uxsort' test/demo Indexed Seqntl Variable length files

input text file for uxsortIDXf3/uxsortIDXf8

 0010.0060 idxtest text file to create Var Lth Indexed files.
 0020.0064 - input to COBOL,uxcp,uxcopy to create IDXf1,3,8 files
 0030.0065 - 01-04=ISAM key, 05=separator, 06-09=size, 10-400 text
 0040.0067 - output record varies from 64 to 256, depending on 06-09
 0050.0080 - here is an 80 byte record------------------------------------------>
 0060.0160 - here is a 160 byte record...........................................
 0070.0240 - here is a 240 byte record (last record in file).....................
 ===============================================================================>

tmp1 subdir listings after uxsortIDXf3w


 #1. -rw-rw-r--    1 uvadm    users     2176 May 14 13:50 idxf3.dat
     -rw-rw-r--    1 uvadm    users     3072 May 14 13:50 idxf3.idx

 #2. -rw-rw-r--    1 uvadm    users     1148 May 14 13:50 idxf3v60.dat
     -rw-rw-r--    1 uvadm    users     3072 May 14 13:50 idxf3v60.idx

 #3. -rw-rw-r--    1 uvadm    users     1148 May 14 13:50 idxf3v60y4.dat
     -rw-rw-r--    1 uvadm    users     3072 May 14 13:50 idxf3v60y4.idx

 #4. -rw-rw-r--    1 uvadm    users     1920 May 14 13:50 idxf3x1

 #5. -rw-rw-r--    1 uvadm    users      892 May 14 13:50 idxf3v60x1

 #6. -rw-rw-r--    1 uvadm    users      892 May 14 13:50 idxf3v60x1y4

tmp1 subdir listings after uxsortIDXf8w


 #1. -rw-rw-r--    1 uvadm    users     4864 May 14 15:26 idxf8.dat

 #2. -rw-rw-r--    1 uvadm    users     4112 May 14 15:26 idxf8v60.dat

 #3. -rw-rw-r--    1 uvadm    users     4112 May 14 15:26 idxf8v60y4.dat

 #4. -rw-rw-r--    1 uvadm    users     1764 May 14 15:26 idxf8x1

 #5. -rw-rw-r--    1 uvadm    users      892 May 14 15:26 idxf8v60x1

 #6. -rw-rw-r--    1 uvadm    users      892 May 14 15:26 idxf8v60x1y4
Note
  • IDXf3 Indexed files have 2 partitions .dat & .idx
  • IDXf8 files have only the .dat partition, but it includes indexes
  • unless option 'x1' is specified for a non-indexed file

Please see 'uvhd' of IDXF3v60 on page 'O2' & IDXf8v60 on page 'O3'.

Please see uvcopy6.htm for IDXf3/IDXf8 test/demos (similar to the above), but for 'uxcopy' vs 'uxcp' or 'uxsort'.

Goto:   Begin this document End this document UVSI Home-Page

P1. uvsort: RDW (Record Descriptor Word) variable length files

RDW (Record Descriptor Word) variable length files are often used to FTP variable length EBCDIC files from the mainframe to unix/linux/windows systems. uvsort provides for the following types of RDW files:

RDWz4
  • 4 byte record prefix with 2 byte binary record size & 2 unused nulls
  • record size in prefix is the total record size including the prefix
  • 2 byte record size is 'BIG-END' binary (like mainframe or SUN,AIX,DEC)
  • 'little-end' machines (Intel,AMD) need to switch ends to perform math
  • RDWz4 is the default (typ=RDW defaults to typ=RDWz4)
RDWz2
  • option 'z2' specifies record prefix only 2 bytes (2 nulls omitted)
  • record size in prefix is the data record size excluding the prefix
RDWz8
  • option 'z8' same as RDWz4, but bypasses any BDW prefixes
RDWn4
  • option 'n4' specifies 4 byte record prefix is Numeric digits (vs binary)
  • record size in prefix is the total record size including the prefix

test/demo files supplied in /home/uvadm/dat1

dat1/testLST
  • text file originally used to create testRDW
dat1/testRDW
  • RDW file created from testLST
dat1/testRDWe
  • RDW file, same as above in EBCDIC

 uvhd dat1/testRDW z4   <-- investigate RDW file (using uvhd)
 ====================     - only 1st of 8 records shown below
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             010044443322246662466222
             080045CC100D045CC09E3E00
Note
  • please see uvcp.htm#K1 thru K6 for extensive illustrations of RDW files
    and conversions of EBCDIC RDW files to ASCII text & fixed length files

Here in uvsort, we will give an example of sorting the ASCII testRDW file to a text file, and then sorting the EBCDIC testRDWe file & translating to ASCII at the same time.

sort RDW ASCII file to a text file


 uvsort "fili1=dat1/testRDW,rcs=64,typ=RDWz4,filo1=tmp/testz4,typ=LSTt,key1=0(6)"
 ================================================================================

text records sorted from RDW file

     DELL00 - Dell Inc.
     HP0000 - Hewlett Packard
     IBM000 - International Business Machines
     MFC000 - Micro Focus COBOL
     MS0000 - Microsoft Corp.
     REDHAT - Red Hat Linux
     SUN000 - Sun Microsystems Ltd
     UVSI00 - UV Software Inc.

Goto:   Begin this document End this document UVSI Home-Page

P2. RDW (Record Descriptor Word) variable length files

ex#2 sort RDW EBCDIC file translating to ASCII

Instead of variable length text, we will create fixed length output records that are required for most COBOL input files. For this demo, we will convert to fixed 64 byte records & insert a LineFeed in the last byte so we can have it both ways (fixed length for COBOL, but LFs allow investigation with vi).


 uvsort "fili1=dat1/testRDWe,rcs=64,typ=RDWz4,filo1=tmp/testz4,typ=RST,tra=0(64)"
 ================================================================================
tra=0(64)
  • translates the records to ASCII.
typ=RST
  • output 'Record Sequential Terminated' (inserts LineFeed in last byte)

confirm conversion OK with uvhd


 uvhd tmp/testz4 r64h2     <-- display output file with uvhd
 =====================       - option r64 for record size
                             - option 'h2' for hex display to see LineFeed
                             - only 1st record shown below
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 DELL10 - Dell Inc.                                             .
             4444332224666246622222222222222222222222222222222222222222222220
             45CC100D045CC09E3E000000000000000000000000000000000000000000000A

Confirmations shown by uvhd

  1. Binary record prefixes have been dropped off, because output typ=RST extracts the data record only & omits the typ=RDW record prefix.

  2. LineFeeds in last byte of each record (x'0A's).

  3. Output is ASCII 1st char 'D' is x'44' here vs x'C4' on page 'P1' Translation performed by instruction 'tra=0(64)'.

  4. See pages uvcp.htm for other instructions you can use to reformat data (mvc, clr, rep, tre, tru, trl, trp, sel, del, etc...).

    see DATAcnv1 for more RDW examples

Note
  • also see DATAcnv1.htm for more examples of data conversions
  • stripping BDW/RDW record prefixes
  • converting variable length files to fixed length
  • translating from EBCDIC to ASCII, allowing for packed & signed fields

Goto:   Begin this document End this document UVSI Home-Page

Q1. uvsort - Data file Utility

file type STL

File 'typ=STL' is the STandard Language file system used by AIX COBOL, added to uvsort in February 2016. These are complex file formats with file headers & record headers. There are 3 subtypes as follows:

typ=STLs
  • STL Sequential files (input & output)
typ=STLi
  • STL Indexed files (input only)
typ=STLr
  • STL Relative record files (input only)

File typ=STL Indexed & Relative is provided for input files only. You can copy them to output file types (LSTt,RSF,RST,RSV,RDW,ISF,IDXf1,& STLs Sequential).

converting STL Sequential,Indexed,& Relative to typ=LSTt

Here are examples sorting the 3 demo files provided in $UV/dat1/... We will convert them to 'typ=LSTt' which can be easily checked with vi.


 uvsort "fili1=dat1/testSTLs,typ=STLs,rcs=4096,filo1=tmp/teststls,typ=LSTt,key1=0(8)"
 ====================================================================================
 - sort 'typ=STLs' Sequential to 'typ=LSTt' Line Sequential

 uvsort "fili1=dat1/testSTLi,typ=STLi,rcs=4096,filo1=tmp/teststli,typ=LSTt,key1=0(8)"
 ====================================================================================
 - sort 'typ=STLi' Indexed to 'typ=LSTt' Line Sequential

 uvsort "fili1=dat1/testSTLr,typ=STLr,rcs=4096,filo1=tmp/teststlr,typ=LSTt,key1=0(8)"
 ====================================================================================
 - sort 'typ=STLr' Relative to 'typ=LSTt' Line Sequential

We specify 'rcs=4096' or a size you know is larger than the largest record expected. Record size specified by rcs=... will init I/O areas to blanks, The actual record size is specified in the file header &/or as a prefix on each record & could vary from record to record.

Output 'typ=LSTt' inserts a LineFeed x'0A' after the last nonblank, truncating records depending on data present. So the output records could vary from 4096 bytes down to 1 byte.

Goto:   Begin this document End this document UVSI Home-Page

Q2. file typ=STL

sorting STL Indexed to typ=RSF


 uvsort "fili1=dat1/testSTLi,typ=STLi,rcs=4096,filo1=tmp/teststli,typ=RSF,key1=0(8)"
 ===================================================================================
 - sort 'typ=STLi' Indexed to 'typ=RSF' Record Sequential Fixed
 In this case the output records would all be 4096 bytes (blank filled on right).
 - regardless of whether Input records are variable or fixed.

 uvsort "fili1=dat1/testSTLi,typ=STLi,rcs=4096,filo1=tmp/teststli,typ=RSF,rcs=200,key1=0(8)"
 ===========================================================================================
 - can specify record size on output side (after filo1=...) as desired

 uvsort "fili1=dat1/testSTLi,typ=STLi,rcs=200,filo1=tmp/teststli,typ=RST,key1=0(8)"
 ==================================================================================
 - or can specify on input side if you know records are <= 200
 - this example specifies output 'typ=RST' which inserts a LineFeed in last byte
   which allows you to use unix tools (vi,etc) on the output file
   (if you know the last byte is unused for data)

sorting STL Indexed (or Relative) to STL Sequential


 uvcp "fili1=dat1/testSTLi,typ=STLi,rcs=4096,filo1=tmp/teststls,typ=STLs,key1=0(8)"
 ==================================================================================
 - sort 'typ=STLi' Indexed to 'typ=STLs' Seqquential
 - output record size is determined by input record size & could be variable

 uvcp "fili1=dat1/testSTLi,typ=STLi,rcs=4096,filo1=tmp/teststls,typ=STLs,rcs=200,key1=0(8)"
 ==========================================================================================
 - But if you specify a record size on the output side (after filo1=...)
   then the output file size will be fixed

STL file formats

See STL file formats documented at http://uvsoftware.ca/uvhd.htm#5J1 or at http://uvsoftware.ca/uvcp.htm#W1

'uvhd' allows you to see the file header records & the record prefixes but 'uvsort' & 'uvcp' will extract only the data records.


 uvhd dat1/testSTLs      <-- will show file header (112 bytes)
 ==================

 uvhd dat1/testSTLs x2   <-- option 'x2' for STL Sequential
 =====================     - use 'x4' for Indexed & 'x8' for Relative
 - file header & system records (for Indexes) will be bypassed
 - you will see only the data records

Goto:   Begin this document End this document UVSI Home-Page

S1. uvsort - unix sort utility with mainframe functionality

CoSort to uvsort converter

In May 2010 we developed a 'CoSort' to 'uvsort' converter. 'cosort2uvsort1' will convert unix/linux CoSort parameters to the equivalent uvsort parameters.

Here is our sample CoSort 'cosort21' & the converted equivalent uvsort 'uvsort21'. On the next page we will present the operating instructions to convert 1 at a time or all CoSorts in a directory.

sample CoSort

 /sort
 /infiles=(SORTIN)
 /length=64
 /field=(custno,  position=1, size=6)
 /field=(slsmn,   position=11, size=2)
 /field=(idate,   position=14, size=6)
 /field=(inv,     position=21, size=8)
 /field=(product, position=31, size=6)
 /field=(qty,     position=39, size=6, numeric)
 /field=(price,   position=46, size=7, numeric)
 /field=(amount,  position=54, size=9, numeric)
 /condition=(oldsale, test=(idate gt "500101" and idate lt "991232"))
 /condition=(slsmn21, test=(slsmn eq "21" and amount gt 20000))
 /condition=(slsmn44, test=(slsmn eq "44" and amount gt 20000))
 /condition=(slsmn65, test=(slsmn eq "65" and amount gt 20000))
 /key=(slsmn,  ascending)
 /key=(amount, descending)
 /omit=(condition=oldsale)
 /outfile=(SORTOF01) /include=(condition=slsmn21)
 /outfile=(SORTOF02) /include=(condition=slsmn44)
 /outfile=(SORTOF03) /include=(condition=slsmn65)

equivalent uvsort

 # cosort21 - uvsort replacing COSORT
 #         - converted at UV_Software on 2010/05/01_16:41:56
 #         - call this parameter file as follows:
 # export SORTIN=...      #<-- define input file
 # export SORTOUT=...     #<-- define output file
 # export PF=...          #<-- define ParamterFile dirctory (in profiles)
 # uvsort $PF/cosort21   #<-- execute uvsort on this parameter file
 # ====================
 fili01=${SORTIN},typ=RSF,rcs=00064
 filo01=${SORTOF01},typ=RSF,rcs=64
 filo02=${SORTOF02},typ=RSF,rcs=64
 filo03=${SORTOF03},typ=RSF,rcs=64
 keys=(10,2,c,a,53,9,c,a)
 del1i00=13(6c)>500101,del2i00=13(6c)<991232
 sel1o01=10(2c):21,sel2o01=53(9z)>20000
 sel1o02=10(2c):44,sel2o02=53(9z)>20000
 sel1o03=10(2c):65,sel2o03=53(9z)>20000

Goto:   Begin this document End this document UVSI Home-Page

S2. CoSort to uvsort converter

converting CoSort to uvsort

If you have the May 2010 version of Vancouver Utilities, you could convert the sample pf/demo/cosort21 to pf/demo/uvsort21, and then execute it on the supplied test file dat1/sales2. The relevant subdirs & files are:

 /home/uvadm/
 :-----bin/                   <-- Vancouver Utility executable programs
 :     :-----uvsort
 :-----pf/                    <-- VU supplied Parameter Files
 :-----pf/IBM/                <-- conversion jobs
 :     :-----cosort2uvsort1   <-- CoSort to uvsort converter
 :-----pf/demo/               <-- demo Parameter Files
 :     :-----cosort21         <-- sample CoSort parameters
 :     :-----uvsort21         <-- converted equivalent uvsort
 :-----dat1/                  <-- Data files subdir
 :     :-----sales2           <-- data file input for converted CoSort
 :-----tmp/                   <-- Temp subdir for output files
 :     :-----sales11
 :     :-----sales44          <-- 3 output files from uvsort21
 :     :-----sales65

 #1. Login uvadm --> /home/uvadm

 #2. uvcopy cosort2uvsort1,fili1=pf/demo/cosort21,filo1=pf/demo/uvsort21
     ===================================================================
     - convert sample CoSort to uvsort equivalent parameters

 #3a. export SORTIN1=dat1/sales2  <-- export input file for uvsort21
      ==========================
 #3b. export OUTDIR=tmp           <-- export output subdir for uvsort21
      =================

 #4. uvsort pf/demo/uvsort21      <-- execute uvsort on parameter file uvsort21
     =======================

You could shorten the 'uvsort' command line if you export a $SYMBOL for the Parameter Files subdir, for example:


 #3c. export PF=/home/uvadm/pf/demo/
      ==============================

 #4a. uvsort $PF/uvsort21      <-- execute uvsort on parameter file uvsort21
      ===================

You would code 'PF' in your profile to define your own Parameter Files subdir. This would allow you to run your uvsorts from any location without having to enter long pathnames.

If you have hundreds of CoSorts, you could convert all CoSorts to equivalent uvsorts as follows:


 uvcopyx cosort2uvsort1 cosortdir uvsortdir uop=q0i7
 ===================================================

Goto:   Begin this document End this document UVSI Home-Page

T1. Sort Timings - uvsort vs mfsort vs unix sort

In this section I will present a sample uvsort & equivalent mfsort, unix sort, and document how I performed some timings between them.

sample uvsort

 # uvsort2 - sample uvsort (UV Software Vancouver Utilities)
 #         - for comparison to 'mfsort' & unix system 'sort'
 #         - by Owen Townsend, UV Software, July 19/2015
 #         - this script stored at /home/uvadm/sf/demo/uvsort2 (in uvadm PATH)
 # See www.uvsoftware.ca/uvsort.htm#T1 doc re demo uvsort vs MFSORT timings
 # - uvsort.htm#T2 shows how to setup big file for sort timings
 echo "uvsort2, input data1/cmbig, Begun at: $(date)"
 uvsort "fili1=data1/cmbig,rcs=256,typ=RSF,filo1=data1/cmbiguvsort,key1=0(6)"
 #============================================================================
 echo "uvsort2, output data1/cmbiguvsort, Ended at: $(date)"

equivalent mfsort

 # mfsort2 - sample MFSORT (Micro Focus standalone sort utility)
 #         - by Owen Townsend, UV Software, July 02/2010
 #         - this script stored at /home/uvadm/sf/demo/mfsort2 (in uvadm PATH)
 # See www.uvsoftware.ca/uvsort.htm#T1 doc re demo MFSORT vs uvsort timings
 # - uvsort.htm#T2 shows how to setup big file for sort timings
 echo "mfsort2, input data1/cmbig, Begun at: $(date)"
 mfsort "sort fields(1,6,ch,a) \
         use data1/cmbig org sq record f,256 \
         give data1/cmbigmfsort"
 echo "mfsort2, output data1/cmbigmf, Ended at: $(date)"

equivalent unix system sort

 # unixsort2 - sample unix system sort for Timing comparison to:
 #             Vancouver Utilities 'uvsort' &  Micro Focus sort 'mfsort'
 #           - by Owen Townsend, UV Software, July 19/2015
 #           - this script stored at /home/uvadm/sf/demo/unixsort2
 # See www.uvsoftware.ca/uvsort.htm#T1 doc re demo uvsort vs unixsort & mfsort
 # - uvsort.htm#T2 shows how to setup big file for sort timings
 echo "unixsort2, input data1/cmbig, Begun at: $(date)"
 sort -k '1.1,1.6' -o data1/cmbigunixsort data1/cmbig
 #===================================================
 echo "unixsort2, output data1/cmbigunixsort, Ended at: $(date)"
 #
 #Timings for 4 gig sort on HP Z420 workstation July 19/2015
 # unix sort = 34 seconds, mfsort = 64 seconds, uvsort = 57 seconds
 # unix sort is faster, but cannot sort mainframe type files because:
 #
 # 1. unix sort cannot sort fixed length record files without LineFeeds
 # 2. unix sort cannot sort on packed decimal fields
 # 3. unix sort cannot sort Indexed files (Micro Focus compatible)
 # 4. unix sort does not have the select/delete/reformat capabilities of
 #    uvsort or mfsort (similar to the mainframe DFSORT or SYNCSORT)

Goto:   Begin this document End this document UVSI Home-Page

T2. sort time comparison - uvsort vs mfsort

script to make BIG file

 #!/bin/ksh
 # makebig1 - make a big file by concatenating spcfd no of times
 echo "makebig1 - make a big file by concatenating spcfd no of times"
 f1="$1"; f2="$2"; integer n="$3"
 if [[ -f "$f1" && -n "$f2" && -n "$n" ]]; then :
    else echo "usage: makebig1 file1 file2 ##  <-- arg1/2=filenames, arg3 number"
         echo "       ======================="
         exit 9; fi
 # if output file does not exist, create empty file now
 if [[ ! -f $f2 ]]; then >$f2; fi
 integer i=0
 while ((i < n))
   do cat $f1 >> $f2
      ((i+=1))
   done
 echo "completed $n concats of $f1 to $f2"
 exit 0

directories for sort time tests

For these tests, I created a separate user 'testbig' with homedir in file system /home4/... where I had lots of space to create & sort BIG files. Here are the relevant directories & files:

 /home/testbig
 :-----data1               <-- subdir for input files
 :     :-----cm1           <-- 8K demo file copied from /home/uvadm/data1/custmas1
 :     :                     - 32 records of 256 bytes = 8192
 :     :-----cmbig         <-- 1 Gig file created by concatenating cm1 125,000 times
 :     :                     - 8192 * 125,000 = 1,024,000,000
 :     :-----cmbiguvsort   <-- output file from uvsort
 :     :-----cmbigmfsort   <-- output file from mfsort
 :     :                     - see scripts mfsort2 & uvsort2 listed on prior page

 uvhd data1/custmas1 r256 <-- use 'uvhd' to display 1st record of test file
 ========================   - customer N&A with 24 * 5 byte packed sales history
                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    604-754-5531                  ..4V|...
             4442222222222442535343222233323332333322222222222222222201357000
             9DF00000000002306931810000604D754D55310000000000000000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ...............................f.....<........f.C
             0000008900000000880000000018000680001300000000694222222222222222
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3000000000000000

Goto:   Begin this document End this document UVSI Home-Page

T3. sort time comparison - uvsort vs mfsort

Hardware/Software for sort timing tests

make 4 gig file & perform timing tests

  1. First setup a user with homedir in a file system with lots of space. We setup userid 'testbig' at /home4/testbig & subdirs as illustrated above. You could setup the subdirs in your homedir if enough space in /home/... Our biggest test is an 8 gig file so you need 24 gig (input, output, work).


 #2. Login as testbig --> /home4/testbig (or login as yourself)
     ================

 #3. cp /home/uvadm/data1/custmas1 data1/cm1 <-- copy 8K demo file
     =======================================   - 32 records * 256 = 8192

 #4. makebig1 data1/cm1 data1/cmbig 500000  <-- run script to make big file
     =====================================    - 8192 * 500000 = 4,096,000,000

 #5a. uvsort2    <-- run uvsort script (see listing on page 'T1' above)
      =======      - see timings below

 #5b. uvsort "fili1=data1/cmbig,rcs=256,typ=RSF,filo1=data1/cmbiguvsort,key1=0(6)"
      ============================================================================
      - alternative, run uvsort directly, all prameters on cmd line (vs script)
 150712:114819:BIGSORT1: uvsort fili1=data1/cmbig,filo1=data1/cmbiguvsort
 150712:114819:BIGSORT1: inrecs  = 1000000 fili01 data1/cmbig
 150712:114822:BIGSORT1: inrecs  = 2000000 fili01 data1/cmbig
    --- 12 lines omitted ---
 150712:114841:BIGSORT1: inrecs  = 15000000 fili01 data1/cmbig
 150712:114844:BIGSORT1: inrecs  = 16000000 fili01 data1/cmbig
 150712:114844:BIGSORT1: EOF fili01 16000000 rds, 4096000000 size; data1/cmbig
 150712:114845:BIGSORT1: sort phase complete, used 512 megabytes, sort output begin
 150712:114846:BIGSORT1: outrecs = 1000000 filo01 data1/cmbiguvsort
 150712:114848:BIGSORT1: outrecs = 2000000 filo01 data1/cmbiguvsort
    --- 12 lines omitted ---
 150712:114915:BIGSORT1: outrecs = 15000000 filo01 data1/cmbiguvsort
 150712:114917:BIGSORT1: outrecs = 16000000 filo01 data1/cmbiguvsort
 150712:114917:BIGSORT1: EOF filo01 16000000 wrts, 4096000000 size; data1/cmbiguvsort
  uvsort start 2015/07/12_11:48:19 end 11:49:17 ELAPSED  TIME 00:58

Goto:   Begin this document End this document UVSI Home-Page

T4. sort time comparison - uvsort vs mfsort


 #6a. mfsort2    <-- run MFSORT script (see listing on page 'T1' above)
      =======

 #6b. mfsort "sort fields(1,6,ch,a) \
              use data1/cmbig org sq record f,256 \
              give data1/cmbigmfsort"
      ==============================================
      - alternative, run mfsort directly, all prameters on cmd line (vs script)
           Micro Focus MFJSORT Utility 3.0.00
           sort fields(1,6,ch,a)         use data1/cmbig org sq record f,256         give
 SORT204I: ********************* Sort Statistics *********************
 SORT205I: INPUT  file 'data1/cmbig'
                           32000000 record(s) input.
                           32000000 record(s) used.
 SORT206I: OUTPUT file 'data1/cmbigmfsort'
                           32000000 record(s) used.
                           32000000 record(s) output.
 SORT399I: End of Micro Focus MFJSORT Utility.
 mfsort2, input data1/cmbig, Begun at: Sun Jul 12 10:51:29 PDT 2015
 mfsort2, output data2/cmbigmfsort, Ended at: Sun Jul 12 10:52:34 PDT 2015
Note
  • MFSORT time = (10:52:34 - 10:51:29) = 65 seconds
  • uvsort time = 55 seconds (almost 20% faster)

specify directory for sort work files


 export TMPDIR=/tmp  <-- optional variable for 'mfsort' work files
 ==================    - defaults to current directory if unspecified

 export SORTDIR=/tmp  <-- optional variable for 'uvsort' work files
 ===================    - defaults to ./tmp/ (tmp subdir in current directory)
Note
  • see uvsort options on pages 'F1' - 'F5'

Goto:   Begin this document End this document UVSI Home-Page

T5. sort time comparison - uvsort vs mfsort

sort times for 8 GIG file


 #1. makebig1 data1/cm1 data1/cmbig 1000000  <-- run script to make big file
     ======================================    - 8192 * 1000000 = 8,192,000,000

 #2. uvsort2    <-- run uvsort script (see listing on page 'T1' above)
     =======      - console display log similar to 4 gig test above
                  - time = 3 minutes, 54 seconds

 #3. mfsort2    <-- run MFSORT script (see listing on page 'T1' above)
     =======      - console display log similar to 4 gig test above
                  - time = 4 minutes, 57 seconds
Note
  • on a 8 gig sort, uvsort is over 20% faster than MFSORT

conclusions

uvsort is 20% faster than mfsort, which is probably not enough to justify buying the Vancouver Utilities if that were the only reason.

'uvsort' is just 1 part of the Vancouver Utilities, which are designed to replace mainframe utilities on Unix/Linux/windows systems (mainframe utilities such as SORT,IDCAMS,IEBGENER,FILEAID,EASYTRIEVE,etc).

Be sure to try out the 1 FREE utility 'uvhd', which you may download from www.uvsoftware.ca/freestuff.htm. uvhd is all my customers favorite utility.

'uvhd' is a an interactive file investigation utility with 25 help screens. It displays files with packed/binary fields in vertical hexadecimal & prompts for commands to browse, search, select, update, print, etc.

Goto:   Begin this document End this document UVSI Home-Page

U1. JCL conversion Mainframe SORT to uvsort

SORT INCLUDE/OMIT converted to uvsort sel/del

The JCL converter converts mainframe SORT steps to the uvsort equivalent. Any 'INCLUDE/OMIT's are converted to 'sel/del's. Complex SORTs with multiple mixed INCLUDEs & OMITs may require manual changes and a warning is given.

Very complex SORT INCLUDE/OMIT's with multiple mixed AND/OR's might be better rewritten in uvcopy, which has the power to do whatever kind of data manipulation and record selection desired. For example, uvcopy has table load & lookup instructions that can dramatically shorten voluminous selection codes and simplify maintenace.

We will present several examples of SORT INCLUDE/OMIT AND/OR conversions - 1st requiring no change, some requiring changes,& some illustrating splitting the record selections out to a 2nd step (rewritten in uvcopy).

These examples will be presented in the next section V1-V9, but first you should understand the mainframe JCL conversion, which is fully documented in MVSJCLtest2.htm#Part_9, with a short version here on pages U2-U5.

summary of SORT INCLUDE AND/OR examples

V1. JAR410.JCL - SORT INCLUDE AND/OR's requiring NO changes
 jar410.ksh - converted to script, calling uvsort with sel/del's
V2. JAR420.JCL - SORT INCLUDE AND/OR's requiring 1 change
 jar420.ksh - converted to script, calling uvsort with sel/del's
V3. JAR430.JCL - SORT INCLUDE AND/OR's requiring SOME changes
 jar430.ksh - converted to script, calling uvsort with sel/del's
V4. JAR440.JCL - SORT INCLUDE AND/OR's requiring MANY changes
 jar440.ksh - converted to script, calling uvsort with sel/del's
V5. jar441.ksh - alternative to jar440, with record selections in 2nd step
 jar441.uvc - uvcopy parameter file called by step 2 of jar441.ksh
  - using compare/skip instructions for each code value
V6. jar442.ksh - 2nd alternative to jar440, with record selections in 2nd step
 jar442.uvc - uvcopy parameter file called by step 2 of jar442.ksh
  - using 'tst' to compare multiple values with 1 instruction
V7. jar443.ksh - 3nd alternative to jar440, with record selections in 2nd step
 jar443.uvc - uvcopy parameter file called by step 2 of jar443.ksh
  - using table load/lookup to shorten coding & simplify maintenance
V8. jar444.ksh - 4th alternative with sort & record selections combined
 jar444.uvc - uvcopy parameter file called by jar444.ksh (now just 1 step)
  - same as jar443.uvc + sort instructions (sxo,sxp,sxs,sxg)
V9. jar445.ksh - 5th alternative to demo read table lookup codes from external file
 jar445.uvc - uvcopy parameter file called by jar445.ksh
  - with 'rtb' to read codes from external file for lookup by 'lok'

Goto:   Begin this document End this document UVSI Home-Page

U2. JCL conversion Mainframe SORT to uvsort

JCL conversion - brief summary

See complete details of JCL conversion at JCLcnv1demo.htm, but here is a summary of the preparations required for JCL conversion:

  1. Install Vancouver Utilities as per install.htm#B1 - B6.
  2. Setup users uvadm & appsadm as per install.htm#B1 - B6.
  3. Compile Vancouver Utilities as per install.htm#C1 - C3.
  4. Setup your userid & profile to use VU as per JCLcnv1demo.htm#3C1.
  5. Understand stub & common profiles at JCLcnv1demo.htm#1C1 & 1C2.
  6. Setup directories required for JCL conversion JCLcnv1demo.htm#Part_3.
  7. Copy demo JCL & data files as per JCLcnv1demo.htm#3D1.
  8. Convert JCL to ksh as per JCLcnv1demo.htm#3F1.

    alternative to 6,7,8

You need more files than are copied by steps 6,7,8 above, so you need to copy the complete directories of JCL conversion demo files as follows. Then you do not have to convert JCL to scripts, the converted scripts will be copied. Assuming you are logged in as yourself & have setup testlibs & testdata in your homedir:


 #6a. cp -r /home/uvadm/mvstest/mvstest1/testlibs1/* testlibs
      =======================================================

 #6b. cp -r /home/uvadm/mvstest/mvstest1/testdata1/* testdata
      =======================================================

2nd alternative to 6,7,8

Actually you could run the following demos by logging in as uvadm and proceeding as follows:

  1. Update the uvadm .bash_profile (was copied from supplied stub_profile). Look for following lines & change as shown below:

      export RUNLIBS=$HOME/testlibs             <-- as supplied
      export RUNDATA=$HOME/testdata
      export RUNLIBS=$HOME/mvstest/mvstest1/testlibs1  <-- change to this
      export RUNDATA=$HOME/mvstest/mvstest1/testdata1
  1. Logoff & back on to make new defs of RUNLIBS & RUNDATA effective You can then change to testlibs1 or testdata1 using alliases:

cdl <-- alias to change to testlibs1 cdd <-- alias to change to testdata1

Goto:   Begin this document End this document UVSI Home-Page

U3. JCL conversion Mainframe SORT to uvsort

Directories relevant to SORT INCLUDE/OMIT conversion demos

Here are the relevant subdirs required to demo conversion of SORT INCLUDE/OMIT's to uvsort sel/del's - selected from the diagrams at JCLcnv1demo.htm#3B2 & 3B3.

 /home/userxx
 :-----testlibs        <-- libraries for JCL conversions
 :     :------jcl0       - original mainframe JCL with SORT INCLUDEs AND/ORs
 :     :------jcls       - converted to Korn shell scripts
 :-----testdata        <-- data files for testing JCL/scripts
 :     :------data       - data files to test/execute Korn shell scripts
 :     :------etc

It would be better to setup superdirs testlibs & testdata with relevant subdirs and copy the files required for the JCL/SOPRT conversions to ksh/uvsort. But if you have login to /home/uvadm, you could run the test/demos there. This requires 'export RUNLIBS=/home/uvadm/mvstest/testlibs' and 'export RUNDATA=/home/uvadm/mvstest/testdata' as shown on the previous page. The relevant subdirs are illustrated below:

Library subdirs for JCL conversions

 /home/uvadm/mvstest/
 :-----testlibs     <-- libraries for JCL conversion to Korn shell scripts
 :     :------cbls       - several directories not relevant to these demos
 :     :------etc          not shown here
 :     :
 :     :------jcl0      <-- original mainframe JCL
 :     :      :----JAR410.JCL,...JAR440.JCL  <-- demo SORT INCLUDE convert
 :     :
 :     :------jcl1,jcl2,jcl3 - intermediate subdirs used in JCL conversion
 :     :
 :     :------jcls      <-- converted to Korn shell scripts
 :     :      :----jar410.ksh,...jar440.ksh  <-- converted to uvsort sel/del
 :     :
 :     :------sf        <-- modified scripts (SORT INCLUDE complex AND/OR's)
 :     :      :           - store in sf/... to avoid overwrite on JCL reconvert
 :     :      :----jar420.ksh,...jar444.ksh  <-- see listings ahead
 :     :------pf        <-- uvcopy jobs to replace complex uvsort sel/del's
 :     :      :           - called as 2nd step in script (after uvsort step)
 :     :      :----jar441.uvc,...jar444.uvc  <-- see listings ahead

Data subdirs for test/demo executions

 /home/uvadm/mvstest/
 :-----testdata      <-- data files for testing JCL/scripts
 :     :------data        - datafiles
 :     :
 :     :------ctl         - GDG control file
 :     :------joblog,jobmsgs,jobtimes,jobtmp,sysout (less relevant subdirs)
 :     :------tmp         - tmp subdir for uvsort & misc use
 :     :------etc         - several less relevant subdirs not shown

Goto:   Begin this document End this document UVSI Home-Page

U4. JCL conversion Mainframe SORT to uvsort

Performing JCL conversion

If you followed instructions on page 'U2', you could execute all steps of JCL conversion as follows: (see complete doc at JCLcnv1demo.htm#3F1).


 jcl2ksh51A all    <-- perform all steps of JCL conversion to ksh scripts
 ==============
 jcl0 -------> jcl1 -----------> jcl2 -------------> jcl3 ------------> jcls
      cleanup       PROC expand      convert to ksh      manual copy 1 at a time
                                                         when ready to test each

JCL conversion is a multi-step process thru several intermediate directories as shown above. The jcl2ksh51A script stops at jcl3/... & we recommend (for your real conversions) that you then copy 1 at a time from jcl3 to jcls when you are ready to debug each script. But for these test/demos we will copy all jcl3/* to jcls/... (jcls/ is in the profile search PATH).


 cp jcl3/* jcls    <-- copy all converted JCL/scripts to search PATH jcls/
 ==============

But if you use the 1st or 2nd alternatives at the bottom of page 'U2', you do not need to convert or copy jcl3 to jcls, because the alternatives copy all subdirs including the converted jcls/...

Here is the 1st of 4 supplied JCLs to demo conversion of SORT/INCLUDEs to uvsort/selects.

JAR410.JCL - 1st of 4 SORT/INCLUDE demos

 //JAR410   JOB  'test SORT INCLUDE AND/OR'
 //STEP1    EXEC PGM=SORT
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=AR.SALES.ITEMS1,DISP=NEW,LRECL=64
 //SYSIN    DD *
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman & product
 INCLUDE COND=(11,2,CH,EQ,C'21',AND,31,6,CH,EQ,C'HAM001',OR,
                11,2,CH,EQ,C'44',AND,31,6,CH,EQ,C'SAW051')
 /*

See the converted script equivalent listed below. Only this 1st JCL/script is listed with all lines shown. All 8 demo JCLs are listed in the next section 'V0' - 'V9', but only the uvsort lines are extracted & listed for the scripts (all you need to compare original SORT/INCLUDEs to converted uvsort/selects).

Goto:   Begin this document End this document UVSI Home-Page

U5. JCL conversion Mainframe SORT to uvsort

jar410.ksh - 1st of 8 uvsort/select demos

 #!/bin/ksh
 ##JAR410   JOB  'test SORT INCLUDE AND/OR'
 export JOBID2=JAR410 jobid2=jar410; scriptpath="$0"; args="$*"
 if [[ -z "$JOBID1" ]]; then export JOBID1=$JOBID2; fi
 for arg in $args; do if [[ "$arg" == *=* ]]; then export $arg; fi; done
 integer JCC=0 SCC=0 LCC=0  # init step status return codes
 autoload jobset51 gdgclear51 jobend51 jobabend51 logmsg1 logmsg2 stepctl51
 autoload exportfile exportgen0 exportgen1 exportgen2 exportgenall exportgenx
 . $APPSADM/env/stub.ini  #<-- for control-M (see notes in env/stub.ini)
 jobset51    # call function for JCL/script initialization
 #gdgclear51 $PUB_... #<-- insert gdgclear51's for exportgen1's to $PUBs
 goto
 S0000=A
 #1======================= begin step#S0010 SORT ========================
 S0010=A
 stepctl51;  #calc StepTimes & check jobstop/goto end
 goto
 export JSTEP=S0010; ((XSTEP+=1)); SCC=0; LCC=0; alias goto="";
 logmsg2 "******** Begin Step $JSTEP sort (#$XSTEP) ********"
 ##STEP1    EXEC PGM=SORT
 export PROGID=sort
 export PARM=""
 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items1
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=(11,2,CH,EQ,C'21',AND,31,6,CH,EQ,C'HAM001',OR,
                11,2,CH,EQ,C'44',AND,31,6,CH,EQ,C'SAW051')
 /*
 logmsg2 "Executing--> uvsort \"fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTO..."
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a),sel1=10(2c):21,sel2=30(6c):HAM001,sel1=10(2c):44\
 ,sel2=30(6c):SAW051"
 #4----------------------------------------------------------------------
 LCC=$?; S0010C=$LCC; ((SCC+=LCC)); ((JCC+=LCC)); S0010R=1; alias goto="";
 if ((S0010C != 0))
    then logmsg2 "ERR: step#$JSTEP sort abterm $SCC"
    alias goto="<<S9900=\A"; fi
 goto
 #8======================================================================
 S9000=A
 jobend51 $RUNDATA
 logmsg2 "JobEnd=Normal, StepsExecuted=$XSTEP, LastStep=$JSTEP"
 exit 0 #ver:20130620 a1b3c0d1e2f3g1i1j0k3l50m4n3o0p0r0s0t1u1v3w0x0y1z0
 #9======================================================================
 S9900=A
 logmsg2 "JobEnd=AbTerm, JCC=$JCC,StepsX/L=$XSTEP/$JSTEP" RV ACK
 jobabend51 $RUNDATA
 exit $JCC

Goto:   Begin this document End this document UVSI Home-Page

U6. JCL conversion Mainframe SORT to uvsort

Executing jcls/... to demo SORT/INCLUDE conversions

You can execute any of the SORT/INCLUDE demo JCL/scripts (listed on page 'U1' or 'V0') as follows, using jar410.ksh for example:


 #1a. Login as yourself if you copied testlibs/testdata to your homedir as on 'U2'.
 #1b. Or Login as uvadm for 2nd alternative documented on page 'U2'.

 #2. cdl   <-- alias to change to $RUNLIBS ($HOME/testlibs/)

 #3a. vi jcl0/JAR410.JCL  <-- inspect original mainframe JCL (before conversion)

 #3b. vi jcls/jar410.ksh  <-- inspect converted script equivalent
                            - see listings ahead on pages 'U5' & 'U6'

 #4. cdd   <-- alias to change to $RUNDATA ($HOME/testdata/)

 #5a. vi data1/ar.sales.items  <-- inspect input data file
                                - will not work, because no LineFeeds

 #5b. uvhd data1/ar.sales.items r64 <-- inspect input data file
                                     - use uvhd to inspect data w/o LineFeeds

 #6. jar410.ksh    <-- execute jcl/script 1st demo for SORT/INCLUDEs
     ==========        (converted to uvsort/selects)

 #7. vi data1/ar.sales.items1 <-- inspect output uvsort/selects (items1 vs items)
     =======================   - works if uvsort filo1 typ=RSF changed to typ=RST

 #7b. uvhd data1/ar.sales.items1 r64  <-- use uvhd if vi does not work
      =============================

Notes re uvsort record selections

  1. There are 20 records in the input file & there should be 2 records in the output file, see listings on the next page.

  2. You should be able to verify the selections by comparing the SORT/INCLUDEs listed on page 'U4' (or the uvsort/selects on page 'U5') to the 20 input records listed below.

Goto:   Begin this document End this document UVSI Home-Page

U7. JCL conversion Mainframe SORT to uvsort

input data for uvsort/select demos


 #4. cdd   <-- alias for cd $HOME/$RUNDATA
             - instructions repeated from page above

 #5c. cat data1/ar.sales.items  <-- display input file
      =======================
          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 990802 IN111001  HAM001  000020 0000101 000002020
 130140    21 000802 IN111001  SCR012  000021 0001001 000021021
 130140    21 010802 IN111001  CHR001  000022 0010001 000220022
 139923    35 980802 IN111002  TAB013  000023 0000201 000004623
 139923    35 990807 IN111002  VAR200  000024 0002001 000048024
 150825    44 010804 IN1122    HAM001  000025 0020001 000500025
 150825    44 020804 IN1122    HAX129  000026 0000301 000007826
 201120    44 030807 CR5234    BBQ001  000027 0003001 000081027
 223240    65 010816 CR955     HAM001  000028 0030001 000840028
 223240    65 020816 IN441     BBQ001  000029 0000101 000002929
 308685    21 020812 IN6605    SAW051  00001p 0000101 00000101p
 308685    21 030812 IN6605    BIT075  00001q 0001001 00001101q
 308685    21 040812 CR8835    TAB013  00001r 0010001 00012001r
 315512    44 040805 IN2251    HAM001  00001s 0000201 00000261s
 315512    44 050805 IN2251    SAW051  00001t 0002001 00002801t
 315512    44 060805 IN2255    BIT075  00001u 0020001 00030001u
 400002    85 990812 CR245     HAX129  00001v 0000101 00000161v
 406082    35 010815 IN33001   BBQ001  00001w 0001001 00001701w
 406082    35 020815 IN33001   TAB013  00001x 0010001 00018001x
 406082    65 030816 IN441     VAR200  00001y 0000101 00000191y

output data from jar410.ksh (1st of 8 demos)


 #6. jar410.ksh  <-- execute 1st demo SORT/INCLUDEs uvsort/selects
     ==========

 #7. cat data1/ar.sales.items1  <-- display selected records
     ========================
          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 cust#   slmn YYMMDD invoice  product    qty   price    amount
 ================================================================
 130140    21 990802 IN111001  HAM001  000020 0000101 000002020
 315512    44 050805 IN2251    SAW051  00001t 0002001 00002801t

Goto:   Begin this document End this document UVSI Home-Page

U8. JCL conversion Mainframe SORT to uvsort

running other SORT/INCLUDE demos

  1. You can run any of the demo SORT/INCLUDE uvsort/selects listed on page 'U1' But better to use the next section 'V0' - 'V9' as your guide, where you can see the original SORT/INCLUDEs & converted uvsort/selects.

  2. Note that only jar441.ksh required no manual changes after automatic conversion. Auto conversion is from jcl0 --> jcl1 --> jcl2 --> jcl3 but then we copied all jcl3/* --> jcls, which would overwrite any manual changes that had been made in jcls/...

  3. For the above reason I stored the manually corrected scripts (jar420-jar444) in subdir sf/... and since sf/ is in the search PATH ahead of jcls/... you will execute the modified scripts when you enter just the name:


      jar420.ksh   <-- executes $HOME/$RUNLIBS/sf/jar420.ksh,
      ==========     - because sf/ precedes jcls/ in the profile $PATH

executing uncorrected vs corrected scripts

  1. You can run both uncorrected & corrected scripts as follows (assuming you are in $HOME/testlibs/ with both sf/ & jcls/ below you).


      jcls/jar420.ksh  <-- run un-corrected script
      ===============

      sf/jar420.ksh    <-- run corrected script
      =============
      jar420.ksh       <-- also runs corrected script
      ==========         - since sf/ in $PATH ahead of jcls/
  1. For un-corrected scripts in jcls/... you should get following results: jar410=02, jar420=08, jar430=09, jar440=13,

  2. For corrected scripts in sf/... you should get following results: jar410=02, jar420=03, jar430=05, jar440=04, jar441=04, jar442=04, jar443=04, jar444=04.

  3. Note, there are NO scripts jar441.ksh,jar442.ksh,jar443.ksh,jar444.ksh in jcls/... because there are NO JCLs JAR441.JCL - JAR444.JCL in jcl0/...

  4. jar441.ksh - jar444.ksh are manually created versions of jar440.ksh to illustrate alternate solutions for manually corrected SORT/INCLUDEs with multiple mixed AND/ORs in ((parens)). See listings on pages 'V5' - 'V9'.

  5. Of course jar440.ksh - jar445.ksh all get same results (4 records) because they are alternate solutions of the same sselections.

Goto:   Begin this document End this document UVSI Home-Page

V0. JCL conversion Mainframe SORT to uvsort

SORT INCLUDE/OMIT converted to uvsort sel/del

The JCL converter converts mainframe SORT steps to the uvsort equivalent. Any 'INCLUDE/OMIT's are converted to 'sel/del's. Complex SORTs with multiple mixed INCLUDEs & OMITs may require manual changes and a warning is given.

Very complex SORT INCLUDE/OMIT's with multiple mixed AND/OR's might be better rewritten in uvcopy, which has the power to do whatever kind of data manipulation and record selection desired. For example, uvcopy has table load & lookup instructions that can dramatically shorten voluminous selection codes and simplify maintenace.

We will present several examples of SORT INCLUDE/OMIT AND/OR conversions - 1st requiring no change, some requiring changes,& some illustrating splitting the record selections out to a 2nd step (rewritten in uvcopy).

This section assumes you already understand the JCL/script & SORT/uvsort conversions and are testing the JCL/scripts & may need to modify some of the script/uvsort/sel/del's converted from the original SORT/INCLUDE/AND/OR's. To understand the JCL/script conversions see the prior section 'U1' - U8.

summary of SORT INCLUDE AND/OR examples

V1. JAR410.JCL - SORT INCLUDE AND/OR's requiring NO changes
 jar410.ksh - converted to script, calling uvsort with sel/del's
V2. JAR420.JCL - SORT INCLUDE AND/OR's requiring 1 change
 jar420.ksh - converted to script, calling uvsort with sel/del's
V3. JAR430.JCL - SORT INCLUDE AND/OR's requiring a FEW changes
 jar430.ksh - converted to script, calling uvsort with sel/del's
V4. JAR440.JCL - SORT INCLUDE AND/OR's requiring MANY changes
 jar440.ksh - converted to script, calling uvsort with sel/del's
V5. jar441.ksh - alternative to jar440, with record selections in 2nd step
 jar441.uvc - uvcopy parameter file called by step 2 of jar441.ksh
  - using compare/skip instructions for each code value
V6. jar442.ksh - 2nd alternative to jar440, with record selections in 2nd step
 jar442.uvc - uvcopy parameter file called by step 2 of jar442.ksh
  - using 'tst' to compare multiple values with 1 instruction
V7. jar443.ksh - 3nd alternative to jar440, with record selections in 2nd step
 jar443.uvc - uvcopy parameter file called by step 2 of jar443.ksh
  - using table load/lookup to shorten coding & simplify maintenance
V8. jar444.ksh - 4th alternative with sort & record selections combined
 jar444.uvc - uvcopy parameter file called by jar444.ksh (now just 1 step)
  - same as jar443.uvc + sort instructions (sxo,sxp,sxs,sxg)
V9. jar445.ksh - 5th alternative to demo read table lookup codes from external file
 jar445.uvc - uvcopy parameter file called by jar445.ksh
  - with 'rtb' to read codes from external file for lookup by 'lok'

Goto:   Begin this document End this document UVSI Home-Page

V1. JCL/SORT/INCLUDE conversions to script/uvsort/select

The following SORT/uvsort conversion examples will show only the relevant lines of INCLUDEs & converted selects - so we can focus on any corrections that may be required for complex INCLUDEs with multiple mixed AND/ORs. See all lines of the 1st sample JCL/script SORT/uvsort on pages 'U4' & 'U5'.

JAR410.JCL - SORT INCLUDE AND/OR's requiring NO changes

 //JAR410   JOB  'test SORT INCLUDE AND/OR'
 //STEP1    EXEC PGM=SORT
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=AR.SALES.ITEMS1,DISP=NEW,LRECL=64
 //SYSIN    DD *
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman & product
 INCLUDE COND=(11,2,CH,EQ,C'21',AND,31,6,CH,EQ,C'HAM001',OR,
                11,2,CH,EQ,C'44',AND,31,6,CH,EQ,C'SAW051')
 /*

jar410.ksh - script/uvsort equivalent selects

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items1
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=(11,2,CH,EQ,C'21',AND,31,6,CH,EQ,C'HAM001',OR,
                11,2,CH,EQ,C'44',AND,31,6,CH,EQ,C'SAW051')
 /*
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RSF,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel2=30(6c):HAM001,sel1=10(2c):44,sel2=30(6c):SAW051"
 #4----------------------------------------------------------------------

Changes for INCLUDEs with mixed AND/ORs

  1. No changes required for INCLUDE/OMITs without mixed AND/ORs.

  2. No changes required for simpler INCLUDE/OMITs with mixed AND/ORs (jar410 above has mixed AND/ORs but requires no changes).

  3. An #ERR/warning is generated when changes are required (see jar420 below, which does require a change).

  4. You can execute any of these 8 demo jobs (jar410.ksh - jar444.ksh) as documented on page 'U6' & you should get the record counts listed on page 'U8'.

Goto:   Begin this document End this document UVSI Home-Page

V2. JCL/SORT/INCLUDE conversions to script/uvsort/select

JAR420.JCL - SORT INCLUDE AND/OR's requiring 1 change

 //JAR420   JOB  'test SORT INCLUDE AND/OR'
 //STEP1    EXEC PGM=SORT
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=AR.SALES.ITEMS2,DISP=NEW,LRECL=64
 //SYSIN    DD *
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman & product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
           AND,(31,6,CH,EQ,C'HAM001'))
 /*

jar420.ksh - script/uvsort/select BEFORE 1 change

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items2
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
           AND,(31,6,CH,EQ,C'HAM001'))
 /*
 #ERR! OR/AND mixed, may need fix, see www.uvsoftware.ca/uvsort.htm#V0
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RSF,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel1=10(2c):44,sel2=30(6c):HAM001"
 #4----------------------------------------------------------------------

jar420.ksh - uvsort/select AFTER 1 change

 #-----------------------------------------------------------------------
 # original above, corrected below for mixed OR/AND problem
 # (JCL converter does not respect multiple parens in INCLUDE COND=((..etc..))
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RST,rcs=46\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel2=30(6c):HAM001,sel1=10(2c):44,sel2=30(6c):HAM001"
 #---------------^^^^^^^^^^^^^^^^^^^----------------------------------

((parens)) not considered in mixed AND/ORs

  1. 'OR' causes uvsort to reset to sel1, 'AND' causes increment sel2,sel3,etc.

  2. So the OR & 2nd AND convert OK, but the AND must be re-coded with the 1st CONDition. I have underlined the inserted AND (sel2)

  3. For these test/demos I changed uvsort fileo1=$SORTOUT,typ=RSF to 'typ=RST', which inserts LineFeeds in output records for easier inspection with vi/cat.

Goto:   Begin this document End this document UVSI Home-Page

V3. JCL/SORT/INCLUDE conversions to script/uvsort/select

JAR430.JCL - SORT INCLUDE AND/OR's requiring a FEW changes

 //JAR430   JOB  'test SORT INCLUDE AND/OR'
 //STEP1    EXEC PGM=SORT
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=AR.SALES.ITEMS3,DISP=NEW,LRECL=64
 //SYSIN    DD *
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman & product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'HAM001',OR,31,6,CH,EQ,C'SAW051'))
 /*

jar430.ksh - script/uvsort/select BEFORE a few changes

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items3
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'HAM001',OR,31,6,CH,EQ,C'SAW051'))
 /*
 #ERR! AND/OR mixed, may need fix, see www.uvsoftware.ca/uvsort.htm#V0
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RSF,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel1=10(2c):44,sel2=30(6c):HAM001,sel1=30(6c):SAW051"
 #4----------------------------------------------------------------------

jar430.ksh - script/uvsort/select AFTER a few changes

 #-----------------------------------------------------------------------
 # original above, corrected below for mixed OR/AND problem
 # (JCL converter does not respect multiple parens in INCLUDE COND=((..etc..))
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel2=30(6c):HAM001,sel1=10(2c):21,sel2=30(6c):SAW051\
 ,sel1=10(2c):44,sel2=30(6c):HAM001,sel1=10(2c):44,sel2=30(6c):SAW051"
 # slsmn '21' (1st sel line) must have sel2 for both HAM001 & SAW051
 # slsmn '44' (2nd sel line) must have sel2 for both HAM001 & SAW051
 # For long complex INCLUDE/OMITs with multiple AND/OR's
 # - recommend re-write in uvcopy, using table load/lookup if many codes
 # - see sf/jar441.ksh & pf/jar441.uvc, sf/jar442.ksh & pf/jar442.uvc, etc
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

V4. JCL/SORT/INCLUDE conversions to script/uvsort/select

JAR440.JCL - SORT INCLUDE AND/OR's requiring MANY changes

 //JAR440   JOB  'test SORT INCLUDE AND/OR'
 //STEP1    EXEC PGM=SORT
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=AR.SALES.ITEMS4,DISP=NEW,LRECL=64
 //SYSIN    DD *
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman & product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'BBQ001',OR,31,6,CH,EQ,C'BIT075',
             OR,31,6,CH,EQ,C'HAM051',OR,31,6,CH,EQ,C'HAX129',
             OR,31,6,CH,EQ,C'TAB013',OR,31,6,CH,EQ,C'VAR200'),
           AND,(39,6,ZD,GT,19))
 /*

jar440.ksh - script/uvsort/select BEFORE many changes

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items4
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'BBQ001',OR,31,6,CH,EQ,C'BIT075',
             OR,31,6,CH,EQ,C'HAM051',OR,31,6,CH,EQ,C'HAX129',
             OR,31,6,CH,EQ,C'TAB013',OR,31,6,CH,EQ,C'VAR200'),
           AND,(39,6,ZD,GT,19))
 /*
 #ERR! OR/AND mixed, may need fix, see www.uvsoftware.ca/uvsort.htm#V0
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RSF,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel1=10(2c):44\
 ,sel2=30(6c):BBQ001,sel1=30(6c):BIT075,sel1=30(6c):HAM051\
 ,sel1=30(6c):HAX129,sel1=30(6c):TAB013,sel1=30(6c):VAR200\
 ,sel2=38(6z)>19"
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

jar440.ksh - uvsort/select AFTER many changes

 #-----------------------------------------------------------------------
 # original above, corrected below, for mixed OR/AND problem
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)\
 ,sel1=10(2c):21,sel2=30(6c):BBQ001,sel3=38(6z)>19
 ,sel1=10(2c):21,sel2=30(6c):BIT075,sel3=38(6z)>19
 ,sel1=10(2c):21,sel2=30(6c):HAM001,sel3=38(6z)>19
 ,sel1=10(2c):21,sel2=30(6c):HAX129,sel3=38(6z)>19
 ,sel1=10(2c):21,sel2=30(6c):TAB013,sel3=38(6z)>19
 ,sel1=10(2c):21,sel2=30(6c):VAR200,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):BBQ001,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):BIT075,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):HAM001,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):HAX129,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):TAB013,sel3=38(6z)>19
 ,sel1=10(2c):44,sel2=30(6c):VAR200,sel3=38(6z)>19"
 # Must add many 'sel's since converter did not respect (parens) & many AND/OR's
 # - each salesman selecting 6 products selling more than 19 units
 # For long complex INCLUDE/OMITs with multiple AND/OR's
 # - recommend re-write in uvcopy, using table load/lookup if many codes
 # - see sf/jar441.ksh & pf/jar441.uvc, sf/jar442.ksh & pf/jar442.uvc, etc
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

V5. JCL/SORT/INCLUDE conversions to script/uvsort/select

jar441.ksh - jar400 alt, step2 call uvcopy cmc record selects

 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'BBQ001',OR,31,6,CH,EQ,C'BIT075',
             OR,31,6,CH,EQ,C'HAM051',OR,31,6,CH,EQ,C'HAX129',
             OR,31,6,CH,EQ,C'TAB013',OR,31,6,CH,EQ,C'VAR200'),
           AND,(39,6,ZD,GT,19))
 /*
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT1,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)"
 #4----------------------------------------------------------------------
 # -- non-relevant lines omitted --
 #1======================= begin step#S0012 uvcopy ========================
 S0012=A
 #Note - uvcopy step inserted manually to perform record selection after uvsort
 # - because original mainframe SORT had mixed AND/OR's with multiple (parens)
 # - requiring too much corrective code in uvsort, can recode in uvcopy shorter
 # - uvcopy parameter file stored in $RUNLIBS/pf/... & called in this 2nd step
 stepctl51;
 goto
 export JSTEP=S0012; ((XSTEP+=1)); SCC=0; LCC=0; alias goto="";
 logmsg2 "******** Begin Step $JSTEP uvcopy (#$XSTEP) ********"
 export PROGID=uvcopy
 exportfile  SORTOUT1 tmp/ar.sales.items4
 exportfile  SORTOUT data1/ar.sales.items4
 #3----------------------------------------------------------------------
 uvcopy $RUNLIBS/pf/jar441.uvc,fili1=$SORTOUT1,filo1=$SORTOUT
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

jar441.uvc - 2nd step uvcopy, cmc/skp for each value

 # jar441.uvc - uvcopy job to perform record selection following uvsort
 # - separate parameter file called from the JCL/script after uvsort
 # - see script sf/jar441.ksh that calls this uvcopy job
 # sf/jar441.ksh & pf/jar441.uvc replace auto-gen jcls/jar440.ksh
 # - from jcl0/JAR440.JCL which had mixed AND/OR's with multiple parens
 fili1=${SORTOUT1},typ=RST,rcs=64
 filo1=${SORTOUT},typ=RST,rcs=64    'typ=RST' inserts LineFeeds, allows 'vi'
 @run
        opn    all                   open files
 man20  get    fili1,a0              get record
        skp>   man90                 (cc set > at EOF)
 #
        cmn    a38(6),19             sale units > 19 ?
        skp<=  man20
 #
        cmc    a10(2),'21'           salesman 21 or 44 ?
        skp=   man30
        cmc    a10(2),'44'
        skp!   man20
 #
 man30  cmc    a30(6),'BBQ001'       any 1 of 6 products ?
        skp=   man60
        cmc    a30(6),'BIT075'
        skp=   man60
        cmc    a30(6),'HAM001'
        skp=   man60
        cmc    a30(6),'HAX129'
        skp=   man60
        cmc    a30(6),'TAB013'
        skp=   man60
        cmc    a30(6),'VAR200'
        skp!   man20
 #
 man60  put    filo1,a0              write selected record
        skp    man20                 return to get next
 man90  cls    all                   close files
        eoj                          end job

Notes re uvcopy instructions

  1. See all uvcopy instructions documented at uvcopy3.htm.

  2. This solution uses 'cmn' (CoMpare Numeric), 'cmc' (CoMpare Character), & 'skp' (SKiP depending on condition =!<>)

  3. Compare this to following shorter solutions jar442.uvc & jar443.uvc.

  4. jar442.uvc uses 'tst' which compares multiple values with 1 instruction to reduce the number of instructions required.

  5. jar443.uvc uses 'lod' & 'lok' (table load & table lookup) to reduce instructions & also to improve maintenance (easier to update tables).

Goto:   Begin this document End this document UVSI Home-Page

V6. JCL/SORT/INCLUDE conversions to script/uvsort/select

jar442.ksh - jar400 alt, step2 call uvcopy tst record selects

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT1 tmp/ar.sales.items4
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'BBQ001',OR,31,6,CH,EQ,C'BIT075',
             OR,31,6,CH,EQ,C'HAM051',OR,31,6,CH,EQ,C'HAX129',
             OR,31,6,CH,EQ,C'TAB013',OR,31,6,CH,EQ,C'VAR200'),
           AND,(39,6,ZD,GT,19))
 /*
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT1,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)"
 #4----------------------------------------------------------------------
 # -- non-relevant lines omitted --
 #1======================= begin step#S0012 uvcopy ========================
 S0012=A
 #Note - uvcopy step inserted manually to perform record selection after uvsort
 # - sf/jar442.ksh same as sf/jar441, but calls pf/jar442.uvc (vs pf/jar441.uvc)
 # - pf/jar442 performs same selections as jar441, but in less code using 'tst'
 #
 # -- non-relevant lines omitted --
 export PROGID=uvcopy
 exportfile  SORTOUT1 tmp/ar.sales.items4
 exportfile  SORTOUT data1/ar.sales.items4
 #3----------------------------------------------------------------------
 uvcopy $RUNLIBS/pf/jar442.uvc,fili1=$SORTOUT1,filo1=$SORTOUT
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

jar442.uvc - uvcopy test multi-values with 1 'tst' instrn

 # jar442.uvc - uvcopy job to perform record selection following uvsort
 # - separate parameter file called from the JCL/script after uvsort
 # - see script sf/jar442.ksh that calls this uvcopy job
 # sf/jar442.ksh & pf/jar442.uvc replace auto-gen jcls/jar440.ksh
 # - from jcl0/JAR440.JCL which had mixed AND/OR's with multiple parens
 # pf/jar442.uvc same function as pf/jar441.uvc
 # - and demos use of 'tst' instrn to save many lines of code
 fili1=${SORTOUT1},typ=RST,rcs=64
 filo1=${SORTOUT},typ=RST,rcs=64    'typ=RST' inserts LineFeeds, allows 'vi'
 @run
        opn    all                   open files
 man20  get    fili1,a0              get record
        skp>   man90                 (cc set > at EOF)
 #
        cmn    a38(6),19             sale units > 19 ?
        skp<=  man20
 #
        tstl2  a10(2),'2144'         salesman 21 or 44 ?
        skp!   man20
 #
 man30  tstl6  a30(6),'BBQ001BIT075HAM001HAX129TAB013VAR200'
        skp!   man20
 #
 man60  put    filo1,a0              write selected record
        skp    man20                 return to get next
 man90  cls    all                   close files
        eoj                          end job

Notes re uvcopy instructions

  1. jar442.uvc uses 'tst' which compares multiple values with 1 instruction to reduce the number of instructions required.

Goto:   Begin this document End this document UVSI Home-Page

V7. JCL/SORT/INCLUDE conversions to script/uvsort/select

jar443.ksh - jar400 alt, step2 call uvcopy lod/lok selects

 exportfile  SORTIN data1/ar.sales.items
 exportfile  SORTOUT1 tmp/ar.sales.items4
 exportfile SYSIN $JTMP/${JSTEP}_${PROGID}_SYSIN
 cat > $SYSIN <</*
 SORT FIELDS=(11,2,CH,A,31,6,CH,A)   Sort on salesman $ product
 INCLUDE COND=((11,2,CH,EQ,C'21',OR,11,2,CH,EQ,C'44'),
          AND,(31,6,CH,EQ,C'BBQ001',OR,31,6,CH,EQ,C'BIT075',
             OR,31,6,CH,EQ,C'HAM051',OR,31,6,CH,EQ,C'HAX129',
             OR,31,6,CH,EQ,C'TAB013',OR,31,6,CH,EQ,C'VAR200'),
           AND,(39,6,ZD,GT,19))
 /*
 #3----------------------------------------------------------------------
 uvsort "fili1=$SORTIN,typ=RSF,rcs=64,filo1=$SORTOUT1,typ=RST,rcs=64\
 ,keys=(10,2,c,a,30,6,c,a)"
 #4----------------------------------------------------------------------
 # -- non-relevant lines omitted --
 #1======================= begin step#S0012 uvcopy ========================
 #Note - uvcopy step inserted manually to perform record selection after uvsort
 # - sf/jar443.ksh same as sf/jar442, but calls pf/jar443.uvc (vs pf/jar442.uvc)
 # - pf/jar443 performs same selections as jar441, but in less code using lod/lok
 # - 'lod' loads table for lookup by 'lok', saves code & easier to maintain
 #
 # -- non-relevant lines omitted --
 export PROGID=uvcopy
 exportfile  SORTOUT1 tmp/ar.sales.items4
 exportfile  SORTOUT data1/ar.sales.items4
 #3----------------------------------------------------------------------
 uvcopy $RUNLIBS/pf/jar443.uvc,fili1=$SORTOUT1,filo1=$SORTOUT
 #4----------------------------------------------------------------------

Goto:   Begin this document End this document UVSI Home-Page

jar443.uvc - uvcopy table load/lookup

 # jar443.uvc - uvcopy job to perform record selection following uvsort
 # - separate parameter file called from the JCL/script after uvsort
 # - see script sf/jar443.ksh that calls this uvcopy job
 # sf/jar443.ksh & pf/jar443.uvc replace auto-gen jcls/jar440.ksh
 # - from jcl0/JAR440.JCL which had mixed AND/OR's with multiple parens
 # pf/jar443.uvc same function as pf/jar442.uvc
 # - and demos use of lod/lok to load tables & lookup tables
 # - saves coding & for easier maintenance
 # - great when you have dozens or hundreds of codes to select
 fili1=${SORTOUT1},typ=RST,rcs=64
 filo1=${SORTOUT},typ=RST,rcs=64    'typ=RST' inserts LineFeeds, allows 'vi'
 lod=c0(6)      <-- load table of 6 byte product codes for 'lok'-up
 BBQ001
 BIT075
 HAM001
 HAX129
 TAB013
 VAR200
 ~~~~~~
 #Note - following #commented out 3 lines same as 8 lines above
 #     - save code by loading all 6 x 6 byte entries as 1 36 byte entry
 #     - but 'lok'up using 6 byte entry length
 # lod=c0(36)
 # BBQ001BIT075HAM001HAX129TAB013VAR200
 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 @run
        opn    all                   open files
 man20  get    fili1,a0              get record
        skp>   man90                 (cc set > at EOF)
 #
        cmn    a38(6),19             sale units > 19 ?
        skp<=  man20
 #
        tstl2  a10(2),'2144'         salesman 21 or 44 ?
        skp!   man20
 #
 man30  lok    c0(6),c0(6),a30(6)   lookup table of product codes
        skp!   man20
 #
 man60  put    filo1,a0              write selected record
        skp    man20                 return to get next
 man90  cls    all                   close files
        eoj                          end job
  1. jar443.uvc uses 'lod' & 'lok' (table load & table lookup) to reduce instructions & also to improve maintenance (easier to update tables).

  2. Could easily add dozens or hundreds of more values to be selected. The solution above uses 1 line for each table value (for clarity). Note 'lod=c0(6)' loads 1 value on each line (for lookup by the 'lok').

  3. Also see the next solution jar444.uvc which 'lod's multiple values on each line to reduce lines required. Note 'lod=c0(36)' loads 6 values on each line, but uses same lookup 'lok c0(6),c0(6),a30(6)'. Op1 defines the table entry length & op2 the compare length to op3. Op1 & op2 are same here 'c0(6)', but this allows multi column tables.

Goto:   Begin this document End this document UVSI Home-Page

V8. JCL/SORT/INCLUDE conversions to script/uvsort/select

jar444.ksh - jar400 alt, 1 step call uvcopy sort+select


 #1======================= begin step#S0012 uvcopy ========================
 export PROGID=uvcopy
 exportfile  SORTIN  data1/ar.sales.items
 exportfile  SORTOUT data1/ar.sales.items4
 #3----------------------------------------------------------------------
 uvcopy $RUNLIBS/pf/jar444.uvc,fili1=$SORTIN,filo1=$SORTOUT
 #4----------------------------------------------------------------------

jar444.uvc - uvcopy sort & select combined

 # jar444.uvc - uvcopy job to perform record selection following uvsort
 # - separate parameter file called from the JCL/script after uvsort
 # - see script sf/jar444.ksh that calls this uvcopy job
 # sf/jar444.ksh & pf/jar444.uvc replace auto-gen jcls/jar440.ksh
 # - from jcl0/JAR440.JCL which had mixed AND/OR's with multiple parens
 # pf/jar444.uvc performs both the sort & the record selection
 # - demos how to code a sort within a uvcopy job
 # sf/jar444.ksh now only 1 step to call this uvcopy job (remove the uvsort)
 fili1=${SORTOUT1},typ=RST,rcs=64
 filo1=${SORTOUT},typ=RST,rcs=64    'typ=RST' inserts LineFeeds, allows 'vi'
 lod=c0(36)
 BBQ001BIT075HAM001HAX129TAB013VAR200
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 # - above saves code, by 'lod'ing all 6 x 6 byte entries as 1 36 byte entry
 # - but 'lok'up below, using 6 byte entry length
 @run
        opn    all                   open files
        sxo    64,'10(2),30(6)'  <-- open sort, recsize 64, keys slsmn&product
 #
 man20  get    fili1,a0              get record
        skp>   man40                 at EOF, go execute sort
        cmn    a38(6),19             sale units > 19 ?
        skp<=  man20
        tstl2  a10(2),'2144'         salesman 21 or 44 ?
        skp!   man20
        lok    c0(6),c0(6),a30(6)    lookup table of product codes
        skp!   man20
        sxp    a0(64)            <-- put to the sort
        skp    man20                 return to get next record
 #
 man40  sxs                      <-- execute the sort
 #
 man50  sxg    b0(64)            <-- get record from sort
        skp>   man90                 (cc set > at end sorted recs)
        put    filo1,b0              write selected record
        skp    man50                 return to get next from sort
 #
 man90  cls    all                   close files
        eoj                          end job
  1. The sort is performed by sxo,sxp,sxs,& sxg (see '<--' markers).

  2. This job has same record selection (table load/lookup) as prior jar443.uvc) but uses 'lod=c0(36)' to load 6 values per line vs 1 'lod=c0(6)'.

Goto:   Begin this document End this document UVSI Home-Page

V9. JCL/SORT/INCLUDE conversions to script/uvsort/select

jar445.ksh - same as jar443.ksh, but calls jar445.uvc

jar445.uvc - jar443.ksh alt, read table from external file

 # jar445.uvc - uvcopy job to perform record selection following uvsort
 # - separate parameter file called from the JCL/script after uvsort
 # - see script sf/jar445.ksh that calls this uvcopy job pf/jar445.uvc
 # - demo load/lookup tables to save coding & enable easier maintenance
 # - great when you have dozens or hundreds of codes to select
 # jar445.uvc - alternate coding for jar443.uvc
 # jar443.uvc - demo'd lookup tables using 'lod' load values coded in job
 # jar445.uvc - same demo, but using 'rtb' to Read TaBle from external file
 fili1=${SORTOUT1},typ=RST,rcs=64
 filo1=${SORTOUT},typ=RST,rcs=64
 fili2=data1/ar.product.select,typ=LST,rcs=80  #<-- products to be selected
 # - fili2 read into pf/jar445.uvc by 'rtbc1' (after 'opn' below)
 # BBQ001   BAR-B-Q
 # BIT075   DRILL BIT       <-- sample records shown here as #comments
 # HAM001   CLAW HAMMER
 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 #      rtbc1  fili2,c0(80),c0(40)  <-- Read TaBle of product codes
 # - reads 80 bytes max, stores 1st 40 bytes in table entries
 #      lok    c0(40),c0(6),a30(6)  <-- see LOoKup instrn below
 # - looks up table of 40 byte entries of table in area 'c'
 # - compares 1st 6 bytes c0(6) to bytes 30-35 a30(6) of input record area 'a'
 @run
        opn    all                      open files
        rtbc1  fili2,c0(80),c0(40)  <-- Read TaBle of product codes
 #
 man20  get    fili1,a0                 get record
        skp>   man90                    (cc set > at EOF)
        cmn    a38(6),19                sale units > 19 ?
        skp<=  man20
        tstl2  a10(2),'2144'            salesman 21 or 44 ?
        skp!   man20
 #
 man30  lok    c0(40),c0(6),a30(6)  <-- lookup table of product codes
        skp!   man20
 #
 man60  put    filo1,a0                 write selected record
        skp    man20                    return to get next
 #
 man90  cls    all                      close files
        eoj                             end job

Goto:   Begin this document End this document UVSI Home-Page

Visitor Counters for ThisYear and LastYear