UVdemos.doc - Contents by Part#


Part_0 Introduction, Overview, Intended Audience
- intended for any Unix/Linux user (vs mainframe conversion users)
- demo Vancouver Utility capabilities far beyond unix/linux system utilities
- Pre-Requisites to run the UVdemos tutorials

Part_1 Installing Vancouver utilities, setup uvadm & appsadm admin accounts
- setup user profiles to execute Vancouver Utilities

Part_2 Basic Vancouver Utilities are uvhd,uvsort,uvcp,& uvlist (vs uvcopy)
Brief descriptions of uvhd,uvsort,uvcp,& uvlist, followed by tutorials
illustrating how they might help you solve your data problems.
uvhd demo - investigating problem files, CR's in scripts, removing
uvhd demo - converting French accented characters (UTF-8) to unaccented
uvcp demo - converting EBCDIC files to ASCII

Part_3 SelectJobs - select lines from 1 or all files in a directory, identified by
Multiple patterns & conditions may be on same line or anywhere in the file.
May identify Begin & End group lines to find desired patterns in between.
Like multiple greps in 1 pass, creates reports named from files & patterns.

Part_4 'table3d' create table summary counts of desired items in text files
- pre-programmed job using uvcopy (powerful data manipulation utility)
- scan all files in directory building table statistics to dump at EOF
- examples - counts of IP#s, Userids, Passwords in FTP scripts

Part_5 Search/replace pre-programmed jobs
scan1d - scan all files in a directory for 1 pattern
- may qualify with 2 other patterns that may or may not be present
rep1d - search all files in directory, replacing 1 pattern with an alternate
  while copying to a 2nd directory
- may qualify with 2 other patterns that may or may not be present
rep2d - search all files in directory, replacing using a search/replace table
  with unlimited number of search & replacement strings
- may qualify search/replaces with 1 present & 1 absent

Part_10 Complex conversions considered too long for presentation earlier.
We may have introduced in previous parts with links to see the full
presentations in Part 10. For example '10A1' is the full documentation
converting UTF-8 French Accents to unaccented equivalents for old applications

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_0 UVdemos.doc - Vancouver Utilities for any Unix/Linux user

Introduction & Overview

This document is intended to make you aware of the most useful utilities supplied in the Vancouver Utilities package. You can run various tutorials using pre-programmed jobs with supplied test data files.

We will illustrate Vancouver Utility capabilities that are not possible with the standard unix/linux system utilities & we hope you can see how these capabilities could apply to your particular data processing projects.

These utilities are 'GENERAL PURPOSE' - they can be used with any data-file & can process all files in a directory. For our demos, we will use files such as FTP scripts, SQL scripts,& JCL, but we hope you can realize how these utilities could apply to your files (data, scripts,& programming languages).

We will minimize using JCL as testdata (often used in mainframe conversion demos). We will maximize the use testdata such as FTP & SQL scripts, which are familiar to most data processing professionals.

The intention here is to give you a short (1 or 2 page) introduction to the various utilities available, and then give you a link to the complete documentation which could be in various other books.

If you have a problem with your files & are wondering how it could be solved with Vancouver Utilities, we invite you to email us, explain the problem, and we will help you solve it (as our time allows).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_1 UVdemos.doc - Vancouver Utilities for any Unix/Linux user

Part 1 - Contents


1A1. Install Vancouver Utilities (if not already installed)

1A2. Vancouver Utility Administrator home directory /home/uvadm
- directories & contents briefly described

1A3. Install Vancouver Utilities (if not already installed)

1B1. Setup user logins and/or modifying profiles to execute Vancouver Utilities
- user profiles call the common_profile stored in /home/appsadm/env/...

1B2. .bash_profile for userxx

1B3. common_profile_abc for VU users

1B4. Advantages of stub_profile,& common_profile

1C1. setup userxx account to run demos

1C2. copy demo files to your $HOME/demo/...

1D1. demo/dat1/... - DATA files
1D2. demo/parms/... - FTP scripts (4 short samples)
1D3. demo/sqls/... - SQL scripts (8 short samples)
1D4. demo/sf/... - bash scripts (llsx & llsy)
demo/pf/... - uvcopy jobs (removeCR_uvcopy)
1D5. demo/jcl2/... - JCL (just 1 of 6 samples listed)

1A1. Install Vancouver Utilities

Before you can begin these Vancouver Utility UVdemos tutorials, you must install the package as documented at www.uvsoftware.ca/install.htm. Here are the more important pages:

Page B2
  • setup Vancouver Utilities administrator account
  • commonly /home/uvadm but could be elsewhere & export UV=location
Page B3
  • Download Vancouver Utilities with login & password supplied & unzip
Page B4
  • setup appsadm account to hold the common_profile called by user profiles
  • so not lost when new versions of Vancouver Utilities installed
Page C1
  • compile the Vancouver Utility C programs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A2. UVdemos.doc - Vancouver Utilities for any Unix/Linux user

Vancouver Utilities homedir contents

 /home/uvadm           <-- $UV
 :-----bin               - binaries (uvcopy,uvsort,uvcp,uvhd,uvlist,etc)
 :-----ctl               - control files for various purposes
 :-----dat1              - test data files to demo various Vancouver Utilities`
 :-----doc               - Vancouver Utilities documentation (text)
 :-----dochtml           - documentation in HTML (same as on www.uvsoftware.ca)
 :-----demo            <-- demo files for this TestDemo tutorial
 :     :                 - to be copied to your $HOME/demo/...
 :     :-----dat1        - data files for uvsort,uvcopy,uvcp,etc
 :     :-----jcl0        - mainframe JCL samples (8)
 :     :-----jcl2        - JCLs cleaned up, lower cased
 :     :-----jcl3        - JCLs after conversion to Korn shell scripts
 :     :-----parms       - SYSIN control cards for various purposes
 :     :-----pf          - Parameter Files for uvcopy
 :     :-----rpts        - reports created by various demos (selects,etc)
 :     :-----sf          - Script Files for bash or ksh
 :     :-----sqls        - SQL scripts for demo table analysis
 :     :-----stats       - outputs from table3d table summaries
 :     :-----tf          - Test Files (similar to dat1/...)
 :     :-----tmp         - temporary files, outputs from demos
 :     :-----tmp1        - may clear tmp files before demos to see only new outputs
 :-----env             <-- profiles for users, PATHs to utilities & other software
 :-----hdr               - hdr files for C compiles
 :-----htmlcode          - merged into dochtml when text converted to HTML
 :-----lib               - libraries for C compiles (subfunctions,DISAM,etc)
 :-----mvstest           - test/demos for MVS JCL/COBOL MainFrame conversions
 :-----mf              <-- small versions of MainFrame testdata files
 :-----pf              <-- Parameter Files for uvcopy & uvqrpg
 :     :-----adm         - administrative jobs
 :     :-----demo        - demo jobs
 :     :-----IBM         - IBM mainframe conversion jobs
 :     :-----util        - utility jobs
 :-----sf              <-- Script Files
 :     :-----adm         - administrative scripts
 :     :-----demo        - demo scripts
 :     :-----IBM         - IBM mainframe conversion scripts
 :     :-----util        - utility scripts
 :-----sfun              - ksh functions used in converted JCL/scripts
 :-----src             <-- Vancouver Utilities C source code
 :-----srcf              - C source for various sub-functions
 :-----tf                - test files for various examples in doc
 :-----tmp               - tmp subdir (test/demo outputs)
 :-----vsetest           - test/demos for VSE JCL/COBOL MainFrame conversions

Note that the uvadm subdirs for 'sf' (Script Files) & 'pf' (uvcopy Parameter Files or uvcopy jobs) are further sub-directoried as shown above.


Note - see http://uvsoftwre.ca/intall.htm to Install Vancouver Utilities
- see a brief summary on the next page

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A3. UVdemos.doc - Installing Vancouver Utilities

Before you can run the UVdemos tutorials, you must install the Vancouver Utilities (if not already installed). See instructions at http://uvsoftwre.ca/intall.htm, but here is a brief summary.

Installing Vancouver Utilities - Summary

  1. Setup user account 'uvadm' to serve as Vancouver Utilities administrator. Usual home directory is /home/uvadm, but could be elsewhere if you change the environmental variable 'export UV=/home/uvadm' in the common_profile.

  2. We have suggested setting up group 'apps' for uvadm & other users who will be working with Vancouver Utilities.

  3. Download the Vancouver Utilities from the UV Software website into the uvadm home directory,& unzip or untar (from archives uvadm.zip or uvadm.tar.gz). To download the complete package, you will need a userid & passwword from UV Software, but some programs are free (such as 'uvhd').

  4. You may need to modify the profiles as required for your site. We recommend setting up user 'appsadm' to store your modified versions of the profiles, so you do not lose them if you install a new version of Vancouver Utilities.

  5. Setup user 'appsadm' & copy /home/uvadm/env/* to /home/appsadm/env Then modify .profiles or .bash_profiles to call the common_profile from /home/appsadm/env/common_profile. See short versions of the stub_profile & common_profile listed on pages '1B2' & '1B3'. The stub_profiles (in user homedirs) calls the common_profile via '.'

  6. Setting up appsadm to store the common_profile is vital if you have multiple users, but could be eliminated if you are installing VU only for yourself. You could then call the common_profile from $UV/env/common_profile, rather than from $APPSADM/env/common_profile, & have backup for your $UV/env/common_profile when you install new versions of Vancouver Utilities.

  7. The C source utility programs are stored in $UV/src & are compiled into $UV/bin. The binaries delivered may or may not work on your Linux version (compile if required).

See complete install & compile instructions at http://uvsoftware.ca/install.htm.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B1. 'profiles' provided in /home/uvadm/env

'profiles' provided in /home/uvadm/env

 /home/uvadm/env         <-- profiles provided here
 :-----stub_profile_uv     - copy/rename to .profile (ksh) or .bash_profile (bash)
 :                         - defines RUNLIBS/RUNDATA for programmers & operators
 :-----common_profile_uv <-- common profile (called by stub_profile)
 :                         - defines PATH's etc using $RUNLIBS/$RUNDATA
 /home/appsadm/env       <-- setup user 'appsadm' & copy from /home/uvadm/env/*
 :-----stub_profile_ABC    - customize & copy to homedirs .profile or .bash_profile
 :-----common_profile_ABC  - common profile (called by stub_profile)

We recommend setting up an application administrator 'appsadm', copying /home/uvadm/env/* to /home/appsadm/env, and customizing the profiles there depending on the site locations of libraries & data.

Do NOT customize profiles in /home/uvadm/env/... because they would be overwritten when a new version of Vancouver Utilities is installed.

We recommend the concept of 'stub' & 'common' profiles. The shell profile in each user's homedir is a 'stub' that calls 'common_profile' which are stored in /home/appsadm/env/...

  1. The supplied 'stub_profile' is copied to homedirs, renamed as '.bash_profile' for bash shell (Linux) & '.profile' for Korn shell (Unix), and then modified depending on whether the user is a programmer or operator. - defines RUNLIBS as testlibs for programmers OR prodlibs for operators - defines RUNDATA as testdata for programmers OR proddata for operators

  2. 'common_profile' then defines the 'PATH's using $RUNLIBS,$COBDIR,$UV,etc For example: export PATH=$PATH:$RUNLIBS/jcls (converted JCL/scripts). Defines software superdirs (uvadm, COBDIR, ORACLE_BASE, ORACLE_HOME, etc)

  3. '$RUNDATA' determines data-file locations indirectly as follows: $RUNDATA defines the superdir housing all data-files.

Note that stub profiles must call 'common_profile' using '.' (dot execution), which means the 'export's made in the common_profile will still be effective on return to the users profile.

This system is a big advantage for any site with multiple users, it means the sysadmin can update common_profile once in 1 place & those changes are effective for all users.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B2. 'profiles' provided in /home/uvadm/env

.bash_profile for userxx

Here is a stripped down version of /home/uvadm/env/stub_profile_uv

 # stub_profile_uv - stub_profile for Vancouver Utilities
 #                 - stub_profile calls 'common_profile'
 #
 # common_profile & stub_profile - distributed in $UV/env/...
 # - copy to $APPSADM/env/... (/home/appsadm/env/...) & modify for your site
 # - do not modify profiles in $UV because new versions of uvadm would overwrite
 # - see more at www.uvsoftware.ca/install.htm
 #
 # stub_profile defines $RUNLIBS & $RUNDATA for common_profile,
 # - which uses them to define PATHs to libraries & data
 #
 export RUNLIBS=$HOME/testlibs  # superdir for libraries
 export RUNDATA=$HOME/testdata  # superdir for datafiles
 #
 echo "Vancouver Utility profile"
 echo "HOSTNAME=$HOSTNAME LoginUser=$LOGNAME"
 echo "RUNLIBS=$RUNLIBS RUNDATA=$RUNDATA"
 #
 . /home/appsadm/env/common_profile_uv  # CHANGED to "appsadm" when profiles copied
 #====================================  # from uvadm/env/ to appsadm/env/
 #
 # - after uvadm install, you should setup appsadm & store env/profiles there
 # - then all user profiles should call common_profile from /home/appsadm/env/...
 # - prevents loss of customized common_profile when new version uvadm installed
 #
 #       ** misc items that user may need to override common_profile defs **
 # export TERM=linux       # TERM - modify depending on your terminal
 # stty erase '^?'         # erase char - modify depending on your terminal
 # stty intr '^C'          # interrupt ^C, (probably already default ?)
 # export UVLPDEST="-dlp0" # default destination for uvlp(uvlist) scripts
 #                         # change to a printer near you & un-comment
 #
 #                       ** user aliases, etc **
 # alias l='ls -l'         # save keystrokes on very often used commands
 # - see common_profile for several more aliases
 # - add more here depending on user preferences

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B3. 'profiles' provided in /home/uvadm/env

common_profile_abc for VU users

Here is a stripped down version of /home/uvadm/env/common_profile_uv

 # common_profile_abc - for users of Vancouver Utilities,COBOL,etc
 #                    - by Owen Townsend, UV Software, Dec 2017
 # - distributed in $UV/env/common_profile_uv
 # - copied/renamed to $APPSADM/env/common_profile_abc for execution by user profiles
 # - see more at www.uvsoftware.ca/install.htm & www.uvsoftware.ca/admjobs.htm
 #
 # common_profile_?? - defines search PATHs based on $RUNLIBS & $RUNDATA
 #                     which must be defined in user's .bash_profile
 #                     before calling this commmon_profile
 #
 # stub_profile_?? - copied to user homedirs & renamed as .profile or .bash_profile
 #                 - defines RUNLIBS & RUNDATA before calling common_profile
 #                 - stub_profile distributed in $UV/env contains following:
 #  export RUNLIBS=$HOME/testlibs
 #  export RUNDATA=$HOME/testdata
 #
 #  . /home/appsadm/env/common_profile_?? #<-- must '.' execute common_profile
 #  =====================================    - store common_profile in appsadm/env
 #
 #Note - profiles distributed in $UV/env/... (usually /home/uvadm/env/...)
 #     - should setup user 'appsadm' & copy $UV/env/... to $APPSADM/env/...
 #     - then change stub_profiles to call common_profile from $APPSADM/env/...
 #       (so you will not lose your changes to profiles when new uvadm installed)
 #
 #                  ** begin code for common_profile **
 #
 export UV=/home/uvadm           # UV homedir symbol used below
 export APPSADM=/home/appsadm    # site applications administrator
 export LOGDIR=$APPSADM          # console logging subdirs log1,log2,log3
 #
 # setup PATH for Vancouver Utilities programs & scripts (uvadm & appsadm)
 # - append onto system PATH, using symbols defined above ($UV, $APPSADM, etc)
 export PATH=$PATH:$HOME/bin:$HOME/sf:$APPSADM/bin:$APPSADM/sf:$RUNLIBS/sf
 export PATH=$PATH:$UV/bin:$UV/sf/adm:$UV/sf/demo:$UV/sf/util:$UV/sf/IBM
 export PATH=$PATH:$UV/help
 export PATH=$PATH:/usr/sbin   # add system dir for sendmail, etc
 #
 # setup 'PFPATH' for uvcopy & uvqrpg interpreter to find Parameter Files (jobs)
 export PFPATH=$RUNLIBS/pf,$CNVDATA/pfx3,$RUNLIBS/ezts,$HOME/pf,$APPSADM/pf
 export PFPATH=$PFPATH,$UV/pf/adm,$UV/pf/demo,$UV/pf/util,$UV/pf/IBM
 # - uvcopy accepts ',' delimiters as well as ':' in case of SFU on Windows
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 #                 ** UV Recommended items **
 umask 002                     # permissions 775 dirs, 664 files
 trm=$(tty)                    # capture terminal device for PS1
 export trmv=${trm#/dev/}      # remove prefix /dev/
 export HOSTNAME               # should already be set
 export HOST1=${HOSTNAME%%.*}  # extract 1st segment of $HOSTNAME
 export PS1='<@$HOST1:$LOGNAME:$PWD> '
 export EDITOR=vi              # for Korn shell history
 export VISUAL=vi              # for Korn shell history
 export HISTSIZE=5000;         # Korn shell history file size
 #
 #                      ** aliases **
 # alias commands to prompt for overwrite (highly recommended)
 # - use option '-f' when you have many files  (rm -f tmp/*, etc)
 alias rm='rm -i'              # confirm removes
 alias mv='mv -i'              # confirm renames
 alias cp='cp -i'              # confirm copy overwrites
 alias l='ls -l'               # save keystrokes
 alias cdl='cd $RUNLIBS'       # quick access to LIBS superdir
 alias cdd='cd $RUNDATA'       # DATA superdir
 #
 #                   ** printer stuff **
 # printer destinations for  laser printing scripts
 # - modify UVLPDEST to the network printer closest to you
 export UVLPDEST="-dMS610"      # default dest for uvlp(uvlist) scripts
 export UVLPOPTN="-onobanner"   # for unix/linux (SFU does not allow)
 export UVHDPRINT=uvlp12        # script for uvhd 'i' immediate print command
 #
 #                  ** Microsoft SQL Server **
 # see www.uvsoftware.ca/sqldemo.htm Part6
 export PATH=$PATH:/opt/mssql-tools/bin
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/microsoft/msodbcsql/lib64
 export ODBCSQL="ODBC Driver 13 for SQL Server"
 export DATABASE=testdb   # $(DATABASE) used in table create & load scripts
 export ODBCINI=/etc/odbc.ini
 export ODBCSYSINI=/etc/  #<-- Directory with ODBC config (not File odbcinst.ini)
 #
 #------------------------- end of common_profile ---------------------------

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B4. profiles for Vancouver Utilities on Unix/Linux

Advantages of stub_profile,& common_profile

  1. You never need to change scripts between TESTING & PRODUCTION.

  2. Programmers execute scripts from $TESTLIBS/scripts that access data from $TESTDATA/... because their profiles set RUNLIBS/RUNDATA as TESTLIBS/TESTDATA

  3. Operators execute scripts from $PRODLIBS/jcls that access data from $PRODDATA/... because their profiles set RUNLIBS/RUNDATA as PRODLIBS/PRODDATA

  4. This system could allow testing & production on the same machine. They are usually on separate machines, and this system provides great flexibility for both these situations.

  5. Different programmers can have their own set of TESTDATA simply by changing the definition of RUNDATA in their profile.

  6. On production machines, different operators could operate on different sets of data for different companies on the same machine, using the same scripts & programs with no changes required.

  7. The system administrator (appsadm) can make changes in 1 place to affect all users vs having to update profiles in all user homedirs.

  8. The sysadmin would control any updates to common_profile. On test systems the programmers would control updates to their stub_profile.

  9. Programmers have the flexibility required. They can select different sets of testdata by redefining RUNDATA in their profile

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C1. UVdemos.doc - Vancouver Utilities for any Unix/Linux user

setup user account to run demos

Here are instructions to setup a user account to run the demos. You could run the demos in the Vancouver Utilities admin account /home/uvadm/... But it is better to setup a user account & copy the demo files into your directories.


 #1. login as 'root'

 #2.  useradd -m -g apps userxx   <-- setup user 'userxx'
      =========================
      - shell defaults depending on system (unix or linux)

 #2a. useradd -m -g apps -s /bin/bash userxx
      ======================================
      - use option '-s' to specify login shell as 'bash'

 #2b. useradd -m -g apps -s /bin/ksh userxx
      =====================================
      - use option '-s' to specify login shell as 'ksh' (Korn shell)

 #3. passwd userxx           <-- setup password desired
     ==============

 #4. chmod 755 /home/userxx  <-- allow copying between your user accounts
     ======================    - optional

 #5. cp $UV/env/stub_profile_uv /home/userxx/.bash_profile
     =====================================================
     - copy supplied profile to homedir renaming to '.bash_profile'
     - calls the common profile from $APPSADM/env/common_profile_uv
     - see profiles listed above on pages '1B2' & '1B3'

Alternative - setup subdir in your home dir

Alternatively, you could use your existing login account. Make a sub-directory $HOME/demo, and copy -r /home/uvadm/demo/* to your $HOME/demo/...

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C2. UVdemos.doc - Vancouver Utilities for any Unix/Linux user

copy demo files to your $HOME/demo


 #1. login userxx --> /home/userxx  # OR use your existing login

 #2. mkdir demo   <-- make demo directory

 #3. cp -r $UV/demo/* demo   <-- copy demo files to your subdir
     =====================
 #4. cd demo  ---> /home/userxx/demo

 #5. llr    <-- alias to List Long Recursive (directories & files)
     ===        alias definition --> alias llr='ls -lR'

llr lists directories 1st

      drwxrwxr-x 2 uvadm apps 4096 Jun 16 09:23 dat1
      drwxrwxr-x 2 uvadm apps 4096 Mar 26 15:59 jcl0
      drwxrwxr-x 2 uvadm apps 4096 Jun  5 09:03 jcl2
      drwxrwxr-x 2 uvadm apps 4096 Jun  5 09:18 jcl3
      drwxrwxr-x 2 uvadm apps 4096 Feb 12 18:08 parms
      drwxrwxr-x 2 uvadm apps 4096 Mar 22 16:45 pf
      drwxrwxr-x 2 uvadm apps 4096 Feb 12 17:33 rpts
      drwxrwxr-x 2 uvadm apps 4096 Jun  9 10:28 selects
      drwxrwxr-x 2 uvadm apps 4096 Feb 25 12:19 sf
      drwxrwxr-x 2 uvadm apps 4096 Jun  3 12:38 sqls
      drwxrwxr-x 2 uvadm apps 4096 Mar 28 11:42 stats
      drwxrwxr-x 2 uvadm apps 4096 Jun 16 09:04 tmp
      drwxrwxr-x 2 uvadm apps 4096 Mar 26 16:05 tmp1
      drwxrwxr-x 2 uvadm apps 4096 Jun  7 11:04 tmp2

llr lists files within directories

      demo/dat1:
      -rw-rw-r-- 1 uvadm apps   821 Jun 16 09:13 accents1
      -rw-rw-r-- 1 uvadm apps   298 Jun 16 09:17 accents2
      -rw-rw-r-- 1 uvadm apps 31920 Jun  5 09:30 camps
      -rw-r--r-- 1 uvadm apps 32503 Jun  5 09:30 camps.csv
      -rw-rw-r-- 1 uvadm apps  8192 Mar 25 17:13 custmas0
      -rw-rw-r-- 1 uvadm apps  8192 Mar 25 17:13 custmas1
      -rw-rw-r-- 1 uvadm apps   671 Jun 16 09:03 IPreplace.tbl
      -rw-rw-r-- 1 uvadm apps   598 Jun  5 09:31 nameadrs1
      -rw-rw-r-- 1 uvadm apps   598 Jun  5 09:31 nameadrs2
      -rw-rw-r-- 1 uvadm apps   450 Feb 22 09:58 namesadrs0e
      -rw-rw-r-- 1 uvadm apps   450 Feb 22 09:58 namesadrs1f
      -rw-rw-r-- 1 uvadm apps   427 Feb 22 09:58 namesadrs2t
      -rw-rw-r-- 1 uvadm apps  1280 Mar 25 17:13 sales2
      -rw-rw-r-- 1 uvadm apps  1280 Mar 25 17:13 sales3
      -rw-rw-r-- 1 uvadm apps   726 Mar 25 17:38 sales3.cpy
      -rw-rw-r-- 1 uvadm apps   576 Mar 25 17:13 sales4
      -rw-rw-r-- 1 uvadm apps   657 Mar 25 17:39 tabtest1
      -rw-rw-r-- 1 uvadm apps   256 Mar 25 17:25 trt256
      -rw-rw-r-- 1 uvadm apps   512 Mar 25 17:26 vendormas0
      -rw-rw-r-- 1 uvadm apps   249 Feb 18 12:29 vendormas3

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

      demo/jcl2:
      -rw-rw-r-- 1 uvadm apps  480 Feb 13 14:56 jar100.jcl
      -rw-rw-r-- 1 uvadm apps  984 Feb 13 14:56 jar200.jcl
      -rw-rw-r-- 1 uvadm apps  777 Feb 13 14:56 jgl100.jcl <-- JCL (before conversion)
      -rw-rw-r-- 1 uvadm apps 1238 Feb 13 14:56 jgl200.jcl
      -rw-rw-r-- 1 uvadm apps 1506 Jun  5 09:03 jgl230.jcl
      -rw-rw-r-- 1 uvadm apps 1591 Jun  5 09:03 jgl232.jcl
      -rw-rw-r-- 1 uvadm apps  932 Feb 13 14:56 jgl320.jcl
      demo/jcl3:
      -rwxrwxr-x 1 uvadm apps 1903 Feb 13 14:57 jar100.ksh
      -rwxrwxr-x 1 uvadm apps 3052 Feb 13 14:57 jar200.ksh
      -rwxrwxr-x 1 uvadm apps 2323 Feb 13 14:57 jgl100.ksh <-- JCL/scripts after conversion
      -rwxrwxr-x 1 uvadm apps 3497 Feb 13 14:57 jgl200.ksh
      -rwxrwxr-x 1 uvadm apps 4015 Feb 13 14:57 jgl230.ksh
      -rwxrwxr-x 1 uvadm apps 3280 Feb 13 14:57 jgl320.ksh
      demo/parms:
      -rw-rw-r-- 1 uvadm apps 138 Feb 12 18:08 ftpdemo1a
      -rw-rw-r-- 1 uvadm apps 135 Feb 12 18:08 ftpdemo1b   <-- parms (FTP scripts)
      -rw-rw-r-- 1 uvadm apps 134 Feb 12 18:08 ftpget1
      -rw-rw-r-- 1 uvadm apps 137 Feb 12 18:08 ftpput1
      demo/pf:                                             <-- uvcopy jobs (Parameter Files)
      -rw-rw-r-- 1 uvadm apps 1118 Mar 22 16:45 removeCR_uvcopy
      demo/selects:
      -rw-rw-r-- 1 uvadm apps 1107 Jun  7 11:06 dat1_nameadrs1_+1@0[20]Townsend:+2@60[20]Canada:-3@0[60]Owen
      -rw-rw-r-- 1 uvadm apps 1357 Jun  9 10:29 jcl2_SYSIN_1@COND:2@AND:3@OR%
      demo/sf:                                             <-- scripts (Script Files)
      -rwxrwxr-x 1 uvadm apps 566 Feb 25 12:23 fixaccents2
      -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:43 llsx
      -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:47 llsx.bak
      -rwxrwxr-x 1 uvadm apps 296 Feb 24 13:25 llsy
      demo/sqls:
      -rw-rw-r-- 1 uvadm apps  56 Feb 16 07:01 select_customers_all
      -rw-rw-r-- 1 uvadm apps  99 Jun  3 08:58 select_customers_basic
      -rw-rw-r-- 1 uvadm apps 150 Jun  3 12:38 select_customers_sales10000
      -rw-rw-r-- 1 uvadm apps 156 Jun  3 12:38 select_customers_sales20000
      -rw-rw-r-- 1 uvadm apps 136 Feb 16 07:20 select_employee_dept123     <-- sql scripts
      -rw-rw-r-- 1 uvadm apps 115 Feb 16 07:17 select_employee_seattle
      -rw-rw-r-- 1 uvadm apps  79 Feb 16 06:57 select_orders_big
      -rw-rw-r-- 1 uvadm apps 110 May 30 13:51 select_patients_over70
      -rw-rw-r-- 1 uvadm apps 109 May 30 13:51 select_patients_over80
      -rw-rw-r-- 1 uvadm apps 110 May 30 13:56 select_patients_over90
      -rw-rw-r-- 1 uvadm apps 129 Feb 16 07:13 select_students_failing
      demo/stats:
      -rw-rw-r-- 1 uvadm apps 1032 Feb 12 18:11 parms_open
      -rw-rw-r-- 1 uvadm apps 1057 Feb 12 18:12 parms_user
      demo/tmp:  <-- tmp dir for mass change output inspections before copyback
      demo/tmp1:   --- temp files created by tutorials ---
      -rw-rw-r-- 1 uvadm apps 1120 Feb 22 09:19 uvfix1.log

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D1. Demo files for Tutorials

demo input files - DATA files


 cat dat1/accents1
 =================
 # dat1/accents1 - French Accents
 11. cedilla      c ---> Ç ---> x'C387' ---> x'63' c
 21. accent acute e ---> é ---> x'C3A9' ---> x'65' e
 31. accent grave a ---> à ---> x'C3A0' ---> x'61' a
 32.              e ---> è ---> x'C3A8' ---> x'65' e
 33.              u ---> ù ---> x'C3B9' ---> x'75' u
 41. dieresis     e ---> ë ---> x'C3AB' ---> x'65' e
 42.              i ---> ï ---> x'C3AF' ---> x'69' i
 43.              u ---> ü ---> x'C3BC' ---> x'75' u
 51. circumflex   a ---> â ---> x'C3A2' ---> x'61' a
 52.              e ---> ê ---> x'C3AA' ---> x'65' e
 53.              i ---> î ---> x'C3AE' ---> x'69' i
 54.              o ---> ô ---> x'C3B4' ---> x'6F' o
 55.              u ---> û ---> x'C3BB' ---> x'75' u
 UTF-8 2 byte codes (1st byte x'C3')
 How can we convert to unaccented characters ?
 Thanks Thérèse

 cat dat1/accents2
 =================

# dat1/accents2 - French Accents (3 of 13) 1. accent grave a ---> à ---> x'C3A0' ---> x'61' a 2. accent acute e ---> é ---> x'C3A9' ---> x'65' e 3. accent grave e ---> è ---> x'C3A8' ---> x'65' e UTF-8 2 byte codes (1st byte x'C3') How can we convert to unaccented characters ? Thanks Thérèse


 cat dat1/nameadrs1    <-- test/demo files for SelectJobs
 ==================
 Owen Townsend       4667 Hoskins Rd     North Vancouver, BC Canada V7K2R3
 Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
 John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6
 John Owens          24 Owen Avenue      Owenville, ON       Canada M5K3B3
 Donald Trump        1600 Pennsylvania   Washingtom, DC      USA 00001
 Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399
 Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D2. Demo files for Tutorials


 cat dat1/namesadrs0e    <-- EBCDIC file
 ====================

ðððñðð@@@@Ö¦…•@㖠... <-- cannot show here, see later conversion to ASCII


 cat dat1/namesadrs2t    <-- ASCII file
 ====================
 000100    Owen Townsend       4667 Hoskins Rd     North Vancouver     BC V7K2R3 Canada
 000200    John Horgan         1234 Government St. Victoria            BC V1P2G3 Canada
 000300    Justin Trudeau      24 Sussex Drive     Ottawa              ON K1Y2L6 Canada
 000400    Donald Trump        1600 Pennsylvania   Washington          DC 00001 USA
 000500    Bill Gates          1 Microsoft Way     Redmond             WA 98052 USA

 cat parms/*      <-- show FTP demo input files
 ===========
      # parms/ftpdemo1a
      open 192.168.0.4
      user user01 user01pw
      lcd data1
      get /home/userxx/ar/sales.items
      get /home/userxx/gl/account.tran1
      quit
      # parms/ftpdemo1b
      open 192.168.0.5
      user user02 user02pw
      lcd data1
      put ar.sales.items   C:\AR\SALES.ITEMS
      put gl.account.tran1 D:\GL\ACCOUNT.TRAN1
      quit
      # parms/ftpget1
      open 192.168.0.6
      user user03 user03pw
      lcd data1
      binary
      cd  /home/userxx/gl
      get gl.acnt.master
      get gl.acnt.trans
      quit
      # parms/ftpput1
      open 192.168.0.6
      user user03 user03pw
      binary
      lcd data1
      cd /home/userxx/data1
      put gl.acnt.master
      put gl.acnt.trans
      quit

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D3. Demo files for Tutorials

demo input files - SQL scripts


 cat sqls/*      <-- show SQL demo input files
 ==========
      /* select_customers_all */
         SELECT * FROM Customers;
      /* select_customers_basic */
         SELECT Name, Phone, Address, City, Zip
                FROM Customers;
      /* select_customers_sales10000 */
         SELECT custno, Name, Phone, Address, City, Zip,
         TYsales, LYsales, FROM
         Customers WHERE LYsales >10000;
      /* select_customers_sales20000 */
         SELECT custno, Name, Phone, Address, City, Zip,
         TYsales, LYsales, FROM
         Customers
         WHERE LYsales >20000;
      /* select_employee_dept123 */
         SELECT EMP_ID, NAME, ADDRESS, CITY, ZIP, DEPT FROM
         EMPLOYEE WHERE DEPT = '123' AND STATUS = 'FULL';
      /* select_employee_seattle */
         SELECT EMP_ID, LAST_NAME FROM EMPLOYEE
         WHERE CITY = 'Seattle' ORDER BY EMP_ID;
      /* select_orders_big */
         SELECT item FROM Orders
         WHERE quantity > 500;
      /* select_patients_over70 */
         SELECT Name, Age FROM Patients WHERE Age > 70
         GROUP BY Age ORDER BY Name;
      /* select_patients_over80 */
         SELECT Name, Age FROM Patients WHERE Age > 80
         GROUP BY Age ORDER BY Name;
      /* select_patients_over90 */
         SELECT Name, Age FROM Patients
         WHERE Age > 90 GROUP BY Age ORDER BY Name;
      /* select_students_failing */
         CREATE VIEW Failing_Students AS
         SELECT S_NAME, Student_ID FROM
         STUDENT WHERE GPA < 30;

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D4. Demo files for Tutorials

demo input files - sf


 cat sf/llsx     <-- show BAD script for uvhd demo
 ===========
      #!/bin/bash
      # llsx - List files in directory, in Size order, Biggest 1st
      echo "llsx - list directory sorted by file-size (biggest files 1st)"
      if  [[ -z "$1" ]] ; then dir="."; else dir=$1; fi
      ls -lS $dir | more
      #=================
      exit 0
      #Note - 'llsx' is a BAD script, But 'vi' will not show the problem !

 cat sf/llsy     <-- show GOOD script for uvhd demo
 ==========
      #!/bin/bash
      # llsy - List files in directory, in Size order, Biggest 1st
      echo "llsy - list directory sorted by file-size (biggest files 1st)"
      if  [[ -z "$1" ]] ; then dir="."; else dir=$1; fi
      ls -lS $dir | more
      #=================
      exit 0
      #Note - 'llsy' is the GOOD script vs 'llsx' the BAD script
Note
  • You can not see any significant difference between BAD & GOOD scripts
    using the usual unix/linux tools (cat, more, vi, lp, etc)
  • You can see the problem using 'uvhd' (display hexadecimal) see '2B1'

demo input files - pf

 # removeCR_uvcopy - copy a file removing any CR's
 #                 - by Owen Townsend, UV Software, Feb 21/2019
 # - documented at http://uvsoftware.ca/uvdemos.htm
 # - demo Vancouver Utilities usefulness to any unix/linux user
 #
 # uvcopy removeCR_uvcopy    <-- execution command
 # ======================
 opr='$jobname - copy a file removing any Carriage Returns'
 fili1=?sf/llsx,rcs=256,typ=LST        #<-- declare input file
 filo1=?tmp/llsx,rcs=256,typ=LSTtp775  #<-- declare output file
 @run
         opn     all                    open files
 # begin loop to read/process/write records until EOF
 man20   get     fili1,a0               get next record into area 'a'
         skp>    man90                  (cc set > at EOF)
         mvc     b0(256),a0             copy input area 'a' to output area 'b'
         rep     b0(256),x'0D',''       remove CRs
         put     filo1,b0               write record to output file
         skp     man20                  return to get next record
 # EOF - close files & end job
 man90   cls     all
         eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D5. Demo files for Tutorials

demo input files - JCL


 cat jcl2/jar200.jcl   <-- sample JCL
 ===================
 //JAR200   JOB  (1234),'TEST/DEMO MVS JCL CONVERT'
 //* SORT AR.SALES.ITEMS BY PRODUCT CODE FOR LISTING
 //STEP010  EXEC PGM=SORT,REGION=2048K
 //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
 //SORTOUT  DD DSN=&&jar200_TEMPSLS,DISP=(NEW,PASS),
 //            DCB=(LRECL=64,BLKSIZE=6400,RECFM=FB),
 //            SPACE=(TRK,(50,50),RLSE),UNIT=DISK
 //SYSIN    DD *
 SORT FIELDS=(31,6,CH,A,1,6,CH,A)
 OMIT COND=((11,2,CH,EQ,C'65'),OR,(COND=(11,2,CH,EQ,C'85'))
 /*
 //*============================ step# 0020 ============================
 //* LIST AR.SALES.ITEMS IN PRODUCT CODE SEQUENCE
 //STEP020  EXEC PGM=CAR200,COND=(4,LT,STEP010),PARM=2006
 //SALEDTL  DD DSN=&&jar200_TEMPSLS,DISP=(OLD,PASS)
 //CUTMAS  DD DSN=AR.CUSTOMER.MASTER.INDEXED,DISP=SHR
 //SALELST  DD DSN=AR.SALES.LIST,DISP=(,CATLG,DELETE),
 //            UNIT=DISK,SPACE=(TRK,(25,25),RLSE),
 //            DCB=(MODEL.DSCB,LRECL=133,BLKSIZE=6118,RECFM=FBA)

 cat jcl2/jar200.jcl   <-- sample JCL for SelectJobs
 ===================     - find SORTs with Both AND & OR CONDitions
 //JGL232   JOB  'alt version of jgl230 to test selectlgfd1 AND 7 OR in same sort'
 //* SORT TRANS, UPDATE GLMASTER,& list GLMASTER
 //* SORT GL TRANSACTIONS BEFORE UPDATING ACCOUNT MASTER
 //*May21/2018 - correct INCLUDE column 55 to 56 to match data
 //STEP010  EXEC PGM=SORT
 //SORTIN   DD DSN=GL.ACCOUNT.TRAN1,DISP=OLD
 //SORTOUT  DD DSN=GL.ACCOUNT.TRANS(+1),DISP=(,CATLG,DELETE),
 //            DCB=(LRECL=80,BLKSIZE=8000,RECFM=FB),
 //            SPACE=(TRK,(50,50),RLSE),UNIT=DISK
 //SYSIN    DD *
 SORT FIELDS=(1,8,CH,A,69,12,CH,A)
 INCLUDE COND=(((9,1,CH,EQ,C'T'),OR,(COND=(9,1,CH,EQ,C'X'))),
                                 AND,(COND=(56,2,CH,EQ,C'AB')))
 /* (only 1st step of jgl232)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_2. UVdemos.doc Vancouver Utility Programming Aids

Part 2 - Basic Vancouver Utilities


2A1. Basic Vancouver Utilities are uvhd,uvsort,uvcp,& uvlist
(vs Advanced Utilities uvcopy & pre-programmed jobs based on uvcopy).
We will give brief descriptions of uvhd,uvsort,uvcp,& uvlist, followed
by tutorials illustrating how they might help you solve data problems.

2B1. uvhd demo - investigating problem files
Shell scripts with CRs will not execute
Detect with uvhd & remove with uvhd or uvcp or script using tr -d

2B5. Alternate solutions (other than uvhd) to remove CR's from text files
- using script 'removeCR' (see /home/uvadm/sf/util/removeCR)
2B6. - using 'uvcp' (Vancouver command line Utility)
2B7. - using 'uvcopy' (Vancouver data manipulation Utility)
2B9. - using 'uvfix1' pre-programmed job prompts for instructions
- saves writing a uvcopy job, uvfix1 fixes 1 file at a time
2B10. - using 'uvfixA' pre-programmed job prompts for instructions
- to fix all files in directory while copying to 2nd directory

2C1. uvhd demo - investigating problem files
 Files with high-bit characters may cause problems for older applications
Example - converting French accents (UTF-8) to unaccented equivalents
 Only a Brief Intro here, see the full presentation later in Part_10

2D1. uvhd demo - investigating unknown files
example - your Linux site receives an EBCDIC file from a mainframe site
- how to identify with uvhd & determine record-size
- mainframe files do not have LineFeeds to mark end of records

2E1. uvcp demo - translate EBCDIC records to ASCII
- and insert LineFeeds at same time.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A1. Basic Vancouver Utilities - Brief Descriptions

These brief descriptions of uvhd,uvcp,uvsort,& uvlist, will be followed by examples/tutorials illustrating how they might help you solve data problems.

uvhd

uvhd is a file investigation utility that displays any file in vertical hexadecimal, and prompts for commands to browse,search,replace,update,select,print,count,etc. uvhd allows you to see characters not shown by text editors (control characters & high-bit characters that can cause problems in text files). uvhd is interactive with 25 help screens, and is the favorite utility of all my customers. See uvhd documentation at http://www.uvsoftware.ca/uvhd.htm.

uvcp

'uvcp' is a command line data utility that can copy files, converting record formats between: fixed, variable, sequential, indexed, text, RDW, etc). uvcp provides record select, reformat, translate, etc. uvcp (& uvsort,uvcopy,etc) are compatible with Micro Focus COBOL & AIX COBOL fixed & variable length sequential, Indexed,& Relative file formats. See uvcp documentation at http://www.uvsoftware.ca/uvcp.htm.

uvsort

'uvsort' is a sort utility for Unix/Linux with mainframe capabilities. The Unix system sort will NOT sort mainframe type files that do not have Linefeeds, may be indexed, and may have packed decimal sort fields. uvsort has the functionality of mainframe SORT - record select, reformat, etc. See uvsort documentation at http://www.uvsoftware.ca/uvsort.htm.

uvlist

'uvlist' is a utility for listing text files on laser printers. uvlist scripts are provided to print files at various cpi,lpi,margins,etc. For example, 'uvlp13LD' will print mainframe reports (132 cols x 66 lines) on 8 1/2 by 11 at 13 cpi, 8 lpi, Landscape, Duplex with margins for 3 hole paper. uvlist is great for listing scripts, programs, etc, because the page headings tell you vital info such as: filename, date, userid, filesize, page#, etc. See uvlist documentation at http://www.uvsoftware.ca/uvlist.htm.

6. uvcopy

'uvcopy' is the most powerful of the Vancouver Utilities and will be discussed in later parts following these basic utilities.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B1. Basic Vancouver Utilities

uvhd demo - investigating problem files

Somebody emails you a handy script to list files in size order. But, when you try to execute it you get


 bash: sf/llsx: /bin/bash^M: bad interpreter
 ===========================================

We are supplying the bad script in demo/sf/llsx & good version in demo/sf/llsy Try the following tutorial to illustrate the problem with sf/llsx & correct it:


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/userxx/demo

 #1. l sf       <-- list the sf (Script Files) directory
     ====
     -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:43 llsx
     -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:47 llsx.bak
     -rwxrwxr-x 1 uvadm apps 296 Feb 24 13:25 llsy

 #2a. sf/llsx sf   <-- execute the BAD script (to list files in sf/...)
      ==========     - get following error:
     "bash: sf/llsx: /bin/bash^M: bad interpreter: No such file"

 #2b. sf/llsy sf   <-- execute the GOOD script to compare results:
      ==========
      llsy - list directory sorted by file-size (biggest files 1st)
      -rwxrwxr-x 1 uvadm apps 296 Feb 24 13:25 llsy
      -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:43 llsx
      -rwxrwxr-x 1 uvadm apps 244 Feb 23 17:47 llsx.bak

 #3a. cat sf/llsx   <-- display the script, to see what's wrong
      ===========       (or could use 'more', 'vi', etc)
      #!/bin/bash
      # llsx - List files in directory, in Size order, Biggest 1st
      echo "llsx - list directory sorted by file-size (biggest files 1st)"
      if  [[ -z "$1" ]] ; then dir="."; else dir=$1; fi
      ls -lS $dir | more
      #=================

We do not see anything wrong with BAD script (sf/llsx), so let's compare to the GOOD script (sf/llsy).


 #3b. cat sf/llsx   <-- display the script, to see what's wrong
      ===========       (or could use 'more', 'vi', etc)
      #!/bin/bash
      # llsy - List files in directory, in Size order, Biggest 1st
      echo "llsy - list directory sorted by file-size (biggest files 1st)"
      if  [[ -z "$1" ]] ; then dir="."; else dir=$1; fi
      ls -lS $dir | more
      #=================

We see NO DIFFERENCE between the BAD & GOOD scripts, using unix system utilities (cat, more, vi, lp, etc). We will try 'uvhd' below:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B2. using 'uvhd' to find problems in files

'uvhd' will display any file in "Vertical Hexadecimal". uvhd expects the filename as arg1 & filetype options as arg2. If you do not know the filetype options, you can first try uvhd without options & hopefully see what the options should be.


 #4. uvhd sf/llsx       <-- NOTE - arg2 filetype options omitted
     ============
     rec#=1 rsize=244 fptr=0 fsize=244 records=0
                  10        20        30        40        50        60
 r#    1 0123456789012345678901234567890123456789012345678901234567890123
       0 #!/bin/bash..# llsx - List files in directory, in Size order, Bi
         2226662667600226677222467726666726626676676772266256762676672246
         31F29EF2138DA30CC380D0C934069C5309E0492534F29C09E039A50F2452C029
      64 ggest 1st..echo "llsx - list directory sorted by file-size (bigg
         6667723770066662266772226677266766767727677662672666627676226666
         775340134DA538F02CC380D0C9340492534F2903F2454029069C5D39A5082977
     128 est files 1st)"..if  [[ -z "$1" ]] ; then dir="."; else dir=$1;
         6772666672377220066225522722232255232766626673222326676266732332
         534069C53013492DA9600BB0DA024120DD0B0485E0492D2E2B05C350492D41B0
     192 fi..ls -lS $dir | more..#=================..exit 0..
         6600672265226672726676002333333333333333330067672300
         69DAC30DC3044920C0DF25DA3DDDDDDDDDDDDDDDDDDA589400DA

When arg2 options are omitted, they default to: 'r256l64'. 'r256' is the default Record-size & 'l64' is default Line display size.

In this case (since the entire script file is less than 256, it displays as 1 block of 4 64 byte segments of 3 vertical hexadecimal line groups.

In the 3 line groups of Vertical Hexadecimal the ASCII characters are displayed on the 1st line, the zones on the 2nd line,& the digits on the 3rd line.

For example, the 1st byte of the file is a '#' & directly below you see the (horizontal) hexadecimal value is x'23' (zone 2 & digit 3). You can easily find an ASCII code chart (on the internet) to confirm this.

For a 2nd example, the 4th byte is 'b', which is x'62' in horizontal hex.

In the display above, the 7 records in the file are all run together in a block of 244 bytes, because we did not give uvhd any filetype options that would separate the records.

But we know that script files are TEXT files, so we need to specify the "text file option" to uvhd, which is simply 't'.

See our next attempt with option 't' on the page below:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B3. using 'uvhd' to find problems in files

Notes re uvhd (display above) with default options

Since we know script files are 'text' files, we will use the 't' option.


 #5. uvhd sf/llsx t  <-- rerun with option 't' separate lines on LineFeeds x'0A'
     ==============
     rec#=1 rsize=13 fptr=0 fsize=244 records=7
                      10
 r#        1 0123456789012
           0 #!/bin/bash..
             2226662667600
             31F29EF2138DA
     rec#=2 rsize=62 fptr=13 fsize=244 records=7
                      10        20        30        40        50        60
 r#        2 01234567890123456789012345678901234567890123456789012345678901
          13 # llsx - List files in directory, in Size order, Biggest 1st..
             22667722246772666672662667667677226625676267667224666677237700
             30CC380D0C934069C5309E0492534F29C09E039A50F2452C029775340134DA
                 --- showing 1st 2 lines, omitting last 5 lines ---

Can you see the problem using uvhd, that we could not see using cat,vi,etc ? We could display the good script demo/sf/llsy & compare:


 #6. uvhd sf/llsy t    <-- display good script & compare to bad
     ==============
     rec#=1 rsize=12 fptr=0 fsize=237 records=7
                      10
 r#        1 012345678901
           0 #!/bin/bash.
             222666266760
             31F29EF2138A
     rec#=2 rsize=61 fptr=12 fsize=237 records=7
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890
          12 # llsy - List files in directory, in Size order, Biggest 1st.
             2266772224677266667266266766767722662567626766722466667723770
             30CC390D0C934069C5309E0492534F29C09E039A50F2452C029775340134A
                 --- showing 1st 2 lines, omitting last 5 lines ---
 You can see the difference:
 - the bad script (sf/llsx) lines end with x'0D' & x'0A'.
 - the good script (sf/llsy) lines end with just x'0A'.
 - it is the x'0D' Carriage returns (CRs) that cause the problem
 - the shell was searching for the script interpreter /bin/bash+x'CR'
 - and gives the errmsg "bash: sf/llsx: /bin/bash^M: bad interpreter"

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B4. using 'uvhd' to find problems in scripts

Fixing the problem - Removing Carriage Returns (CRs)


 #7. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
     ======================

 #8. uvhd sf/llsx tu  <-- run with option 't' & 'u' to allow Update
     ===============
 r#    1 0123456789012
       0 #!/bin/bash..
         2226662667600
         31F29EF2138DA
 rec#=1 rsize=13 fptr=0 fsize=244 records=7
 Enter command, or ?=Help, q=quit, null=NextRecord -->

 #8a. --> v99 x'0D',x'20'  <-- search/replace x'0D' with x'20' (space)
          ===============
          v99 x'0D',' '    <-- could code space as ' ' vs x'20'

 #8b. --> 1  <-- enter 1 to return to 1st record in file (to compare)
          ===
 r#    1 012345678901
       0 #!/bin/bash.
         222666266760      <-- Note CR x'0D' now removed
         31F29EF2138A

 #8c. --> q       <-- reply 'q' to quit
          ===

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B5. using 'uvhd' to find problems in scripts

Notes re Updating with uvhd

  1. Must enter option 'u' on the command line if intent to update

  2. 'v' is command to search/replace, 99 of 'v99' is the number of records to process, but '99' has special meaning to process the entire file, resets file pointer to 1st record & processes all records until EOF.

  3. You could remove the CR by replacing with null as follows:

 #7a. --> v99 x'0D',''   <-- search/replace x'0D' with null ''
          ============
  1. Replacing with null will work in this case, but should normally be avoided. Normally, you should replace with an equal length string, because uvhd updates "in place", so the record rewritten will be shifted left at the replace point, inserting a blank at the end of the record, following the LineFeed x'0A', so we will get a blank ' ' at the begining of all records after 1st update, which will not be a problem for script execution, because scripts bypass blanks. But this would likely be a problem for most data files.

  2. See next page for an alternate solution using 'uvcp' which can remove the CR x'0D' & shift the LF x'0A' left 1 byte, shortening the records. (vs uvhd which always rewrites same size records). uvcp can do this because it copies the file (vs uvhd update in place).

  3. See all uvhd documentation at http://uvsoftware.ca/uvhd.htm, OR enter '?' at the uvhd prompt for 25 HELP screens.

  4. You can change the default options by adding export UVHDROP=... to your profile (or the common_profile), for example, I often use 'export UVHDROP=l100' for Line length 100 bytes.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B6. using 'uvhd' to find problems in scripts

Alternate solutions (vs uvhd) to Remove CRs from files

There are several alternate ways (vs uvhd shown above) to remove CR's from files: You may need solution #5 to fix all files in a directory.

  1. 'removeCR' script to copy file using --> tr -d '\r' <filein >fileout <-- - you can see this script at /home/uvadm/sf/util/removeCR

  2. 'uvcp' to copy the file using instruction --> rep=0(256):0x0D:: <--

  3. 'uvcopy', copies file same as uvcp, instruction --> rep 0(256),x'0D','' <--

  4. 'uvfix1 - pre-programmed uvcopy job prompts for instructions (rep above) - saves you from having to write the uvcopy file I/O framework - you can see this script at /home/uvadm/sf/util/uvfix1

  5. 'uvfixA - same as uvfix1, but copies All files from 1 directory to a 2nd directory applying the instructions entered at the prompt. - you can see this script at /home/uvadm/sf/util/uvfix1

    1. Remove CRs using script 'removeCR'

We will use 'removeCR' to copy the bad script from demo/sf/llsx to tmp/llsx while removing the CR's & then prompt to copy back overwriting original. You could inspect in tmp/llsx before copying back (but you can be confident).


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/userxx/demo

 #1a. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
      ======================

 #1b. removeCR sf/llsx    <-- call removeCR to remove CR's
      ================      - log listed below:
      removeCR  - copy a file to tmp/... removing CRs & nulls, prompt to copy back
      input  lines & bytes:           7 sf/llsx 244	sf/llsx
      output lines & bytes:           7 tmp/llsx 237	tmp/llsx
      copy back overwriting original y/n ? --> y <-- reply 'y'
      copied back to sf/llsx
      -rwxrwxr-x 1 uvadm apps 244 Feb 21 17:28 sf/llsx

 #1c. uvhd sf/llsx t      <-- check results (uvhd display not shown)
      ==============        - same as #8b on page '2B4'
Note

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B7. Alternate solutions (vs uvhd) to Remove CRs from files

2. Remove CRs using 'uvcp'

We will use 'uvcp' to copy the bad script from demo/sf/llsx to tmp/llsx, while removing CR's. You could inspect with 'uvhd tmp/llsx t' before copying back with 'cp'.


 #2a. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
      ======================

 #2b. uvcp "fili1=sf/llsx,rcs=256,typ=LSTt,filo1=tmp/llsx,rep=0(256),x'0D'::"
      =======================================================================

 #2c. uvhd tmp/llsx t      <-- inspect with uvhd, confirm CR's removed
      ===============
      r#    1 012345678901
            0 #!/bin/bash.
              222666266760      <-- Note CR x'0D' now removed
              31F29EF2138A

 #2d. cp tmp/llsx sf/      <-- copy back to sf/...
      ===============

 #2e. chmod 775 sf/llsx    <-- restore execute bits on script file
      =================

We need to restore the execute bits on all 5 solutions, except #3, since uvcopy allows file typ p775 to set output permissions.

See 'uvcp' documentation at http://www.uvsoftware.ca/uvcp.htm

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B8. Alternate solutions (vs uvhd) to Remove CRs from files

3. Remove CRs using 'uvcopy'

'uvcopy' is similar to 'uvcp', much more powerful, but does require a separate "Parameter File" (or "uvcopy job") vs uvcp which accepts all instructions on the command line. We will 1st edit a "uvcopy job" & then execute with uvcopy.


 #3a. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
      ======================

 #3b. vi pf/removeCR_uvcopy   <-- create uvcopy job to remove CR's
      =====================     - listed below:
      # removeCR_uvcopy - copy a file removing any CR's
      #                 - by Owen Townsend, UV Software, Feb 21/2019
      # - documented at http://uvsoftware.ca/uvdemos.htm
      # - demo Vancouver Utilities usefulness to any unix/linux user
      #
      # uvcopy pf/removeCR_uvcopy    <-- execution command
      # =========================
      opr='$jobname - copy a file removing any Carriage Returns'
      fili1=?sf/llsx,rcs=256,typ=LST        #<-- declare input file
      filo1=?tmp/llsx,rcs=256,typ=LSTtp775  #<-- declare output file
      @run
              opn     all                    open files
      # begin loop to read/process/write records until EOF
      man20   get     fili1,a0               get next record into area 'a'
              skp>    man90                  (cc set > at EOF)
              mvc     b0(256),a0             copy input area 'a' to output area 'b'
              rep     b0(256),x'0D',''       remove CRs
              put     filo1,b0               write record to output file
              skp     man20                  return to get next record
      # EOF - close files & end job
      man90   cls     all
              eoj

 #3c. uvcopy pf/removeCR_uvcopy     <-- execute uvcopy job
      =========================
      - no need to define files on command line, since coded in job as the defaults

      uvcopy pf/removeCR_uvcopy,fili1=sf/llsx,filo1=tmp/llsx
      ======================================================
      - but could define on command line when different than the defaults
Note
  • the '?'s on the file declarations (fili1=?sf/llsx,... & filo1=?tmp1/llsx).
    cause prompts for alternate filename keyin, null entry for default coded.

 #3d. uvhd tmp/llsx t    <-- inspect with uvhd, confirm CR's removed
      ===============      - display same as 2c on page '2B7'

 #3e. cp tmp/llsx sf/    <-- copy back to sf/...
      ===============

 #3f. chmod 775 sf/llsx  <-- restore execute bits <-- do NOT need
      =================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B9. Alternate solutions (vs uvhd) to Remove CRs from files

Notes re uvcopy jobs

  1. Here is the one vital instruction in the uvcopy job listed above.

         rep     b0(256),x'0D',''       remove CRs
    ==============================================
  1. The other instructions are the framework to define the files, and to read & write the records.

  2. Solution #4 will use script 'uvfix1' to supply the framework & the only instruction you have to enter is the 'rep' (as shown above).

  3. Note that we stored our uvcopy job in subdir 'pf' (Parameter Files) & the execution command was:

      uvcopy pf/removeCR_uvcopy     <-- execute uvcopy job
      =========================
  1. We had to code the subdir because /home/userxx/demo/pf/ is not defined in the common_profile as part of export PFPATH=...

  2. You do not need to code the directories on the many pre-programmed uvcopy jobs mentioned in this documentation, because they are in PFPATH. Here is the definition of PFPATH in $APPSADM/env/common_profile_uv:

    export PFPATH=$UV/pf/adm:$UV/pf/demo:$UV/pf/util:$UV/pf/IBM:$HOME/pf
    ====================================================================
  1. You can see that this is similar to export PATH=... used in profiles to define the directories for system & user programs.

  2. PFPATH includes $HOME/pf/, but not $HOME/demo/pf/, so that is why we needed to code pf/... on our execution --> uvcopy pf/removeCR_uvcopy

  3. Alternatively, we could add $HOME/demo/pf to PFPATH in the common_profile.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B10. Alternate solutions (vs uvhd) to Remove CRs from files

4. Remove CRs with 'uvfix1' pre-programmed uvcopy job

'uvfix1' is a pre-programmed uvcopy job that prompts for instructions & saves you from having to write the uvcopy file I/O framework.

'uvfix1' works the same way as above solutions, it will copy the bad script from sf/llsx to tmp/llsx (while applying the 'rep' instruction). You can then inspect the result in tmp/llsx & if satisfied, copy back from tmp/llsx to sf/llsx.


 #4a. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
      ======================

 #4b. uvfix1 sf/llsx    <-- call uvfix1, arg1 specifies the file to be modified
      ==============      - console log with prompts shown below:
      uvfix1 - copy a text file & modify via uvcopy instruction keyins
             - input rec in area a has already been copied to output rec in b
      uop=q1c0n99r8192 - user option defaults
            c1         - pass thru any "#" col1 #comment records
              n99      - output record count limit (high# copies to EOF)
                 r8192 - recsize ONLY for Fixed-Length OUTPUT records if desired
                       - for Text files: accept default recsize & default typ=LSTt
      User OPtion (uop) defaults  = q1c0n999999999r8192
 #4c. -->null to accept or enter/override -->  <-- enter null to accept default options
      typ=LSTt default, null accept or enter new typ -->   <-- null accept default typ
       mvc  b27(25),a0  <-- sample instrns, dup cols 1-25 to 27-51
                          - moves from op2 area a to op1 area b
      enter a uvcopy instruction (period "." terminates entries)

 #4d. ---> rep b0(256),x'0D',''   <--- ENTER rep instrn
           ====================
 #4e. ---> .                      <--- ENTER '.' to continue
      190222:092255:uvfix1: EOF fili01 rds=7 size=244: sf/llsx
      190222:092255:uvfix1: EOF filo01 wrts=7 size=237: tmp/llsx
      check output in tmp/llsx, then move back to sf/llsx ?

 #4f. uvhd tmp/llsx t        <-- check output in tmp/llsx, see if OK ?
      ===============
      r#    1 012345678901
            0 #!/bin/bash.
              222666266760   <-- Yes, OK CR x'0D' removed
              31F29EF2138A

 #4g. cp tmp/llsx sf/        <-- copy back overwriting bad script
      ================
Note

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B11. Alternate solutions (vs uvhd) to Remove CRs from files

5. Remove CRs with 'uvfixA' pre-programmed uvcopy job

'uvfixA' is similar to 'uvfix1', But it processes ALL files in a directory. You must specify arg1=InDir, arg2=OutDir,& arg3=filetype (LSTt in this case).

You could run the same fix as above (removing CRs from scripts) & it would not change anything in scripts that did not have CRs. We will use tmp1 as output directory.


 #5a. cp sf/llsx.bak sf/llsx  <-- restore bad file (in case fixed by prior run)
      ======================

 #5b. mkdir tmp1        <-- make output directory if not already present
      ==========
      rm -f tmp1/*      <-- OR, remove all files if tmp1 already present
      ============

 #5c. uvfixA sf tmp1 LSTt   <-- call uvfixA arg1/arg2 I/O directories & arg3 file type
      ===================     - console log with prompts shown below:
      uvfixA - copy/modify all text files from 1 directory to a 2nd directory
             - modifying the files via uvcopy instruction keyins
             - input rec in area a has already been copied to output rec in b
      uop=q1c0n99r8192 - user option defaults
            c1         - pass thru any "#" col1 #comment records
              n99      - output record count limit (high# copies to EOF)
                 r8192 - recsize ONLY for Fixed-Length OUTPUT records if desired
                       - for Text files: accept default recsize & default typ=LSTt
      User OPtion (uop) defaults  = q1c0n999999999r8192
 #5d. -->null accept or enter/override -->    <-- enter null (accept default options)
 #5e. did you create outdir or remove files ? --> y  <--ENTER "y"
          clr  b0(6)," "     <-- sample, clear 1-6 (use single quotesnot doubles)
          add  $ca1,1        <-- sample, increment counter #1
          mvn  b0(4),$ca1    <-- sample, insert sequence# in 1-4
      enter a uvcopy instruction (period "." terminates entries)

 #5f. ---> rep b0(256),x'0D',''   <--- ENTER rep instrn
           ====================
 #5g. ---> .                      <--- ENTER '.' to continue
      190222:101902:uvfixA: EOF fili01 rds=7 size=244: sf/llsx
      190222:101902:uvfixA: EOF filo02 wrts=7 size=237: tmp1/llsx
      190222:101902:uvfixA: EOF fili01 rds=7 size=244: sf/llsy
      190222:101902:uvfixA: EOF filo02 wrts=7 size=237: tmp1/llsy
      190222:101902:uvfixA: EOF fild02 size=4096: tmp1

 #5h. uvhd tmp1/llsx t   <-- check 1 or more in tmp1/llsx, see if OK ?
      ================

 #5i. cp tmp1/* sf/      <-- copy All scripts back overwriting originals
      =============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2C1. uvhd demo - investigating problem files

high-bit characters may cause problems for old applications

Files with high-bit characters may cause problems for older applications. For Example - files with UTF-8 characters input to legacy applications, not designed to handle them. This demo will show you how to identify them with uvhd & convert them to unaccented characters with uvhd or uvcp.

Brief Intro here, More demos later

The original documentation of High-Bit Accented character conversions was moved to '10A1' in Part 10, because it was too long & complex for presentation early in the UVdemos documentation. Here is a quick demo of using 'uvhd' to see the hex coding of some French accents, see all French accents later in Part_10.

dat1/accents3 - UTF-8 accents, short test file


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/userxx/demo

 #1. cat dat1/accents3      <-- display short test file with French accents
     =================
      déjà vu Thérèse

 #2. uvhd dat1/accents3 t   <-- use uvhd to see hex chars of French accents
     ====================
 rec#=1 rsize=20 fptr=0 fsize=20 records=1
                      10        2
 r#        1 01234567890123456789
           0 d..j.. vu Th..r..se.
             6CA6CA277256CA7CA760
             439A300650483923835A

uvhd shows the French accents are coded in 2 bytes (UTF-8 coding). Here are hte hex codes using horizontal hex vs uvhd vertical hex:

      accent grave a ---> à ---> x'C3A0' (vs unaccented a x'61')
      accent acute e ---> é ---> x'C3A9' (vs unaccented e x'65')
      accent grave e ---> è ---> x'C3A8' (vs unaccented e x'65')

If you have a printed copy of this documentation, you will probably see that the UTF-8 accented chars print differently than the screen display, probably because the printer is not setup for UTF-8 characters.

See the full documentation converting French Accents to unaccented equivalents later in Part_10.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D1. Basic Vancouver Utilities

uvhd demo - investigating unknown files

'uvhd' should be the 1st tool you think of when you receive a file of unknown format. Our 1st example assumes that your organization has both mainframes & unix/linux. Someone sends you a mainframe file in EBCDIC & you attempt to investigate with vi. We have provided such a file in demo/dat1/namesadrs0e. Try the following:


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1. vi dat1/namesadrs0e
     ===================
 ðððñðð@@@@Ö¦…•@㖦•¢…------ etc, 'vi' display all garbage ----->

Now let's try uvhd (still not knowing the file is in EBCDIC): 'uvhd' expects you to specify file-type options following the filename. But we do not yet know the file-type & the default is to assume ASCII with record-size 256.


 #2. uvhd dat1/namesadrs0e
     =====================
                  10        20        30        40        50        60
 r#    1 0123456789012345678901234567890123456789012345678901234567890123
       0 ......@@@@....@........@@@@@@@....@.......@..@@@@@.....@........
         FFFFFF4444DA894E9A9A8984444444FFFF4C9A989A4D844444D99A84E8989AA8
         0001000000665503665255400000004667086229520940000056938051536455

uvhd displays in 'vertical hexadecimal' 3 line groups with characters on 1st line, hex zones on the 2nd line, & hex digits on the 3rd line. Here we are showing only the 1st 64 bytes (vs actual 256 bytes in 4 groups of 3 lines of 64 bytes each).

The display is still garbage, but we can see that it is an EBCDIC file. The 1st byte is x'F0' (here in horizontal-hex vs vertical-hex above) & x'F0' should be recognized by most professionals as an EBCDIC zero. And bytes 6-9 are x'40's which are EBCDIC blanks.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D2. uvhd demo - investigating unknown files

The next step should be use option 'a' which tells uvhd to translate the character line to ASCII (but uvhd will still display EBCDIC codes for zone & digit lines).


 #3. uvhd dat1/namesadrs0e a    <-- option 'a' translates EBCDIC to ASCII
     =======================
                  10        20        30        40        50        60
 r#    1 0123456789012345678901234567890123456789012345678901234567890123
       0 000100    Owen Townsend       4667 Hoskins Rd     North Vancouve
         FFFFFF4444DA894E9A9A8984444444FFFF4C9A989A4D844444D99A84E8989AA8
         0001000000665503665255400000004667086229520940000056938051536455
      64 r     BC V5P3V8 Canada    000200    John Horgan         1234 Gov
         944444CC4EFDFEF4C898884444FFFFFF444499894C99889444444444FFFF4C9A
         9000002305573580315141000000020000001685086971500000000012340765
     128 ernment St. Victoria            BC V1P2G3 Canada    000300    Ju
         899989A4EA44E88A9988444444444444CC4EFDFCF4C898884444FFFFFF4444DA
         5954553023B05933699100000000000023051727303151410000000300000014
     192 stin Trudeau      24 Sussex Drive     Ottawa              ON K1Y
         AA894E9A888A444444FF4EAAA8A4C98A844444DAA8A844444444444444DD4DFE
         2395039445140000002402422570499550000063316100000000000000650218

Now you should be able to see that the record-size is 90 bytes because data repeats at 90 bytes - 1st record id# 000100 & 2nd record id# 00020 is 90 bytes higher (64 on 1st line + 26 on 2nd line) (scales are zero-relative).

Notes Re: option 'a' translate EBCDIC to ASCII

  1. uvhd option 'a' translates the character line of the 3 line vertical hexadecimal groups from EBCDIC to ASCII, so you can read it before actual data translated.

  2. The actual data is unchanged, the zones & digits lines are still EBCDIC. Example#1 1st byte is '0' & below you see the zone/digit as x'F0' (in horizontal hex vs vertical hex in display above). Example#2 'O' of Owen Townsend is x'D6' in EBCDIC (would be x'4F' in ASCII).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D3. uvhd demo - investigating unknown files

So now we can give uvhd the correct record-size as option 'r90' and can also specify line display size as option 'l90'.


 #4. uvhd dat1/namesadrs0e ar90l90
     =============================
 rec#=1 rsize=90 fptr=0 fsize=450 records=5
                 10        20        30        40        50        60        70        80        9
 r#   1 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
      0 000100    Owen Townsend       4667 Hoskins Rd     North Vancouver     BC V5P3V8 Canada
        FFFFFF4444DA894E9A9A8984444444FFFF4C9A989A4D844444D99A84E8989AA8944444CC4EFDFEF4C898884444
        000100000066550366525540000000466708622952094000005693805153645590000023055735803151410000
 rec#=2 rsize=90 fptr=90 fsize=450 records=5
                 10        20        30        40        50        60        70        80        9
 r#   2 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
     90 000200    John Horgan         1234 Government St. Victoria            BC V1P2G3 Canada
        FFFFFF4444D9894C99889444444444FFFF4C9A899989A4EA44E88A9988444444444444CC4EFDFCF4C898884444
        000200000016850869715000000000123407655954553023B05933699100000000000023051727303151410000
 rec#=3 rsize=90 fptr=180 fsize=450 records=5
                 10        20        30        40        50        60        70        80        9
 r#   3 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
    180 000300    Justin Trudeau      24 Sussex Drive     Ottawa              ON K1Y2L6 Canada
        FFFFFF4444DAAA894E9A888A444444FF4EAAA8A4C98A844444DAA8A844444444444444DD4DFEFDF4C898884444
        000300000014239503944514000000240242257049955000006331610000000000000065021823603151410000
 rec#=4 rsize=90 fptr=270 fsize=450 records=5
                 10        20        30        40        50        60        70        80        9
 r#   4 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
    270 000400    Donald Trump        1600 Pennsylvannia  Washington          DC 00001 USA
        FFFFFF4444C998984E9A9944444444FFFF4D899AA9A8998844E8A8898A994444444444CC4FFFFF4EEC44444444
        000400000046513403944700000000160007555283515591006128957365000000000043000001042100000000
 rec#=5 rsize=90 fptr=360 fsize=450 records=5
                 10        20        30        40        50        60        70        80        9
 r#   5 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
    360 000500    Bill Gates          1 Microsoft Way     Redmond             WA 98052 USA
        FFFFFF4444C8994C8A8A4444444444F4D8899A98A4E8A44444D8899984444444444444EC4FFFFF4EEC44444444
        000500000029330713520000000000104939626630618000009544654000000000000061098052042100000000

Now that we know the record size of our file, we can process with 'uvcp' (below).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2E1. Basic Vancouver Utilities

uvcp demo - translate to ASCII (& insert LineFeeds)


 #5. uvcp "fili1=dat1/namesadrs0e,typ=RSF,rcs=90,filo1=tmp1/namesadrs1f,tra=0(90)"
     =============================================================================
      - copies dat1/namesadrs0e to tmp1/namesadrs1f, translating EBCDIC to ASCII
Note1
Note2
  • We will not show the output from only translating to ASCII because
  • It makes more sense to combine translation with inserting LineFeeds

uvcp - translate to ASCII & insert LineFeeds at same time

Most linux users would like to work with text files (records ended with LineFeeds) because just about all unix utilities & editors expect text records with LineFeeds.


 #6. uvcp "fili1=dat1/namesadrs0e,typ=RSF,rcs=90,filo1=tmp1/namesadrs2t,typ=LSTt"
     ============================================================================

 uvhd /home/uvadm/dat1/namesadrs2t t
 ===================================
 rec#=1 rsize=87 fptr=0 fsize=427 records=5
                  10        20        30        40        50        60        70        80
 r#    1 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
       0 000100    Owen Townsend       4667 Hoskins Rd     North Vancouver     BC V7K2R3 Canada.
         333333222247662567676662222222333324676667256222224677625666677672222244253535324666660
         0001000000F75E04F7E35E40000000466708F3B9E302400000EF248061E3F565200000230650368031E141A
 rec#=2 rsize=87 fptr=87 fsize=427 records=5
                  10        20        30        40        50        60        70        80
 r#    2 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
      87 000200    John Horgan         1234 Government St. Victoria            BC V1P2G3 Canada.
         333333222266662467666222222222333324676766667257225667676622222222222244253534324666660
         0002000000AF8E08F271E000000000123407F652ED5E4034E06934F291000000000000230610273031E141A
 rec#=3 rsize=87 fptr=174 fsize=427 records=5
                  10        20        30        40        50        60        70        80
 r#    3 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
     174 000300    Justin Trudeau      24 Sussex Drive     Ottawa              ON K1Y2L6 Canada.
         333333222247776625776667222222332577767247676222224776762222222222222244243534324666660
         0003000000A5349E0425451500000024035335804296500000F4417100000000000000FE0B192C6031E141A
 rec#=4 rsize=83 fptr=261 fsize=427 records=5
                  10        20        30        40        50        60        70        80
 r#    4 01234567890123456789012345678901234567890123456789012345678901234567890123456789012
     261 000400    Donald Trump        1600 Pennsylvannia  Washington          DC 00001 USA.
         33333322224666662577672222222233332566677676666622567666676622222222224423333325540
         00040000004FE1C40425D0000000001600005EE39C61EE910071389E74FE0000000000430000010531A
 rec#=5 rsize=83 fptr=344 fsize=427 records=5
                  10        20        30        40        50        60        70        80
 r#    5 01234567890123456789012345678901234567890123456789012345678901234567890123456789012
     344 000500    Bill Gates          1 Microsoft Way     Redmond             WA 98052 USA.
         33333322224666246767222222222232466767667256722222566666622222222222225423333325540
         000500000029CC071453000000000010D932F3F64071900000254DFE40000000000000710980520531A

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_3. Vancouver Utility Programming Aids - SelectJobs

UVdemos Part 3 SelectJobs - Contents


3A1. Select Jobs - Introduction & Practical Examples

3B1. Vancouver Utilities - subdirs relevant to SelectJobs

3B2. Copy $UV/demo/* to demo/ subdir in your homedir
- so you can run SELECT job demos from your demo/... subdir

3C1. selectlf1 - select lines from 1 file identified by up to 9 patterns
matching and/or not-matching datafile contents
- First Example to serve as basis for pattern definition Rules

3D1. Test/Demo selectlf1 to illustrate Pattern Rules (common to all select jobs)
- 10 sample patterns for use with testfile dat1/nameadrs1
  (testdata only 8 records so you can easily check selections correct or not)
- prefix patterns with '+' for presence (default) & '-' for absence
- may specify multiple patterns for 1 selection via sequence#
  example: arg1=+1@Townsend:+2@Canada,-3@Owen
- may restrict search area: arg1=@0[20]Townsend:@20[60]Owen
- output reports created in ./selects/... subdir
  named by concatenating input directory & patterns

3D3. testselectlf1 script to run above 10 test/demos automatically (vs keyins)

3D4. Special Character substitutions
- use '^' circumflex for space & '%' percent for commas
- required when patterns coded on command line arg1=...
- not required if you enter patterns at the prompt
 Reports automatically written to selects/ subdir
- named by concatenating input filename & patterns

3E1. Test/Demo selectlf1 using dat1/camp Canadian Members of Parliament (336 records)
- higher volume demo file (vs 8 record dat1/nameadrs1)
- a few suggested patterns, then make up your own patterns

3F1. selectlfd1 - select lines from all files in a directory
- same selection rules as for selectlf1

3G1. selectlgfd1 - select line Groups from all Files in a Directory
- same selection rules as for selectlf1
- begin group patterns specified via arg1=...
- end group patterns specified via arg2=...
- optional group qualify patterns specified via arg3=...
(selects all lines of group if arg3 absent)

3G2. selectlgfd2 - same as selectlgfd1, but arg3=... searches for patterns
on any line in the group (vs on the same line)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 3 SelectJobs - Contents (continued)


3H1. selectlgfd1 - Example#2 select JCL SYSINs with AND & OR conditions
- We need to inspect SORT INCLUDE/OMIT CONDitions with both "AND"s & "OR"s
  which may need manual corrections.

3H2. selectlgfd2 - Example#2 select JCL SYSINs with AND & OR conditions (BETTER)
- selectlgfd2 can select ONLY the SYSINs with BOTH ANDs & ORs
  vs selectlgfd1 which could only select SYSINs with EITHER AND or OR

3I1. selectfdd1 - select Files from 1 directory to a 2nd directory
- based on multiple patterns on the same line or any line
  Note difference 'selectfdd1' (vs prior selectlf1,selectlfd1,selectlgfd1/2)
- selectfdd1 selects the entire file to an output directory
  vs prior select jobs that created reports in the selects/... directory
Note
  • This document SelectJobs.htm is the same as Part 3 of UVdemos.htm,
  • separate document created as UVdemos.doc was being developed
  • considered to be 1 of the most useful parts of UVdemos.doc
Note
  • Part 3 of UVdemos.htm also available as a separate document 'SelectJobs'
  • separate document created as UVdemos.doc was being developed
  • considered to be 1 of the most useful parts of 'UVdemos'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A1. UVdemos Vancouver Utility Programming Aids

SelectJobs - Introduction & Overview

Select Jobs (selectlf1,selectlfd1,selectlgfd1,etc)are pre-programmed uvcopy jobs to select lines from 1 or all files in a directory, lines identified by up to 9 patterns, matching and/or not-matching datafile contents.

You may prefix patterns with '+' for presence & '-' for absence. You may specify multiple patterns for 1 selection via a sequence#, for example:


 arg1=+1@Townsend:+2@Canada,-3@Owen   <-- multi conditions +present & -absent
 ==================================

You may restrict the search area by displacement[length], for example:


 arg1=+1@0[20]Townsend:+2@20[60]Owen  <-- Townsend in cols 1-20 & Owen in 21-80
 ===================================

This document provides demo files & several suggested search patterns. Please run the demos & check your results to the expected.

We think you will appreciate that these select jobs are much more powerful than 'grep' because they can search for multiple conditional patterns with 1 pass. After the demos & understanding how the patterns & conditions are specified, please try out these select jobs on your files.

You might have a directory of thousands of scripts & you need to select some scripts (for investigation & correction) based on some combination of patterns which may be on the same line or on any line in the script.

Use 'selectlfd1' if the multiple patterns must be on the same line, or use 'selectlgfd1' if the multiple patterns could be anywhere in the file, or if the patterns must be between 2 other lines - start search & stop search lines which can also be identified by up to 9 patterns

For example, we can use selectlgfd1 to search SQL 'SELECT' statements which can be multi-lines, but known to end with a ';' semi-colon. arg1=SELECT,arg2=; & arg3=desired search patterns between arg1 & arg2 (begin & end).

If you have a problem with your files & are wondering how it could be solved with Vancouver Utilities, we invite you to email us, explain the problem, and we will help you solve it (as our time allows).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A2. SelectJobs - Introduction & Overview

Example to Illustrate SelectJob Capabilities vs grep

JCL conversions of complex SORTs with INCLUDE/OMIT AND/OR conditions may need manual corrections & need to be inspected. I wanted to scan hundreds of JCLs extracting only the SORT SYSIN parameters with both "AND" & "OR" CONDitions.

'grep' can not do this because the ANDs & ORs could be on different lines. Here is the command for 'selectlgfd2' which can look for //SYSIN to /* groups with both "AND" & "OR" in between.


 uvcopy selectlgfd2,fild1=jcl2,arg1=@//SYSIN,arg2=/*,arg3=1@COND:2@AND:3@OR%
 ============================================================================
  - scan for //SYSIN to /* line groups with "AND", "OR", "CONDitions"
  - creates report in subdir selects/... named from directory + patterns as shown below:
 # Report: selects/jcl2_SYSIN_1@COND:2@AND:3@OR%
 # ---> uvcopy selectlgfd2,fild1=jcl2,arg1=//SYSIN,arg2=/*,arg3=1@COND:2@AND:3@OR%
 #      =========================================================================
 # format: uvcopy selectlgfd2,fild1=directory,arg1=...,arg2=...,arg3=...,arg4=...,uop=abcdgi
 # arg1 patterns identify begin-group line, arg2 patterns identify end-group line
 # arg3 patterns qualify group selection, arg4 patterns qualify lines to output/print
 #  - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 #  - may prefix patterns: '-' for absence, '+' for presence (default)
 #  - may restrict search area: arg2=@0[2]/*  <-- "/*" must be in 1st 2 bytes
 #  - multiple-condition-series: arg3=1@SORT:2@COND:3@AND:4@OR <-- ALL patterns must match
 # Date=2019/06/09_10:29:01, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 //SYSIN    DD *
 SORT FIELDS=(1,8,CH,A,69,12,CH,A)
 INCLUDE COND=(((9,1,CH,EQ,C'T'),OR,(COND=(9,1,CH,EQ,C'X'))),
                                 AND,(COND=(56,2,CH,EQ,C'AB')))
 /*
                          EOG: #1, 5 lines selected from jcl2/jgl232
                          EOF:     5 selected from 35 in jcl2/jgl232
 EOD: 7 files, 168 lines, selected 1 groups, 5 lines to selects/jcl2_SYSIN_1@COND:2@AND:3@OR%

SelectJob Advantages

  1. Can select lines based on up to 9 patterns that must be present or absent.

  2. Can select groups of lines (identified by patterns on a start-line & an end-line), that include multiple patterns on any line between the start & end lines.

  3. SelectJobs create reports in subdir selects/... automatically named from the file or directory concatenated with patterns used. This allows you to review your past searches & see the command used for easy modify & repeat.

  4. You may restrict pattern searches to record areas, for example "arg1=1@0[20]Owen,2@40[40]Vancouver" searches for "Owen" anywhere in 1st 20 bytes and "Vancouver" anywhere in columns 41-80. For the JCL/SORT example we could have "arg1=@0[7]//SYSIN,arg2=@0[2]/*" since we know these patterns start in column 1.

  5. SelectJobs can use '>' & '<' conditions, but only if the field is in a fixed position. See examples later in this documentation on page '3E3'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B1. UVdemos Vancouver Utility Programming Aids

Vancouver Utilities - subdirs relevant to SelectJobs

Here is the Vancouver Utilities homedir showing only the most relevant subdirs required for the SelectJob demos documented in this section of UVdemos.htm.

You could run the demos logged in as 'uvadm', but if you are sharing the system with other uvadm users, it would be better to copy the demo/subdirs/files to your own homedir. See the copy instructions below & an expanded view of the demo/... files on the next page.

 /home/uvadm             <-- $UV
 :-----bin                 - UV binaries (uvcopy,uvsort,uvcp,uvhd,uvlist,etc)
 :-----dat1                - test data files to demo various Vancouver Utilities
 :     :-----...         <-- 200 testdata files for all uvcopy job demos
 :     :                   - SelectJobs testdata files copied to demo/dat1/...
 :-----doc                 - Vancouver Utilities documentation (text)
 :     :-----UVdemos.doc   - this documentation (1 of 150 files)
 :-----dochtml             - documentation in HTML (uploaded to www.uvsoftware.ca)
 :-***-demo              <-- demo files for TestDemo tutorials, copy to your homedir -->
 :     :                   - copy /home/uvadm/demo/* to your /home/userxx/demo/...
 :     :-----dat1          - data files for SelectJob demos, see details on next page
 :     :-----jcl2          - mainframe JCL samples (8)
 :     :-----parms         - SYSIN control cards for various purposes
 :     :-----sqls          - SQL scripts for SelectJob demos, see next page -->
 :-----pf                <-- Parameter Files for uvcopy
 :     :-----util              - uvcopy SelectJobs
 :     :     :-----selectlf1     - select Lines from 1 File
 :     :     :-----selectlfd1    - select Lines from all Files in a directory
 :     :     :-----selectlgfd1   - select Line Groups from all Files in directory
 :     :     :-----selectlgfd2   - select Line Groups from all Files in directory
 :     :     :-----selectfdd1    - select Files from 1 Directory to a 2nd Directory
 :     :-----adm               - uvcopy jobs for uvadm admin
 :     :-----demo              - uvcopy jobs for various demos
 :     :-----IBM               - uvcopy jobs for Mainframe conversions
 :-----sf                <-- Script Files (bash or Korn shell)
 :     :-----adm               - scripts for uvadm admin
 :     :-----demo              - demo scripts
 :     :-----util              - utility scripts
 :     :-----IBM               - scripts for mainframe conversions
 :-----src               <-- Vancouver Utilities C source code (uvcopy,uvsort,etc)
 :-----tf                  - test files for various demos (similar to dat1/...)
 :-----tmp                 - tmp subdir (test/demo outputs)
 :-----tmp1                - misc, outdir for some utility scripts (like listall1)
 :-----tmp2

Please copy the demo/... files to your homedir to avoid conflict with other users of Vancouver Utilities & to avoid losing your files when new versions of Vancouver Utilities are installed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B2. UVdemos Vancouver Utility Programming Aids

copy $UV/demo to your $HOME/demo/...


 #1. Login userxx --> /home/userxx

 #2. mkdir demo   <-- make directory for copy of demo files in your homedir

 #3. cp -r /home/uvadm/demo/* demo/   <-- copy demo files to your homedir
     ==============================

 #4. cd demo    <-- change into demo/ directory
     =======

SelectJobs demo files copied to your homedir

Here is an expanded view of the most relevant subdirs/files required for the SelectJob demos (omitting subdirs/files used for other parts of UVdemos.doc).

 /home/userxx/demo/              - copy to your homedir, cd demo/ & run jobs there
 :-----dat1                    <-- testdata files for demos
 :     :-----nameadrs1           - 8 line testdatafile to demo pattern rules
 :     :-----CAMPs.csv           - CAnadian Members of Parliament Names & constituencies
 :     :-----camps               - text files in fixed fields extracted from .csv file
 :     :
 :-----jcl2                    <-- JCLs to demo selectlgfd1/2
 :     :-----jar200.jcl
 :     :-----...etc...
 :-----pf                      <-- uvcopy job Parameter Files, see note below
 :     :-----                    - you could create your own uvcopy jobs here
 :     :
 :-----selects                 <-- output directory for select job reports
 :     :-----dat1_nameadrs1_+1@Townsend:+2@Canada:-3@Owen
 :     :-----...                 - sample above from selectlf1 1st demo
 :     :-----...                 - concatenation of directory_file_patterns
 :-----sqls                    <-- SQL scripts for SelectJob demos
 :     :-----select_Customers_all
 :     :-----select_Customers_sales
 :     :-----...etc...
 :-----tmp
 :-----tmp1                    <-- tmp dirs for temp files, sorts, etc
 :-----tmp2

uvcopy job Locations


 uvcopy selectlf1,fili1=dat1/nameadrs1,arg1=+1@Townsend:-2@Owen
 ==============================================================
 - sample uvcopy execution of 'selectlf1' Parameter/Instruction File

How does uvcopy know where to find its Instruction files (Parameter Files) ?

uvcopy looks for the Parameter/Instruction File via $PFPATH, similar to scripts using $PATH. The SELECT jobs are stored in /home/uvadm/pf/util/... (as shown on the previous page) & the common_profile includes:


 export PFPATH=$UV/pf/demo:$UV/pf/IBM:$UV/pf/util:$HOME/pf:---etc---
 ====================================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C1. UVdemos Programming Aids - SelectJobs

dat1/nameadrs1 - testdata file for 1st demo

          1         2         3         4         5         6         7         8
 12345678901234567890123456789012345678901234567890123456789012345678901234567890
 Owen Townsend       4667 Hoskins Rd     North Vancouver, BC Canada V7K2R3
 Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
 John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6
 John Owens          24 Townsend Dr.     Owenville, ON       Canada M5K3B3
 Donald Trump        1600 Pennsylvannia  Washingtom, DC      USA 00001
 Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399
 Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ
Note
  • "Owen" & "Townsend" are repeated in various name & address fields
    so we can practise selecting various combinations from various fields

selectlf1 - 1st Example for Pattern Rules following


 #0b. Login uvadm  --> /home/uvadm  <-- could use uvadm if you are only user
 #0a. Login userxx --> /home/userxx <-- better/safer to copy files to your homedir
 #1.  cd demo            <-- change into demo/ directory
 #2a. rm -f selects/*    <-- remove old selection reports
 #2b. rm -f tmp*/*       <-- remove old temporary files (from tmp/, tmp1/, tmp2/)

 #2. uvcopy selectlf1,fili1=dat1/nameadrs1,arg1=+1@Townsend:+2@Canada:-3@Owen
     ========================================================================
     - select lines from dat1/nameadrs1 with "Townsend" & "Canada", but NOT "Owen"

 #2a. cat    <-- End of Job prompt for command to view output file
      ===      - enter command only (cat,more,vi,etc), no need for filename
               - Result (2 lines) as follows:
 Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
Note
  • Can you confirm that the 2 lines reported (Jenny Townsend & John Horgan)
    are the only 2 of the 8 input lines that satisfy the selection patterns ???
  • See the dat1/nameadrs1 testdatafile listed at the top of this page.
Note
  • output reports are created in the ./selects/ subdir of current directory
  • named by concatenating input directory + filename + arg1 patterns
  • any characters invalid in filenames such as '/' converted to '_' underscores
  • You may view separately at any time with vi,cat,more,lp,uvlp12,etc

 #3a. vi selects/dat1_nameadrs1_+1@Townsend:+2@Canada:-3@Owen  <-- using 'vi'
      =======================================================

 #3b. cat selects/dat1_nameadrs1_+1@Townsend:+2@Canada:-3@Owen  <-- using 'cat'
      ========================================================
      - 'cat' is convenient when you know the result is less than a screenfull

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C2. UVdemos Programming Aids - SelectJobs

Output Report from selectlf1

The prior page showed only the 2 lines selected from the 8 line input file.

Actually, selectlf1 creates a Report with command parameters used, which are useful when you need to review the results or rerun the selection with different parameters. Here is the full report, with the selected lines & stats at the end

selects/dat1_nameadrs1_+1@Townsend:+2@Canada:-3@Owen

 # Report: selects/dat1_nameadrs1_+1@Townsend:+2@Canada:-3@Owen
 # ---> uvcopy selectlf1,fili1=dat1/nameadrs1,arg1=+1@Townsend:+2@Canada:-3@Owen
 #      =========================================================================
 #format: uvcopy selectlf1,fili1=inputfile,arg1=pattern1:pattern2,etc,uop=i0/i1
 # arg1 - line selection conditions (up to 9 patterns to match or not-match)
 # - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 # - may prefix patterns with '-' for absence, '+' for presence (default)
 # - may restrict search area, example: arg1=@0[20]Townsend:@20[60]Owen
 # - multiple conditions: arg1=+1@Townsend:+2@Canada,-1@Owen,-2@BC
 # - if no seq#s (+1,+2,etc) default selects if any +pattern & no -pattern
 # - combination ex: arg1=+1@0[20]Townsend:+2@60[60]Canada:-@0[20]Owen
 # Date=2019/05/12_16:14:14, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
 # EOF - 2 lines selected from dat1/nameadrs1

Report Format for SelectJobs

  1. 1st line identifies the report filename (in selects/... subdir by default)

  2. 2nd line shows the uvcopy SelectJob command line arguments

  3. Lines 3-11 are SELECT job format rules & help info

  4. Line 12 shows current date, sitename, hostname, username,& options

  5. Lines 14+ are the selected data lines (only 2 lines in this example)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D1. Select Jobs - selectlf1, selectlfd1, selectlgfd1

Test/Demos to illustrate Pattern Rules

The Pattern Rules are common to all Select Jobs (selectlf1,selectlfd1,selectlgfd1), but we will use selectlf1 for most examples below. You may run the commands below & confirm your results match our documented selection counts.

Patterns may be entered on the command line via arg1=... OR at the prompt, They are best entered on the command line for easy rerun from command history.

dat1/nameadrs1 - compare demo outputs to these 8 records

          1         2         3         4         5         6         7         8
 12345678901234567890123456789012345678901234567890123456789012345678901234567890
 Owen Townsend       4667 Hoskins Rd     North Vancouver, BC Canada V7K2R3
 Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
 John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6
 John Owens          24 Townsend Dr.     Owenville, ON       Canada M5K3B3
 Donald Trump        1600 Pennsylvannia  Washingtom, DC      USA 00001
 Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399
 Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ

 #1. uvcopy selectlf1,fili1=dat1/nameadrs1,arg1=Townsend:Owen
     ========================================================
     - select lines with "Townsend" OR "Owen" anywhere on the line
      --> enter patterns if not on command line arg1=...
      selects/dat1_nameadrs1_Townsend:Owen default filo02 -->     <-- null accept
      default command null, OR enter: vi,cat,more,etc     --> cat <-- enter cat
           ---------------- 6 of 8 lines are selected --------------
     Owen Townsend       4667 Hoskins Rd     North Vancouver, BC Canada V7K2R3
     Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
     John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
     John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6
     John Owens          24 Townsend Dr.     Owenville, ON       Canada M5K3B3
     Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ
Note
  • #1 above specifies the input file (fili1=dat1/nameadrs1)
  • But, we will omit on following examples, since it is the default
  • Here is the revised command, the prompts,& the recommend replies:
  • null to accept the I/O files & 'cat' to display the selected records

 #1. uvcopy selectlf1,arg1=Townsend:Owen  <-- can omit fili1=dat1/nameadrs1 (default)
     ===================================
     - select lines with "Townsend" OR "Owen" anywhere on the line
      dat1/nameadrs1 = default fili01                     -->     <-- null accept
      --> enter patterns if not on command line arg1=...
      selects/dat1_nameadrs1_Townsend:Owen default filo02 -->     <-- null accept
      default command null, OR enter: vi,cat,more,etc     --> cat <-- enter cat
Note
  • We will not show these prompts & replies on the following examples
  • But please enter the 2 nulls & 'cat' when prompted

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D2. Select Jobs - Test/Demos to illustrate Pattern Rules


 #2. uvcopy selectlf1,arg1=+Townsend:+Canada:-Owen
     =============================================
     - select lines with "Townsend" OR "Canada", but NOT "Owen"   <-- ERROR, INVALID

You may code 1st byte of pattern as '+' (default) to indicate the pattern must be present and '-' to indicate the pattern must be absent (NOT found). But, this example is INVALID, cannot mix '+' & '-' conditions, unless the '-' conditions are at the end of a conditional series, for example: +1...,+2...,-3...,-4...


 #3. uvcopy selectlf1,arg1=+1@Townsend:+2@Canada:-3@Owen <-- CONDITION SERIES
     ===================================================
     - select lines with "Townsend" AND "Canada", but NOT "Owen"
           ---------------- 2 of 8 lines are selected --------------
     Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
     John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3

The '+1' indicates the First condition & '+2' the Second condition of a series (9 max). Match assumed if any '+' pattern found, then negated if any '-' pattern found. You must code '+' patterns prior to any '-' patterns in a series 1/2/3. You could omit the '+' condition prefixes since that is the default


 #4. arg1=+@0[60]Townsend:+@60[20]Canada:-@0[20]Owen  <-- RESTRICTED SEARCH AREAS
     ===============================================
     - match if "Townsend" in 0-59 OR "Canada" in 60-79, But not if "Owen" in 0-19.
           ---------------- 3 of 8 lines are selected --------------
     Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
     John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
     John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6

You may specify "Restricted Search Areas" by coding the Displacement[Length] following the '@' identifier.


 #5. arg1=+1@Townsend:+2@Canada:-3@Owen:+1@Townsend:+2@UK:-3@Owen <-- MULTI-SERIES 1,2,3,1,2,3
     ===========================================================
     - match if "Townsend" AND "Canada" & NOT "Owen"
     - AND   if "Townsend" AND "UK" & NOT "Owen"
           ---------------- 2 of 8 lines are selected --------------
     Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
     John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
     Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ

 #6. uvcopy selectlf1,arg1=-Townsend           <-- NOT present
     ===============================
     - select lines WITHOUT "Townsend" (anywhere in record)
           ---------------- 3 of 8 lines are selected --------------
     John Owens          24 Townsend Dr.     Owenville, ON       Canada M5K3B3
     Donald Trump        1600 Pennsylvannia  Washingtom, DC      USA 00001
     Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399

 #7. uvcopy selectlf1,arg1=-1@Townsend:-2@Owen  <-- SERIES NOT present
     =========================================
     - select lines WITHOUT "Townsend" & without "Owen"
           ---------------- 3 of 8 lines are selected --------------
     Donald Trump        1600 Pennsylvannia  Washingtom, DC      USA 00001
     Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D3. Select Jobs - Test/Demos to illustrate Pattern Rules

script to run 10 selectlf1 test/demos

 # testselectlf1 - script stored at $UV/sf/adm/testselectlf1
 #               --> to test uvcopy $UV/pf/util/selectlf1
 #               - by Owen Townsend, UV Software, May13/2019
 # - run selectlf1 with various selection patterns, reports collectedin ./selects/...
 #              --------- testdatafile -----------
 # Owen Townsend       4667 Hoskins Rd     North Vancouver, BC Canada V7K2R3
 # Jenny Townsend      21 Canada Place     Vancouver, BC       Canada V5P3V8
 # John Horgan         123 Townsend Ave.   Townsend City, BC   Canada V1P2G3
 # John Townsend       2121 Owen Drive.    St. John, NB        Canada M5W1E6
 # John Owens          24 Owen Avenue      Owenville, ON       Canada M5K3B3
 # Donald Trump        1600 Pennsylvannia  Washingtom, DC      USA 00001
 # Bill Gates          1 Microsoft Way     Seattle, WA         USA 98052-6399
 # Peter Townsend      24 Johnson Road     London, England     UK EC1A 1HQ
 #
 export UVCOPYROP=q0i7r0  # inhibit prompts
 uvcopy selectlf1,arg1=Townsend:Owen                  #1 - any match
 uvcopy selectlf1,arg1=+1@Townsend:+2@Canada:-3@Owen  #2 - multi-condition match
 uvcopy selectlf1,arg1=+1@0[20]Townsend:+2@60[20]Canada:-3@0[60]Owen #3(1) - restricted areas
 uvcopy selectlf1,arg1=+1@0[20]Townsend:+2@60[20]Canada:-3@0[60]Owen\
 :+1@0[20]Townsend:+2@60[20]UK:-3@0[60]Owen           #5 - multi-condition-series
 uvcopy selectlf1,arg1=-Townsend                      #6 - NOT condition
 uvcopy selectlf1,arg1=-1@Townsend:-2@Owen            #7 - NOT multi-conditions
 uvcopy selectlf1,arg1=@0[4]>Owen                     #8 - > or < condition
 # following will be rejected with errmsgs
 uvcopy selectlf1,arg1=+Townsend:+Canada:-Owen        #1 ERR: mixed +/-, - not end series
 uvcopy selectlf1,arg1=-Townsend:+Owen                #2 ERR: mixed #1 '-' & any '+'
 uvcopy selectlf1,arg1=-Townsend:-Owen                #3 ERR: multi #1 '-'s
 uvcopy selectlf1,arg1=-1@Townsend:+2@Owen            #4 ERR: #1 '-' with any '+'

executing sf/adm/testselectlf1


 #0b. Login uvadm  --> /home/uvadm  <-- could use uvadm if you are only user
 #0a. Login userxx --> /home/userxx <-- better/safer to copy files to your homedir

 #1. rm -f selects/*    <-- remove old selection reports

 #2. testselectlf1      <-- execute script (no arguments required)
     =============        - will get multiple prompts for input file accept/change
      dat1/nameadrs1 = default fili01                     -->     <-- null accept
      --> enter patterns if not on command line arg1=...
      selects/dat1_nameadrs1_Townsend:Owen default filo02 -->     <-- null accept
      default command null, OR enter: vi,cat,more,etc     --> cat <-- enter cat

For each of the 10 runs in the script, you can reply null to accept the default filenames and then reply "cat" to see the output records selected.

OR, you could reply null to all prompts & then inspect output reports in selects/...


 #3. vi selects/*       <-- view all 10 output reports
     ============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D4. Select Jobs - Test/Demos to illustrate Pattern Rules

Special Characters in arg1 patterns

  1. You can not code ' ' spaces in the arg1 patterns (for testsst2 & select jobs below), because these jobs convert spaces to tildes when storing patterns in the table, & because the uvcopy command line does not allow embedded spaces. BUT, you may code ' ' spaces as '^' circumflexes, which will be converted to ' ' spaces.

  2. You can not code ',' commas in arg1 patterns, because the uvcopy command line uses ',' commas to separate arguments. BUT, you may code commas as '%' percents, which will be converted to ',' commas.

    reports written to selects/... subdir

  3. By default, the SELECT jobs create their output reports in ./selects/... (subdir in the current directory).

  4. The output report filename defaults to a concatenation of the input directory, and the arg1/2/3 patterns, with '/'s converted to '_' underscores. Note in the above example, the report is 'selects/jcl2_SYSIN_1@cond:2@and:3@or'.

  5. If desired, you could specify an alternate output report filename, but the default is very convenient & creates unique filenames identifying the directory searched & the search patterns used.

  6. Note the 14 line report prefix which includes the Report: filename, the uvcopy command that created it, several lines of "help" information, and the current date=..., Site=..., Host=..., User=...,& Options=...

    Try selectlf1 with YOUR PATTERNS

Please try out selectlf1 with your own patterns, You might vary the suggested patterns listed on the previous page in the 'testselectlf1' script. See if you can find a pattern that does not give the expected results.


 uvcopy selectlf1,fili1=dat1/nameadrs1,arg1=...YOURPATTERNS...
 =============================================================

Try selectlf1 with YOUR DATA files & PATTERNS

Then try out selectlf1 with your own files & patterns.


 #1. cd your working directory (above your directory of files)

 #2. mkdir selects   <-- make directory for output reports
     =============

 #3. uvcopy selectlf1,fili1=yourdir/YOURDATAFILE,arg1=...YOURPATTERNS...
     ===================================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3E1. Select Jobs - *selectlf1, selectlfd1, selectlgfd1

dat1/camps - larger test file to demo Select Jobs

$UV/dat1/camps is the CAnadian Members of Parliament data file that we can use for tutorials & you can use for self training using these Select Jobs.

Here are the samples of the 336 records with a column scale in case you wish to test select job searches with restricted areas.

          1         2         3         4         5         6         7         8         9       100
 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
 001  Ziad           Aboultaif           Edmonton Manning              AB Conservative     2015
 002  Dan            Albas               Central Okanagan/Similkameen/ BC Conservative     2015
 003  Harold         Albrecht            Kitchener/Conestoga           ON Conservative     2015
 004  John           Aldag               Cloverdale/Langley City       BC Liberal          2015
 005  Omar           Alghabra            Mississauga Centre            ON Liberal          2015
               --------------- samples of all parties --------------
 011  Charlie        Angus               Timmins/James Bay             ON NDP              2015
 023  Xavier         Barsalou-Duval      Pierre-Boucher/Les Patriotes/ QC Bloc Québécois   2015
 032  Maxime         Bernier             Beauce                        QC People's Party   2015
 202  Elizabeth      May                 Saanich/Gulf Islands          BC Green Party      2015
 244  Jane           Philpott            Markham/Stouffville           ON Independent      2015
 329  Jody           Wilson-Raybould     Vancouver Granville           BC Independent      2015
               ----------------- last 3 of 336 MP's ------------------
 334  David          Yurdiga             Fort McMurray/Cold Lake       AB Conservative     2015
 335  Salma          Zahid               Scarborough Centre            ON Liberal          2015
 336  Bob            Zimmer              Prince George/Peace River/Nor BC Conservative     2015

sample selection from CAnadian Members of Parliament


 uvcopy selectlf1,fili1=dat1/camps,arg1=1@AB:2@liberal
 =====================================================
  - select "Liberal" party members from "AB" (Alberta)
Note
  • report created in selects/... named by concatenating directory + patterns
 # Report: selects/dat1_camps_1@AB:2@Liberal
 # ---> uvcopy selectlf1,fili1=dat1/camps,arg1=1@AB:2@Liberal
 #      =========================================================================
 #format: uvcopy selectlf1,fili1=inputfile,arg1=pattern1:pattern2,etc,uop=i0/i1
 # arg1 - line selection conditions (up to 9 patterns to match or not-match)
 # - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 # - may prefix patterns with '-' for absence, '+' for presence (default)
 # - may restrict search area, example: arg1=@0[20]Townsend:@20[60]Owen
 # - multiple conditions: arg1=+1@Townsend:+2@Canada,-1@Owen,-2@BC
 # - if no seq#s (+1,+2,etc) default selects if any +pattern & no -pattern
 # - combination ex: arg1=+1@0[20]Townsend:+2@60[60]Canada:-@0[20]Owen
 # Date=2019/05/30_11:30:46, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 042  Randy          Boissonnault        Edmonton Centre               AB Liberal          2015
 140  Kent           Hehr                Calgary Centre                AB Liberal          2015
 294  Amarjeet       Sohi                Edmonton Mill Woods           AB Liberal          2015
 # EOF - 3 lines selected from dat1/camps

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3E2. Select Jobs - *selectlf1, selectlfd1, selectlgfd1

suggested patterns for searching dat1/camps

Here are some suggested search patterns you might run on the CAnadian Members of Parliament file (for tutorials & self training on Select Jobs).

Here are 1st 3 of the 336 records with a column scale in case you wish to test select job searches with restricted areas. Note scale is 1 relative, but you must code the zero relative values for restricted area searches.

          1         2         3         4         5         6         7         8         9       100
 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
 001  Ziad           Aboultaif           Edmonton Manning              AB Conservative     2015
 002  Dan            Albas               Central Okanagan/Similkameen/ BC Conservative     2015
 003  Harold         Albrecht            Kitchener/Conestoga           ON Conservative     2015

search CA MP file for Liberals in Vancouver


 uvcopy selectlf1,fili1=dat1/camps,arg1=1@70[2]BC:2@Liberal:3@Vancouver
 ======================================================================
 - select MP's from "BC" who are "Liberal"s from "Vancouver" --> result 5 records
 117  Hedy           Fry                 Vancouver Centre              BC Liberal          2015
 128  Pamela         Goldsmith-Jones     West Vancouver/Sunshine Coast BC Liberal          2015
 223  Joyce          Murray              Vancouver Quadra              BC Liberal          2015
 272  Harjit S.      Sajjan              Vancouver South               BC Liberal          2015
 328  Jonathan       Wilkinson           North Vancouver               BC Liberal          2015

Other searches showing only number of matches expected


 uvcopy selectlf1,fili1=dat1/camps,arg1=AB
 =========================================
 - select all 'MP's from Alberta --> result = 34 records
 - see if you get the same result ?

 uvcopy selectlf1,fili1=dat1/camps,arg1=@70[2]AB
 ===============================================
 - restrict search for "AB" to columns 71-72 (in case AB occurring elsewhere)
 - we got the same number of records (34), so "AB" did not occur elsewhere
 - Next, let's try case insensitive to see if "ab" does occur elsewhere

 uvcopy selectlf1,fili1=dat1/camps,arg1=ab
 =========================================
 - search for "ab" vs "AB", will not work, since default is case sensitive
 --> gets 15 unrelated records with "ab" as parts of names

 uvcopy selectlf1,fili1=dat1/camps,arg1=ab,uop=s1  - case insensitive option 's1'
 ================================================  - "ab" same as "AB"
 - option 's1' (case insensitive) gets --> 52 records (18 more)

 uvcopy selectlf1,fili1=dat1/camps,arg1=...
 ==========================================
 - continue testing with your own patterns

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3E3. Select Jobs - *selectlf1, selectlfd1, selectlgfd1

searching with > & < conditions

Searching for > & < can only be used on fields with fixed-positions. We will demo using dat1/camps which has the year elected in columns 91-94 (zero relative 90-93). Here are 1st 3 of 336 records with a column scale.

          1         2         3         4         5         6         7         8         9       100
 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
 001  Ziad           Aboultaif           Edmonton Manning              AB Conservative     2015
 002  Dan            Albas               Central Okanagan/Similkameen/ BC Conservative     2015
 003  Harold         Albrecht            Kitchener/Conestoga           ON Conservative     2015

 #1. uvcopy "selectlf1,fili1=dat1/camps,arg1=1@Conservative:2@90[4]>2015"
     ====================================================================
     - search for Conservatives elected since 2015
     - must enclose command in quotes when '<' or '>' used

 #2. cat selects/dat1_camps_1@Conservative:2@90[4]_2015
     ==================================================
     - display output report (or could have replied just 'cat' at the prompt)
 # Report: selects/dat1_camps_1@Conservative:2@90[4]_2015
 # ---> uvcopy selectlf1,fili1=dat1/camps,arg1=1@Conservative:2@90[4]>2015
 #      =========================================================================
 #format: uvcopy selectlf1,fili1=inputfile,arg1=pattern1:pattern2,etc,uop=i0/i1
 # arg1 - line selection conditions (up to 9 patterns to match or not-match)
 # - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 # - may prefix patterns with '-' for absence, '+' for presence (default)
 # - may restrict search area, example: arg1=@0[20]Townsend:@20[60]Owen
 # - multiple conditions: arg1=+1@Townsend:+2@Canada,-1@Owen,-2@BC
 # - if no seq#s (+1,+2,etc) default selects if any +pattern & no -pattern
 # - combination ex: arg1=+1@0[20]Townsend:+2@60[60]Canada:-@0[20]Owen
 # Date=2019/06/09_09:35:02, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 022  Michael        Barrett             Leeds/Grenville/Thousand Isla ON Conservative     2018
 030  Bob            Benzen              Calgary Heritage              AB Conservative     2017
 074  Scot           Davidson            York/Simcoe                   ON Conservative     2019
 102  Rosemarie      Falk                Battlefords/Lloydminster      SK Conservative     2017
 164  Stephanie      Kusie               Calgary Midnapore             AB Conservative     2017
 182  Dane           Lloyd               Sturgeon River/Parkland       AB Conservative     2017
 197  Richard        Martel              Chicoutimi/Le Fjord           QC Conservative     2018
 222  Glen           Motz                Medicine Hat/Cardston/Warner  AB Conservative     2016
 # EOF - 8 lines selected from dat1/camps

Try some other commands such as:


 uvcopy "selectlf1,fili1=dat1/camps,arg1=1@Green:2@90[4]>2015:1@NDP:2@90[4]>2015"
 ================================================================================
 195  Paul           Manly               Nanaimo/Ladysmith             BC Green Party      2019
 293  Jagmeet        Singh               Burnaby South                 BC NDP              2019

 uvcopy "selectlf1,fili1=dat1/camps,arg1=1@90[4]>2015:2@90[4]<2020:3@Green"
 ==========================================================================
 195  Paul           Manly               Nanaimo/Ladysmith             BC Green Party      2019

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3F1. SelectJobs: *selectlf1* ,selectlfd1,selectlgfd1,selectlgfd2,selectfdd1

selectlfd1 - SELECT from all files in a directory

'selectlfd1' will select lines from all files in a directory, vs 'selectlf1' which selects lines for only 1 file at a time. We will demo using a directory of SQL scripts $UV/demo/sqls/... (copied to /home/userxx/demo/sqls/...)

4 SQL "customer" scripts in sqls/...

/* select_customers_all */ SELECT * FROM Customers;

 /* select_customers_basic */
     SELECT Name, Phone, Address, City, Zip
            FROM Customers;

/* select_customers_sales10000 */ SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >10000;

/* select_customers_sales20000 */ SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >20000;

select "customer" references from all scripts in directory


 uvcopy selectlfd1,fild1=sqls,arg1=1@SELECT:2@FROM:3@Customers,uop=s1
 ====================================================================
 - select lines with "SELECT","FROM",& "Customers" from all files in sqls/*
 - best to use option 's1' case insensitive
 # Report: selects/sqls_1@SELECT:2@FROM:3@Customers
 # ---> uvcopy selectlfd1,fild1=sqls,arg1=1@SELECT:2@FROM:3@Customers,uop=s1
 #      ====================================================================
 #format: uvcopy selectlf1,fili1=inputfile,arg1=pattern1:pattern2,etc,uop=i0/i1
 # Date=2019/06/03_17:06:11, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=s1
 #=================================================================================
     SELECT * FROM Customers;
                               EOF - 1 lines selected from sqls/select_customers_all
 # EOD - 1 lines selected from 11 files in directory sqls

PROBLEM Notes

  1. File#1 was the only file with all 3 words on the same line (as required by selectlfd1). It would be better to select the Multi-Line SQL statement vs 1 matching line.

  2. See alternative jobs 'selectlgfd1' on the next page to select line groups from arg1="SELECT" until arg2=";" ending SQL statements qualified by arg3 match to 1 or multiple words on 1 line. Or see 'selectlgfd2' which can qualify by matching multiple words on any line between arg1 & arg3.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3G1. SelectJobs: selectlf1,selectlfd1, *selectlgfd1*, selectlgfd2,selectfdd1

selectlgfd1 - Multi-Condition-Series on any ONE line

'selectlgfd1' will select line groups from arg1=... until arg2=... qualified by arg3 matches to 1 or multiple words on 1 line. Here is an example where arg1=SELECT, & arg2=; (begin & end of SQL SELECT statement), qualified by "Customers" & "LYsales", which must appear on the same line for selectlgfd1 (vs selectlgfd2 any line).


 uvcopy "selectlgfd1,fild1=sqls,arg1=SELECT,arg2=;,arg3=1@Customers:2@LYsales,uop=s1"
 ====================================================================================
 # # Report: selects/sqls_SELECT_1@Customers:2@LYsales
 # # ---> uvcopy selectlgfd1,fild1=sqls,arg1=SELECT,arg2=;,arg3=1@Customers:2@LYsales
 # #      ===========================================================================
 # # format: uvcopy selectlgfd1,fild1=directory,arg1=...,arg2=...,arg3=...,arg4=...,uop=abcdgi
 # # arg1 patterns identify begin-group line, arg2 patterns identify end-group line
 # # arg3 patterns qualify group selection, arg4 patterns qualify lines to output/print
 # #  - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 # #  - may prefix patterns: '-' for absence, '+' for presence (default)
 # #  - may restrict search area: arg2=@0[2]/*  <-- "/*" must be in 1st 2 bytes
 # #  - multiple-condition-series: arg3=1@SORT:2@COND:3@AND:4@OR <-- ALL patterns must match
 # # Date=2019/06/03_12:40:18, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 # #===============================================================================
 #     SELECT custno, Name, Phone, Address, City, Zip,
 #     TYsales, LYsales, FROM
 #     Customers WHERE LYsales >10000;
 #                          EOG: #1, 3 lines selected from sqls/select_customers_sales10000
 #                          EOF:     3 selected from 4 in sqls/select_customers_sales10000
 # EOD: 11 files, 36 lines, selected 1 groups, 3 lines to selects/sqls_SELECT_1@Customers:2@LYsales

Notes Re: *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectlfd1

  1. This 'selectlgfd1' shows the complete SQL statement (from "SELECT" to ending ";") (vs 'selectlf1' & 'selectlfd1' which showed only the lines with the search argument)

  2. 'selectlgf1' qualifies the group selection by arg3 patterns, but multiple conditional series patterns (1,2,3,etc) must appear on the same line.

  3. For the 'selectlgfd1' above, we get only 1 of 4 statements (SELECT FROM Customers) because only file#3 had both "Customers" & "LYsales" on the same line.

  4. See next page example 'selectlgfd2', which can qualify the group selection by arg3 conditional series patterns (1,2,3,etc) using multiple words that can appear on any line between the arg1/arg2 begin/end patterns.

  5. Note that we need to enclose the uvcopy "selectlgfd1...arg2=;,..." command in quotes when the arguments include characters such as ';'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3G2. SelectJobs: selectlf1,selectlfd1,selectlgfd1, *selectlgfd2*, selectfdd1

selectlgfd2 - Multi-Condition-Series on MULTI lines (vs 1 line)

'selectlgf2' will select line groups from arg1=... until arg2=... qualified by arg3 matches to 1 or multiple words on ANY line in the Group. Here is an example where arg1=SELECT, & arg2=; (begin & end of SQL SELECT statement), qualified by "Customers" & "LYsales", which may appear on ANY line (vs selectlgfd1 one line).


 uvcopy "selectlgfd2,fild1=sqls,arg1=SELECT,arg2=;,arg3=1@Customers:2@LYsales,uop=s1"
 ====================================================================================
 # # Report: selects/sqls_SELECT_1@Customers:2@LYsales
 # # ---> uvcopy selectlgfd2,fild1=sqls,arg1=SELECT,arg2=;,arg3=1@Customers:2@LYsales,uop=s1
 # #      ==================================================================================
 # # format: uvcopy selectlgfd2,fild1=directory,arg1=...,arg2=...,arg3=...,arg4=...,uop=abcdgi
 # # arg1 patterns identify begin-group line, arg2 patterns identify end-group line
 # # arg3 patterns qualify group selection, arg4 patterns qualify lines to output/print
 # #  - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 # #  - may prefix patterns: '-' for absence, '+' for presence (default)
 # #  - may restrict search area: arg2=@0[2]/*  <-- "/*" must be in 1st 2 bytes
 # #  - multiple-condition-series: arg3=1@SORT:2@COND:3@AND:4@OR <-- ALL patterns must match
 # # Date=2019/06/03_12:43:54, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 # #===============================================================================
 #     SELECT custno, Name, Phone, Address, City, Zip,
 #     TYsales, LYsales, FROM
 #     Customers WHERE LYsales >10000;
 #                          EOG: #1, 3 lines selected from sqls/select_customers_sales10000
 #                          EOF:     3 selected from 4 in sqls/select_customers_sales10000
 #     SELECT custno, Name, Phone, Address, City, Zip,
 #     TYsales, LYsales, FROM
 #     Customers
 #     WHERE LYsales >20000;
 #                          EOG: #1, 4 lines selected from sqls/select_customers_sales20000
 #                          EOF:     4 selected from 5 in sqls/select_customers_sales20000
 # EOD: 11 files, 36 lines, selected 2 groups, 7 lines to selects/sqls_SELECT_1@Customers:2@LYsales

Notes Re: *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectlfd1

  1. Both selectlgfd1 & *selectlgfd2 shows complete SQL statements ("SELECT" to ending ";") (vs 'selectlf1' & 'selectlfd1' which showed only the lines with the search argument).

  2. 'selectlgf1' qualifies the group selection by arg3 patterns, but multiple conditional series patterns (1,2,3,etc) must appear on the same line. For 'selectlgfd1', we got only 1 of 4 statements because only file#3 had both "Customers" & "LYsales" on the same line.

  3. This 'selectlgfd2' can qualify the group selection by arg3 conditional series patterns using multiple words that can appear on any line between the arg1/arg2 begin/end patterns.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3H1. SelectJobs: selectlf1,selectlfd1, *selectlgfd1*, selectlgfd2,selectfdd1

selectlgfd1 - select JCL SYSINs with AND & OR conditions

We need to inspect SORT INCLUDE/OMIT CONDitions with both "AND"s & "OR"s which may need manual corrections. Here is the command line, followed by the report.


 uvcopy selectlgfd1,fild1=jcl2,arg1=//SYSIN,arg2=/*,arg3=1@COND:2@AND:1@COND:2@OR
 ================================================================================
 # Report: selects/jcl2_SYSIN_1@COND:2@AND:1@COND:2@OR%
 # ---> uvcopy selectlgfd1,fild1=jcl2,arg1=//SYSIN,arg2=/*,arg3=1@COND:2@AND:1@COND:2@OR%
 #      =========================================================================
 # format: uvcopy selectlgfd1,fild1=directory,arg1=...,arg2=...,arg3=...,arg4=...,uop=abcdgi
 # arg1 patterns identify begin-group line, arg2 patterns identify end-group line
 # arg3 patterns qualify group selection, arg4 patterns qualify lines to output/print
 #  - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 #  - may prefix patterns: '-' for absence, '+' for presence (default)
 #  - may restrict search area: arg2=@0[2]/*  <-- "/*" must be in 1st 2 bytes
 #  - multiple-condition-series: arg3=1@SORT:2@COND:3@AND:4@OR <-- ALL patterns must match
 # Date=2019/06/09_10:26:58, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 //SYSIN    DD *
 SORT FIELDS=(31,6,CH,A,1,6,CH,A)
 OMIT COND=((11,2,CH,EQ,C'65'),OR,(COND=(11,2,CH,EQ,C'85'))
 /*
                          EOG: #1, 4 lines selected from jcl2/jar200
                          EOF:     4 selected from 23 in jcl2/jar200
 //SYSIN    DD *
 SORT FIELDS=(1,8,CH,A,69,12,CH,A)
 INCLUDE COND=((9,1,CH,EQ,C'T'),AND,(COND=(56,2,CH,EQ,C'BC'))
 /*
                          EOG: #1, 4 lines selected from jcl2/jgl230
                          EOF:     4 selected from 34 in jcl2/jgl230
 //SYSIN    DD *
 SORT FIELDS=(1,8,CH,A,69,12,CH,A)
 INCLUDE COND=(((9,1,CH,EQ,C'T'),OR,(COND=(9,1,CH,EQ,C'X'))),
                                 AND,(COND=(56,2,CH,EQ,C'AB')))
 /*
                          EOG: #1, 5 lines selected from jcl2/jgl232
                          EOF:     5 selected from 35 in jcl2/jgl232
 EOD: 7 files, 168 lines, selected 3 groups, 13 lines to selects/jcl2_SYSIN_1@COND:2@AND:1@COND:2@OR%

Notes re tts & selectlgfd1 demo

  1. demo/jcl2/* contains 7 demo JCLs with 5 SYSINs (3 of them sorts). 'arg1=//SYSIN' defines the group starting line & arg2=/*' defines the group ending line.

  2. 'arg3=1@COND:2@AND:1@COND:2@OR' defines the select group conditions. This arg3 selects groups with lines that contain ("COND" & "AND") OR ("COND" & "OR"). The pattern rules are the same as for the 'sst' instruction (preceding).

  3. Better if we could select ONLY the groups with BOTH AND & OR, since it is only those that need correction, But 'selectlgfd1' (using 'tts' instruction) can not do this, Because its multi conditions apply to single lines & not to all lines in the group

  4. See 'selectlgfd2 (using the 'ttx' instruction), which searches for all lines in the group to satisfy multi condition series patterns (1,2,3,etc).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3H2. SelectJobs: selectlf1,selectlfd1,selectlgfd1, *selectlgfd2*, selectfdd1

selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)

'selectlgfd2' can do a BETTER job of selecting ONLY the SYSINs with BOTH ANDs & ORs (vs 'selectlgfd1 which could only select SYSINs with EITHER AND or OR)


 uvcopy selectlgfd2,fild1=jcl2,arg1=1@//SYSIN,arg2=/*,arg3=1@COND:2@AND:3@OR%
 ============================================================================
 # Report: selects/jcl2_SYSIN_1@COND:2@AND:3@OR%
 # ---> uvcopy selectlgfd2,fild1=jcl2,arg1=//SYSIN,arg2=/*,arg3=1@COND:2@AND:3@OR%
 #      =========================================================================
 # format: uvcopy selectlgfd2,fild1=directory,arg1=...,arg2=...,arg3=...,arg4=...,uop=abcdgi
 # arg1 patterns identify begin-group line, arg2 patterns identify end-group line
 # arg3 patterns qualify group selection, arg4 patterns qualify lines to output/print
 #  - code '^' circumflex for ' ' spaces and '%' percent for ',' commas in patterns
 #  - may prefix patterns: '-' for absence, '+' for presence (default)
 #  - may restrict search area: arg2=@0[2]/*  <-- "/*" must be in 1st 2 bytes
 #  - multiple-condition-series: arg3=1@SORT:2@COND:3@AND:4@OR <-- ALL patterns must match
 # Date=2019/06/09_10:29:01, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=
 #===============================================================================
 //SYSIN    DD *
 SORT FIELDS=(1,8,CH,A,69,12,CH,A)
 INCLUDE COND=(((9,1,CH,EQ,C'T'),OR,(COND=(9,1,CH,EQ,C'X'))),
                                 AND,(COND=(56,2,CH,EQ,C'AB')))
 /*
                          EOG: #1, 5 lines selected from jcl2/jgl232
                          EOF:     5 selected from 35 in jcl2/jgl232
 EOD: 7 files, 168 lines, selected 1 groups, 5 lines to selects/jcl2_SYSIN_1@COND:2@AND:3@OR%

Notes re ttx vs tts & selectlgfd2 vs selectlgfd1

  1. Compare this selectlgfd2 (using the 'ttx' instruction) search result to the previous selectlgfd1 (using the 'tts' instruction).

  2. selectlgfd1/tts selection was 'arg3=1@COND:2@AND:1@COND:2@OR', which selected groups with (COND & AND) or (COND & OR). The result would include extra groups that do not need correction - ONLY groups with BOTH (AND & OR) may need correction.

  3. selectlgfd2/ttx selection is 'arg3=1@COND:2@AND:3@OR', which selects ONLY groups with BOTH ("AND" & "OR"), ie only those that may need correction.

  4. The 'tts' instruction), bases multi condition series patterns (1,2,3,etc) only on patterns found on any 1 line in the group.

  5. The 'ttx' instruction), bases multi condition series patterns (1,2,3,etc) on patterns found on any line in the group.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3I1. SelectJobs: selectlf1,selectlfd1,selectlgfd1,selectlgfd2, *selectfdd1*

select ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line

You could use 'selectfdd1' to select the entire SQL script. 'selectfdd1' selects entire files from the input directory to an output directory.

 'selectfdd1' can be use with BOTH multi-pattern-conditional-series-selection methods
 - based on arg1=multi-patterns on any 1 line AND/OR on arg2=multi-patterns on any line
   in the entire file.

Example#1 will test the arg1 method which selects multi-patterns on ONE line. (Also see Ex#2 on the next page which selects multi-patterns on any line in the file)

selectfdd1 Example#1 - multi-patterns on 1 line (arg1)


 #1. mkdir tmp2 OR rm -f tmp2/*   <-- make an output dir OR clear existing dir
     ==========================

 #2. uvcopy "selectfdd1,fild1=sqls,fild2=tmp2,arg1=1@Customers:2@LYsales,uop=s1t1"
     =========================================****================================
      - select files from sqls/* referencing "Customers" & "LYsales"
        (appearing on ONE line in the file)

stats Report written to selects/...

 # Report: selects/sqls_tmp2_sqls_1@customers:2@lysales
 # ---> uvcopy selectfdd1,fild1=sqls,fild2=tmp2,arg1=1@customers:2@lysales,uop=s1t1
 #      ===========================================================================
 #format: uvcopy selectfdd1,fild1=indir,fild2=outdir,arg1=LinePatterns,arg2=FilePatterns
 # Date=2019/06/03_20:56:02, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=s1t1
 #============----===================================================================
   EOF - 4 lines written to tmp2/select_customers_sales10000
 **EOD Input  - 36 lines  read from 11 files in directory sqls
 **EOD Output - 4 lines written to 1 files in directory tmp2

 #3. cat tmp2/*     <-- display output files
     ==========

/* select_customers_sales10000 */ SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >10000;

Notes Re selectfdd1 selections by *arg1 or arg2

  1. Using "arg1=1@Customers:2@LYsales" we get only 1 of the 2 files that have both "Customers" & "LYsales" - because the other file has these patterns on different lines.

  2. See Ex#2 on the next page which selects multi-patterns on any line in the file. - coding search args on arg2=... finds multi-patterns on any line in the file (vs arg1=... which looks for multi-patterns on the same line).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3I2. SelectJobs: selectlf1,selectlfd1,selectlgfd1,selectlgfd2, *selectfdd1*

select ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line

 'selectfdd1' can be use with BOTH multi-pattern-conditional-series-selection methods
 - based on arg1=multi-patterns on any 1 line AND/OR on arg2=multi-patterns on any line
   in the entire file.

Example#1 (on previous page) used 'arg1=...' which looks for multi-patterns on ONE line. This Example#2 uses 'arg2=...' which looks for multi-patterns on any line in the file.

selectfdd1 Example#2 - multi-patterns on ANY line (arg2)


 #1. mkdir tmp2 OR rm -f tmp2/*   <-- make an output dir OR clear existing dir
     ==========================

 #2. uvcopy "selectfdd1,fild1=sqls,fild2=tmp2,arg2=Customers:LYsales"
     =========================================****===================
      - select files from sqls/* referencing "Customers" & "LYsales"
        (appearing on ANY line in the file)

stats Report written to selects/...

 # Report: selects/sqls_tmp2_sqls_1@select:2@customers:3@lysales
 # ---> uvcopy selectfdd1,fild1=sqls,fild2=tmp2,arg2=1@select:2@customers:3@lysales,uop=s1t1
 #      ====================================================================================
 #format: uvcopy selectfdd1,fild1=indir,fild2=outdir,arg1=LinePatterns,arg2=FilePatterns
 # Date=2019/06/03_20:34:37, Site=UV_Software, Host=uvsoft5, User=uvadm, Options=s1t1
 #===================================================================================
   EOF - 4 lines written to tmp2/select_customers_sales10000
   EOF - 5 lines written to tmp2/select_customers_sales20000
 **EOD Input  - 36 lines  read from 11 files in directory sqls
 **EOD Output - 9 lines written to 2 files in directory tmp2

 #3. cat tmp2/*     <-- display output files
     ==========

/* select_customers_sales10000 */ SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >10000;

/* select_customers_sales20000 */ SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >20000;

Notes selections by arg1 or *arg2 & options 's1t1'

  1. Example#2 using arg2=..., gets both the files that have "Customers" & "LYsales" vs prior Example#1 which got only 1 file with "Customers" & "LYsales" on same line.

  2. To use option s1 with selectlgfd2, you need to also specify option 't1' which translates the data file to lower case when loaded into memory.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_4. UVdemos.doc Vancouver Utility Programming Aids

Part 4 - Contents


4A1. table3d - create table summary counts of desired items in text files
- pre-programmed job using uvcopy (powerful data manipulation utility)
- scan all files in directory building table statistics to dump at EOF
- table3d example#1 - counts of IP#s in FTP scripts

4B1. table3d user options --> uop=a0b3c0d0f0j0l0m0p1s0t0x0w1 <-- defaults
Many options allow you to define the items to be tabled compensating
for complex situations in your data files.

4C1. Most useful options - option 'w#'
Option w specifies the word# offset of the TargetWord from the KeyWord.
It defaults to 'w1' meaning the TargetWord to be tabled & counted is
the 1st word following the search KeyWord specified by arg1=...

4C2. Most useful options - option 'w#'
Option 'y#' specifies the number of keywords to be tabled (default 1).
For example: summarize both userid & password following keyword 'user'.

4D1. Test files to demo table3d
- JCLs, FTP scripts, SQL scripts,

4D2. Table Summarize program names in JCL
Key-Word PGM, Target-Word program-name following PGM=...

4E1. More about table3d, pre-programmed jobs vs specific uvcopy jobs

4E2. tableIP - custom written table summary demo (only 14 instructions)
- creates same table summary as table3d
- but only does that 1 specific summary (vs general purpose)

4E3. Notes uvcopy table summarize demo job 'tableIP'

4F1. table3d - create summaries of TABLEs in SQL scripts
Key-Word FROM, Target-Word table-name following FROM ...

4F4. Problem if Target-Word (table-name) not on same line as Key_Word (FROM)
Could use utility 'combine1d' to combine SQL statements prior to table3d

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A1. UVdemos.doc Vancouver Utility Programming Aids

TableJobs - create Table Summary Counts from Keywords/Targetwords

TableJobs build summary tables of counts & values to be dumped to a report file at EOF. There are various TableJobs for various file-types (text,fixed,variable), but we think you will find 'table3d' most useful to quickly supply you with statistics from directories of programs, scripts, parameter files, etc.

For example, 'table3d' can read all files in the parms/ directory, searching for KeyWord "open" & count occurrences of the following TargetWord (the IP#).

Example#1 - table summarize all IP#s in FTP parms


 uvcopy table3d,fild1=sqls,arg1=open
 ===================================
  - read parms/* search for "open" & count occurrences of following IP#
  - output report created in subdir stats/... & automatically named by concatenating
    directory + keywords separated by underscores

table3d report created in stats/...

 # /home/uvadm/demo/stats/parms_open  <-- This report created by uvcopy:
 # uvcopy table3d,fild1=parms,arg1=open
 # =====================================================================
 # - scan all files in directory for a keyword & table counts of following word
 # InDir=parms  Keyword=open  Qual1=   Qual2=
 # WordSepsBlank=  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1
 # - default selects following word, use option w2 to table 2nd word, etc
 # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
 # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
 # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
 # Date=2019/06/17_12:18:57, Site=UV_Software, Host=uvsoft5, User=uvadm
 #===============================================================================
 #---> uvcopy table3d,fild1=parms,arg1=open
 table3d  2019/06/17_12:19:01  Counts by Targetword following specified Keyword
 tbl#0001  tblt1f7 e0(48)         argument
 line#   1strec#  %      count  target-word
     1         2  25         1   192.168.0.4
     2         2  25         1   192.168.0.5
     3         2  50         2   192.168.0.6
                 100         4*  *TOTAL*

Notes, Benefits of table3d reports

  1. Output reports are automatically created in subdir stats/... named by concatenating the input directory & keywords separated by '_' underscores. The filename is coded on the 1st line of the report.

  2. The uvcopy command used to create the report is coded on the 2nd line, so you can look back at previous runs & repeat them (with modifications as desired).

  3. The reports are date/time stamped with all relevent info (site,host,user,options).

  4. See other examples in the tutorials ahead - userids & passwords in FTP parms, programs called by all JCLs, table-names used by all SQL scripts, etc.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B1. TableJobs - Table Summary Counts from Keywords/Targetwords

table3d options

The example on the previous page did not show the options screen displayed before table3d is executed & the report written to the stats/ directory. Here are the options which we will omit from following examples. Note that we


 uvcopy table3d,fild1=dir,arg1=keyword,arg2=qual1,arg3=qual2,arg4=chars2blank,uop=...
 ====================================================================================
 - use : delimiter for multiple keywords (keyword1:keyword2:keyword3:etc)
 uop=a0b3c0d0f0j0l0m0p1s0t0x0w1 - option defaults
     a1                     - "&" variable end "#", insert space after "#" force endword
     a2                     - discard variables not ending with "#"
       b3                   - assume 1 blank before & after keyword (default)
       b1                   - assume 1 blank before & no blank after
       b2                   - assume no blank before & 1 blank after
       b0                   - assume no blank before or after
         c1                 - bypass comments in COBOL programs (* column 7)
         c2                 - bypass comments in shell scripts (# column 1)
         c4                 - bypass comments in JCL //* cols 1-3
           d1               - insert blank before "$", allow arg1=$
             f0             - do not prepend/append table entry with filename
             f1             - prepend table argument with filename
             f2             - append table argument with filename
               j1           - insert blank before search word in data
               j2           - insert blank after search word in data
               j3           - insert blank & after search word in data
                 l0         - translate lower case before matching patterns
                   m1       - look for multiple keywords on 1 line
                   m2       - look for multiple keywords & prepend targetword
                   m4       - look for multiple keywords & append targetword
                     p1     - qualifier#1 present & qualifier#2 absent
                     p2     - qualifier#1 present & qualifier#2 present
                     p4     - qualifier#1 absent  & qualifier#2 absent
                       s1   - case insensitive translate patterns to lowercase
                         t1 - translate data to lower-case (to match patterns)
                         t2 - translate data to UPPER-case
                     w1     - target word is 1st word following keyword
                     w2     - target word is 2nd word following keyword
                     w0     - target word is same as keyword (partial def)
                   x1       - table w# word (search will ignore keyword)
                            - specified keyword will be inserted in outfilename
                 y#         - table multiple target words (4 max)
                 y3         - would table target word + following 2 words
               z1           - convert x"A3" to "@" in data for keyword "@"
 arg4 special chars to separate words, use with option w# offset from search word
 ------> &\,$=/>-<:(.`);_*[|]" <--- arg4 punctuation chars to blank
 ------> abcdefghjklpqrsuALPRQ <--- OR enter equivalents OR enter "all"
 User OPtion defaults=q1a0b3c0d0f0j0l0m0p1s0t0x0w1 (null accept or enter overrides)
 enter arg1 search keyword ------------>  <-- prompt inhibited if arg1 on cmdline
 enter arg2 qualifier#1 (or null) ----->
 enter arg3 qualifier#2 (or null) ----->
 enter arg4 chars to blank before word sep (comma,equal,etc) -->
 enter strings to be removed (use ":" sep if multiple) --->
    ----- I/O filenames & record counts -----
 EOJ, Output File written to: stats/parms_open
 default command = null, OR enter: vi,cat,more,lp,uvlp12,etc -->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C1. TableJobs - Table Summary Counts from Keywords/Targetwords

most useful options - option 'w#'

Option 'w' is probably the most useful options you are likely to need. Option w specifies the word# offset of the TargetWord from the KeyWord. It defaults to 'w1' meaning the TargetWord to be tabled & counted is the 1st word following the search KeyWord specified by arg1=...

Here is an example using option w2 (2nd word following search KeyWord) to table summarize the output filenames on FTP 'put' commands. First we will show the relevant FTP script (parm file)

# parms/ftpdemo1b open 192.168.0.5 user user02 user02pw lcd data1 put ar.sales.items C:\AR\SALES.ITEMS put gl.account.tran1 D:\GL\ACCOUNT.TRAN1

Example#2 - option 'w#' TargetWord offset from KeyWord


 #2. uvcopy table3d,fild1=parms,arg1=put,uop=w2
     ==========================================
      - search parms/* for keyword 'put' & table 2nd word following
Note
  • you will get prompts for unused arguments, enter null to continue
  • but I suggest reply 'cat' for the last prompt (to display output report)
      ---> cat <-- reply 'cat' to display report

 #2a. vi stats/parms_put_w2  <-- OR, enter separate command if job already ended
      =====================
 # /home/uvadm/demo/stats/parms_put_w2  <-- This report created by uvcopy:
 # uvcopy table3d,fild1=parms,arg1=put,uop=w2
 # =====================================================================
 # - scan all files in directory for a keyword & table counts of following word
 # InDir=parms  Keyword=put  Qual1=   Qual2=
 # WordSepsBlank=  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1w2
 # - default selects following word, use option w2 to table 2nd word, etc
 # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
 # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
 # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
 # Date=2019/06/17_15:11:03, Site=UV_Software, Host=uvsoft5, User=uvadm
 #===============================================================================
 #---> uvcopy table3d,fild1=parms,arg1=put,uop=w2
 table3d  2019/06/17_15:11:05  Counts by Targetword following specified Keyword
 tbl#0001  tblt1f7 e0(48)         argument
 line#   1strec#  %      count  target-word
     1         6  50         1   C:\AR\SALES.ITEMS
     2         7  50         1   D:\GL\ACCOUNT.TRAN1
                 100         2*  *TOTAL*

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C2. TableJobs - Table Summary Counts from Keywords/Targetwords

most useful options - option 'y#'

Option 'y#' specifies the number of keywords to be tabled (default 1). For example, we can table summarize both userid & password following keyword 'user'. Here is a sample using the FTP parms/* but showing only 1st file:

# parms/ftpdemo1a open 192.168.0.4 user user01 user01pw lcd data1 get /home/userxx/ar/sales.items get /home/userxx/gl/account.tran1

Example#3 - option 'y' no of words to table


 #3. uvcopy table3d,fild1=parms,arg1=user,uop=y2
     ===========================================
     - search parms/* for keyword 'user' & table following 2 words
     --> cat <-- enter at last prompt to display report
 # /home/uvadm/demo/stats/parms_user_y2  <-- This report created by uvcopy:
 # uvcopy table3d,fild1=parms,arg1=user,uop=y2
 # =====================================================================
 # - scan all files in directory for a keyword & table counts of following word
 # InDir=parms  Keyword=user  Qual1=   Qual2=
 # WordSepsBlank=  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1y2
 # - default selects following word, use option w2 to table 2nd word, etc
 # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
 # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
 # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
 # Date=2019/06/17_15:31:41, Site=UV_Software, Host=uvsoft5, User=uvadm
 #===============================================================================
 #---> uvcopy table3d,fild1=parms,arg1=user,uop=y2
 table3d  2019/06/17_15:31:44  Counts by Targetword following specified Keyword
 tbl#0001  tblt1f7 e0(48)         argument
 line#   1strec#  %      count  target-word
     1         3  25         1  user01 user01pw
     2         3  25         1  user02 user02pw
     3         3  50         2  user03 user03pw
                 100         4*  *TOTAL*

Notes

  1. Note that any options entered are concatenated onto the output filename (parms_user_y2 for the example above). The 1st line of the report file shows you the full path name of the report file (see above).

  2. For these demos, we are coding all arguments & options on the command-line, but you could enter only 'uvcopy table3d' & enter relevant info at the prompts. When we enter all relevant info on the cmdline, we can just make null replies to all following prompts.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4D1. UVdemos.doc Vancouver Utility Programming Aids

table3d - create summary table of programs in JCL

This the 4th demo of table3d, now using JCL as input


 #40a. login userxx --> /home/userxx
 #40b. cd demo      --> /home/usrxx/demo

 #41. l    <-- list files in current directory /home/userxx/demo
     ===
      drwxrwxr-x 2 userxx apps 4096 Feb 13 15:01 jcl2
      drwxrwxr-x 2 userxx apps 4096 Feb 13 15:01 jcl3
      drwxrwxr-x 2 userxx apps 4096 Feb 13 06:47 parms
      drwxrwxr-x 2 userxx apps 4096 Feb 13 06:47 rpts
      drwxrwxr-x 2 userxx apps 4096 Feb 13 06:47 sqls
      drwxrwxr-x 2 userxx apps 4096 Feb 13 15:04 stats

 #42. l jcl2   <-- list jcl2/* input files for 3rd demo of table3d
      ======
      -rw-rw-r-- 1 userxx apps  480 Feb 13 15:01 jar100.jcl
      -rw-rw-r-- 1 userxx apps  613 Feb 13 15:01 jar120.jcl
      -rw-rw-r-- 1 userxx apps  984 Feb 13 15:01 jar200.jcl
      -rw-rw-r-- 1 userxx apps  777 Feb 13 15:01 jgl100.jcl
      -rw-rw-r-- 1 userxx apps 1238 Feb 13 15:01 jgl200.jcl  <-- listed below
      -rw-rw-r-- 1 userxx apps  617 Feb 13 15:01 jgl220.jcl
      -rw-rw-r-- 1 userxx apps 1477 Feb 13 15:01 jgl230.jcl
      -rw-rw-r-- 1 userxx apps  932 Feb 13 15:01 jgl320.jcl

 #43a. cat jcl2/jar200.jcl
       ===================
      //JAR200   JOB  (1234),'TEST/DEMO MVS JCL CONVERT'
      //* SORT AR.SALES.ITEMS BY PRODUCT CODE FOR LISTING
      //STEP010  EXEC PGM=SORT,REGION=2048K                      <--Note PGM=SORT
      //SORTIN   DD DSN=AR.SALES.ITEMS,DISP=SHR
      //SORTOUT  DD DSN=&&jar200_TEMPSLS,DISP=(NEW,PASS),...
      //SYSIN    DD *
      SORT FIELDS=(31,6,CH,A,1,6,CH,A)
      OMIT COND=((11,2,CH,EQ,C'65'),OR,(COND=(11,2,CH,EQ,C'85'))
      /*
      //*=================== step# 0020 ========================
      //* LIST AR.SALES.ITEMS IN PRODUCT CODE SEQUENCE
      //STEP020  EXEC PGM=CAR200,COND=(4,LT,STEP010),PARM=2006   <--Note PGM=CAR200
      //SALEDTL  DD DSN=&&jar200_TEMPSLS,DISP=(OLD,PASS)
      //CUSTMAS  DD DSN=AR.CUSTOMER.MASTER.INDEXED,DISP=SHR
      //SALELST  DD DSN=AR.SALES.LIST,DISP=(,CATLG,DELETE),...

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

table summarize PGM names in JCL


 #44. uvcopy table3d,fild1=jcl2,arg1=PGM,arg4=ec,uop=w1q0i7
      =====================================================
      - read following notes before executing above command line

Notes re arguments & options

We will use 'PGM' as the keyword to table summarize the following word (program-name).

Re: 'arg4=ec', arg4 specifies characters to blank before word separation. Some special characters are a problem when entered using arg4=... on the command line so we allow you to use an alpha character to represent & show the equivalents as follows:

&\,$=/-:(.`);_*|" <--- arg4=special chars to blank abcdefhklpqrsuAPQ <--- equivalents, easier to enter

 //STEP010  EXEC PGM=CAR100,PARM=2008

For the above example, we need to blank the '=' & the ',' to separate the words. We used 'arg4=ec' vs 'arg4==,', because the ',' would confuse command parsing.

You could omit arg4=... from command line & enter actual characters at the prompt, but we prefer entering on the command line for this documentation.

Re: options 'uop=w1q0i7'. 'w1' means the target-word is 1st word following the keyword & we did not need to specify since it is the default.

Option 'q0' inhibits prompting for options (not needed if on the command line).

Option 'i7' inhibits displaying the 'opr' commands describing options, etc.

now execute the table3d command


 #44. uvcopy table3d,fild1=jcl2,arg1=PGM,arg4=ec,uop=q0i7
      ===================================================
 enter arg2 qualifier#1 (null=disable) ----->
 enter arg3 qualifier#2 (null=disable) ----->
 enter strings to be removed (use ":" sep if multiple) --->
 190216:160322:table3d: EOF fili01 rds=34 size=1477: demo/jcl2/jgl230.jcl hits=3
 190216:160322:table3d: EOF fili01 rds=23 size=984: demo/jcl2/jar200.jcl hits=2
 190216:160322:table3d: EOF fili01 rds=11 size=480: demo/jcl2/jar100.jcl hits=1
 190216:160322:table3d: EOF fili01 rds=17 size=777: demo/jcl2/jgl100.jcl hits=1
 190216:160322:table3d: EOF fili01 rds=26 size=1238: demo/jcl2/jgl200.jcl hits=2
 190216:160322:table3d: EOF fili01 rds=22 size=932: demo/jcl2/jgl320.jcl hits=2
 190216:160322:table3d: EOF fild01 rds=8 size=4096: demo/jcl2 hits=11
 190216:160322:table3d: EOF filo01 wrts=20 size=1322: stats/jcl2_PGM_ec_w1q0i7
 EOJ, Output File written to: stats/jcl2_PGM_ec_w1q0i7
 default command = null, OR enter: more,print,uvlpr12,edit,null -->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

table3d output report for 3rd demo JCL programs


 #44a. --> cat <-- suggest 'cat' to view outputs (when known to be small)
           ===   - alternative, null reply & use #35. below (vi stats/parms_open)
 # /home/uvadm/demo/stats/jcl2_PGM_ec_w1q0i7  <-- This report created by uvcopy:
 # uvcopy table3d,fild1=jcl2,arg1=PGM,arg4=ec,uop=w1q0i7
 # =====================================================================
 # - scan all files in directory for a keyword & table counts of following word
 # InDir=jcl2  Keyword=PGM  Qual1=   Qual2=
 # WordSepsBlank==,  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1w1q0i7
 # - default selects following word, use option w2 to table 2nd word, etc
 # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
 # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
 # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
 # Date=2019/06/17_16:06:56, Site=UV_Software, Host=uvsoft5, User=uvadm
 #===============================================================================
 #---> uvcopy table3d,fild1=jcl2,arg1=PGM,arg4=ec,uop=w1q0i7
 table3d  2019/06/17_16:06:59  Counts by Targetword following specified Keyword
 tbl#0001  tblt1f7 e0(48)         argument
 line#   1strec#  %      count  target-word
     1         3   7         1   CAR100
     2        15   7         1   CAR200
     3        26  21         3   CGL100
     4        17  21         3   CGL200
     5        12   7         1   IDCAMS
     6         6   7         1   IEBGENER
     7         5  28         4   SORT
                 100        14*  *TOTAL*

 #45. vi stats/jcl2_PGM_ec_q0i7  <-- view report separately after running table3d
      =========================
Note
  • the report is named for the directory + keyword + separators + options
  • so you do not have to make up a unique name to avoid overwriting prior report

General Purpose Utilities

The example above used JCL to test/demo table3d. You may not have JCL, but we hope you can see how table3d might be used with whatever files, scripts,& programming languages you have.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4E1. UVdemos.doc Vancouver Utility Programming Aids

more about table3d, pre-programmed jobs vs specific uvcopy jobs

Pre-programmed jobs make it easy to apply the power of uvcopy to your data files - analyzing, selecting, searching/replacing, etc with more options & qualifications than are provided by the standard unix/linux system utilities.

You do not need to know the uvcopy instruction set to use general purpose pre-programmed jobs such as 'table3d', but we hope you will eventually learn the uvcopy instruction set, in case you find situations where you need more specific custom processing.

You can see the instruction file for 'table3d' at /home/uvadm/pf/util/table3d. It has over 250 instructions because it processes all files in a directory & provides many options to search for & qualify keywords & build summary tables of following targetwords.

We will show you here a much simpler table summary job (tableIP) with only 14 essential instructions that creates the same table summary as did the previous 'table3d', but it is written specifically to table summarize IP#s following the word 'open' (vs the general purpose pre-programmed table3d which accepts arguments to summarize any desired item following any desired keyword for all files in directory).

'tableIP' is stored at /home/uvadm/pf/demo/tableIP & you can run as follows:


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1. cat parms/* >tmp1/parms_all   <-- concat all parms/* into 1 file for tableIP
     ===========================

 #2. uvcopy tableIP,fili1=tmp1/parms_all,filo1=stats/parms_ftp_tableIP
     =================================================================
     - run demo job to summarize IP#s in 1 file & create table same as previous table3d
 tableIP  2019/06/17_16:22:09  table summarize TCP/IP#s in FTP scripts
 tbl#0001  tblt1   ax6(16)        argument
 line#   1strec#  %      count  FTP IP#
     1         2  25         1  192.168.0.4
     2         9  25         1  192.168.0.5
     3        17  50         2  192.168.0.6
                 100         4*  *TOTAL*

Since this reads only 1 file (vs all files in directory), we concatenate the parms/* files used for the previous table3d demo into 1 file stored in tmp1/parms_all for input to uvcopy tableIP.

See the 'tableIP' demo job listed on the next page:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4E2. more about table3d, pre-programmed jobs vs specific uvcopy jobs

tableIP - simplified table summary demo

 # tableIP - table summarize counts of unique TCP/IP#s in FTP scripts
 #         - uvcopy demo job, by Owen Townsend, UV Software, June 2019
 # - demo job used in documentation www.uvsoftware.ca/uvdemos.htm#2C1
 # - Normally use pre-programmed job 'table3d' to crate summary table counts
 # - This illustrates writing the uvcopy code for a custom table summary
 # - to keep it simple we will 1st concat all parm files into 1 tmp file
 #
 # cat parms/* >tmp1/parms_all
 #
 # uvcopy tableIP,fili1=tmp1/parms_all,filo1=stats/parms_ftp_tableIP
 # =================================================================
 # - read input fili1=..., build table of IP#s, write to output filo1=... at EOF
 #
 # uvcopy tableIP    <-- same as above, I/O files default as above
 # ==============
 #
 # 'tableIP' is a simplified version of 'table3d' general purpose pre-programmed job
 #   - to create table summaries of ANY item from ALL files in a directory
 #   - without needing to write any uvcopy instructions
 # 'tableIP' builds a table of 1 specific item from only 1 file
 #   - intended to demo the power of uvcopy & encourage you to learn uvcopy
 #   - will scan for lines with word 'open' known to precede the IP#
 #     open 192.168.0.4   <-- scan for lines with 'open' known to precede the IP#
 #
 opr='$jobname - table summarize TCP/IP#s in FTP scripts'
 rop=r1  # option for EOF prompt to view output file
 fili1=?tmp1/parms_all,rcs=256,typ=LST
 filo1=?stats/parms_ftp_tableIP,rcs=256,typ=LSTt
 @run
         opn     all
 # begin loop to get records, build table until EOF
 man20   get     fili1,a0               get next record into area 'a'
         skp>    man90                  (cc set > at EOF)
         sqzl1c1 a0(80),' '             ensure column 1 blank & 1 blank between words
         scn     a0(80),' open '        scan for ' open ' preceding IP#
         skp!    man20                  if not found - return to get next record
         tblt1   ax6(16),'FTP IP#'      build table counts in memory
         skp     man20                  return to get next record
 # EOF - dump table to output file, close files, end job
 man90   tbpt1  filo1,'table summarize TCP/IP#s in FTP scripts'
         cls    all
         eoj
 #            ** output - stats/parms_ftps_tableIP **
 # tableIP  2019/06/17_16:22:09  table summarize TCP/IP#s in FTP scripts
 # tbl#0001  tblt1   ax6(16)        argument
 # line#   1strec#  %      count  FTP IP#
 #     1         2  25         1  192.168.0.4
 #     2         9  25         1  192.168.0.5
 #     3        17  50         2  192.168.0.6
 #                 100         4*  *TOTAL*

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4E3. more about table3d, pre-programmed jobs vs specific uvcopy jobs

Notes re uvcopy demo job 'tableIP'

  1. You could use this demo job to learn a lot about the uvcopy instruction set.

  2. Some uvcopy instructions are similar to the IBM 360 Z/OS instruction set, but uvcopy instructions are interpretive, no compile required, just edit & run.

  3. uvcopy gives you the power of assembler in an easy to use framework. In particular, this example will illustrate the power of index registers. uvcopy operands consist of an area (a-z), optionally an index register (a-z), and a displacement. For example op1 of 'tbl' is 'ax6' which is area 'a' + the displacement in register 'x' + the displacement coded ('6') in this example.

  4. 'scn' finds the word ' open ' & stores its displacement in index register 'x', which is then used as by 'tbl' to address the following word for tabling.

  5. All instructions are documented at https://uvsoftware.ca#uvcopy3.htm. The 1st page is an index with links to the various instructions. Here are direct links to some of the more interesting instructions in tableIP.
      https://uvsoftware.ca#uvcopy3.htm#sqz
      https://uvsoftware.ca#uvcopy3.htm#scn
      https://uvsoftware.ca#uvcopy3.htm#tbl
      https://uvsoftware.ca#uvcopy3.htm#tbp
  1. Here are a few notes about these 4 instructions use in tableIP & you can click on the links to see the complete doc on any of these.

 #7a. sqzl1c1 a0(80),' '             ensure column 1 blank & 1 blank between words
      =================
      - option 'l1' of sqzl1c1 ensures 1 blank in column 1
      - option 'c1' of sqzl1c1 ensures 1 blank between words
        so we can know that the word following ' open ' is exactly 6 bytes higher

 #7b. scn     a0(80),' open '        scan for ' open ' preceding IP#
      =======================
      - scn stores the zero relative displacement of ' open ' in register x.

 #7c. tblt1   ax6(16),'FTP IP#'      build table counts in memory
      ========================
      - tblt1 builds table#1 based on the 16 bytes located in area 'a'
        + displacement in register 'x' + 6 = area 'a' + 0 + 6
      - in this example, register 'x' will be 0 for ' open 192.168.0.4 ' since 'open'
        was 1st word on the line & sqzl1c1 has ensured the leading ' ' matching ' open '

 #7d. tbpt1  filo1,'table summarize TCP/IP#s in FTP scripts'
      ======================================================
      - tbpt1 (at EOF) dumps (Prints) the table to the output file (filo1=... above)
      - operand 2 supplies an extra heading for the table.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F1. UVdemos.doc Vancouver Utility Programming Aids

table3d - create summaries of TABLEs in SQL scripts


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1a. l        <-- list subdirs in /home/userxx/demo
      ===
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 ctl
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 jcl2
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 jcl3
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 parms
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 rpts
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 sqls     <-- SQL scripts
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 stats
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 tmp
      drwxrwxr-x 2 userxx apps 4096 Feb 16 17:16 tmp1

 #1b. l sqls     <-- list files in sqls/...
 ===========
      -rw-rw-r-- 1 userxx apps  56 Feb 16 17:16 select_customers_all    <-- simple example
      -rw-rw-r-- 1 userxx apps  97 Feb 16 17:16 select_customers_basic
      -rw-rw-r-- 1 userxx apps 152 Feb 16 17:16 select_customers_WA     <-- PROBLEM
      -rw-rw-r-- 1 userxx apps 136 Feb 16 17:16 select_employee_dept123
      -rw-rw-r-- 1 userxx apps 115 Feb 16 17:16 select_employee_seattle
      -rw-rw-r-- 1 userxx apps  79 Feb 16 17:16 select_orders_big
      -rw-rw-r-- 1 userxx apps 306 Feb 16 17:16 select_patients_old
      -rw-rw-r-- 1 userxx apps 129 Feb 16 17:16 select_students_failing

We can use the known keyword 'FROM' to summarize the following SQL TABLE-name. For example here is the SELECT stmnt from the 1st of the 8 listed on the enxt page.

      SELECT * FROM Customers;

PROBLEM in SQL scripts for table3d

table3d expects the target-word (to be summarized) to follow the known key-word on the same line, as above, but consider the SELECT in the 3rd SQL script:

       SELECT custno, Name, Birthday, Phone,
       Address, City, Zip, LYsales,Discount, FROM   <-- PROBLEM
       Customers WHERE LYsales >5000;

Our solution is to precede 'table3d' with 'combine1d' which will combine the SQL SELECT statements onto 1 line using the ';' semi-colons that end SQL statements.

Note that combine1d is a general purpose utility & could be used for other file types that you may have. 'arg1=...' defines optional statement begin patterns (up to 8 words) and 'arg4=..." defines the END patterns (up to 8 words, not necessarily 1 character).

'arg2=...' defines up to 8 words that must be present for the stmnt to be selected and 'arg3=...' defines up to 8 words that must be absent for the stmnt to be selected.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F2. table3d - create summaries of TABLEs in SQL scripts


 #1c. cat sqls/*  <-- display contents of files in sqls/...
      ==========
      /* select_customers_all */
         SELECT * FROM Customers;
      /* select_customers_basic */
         SELECT Name, Phone, Address, City, Zip
                FROM Customers;
      /* select_customers_sales10000 */
         SELECT custno, Name, Phone, Address, City, Zip,
         TYsales, LYsales, FROM
         Customers WHERE LYsales >10000;
      /* select_customers_sales20000 */
         SELECT custno, Name, Phone, Address, City, Zip,
         TYsales, LYsales, FROM
         Customers
         WHERE LYsales >20000;
      /* select_employee_dept123 */
         SELECT EMP_ID, NAME, ADDRESS, CITY, ZIP, DEPT FROM
         EMPLOYEE WHERE DEPT = '123' AND STATUS = 'FULL';
      /* select_employee_seattle */
         SELECT EMP_ID, LAST_NAME FROM EMPLOYEE
         WHERE CITY = 'Seattle' ORDER BY EMP_ID;
      /* select_orders_big */
         SELECT item FROM Orders
         WHERE quantity > 500;
      /* select_patients_over70 */
         SELECT Name, Age FROM Patients WHERE Age > 70
         GROUP BY Age ORDER BY Name;
      /* select_patients_over80 */
         SELECT Name, Age FROM Patients WHERE Age > 80
         GROUP BY Age ORDER BY Name;
      /* select_patients_over90 */
         SELECT Name, Age FROM Patients
         WHERE Age > 90 GROUP BY Age ORDER BY Name;
      /* select_students_failing */
         CREATE VIEW Failing_Students AS
         SELECT S_NAME, Student_ID FROM
         STUDENT WHERE GPA < 30;

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F3. table3d - create summaries of TABLEs in SQL scripts

Here is our 1st attempt, will be incorrect, as explained above.


 #2. uvcopy table3d,fild1=sqls,arg1=FROM,uop=q0i7
     ============================================
      enter arg2 qualifier#1 (null=disable) ----->
      enter arg3 qualifier#2 (null=disable) ----->
      enter arg4 chars to blank before word sep (comma,equal,etc) -->
      190617:173959:table3d: EOF fili01 rds=3 size=77: sqls/select_orders_big hits=1
      190617:173959:table3d: EOF fili01 rds=3 size=109: sqls/select_patients_over80 hits=1
      190617:173959:table3d: EOF fili01 rds=2 size=55: sqls/select_customers_all hits=1
      190617:173959:table3d: EOF fili01 rds=3 size=97: sqls/select_customers_basic hits=1
      190617:173959:table3d: EOF fili01 rds=3 size=115: sqls/select_employee_seattle hits=1
      190617:173959:table3d: EOF fili01 rds=3 size=110: sqls/select_patients_over70 hits=1
      190617:173959:table3d: EOF fili01 rds=5 size=152: sqls/select_customers_sales20000
      190617:173959:table3d: EOF fili01 rds=3 size=110: sqls/select_patients_over90 hits=1
      190617:173959:table3d: EOF fili01 rds=4 size=147: sqls/select_customers_sales10000
      190617:173959:table3d: EOF fili01 rds=4 size=126: sqls/select_students_failing
      190617:173959:table3d: EOF fili01 rds=3 size=136: sqls/select_employee_dept123
      190617:173959:table3d: EOF fild01 rds=13 size=4096: sqls hits=7
      190617:173959:table3d: EOF filo01 wrts=17 size=1240: stats/sqls_FROM_q0i7
      EOJ, Output File written to: stats/sqls_FROM_q0i7
      default command = null, OR enter: more,print,uvlpr12,edit,null --> cat
      # /home/uvadm/demo/stats/sqls_FROM  <-- This report created by uvcopy:
      # uvcopy table3d,fild1=sqls,arg1=FROM
      # =====================================================================
      # - scan all files in directory for a keyword & table counts of following word
      # InDir=sqls  Keyword=FROM  Qual1=   Qual2=
      # WordSepsBlank=  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1
      # - default selects following word, use option w2 to table 2nd word, etc
      # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
      # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
      # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
      # Date=2019/06/17_11:45:06, Site=UV_Software, Host=uvsoft5, User=uvadm
      #===============================================================================
      #---> uvcopy table3d,fild1=sqls,arg1=FROM
      table3d  2019/06/17_11:45:09  Counts by Targetword following specified Keyword
      tbl#0001  tblt1f7 e0(48)         argument
      line#   1strec#  %      count  target-word
          1         2  28         2   Customers;
          2         2  14         1   EMPLOYEE
          3         2  14         1   Orders
          4         2  42         3   Patients
                      100         7*  *TOTAL*

PROBLEM - some TABLE names missing ?

If you compare our 1st attempt above to the 8 sql files listed on page '4F2', you will see that there are some missing. This 1st attempt found only 4 TABLE-names, & 6 occurrences, but we know there are 5 TABLE-names & 8 occurrences.

'table3d' expects the target-word to be summarized (SQL TABLEname in this case) to directly follow the keyword ('FROM' in this case), BUT in 4 of the 8 sql files, the keyword 'FROM' is on the line preceding the line with the target-word (TABLEname).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F4. table3d - create summaries of TABLEs in SQL scripts

Solution - combine SQL statements onto 1 line

We will use the uvcopy 'combine1d' utility to combine SQL statements onto 1 line. 'arg1=SELECT' will start combining lines until the 'arg4=;' end-of-stmnt marker found and 'arg2=FROM' must occur somewhere within the combined statement.

We will use 'combine1d' to combine the SQL stmnts from all files in sqls/... and write the output combined stmnts into the tmp1/... directory as follows:


 #2. uvcopy "combine1d,fild1=sqls,fild2=tmp1,arg1=SELECT,arg2=FROM,arg4=;,uop=d1r1"
     ==============================================================================
      - copy all files in sqls/* to tmp1/... combining multi-line SQL stmnts onto 1 line
      - we use option 'd1' to drop /*...comments...*/ & option 'r1' to remove the ';' marker
      uop=d1r0t0 - option defaults
          d1     - drop /* ...comments... */
            r1   - remove stmt end pattern
              t0 - no case translation (t1 lower, t2 UPPER)
 190617:172639:combine1d: EOF fili01 rds=2 size=55: sqls/select_customers_all
 190617:172639:combine1d: EOF filo02 wrts=1 size=24: tmp1/select_customers_all
 190617:172639:combine1d: EOF fili01 rds=3 size=97: sqls/select_customers_basic
 190617:172639:combine1d: EOF filo02 wrts=1 size=54: tmp1/select_customers_basic
 190617:172639:combine1d: EOF fili01 rds=4 size=147: sqls/select_customers_sales10000
 190617:172639:combine1d: EOF filo02 wrts=1 size=102: tmp1/select_customers_sales10000
 190617:172639:combine1d: EOF fili01 rds=5 size=152: sqls/select_customers_sales20000
 190617:172639:combine1d: EOF filo02 wrts=1 size=102: tmp1/select_customers_sales20000
 190617:172639:combine1d: EOF fili01 rds=3 size=136: sqls/select_employee_dept123
 190617:172639:combine1d: EOF filo02 wrts=1 size=99: tmp1/select_employee_dept123
 190617:172639:combine1d: EOF fili01 rds=3 size=115: sqls/select_employee_seattle
 190617:172639:combine1d: EOF filo02 wrts=1 size=78: tmp1/select_employee_seattle
 190617:172639:combine1d: EOF fili01 rds=3 size=77: sqls/select_orders_big
 190617:172639:combine1d: EOF filo02 wrts=1 size=45: tmp1/select_orders_big
 190617:172639:combine1d: EOF fili01 rds=3 size=110: sqls/select_patients_over70
 190617:172639:combine1d: EOF filo02 wrts=1 size=73: tmp1/select_patients_over70
 190617:172639:combine1d: EOF fili01 rds=3 size=109: sqls/select_patients_over80
 190617:172639:combine1d: EOF filo02 wrts=1 size=73: tmp1/select_patients_over80
 190617:172639:combine1d: EOF fili01 rds=3 size=110: sqls/select_patients_over90
 190617:172639:combine1d: EOF filo02 wrts=1 size=73: tmp1/select_patients_over90
 190617:172639:combine1d: EOF fili01 rds=4 size=126: sqls/select_students_failing
 190617:172639:combine1d: EOF filo02 wrts=1 size=54: tmp1/select_students_failing
 190617:172639:combine1d: EOF fili03 rds=11 size=259: tmp/infiles
 190617:172639:combine1d: EOF fild01 size=4096: sqls
 190617:172639:combine1d: EOF fild02 size=4096: tmp1

 #2a. cat tmp1/*    <-- list contents of all output files from combine1d
      ==========
      SELECT * FROM Customers
      SELECT Name, Phone, Address, City, Zip FROM Customers
      SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >10000
      SELECT custno, Name, Phone, Address, City, Zip, TYsales, LYsales, FROM Customers WHERE LYsales >20000
      SELECT EMP_ID, NAME, ADDRESS, CITY, ZIP, DEPT FROM EMPLOYEE WHERE DEPT = '123' AND STATUS = 'FULL'
      SELECT EMP_ID, LAST_NAME FROM EMPLOYEE WHERE CITY = 'Seattle' ORDER BY EMP_ID
      SELECT item FROM Orders WHERE quantity > 500
      SELECT Name, Age FROM Patients WHERE Age > 70 GROUP BY Age ORDER BY Name
      SELECT Name, Age FROM Patients WHERE Age > 80 GROUP BY Age ORDER BY Name
      SELECT Name, Age FROM Patients WHERE Age > 90 GROUP BY Age ORDER BY Name
      SELECT S_NAME, Student_ID FROM STUDENT WHERE GPA < 30
Note
  • options default to d1 (drop /*...comments...*/) & r1 (remove end stmnt ';')
  • you might rerun using option d0 (to NOT drop /*...cmts...*/)
  • which would report SELECT stmnt TABLEs found in /*...cmts...*/

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F5. table3d - create summaries of TABLEs in SQL scripts

now, use table3d to create correct table summary

We will specify the input directory as tmp1/... (output from combine1d above), and the report output of table3d defaults to stats/... Note that table3d names the output report by concatenating the input directory (now tmp1) + the keyword, so in this case the output report will be 'stats/tmp1_FROM'. You could name the output file by adding 'filo1=sqls_FROM' to the command below:


 #3. uvcopy table3d,fild1=tmp1,arg1=FROM,uop=q0i7
     ============================================
      enter arg2 qualifier#1 (null=disable) ----->
      enter arg3 qualifier#2 (null=disable) ----->
      enter arg4 chars to blank before word sep (comma,equal,etc) -->
      190617:173015:table3d: EOF fili01 rds=1 size=45: tmp1/select_orders_big hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=73: tmp1/select_patients_over80 hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=24: tmp1/select_customers_all hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=54: tmp1/select_customers_basic hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=78: tmp1/select_employee_seattle hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=73: tmp1/select_patients_over70 hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=102: tmp1/select_customers_sales20000 hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=73: tmp1/select_patients_over90 hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=102: tmp1/select_customers_sales10000 hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=54: tmp1/select_students_failing hits=1
      190617:173015:table3d: EOF fili01 rds=1 size=99: tmp1/select_employee_dept123 hits=1
      190617:173015:table3d: EOF fild01 rds=13 size=4096: tmp1 hits=11
      190617:173015:table3d: EOF filo01 wrts=18 size=1279: stats/tmp1_FROM_q0i7
      EOJ, Output File written to: stats/tmp1_FROM_q0i7
      default command = null, OR enter: vi,cat,more,lp,uvlp12,etc -->
      # /home/uvadm/demo/stats/tmp1_FROM  <-- This report created by uvcopy:
      # uvcopy table3d,fild1=tmp1,arg1=FROM
      # =====================================================================
      # - scan all files in directory for a keyword & table counts of following word
      # InDir=tmp1  Keyword=FROM  Qual1=   Qual2=
      # WordSepsBlank=  Options=q1a0b3c0d0f0j0l0m0p1s0t0x0w1
      # - default selects following word, use option w2 to table 2nd word, etc
      # - specify arg1=Keyword,arg2=Qualifier-present,arg2=Qualifier-absent
      # &\,$=/>-<:(.');_*[|]"<--- arg4=punctuation to blank for word separation
      # abcdefghjklpqrsuALPRQ<--- equivalents, easier to enter, may enter "all"
      # Date=2019/06/17_11:52:30, Site=UV_Software, Host=uvsoft5, User=uvadm
      #===============================================================================
      #---> uvcopy table3d,fild1=tmp1,arg1=FROM
      table3d  2019/06/17_11:52:34  Counts by Targetword following specified Keyword
      tbl#0001  tblt1f7 e0(48)         argument
      line#   1strec#  %      count  target-word
          1         1  36         4   Customers
          2         1  18         2   EMPLOYEE
          3         1   9         1   Orders
          4         1  27         3   Patients
          5         1   9         1   STUDENT
                      100        11*  *TOTAL*

Now you can see that we are reporting TABLE-names found in all 8 SQL scripts (3 have Customers, 2 have EMPLOYEE,& others have only 1 TABLE-name in the script).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_5. UVdemos.doc Vancouver Utility Programming Aids

Part 5 - Search/Replace - Contents


5A1. scan1d pre-programmed job to scan all files in a directory
- searching for 1 pattern & creating a report matching lines

5B1. rep1d pre-programmed job to search for 1 pattern & replace with an alternate
- while copying all files from 1 directory to a 2nd (empty) directory

5B2. rep2d pre-programmed job to search for multiple patterns & replace with alternates
- uses a pre-edited table of search/replace patterns
- while copying all files from 1 directory to a 2nd (empty) directory

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A1. UVdemos.doc Vancouver Utility Programming Aids

scan1d - scan all files in directory reporting matches to 1 pattern

'scan1d' is a pre-programmed uvcopy job to scan all files in a directory searching for 1 pattern & reporting matches. Of course you would usually do this with 'grep' but you could use scan1d when you want a nicer report or you want to qualify the matches by 1 or 2 other patterns that must be present or absent on the same line.

We will demo scan1d using the FTP demo files in demo/parms/... All 4 files are listed on page '1C3', but here is the 1st 1 as a reminder.

      # parms/ftpdemo1a
      open 192.168.0.4
      user user01 user01pw
      lcd data1
      get /home/userxx/ar/sales.items

 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1. uvcopy scan1d,fild1=parms,arg1=open,filo1=tmp/parms_open
     ========================================================
      - scan all files in parms/... reporting matches to 'open'
      - output report as follows:
      Job: scan1d  Dir: parms  Search: open  Qual1:   Qual2:
       SearchBgn:    SearchEnd:    MatchOps:   UserOps: q1a1b99999c0d256e0p1
      ====================================================== 2019/02/15_11:17:53
      00002 open 192.168.0.4
      00007                   1 hits @EOF: parms/ftpdemo1a
      00002 open 192.168.0.5
      00008                   1 hits @EOF: parms/ftpdemo1b
      00002 open 192.168.0.6
      00009                   1 hits @EOF: parms/ftpget1
      00002 open 192.168.0.6
      00009                   1 hits @EOF: parms/ftpput1
      ====================================================== 2019/02/15_11:17:53
       EOD: 00004 hits in 0004 files of 0004 total (00033 lines)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B1. UVdemos.doc Vancouver Utility Programming Aids

rep1d - search/replace 1 pattern copying all files to 2nd directory

'rep1d' is a pre-programmed uvcopy job to scan all files in a directory searching for 1 pattern & replacing with an alternate. You would usually use 'sed' for this, but rep1d could be used when you need to qualify the replacements by 1 or 2 other patterns that must be present or absent on the same line.


 #1. uvcopy rep1d,fild1=parms,fild2=tmp1,arg1=192.168.0.6,arg2=\$IP6
                             ,filo1=tmp/parms_192.168.0.6
     ===============================================================
      - copy files from parms/* to tmp1/... replacing IP '192.168.0.6' with '$IP6'
      - output dir defaults to tmp1/... & audit file defaults to tmp/${fild1}_$arg1
      - the audit file shows the line after the replacement has been made

output audit file

      Job: rep1d InDir: parms OutDir: tmp1 Search: 192.168.0.6 Replace: $IP6
      QualPresent:   Absent:   SearchBgn:    SearchEnd:
      MatchOptions:   UserOptions: q1a1b99999c0d256e0p1
      ====================================================== 2019/02/15_11:21:34
      00002 open $IP6
      00009                   1 reps @EOF: tmp1/ftpget1
      00002 open $IP6
      00009                   1 reps @EOF: tmp1/ftpput1
      ====================================================== 2019/02/15_11:21:34
      EOD: 2 reps in 2 files of 4 total (33 lines)
      Audit filename: tmp/parms_192.168.0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B2. UVdemos.doc Vancouver Utility Programming Aids

rep2d search/replace multiple patterns copying all files to 2nd dir

'rep2d' is a pre-programmed uvcopy job to copy all files in a directory to a 2nd directory, searching/replacing via a table of multiple search/replace patterns, optionally qualified by patterns that must be present or absent on the same line.

You would pre-edit the search/replace table, as shown in the demo below: I suggest you use the demo/ctl/ subdir to store your search/replace tables.


 #1. vi ctl/IPreplace.tbl   <-- edit search/replace table for multiple IP#s
     ====================
      # IPreplace.tbl - table of search/replace IP#s stored in /home/uvadm/demo/ctl/IPreplace.tbl
      #               - to demo uvcopy 'rep2d' search/replace by table of old/new patterns
      #               - by Owen Townsend, UV Software, Feb15/2019
      #               - see www.uvsoftware.ca/uvdemos.htm
      # 01-30=search, 31-60=replace,61-80=qualifier_present,81-100=qualifier_absent
      #
      192.168.0.4~~~~~~~~~~~~~~~~~~~$IP4~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      192.168.0.5~~~~~~~~~~~~~~~~~~~$IP5~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      192.168.0.6~~~~~~~~~~~~~~~~~~~$IP6~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note
  • the # comment lines will be ignored by the 'rep2d' job.
  • the non-comment search/replace patterns will be shown in the audit file

 #2. uvcopy rep2d,fild1=parms,fild2=tmp1,fili3=ctl/IPreplace.tbl
                             ,filo1=tmp/parms_IPreplace.tbl
     ===============================================================
      - copy all files from parms/* to tmp1/... searching/replacing via the table
        of multiple patterns stored in ctl/IPreplace.tbl (specify with fili3=...)
      - here is the audit file, showing the search/replace table used & only
        the modified lines after replacements made
      Job: rep2d  InDir: parms  OutDir: tmp1  Table: ctl/IPreplace.tbl
      SearchBgn:    SearchEnd:    MatchOptns: df2g2  UserOptns: q1a1b99999c0d256e0p1
      ====================================================== 2019/02/15_11:41:25
      192.168.0.4~~~~~~~~~~~~~~~~~~~$IP4~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      192.168.0.5~~~~~~~~~~~~~~~~~~~$IP5~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      192.168.0.6~~~~~~~~~~~~~~~~~~~$IP6~~~~~~~~~~~~~~~~~~~~~~~~~~open~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      ====================================================== 2019/02/15_11:41:25
      00002 open $IP4
      00007                   1 reps @EOF: tmp1/ftpdemo1a
      00002 open $IP5
      00008                   1 reps @EOF: tmp1/ftpdemo1b
      00002 open $IP6
      00009                   1 reps @EOF: tmp1/ftpget1
      00002 open $IP6
      00009                   1 reps @EOF: tmp1/ftpput1
      ====================================================== 2019/02/15_11:41:25
       EOD: 00004 reps in 0004 files of 0004 total (00033 lines)
      Audit filename: tmp/parms_IPreplace.tbl

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_10 UVdemos.doc - Vancouver Utilities for any Unix/Linux user

Part 10 Contents

Part_10 documents items that were considered too long & complex for presentation earlier in the UVdemos documentation. We may have made brief references to these with links to see the full presentations here in Part_10.


10A1. Converting UTF-8 French Accents to unaccented equivalents for old applications
Files with high-bit characters may cause problems for older applications.
For Example - files with UTF-8 characters input to legacy appliciations,
not designed to handle them. This demo will show you how to identify them
with uvhd & convert them to unaccented characters with uvhd or uvcp.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A1. uvhd demo - investigating problem files

high-bit characters may cause problems for old applications

Files with high-bit characters may cause problems for older applications. For Example - files with UTF-8 characters input to legacy appliciations, not designed to handle them. This demo will show you how to identify them with uvhd & convert them to unaccented characters with uvhd or uvcp.

Let's suppose I get an email from Therese asking me how to convert accented French characters to unaccented equivalents. She knows the problem is that the new files are using UTF-8 character encoding & has identified the French accented characters & their UTF-8 codes in hexadecimal.

dat1/accents1 - French UTF-8 accented characters


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1. cat dat1/accents1   <-- display sample file with UTF-8 chars
     =================
      Dear Owen: Re: French Accents (dat1/accents1)
      11. cedilla      c ---> Ç ---> x'C387' ---> x'63' c
      21. accent acute e ---> é ---> x'C3A9' ---> x'65' e
      31. accent grave a ---> à ---> x'C3A0' ---> x'61' a
      32.              e ---> è ---> x'C3A8' ---> x'65' e
      33.              u ---> ù ---> x'C3B9' ---> x'75' u
      41. dieresis     e ---> ë ---> x'C3AB' ---> x'65' e
      42.              i ---> ï ---> x'C3AF' ---> x'69' i
      43.              u ---> ü ---> x'C3BC' ---> x'75' u
      51. circumflex   a ---> â ---> x'C3A2' ---> x'61' a
      52.              e ---> ê ---> x'C3AA' ---> x'65' e
      53.              i ---> î ---> x'C3AE' ---> x'69' i
      54.              o ---> ô ---> x'C3B4' ---> x'6F' o
      55.              u ---> û ---> x'C3BB' ---> x'75' u
      UTF-8 2 byte codes (1st byte x'C3')
      How can we convert to unaccented characters ?
      Thanks Thérèse
Note
  • If you have a printed copy of this documentation, you will probably see
    that the UTF-8 accented chars print differently than the screen display,
    probably because the printer is not setup for UTF-8 characters.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A2. High-Bit characters may cause problems for Legacy Applications

Investigating Accented Characters with 'uvhd'

Note
  • The file shown above would take too many lines in this documentation
  • so we will use a shorter version (dat1/accents2 vs dat1/accents1).

 #2. cat dat1/accents2   <-- display short version of UTF-8 sample file
     =================
      Dear Owen: Re: French Accents (dat1/accents2)
      1. accent grave a ---> à ---> x'C3A0' ---> x'61' a
      2. accent acute e ---> é ---> x'C3A9' ---> x'65' e
      3. accent grave e ---> è ---> x'C3A8' ---> x'65' e
      How can we convert to unaccented characters ?
      Thanks Thérèse

 #3. uvhd dat1/accents2 t  <-- investigate with 'uvhd'
     ====================    - using option 't' for Text files
 rec#=1 rsize=46 fptr=0 fsize=265 records=6
                   10        20        30        40
 r#     1 0123456789012345678901234567890123456789012345
        0 Dear Owen: Re: French Accents (dat1/accents2).
          4667247663256324766662466667722667326666677320
          45120F75EA025A0625E3801335E43084141F1335E4329A
 rec#=2 rsize=52 fptr=46 fsize=265 records=6
                   10        20        30        40        50
 r#     2 0123456789012345678901234567890123456789012345678901
       46 1. accent grave a ---> .. ---> x'C3A0' ---> x'61' a.
          32266666726767626222232CA222232724343222223272332260
          1E01335E4072165010DDDE0300DDDE087331070DDDE08761701A
 rec#=3 rsize=52 fptr=98 fsize=265 records=6
                   10        20        30        40        50
 r#     3 0123456789012345678901234567890123456789012345678901
       98 2. accent acute e ---> .. ---> x'C3A9' ---> x'65' e.
          32266666726677626222232CA222232724343222223272332260
          2E01335E4013545050DDDE0390DDDE087331970DDDE08765705A
 rec#=4 rsize=52 fptr=150 fsize=265 records=6
                   10        20        30        40        50
 r#     4 0123456789012345678901234567890123456789012345678901
      150 3. accent grave e ---> .. ---> x'C3A8' ---> x'65' e.
          32266666726767626222232CA222232724343222223272332260
          3E01335E4072165050DDDE0380DDDE087331870DDDE08765705A
 rec#=5 rsize=46 fptr=202 fsize=265 records=6
                   10        20        30        40
 r#     5 0123456789012345678901234567890123456789012345
      202 How can we convert to unaccented characters ?.
          4672666276266676772762766666676626667667677230
          8F7031E07503FE652404F05E1335E454038121345230FA
 rec#=6 rsize=17 fptr=248 fsize=265 records=6
                   10
 r#     6 01234567890123456
      248 Thanks Th..r..se.
          566667256CA7CA760
          481EB30483923835A

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A3. Investigating Accented Characters with 'uvhd'

Notes re UTF-8 French Accented Characters

We will use record #2 (isolated from 6 uvhd records above), as the basis for the notes following.

                   10        20        30        40        50
 r#     2 0123456789012345678901234567890123456789012345678901
       46 1. accent grave a ---> .. ---> x'C3A0' ---> x'61' a.
          32266666726767626222232CA222232724343222223272332260
          1E01335E4072165010DDDE0300DDDE087331070DDDE08761701A
  1. The vertical hex for grave 'a' under scale bytes 23 & 24 are shown again in horizontal hex under bytes 33-36 as x'C3A0', followed by the unaccented equivalent 'a' under scale 46-47 as x'61'.

  2. x'C3A0' is UTF-8 coding for grave 'a'. UTF-8 uses just 1 byte for the 1st 128 bytes of the ASCII code table. The next 128 (hi-bit) ASCII codes use 2 bytes & the 1st byte is x'C2' for 128-191 & x'C3' for 192-255. The French accents all have 1st byte x'C3'.

  3. So, to convert the French accented characters to unaccented equivalents, we could remove the 1st byte x'C3' & convert the 2nd byte to the unaccented equivalent. For this example grave a 2nd byte x'A0' to x'61' unaccented 'a'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A4. Updating files with uvhd

converting accented characters to unaccented equivalents

For this demo, we will convert only the subset of 3 French characters. We will use uvhd to convert the x'C3' to space x'20' & the 3 French accents as shown below:

      1. accent grave a ---> à ---> x'C3A0' ---> x'61' a
      2. accent acute e ---> é ---> x'C3A9' ---> x'65' e
      3. accent grave e ---> è ---> x'C3A8' ---> x'65' e

 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo
 #0c. cp dat1/accents2.bak dat1/accents2  <-- restore file if re-testing
      =================================

 #1. uvhd dat1/accents2 tu
     =====================
                  10        20        30        40
      r#    1 0123456789012345678901234567890123456789012345
            0 Dear Owen: Re: French Accents (dat1/accents2).
              4667247663256324766662466667722667326666677320
              45120F75EA025A0625E3801335E43084141F1335E4329A
              --- uvhd shows 1st record & prompts for your command:

 #1a. v99 x'C3',x'20'    <-- search/replace 1st byte UTF-8 to space
      ===============
      v99 x'C3',' '      <-- can code the space as character vs hex
      r#    6 01234567890123456
          248 Thanks Th .r .se.
              5666672562A72A760
              481EB30480920835A
      EOF, 6 read, 5 reps in 4 records, v99 x'C3',x'20'
      --- v99 searches/replaces entire file, shows last record updated,
          reports results as shown above, & prompts for next command

 #1b. v99 x'A0','a'  <-- search/replace grave 'a' with unaccented 'a'
      =============
      r#    2 0123456789012345678901234567890123456789012345678901
           46 1. accent grave a --->  a ---> x'C3A0' ---> x'61' a.
              3226666672676762622223226222232724343222223272332260
              1E01335E4072165010DDDE0010DDDE087331070DDDE08761701A
      EOF, 6 read, 1 reps in 1 records, v99 x'A0','a'

 #1c. v99 x'A9','e'  <-- search/replace acute 'e' with unaccented 'e'
      =============

 #1d. v99 x'A8','e'  <-- search/replace grave 'e' with unaccented 'e'
      =============

 #1e. q              <-- quit uvhd
      ===
Note
  • we have omitted showing results of #1c & #1d updates to save space
  • see the updated file on the page below:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A5. converting accented characters to unaccented equivalents

Results Before & After conversion


 #2. cat dat1/accents2.bak   <-- show demo file Before updates
     =====================
      Dear Owen: Re: French Accents (dat1/accents2)
      1. accent grave a ---> à ---> x'C3A0' ---> x'61' a
      2. accent acute e ---> é ---> x'C3A9' ---> x'65' e
      3. accent grave e ---> è ---> x'C3A8' ---> x'65' e
      How can we convert to unaccented characters ?
      Thanks Thérèse

 #3. cat dat1/accents2        <-- show demo file After updates
     =================
      Dear Owen: Re: French Accents (dat1/accents2)
      1. accent grave a --->  a ---> x'C3A0' ---> x'61' a
      2. accent acute e --->  e ---> x'C3A9' ---> x'65' e
      3. accent grave e --->  e ---> x'C3A8' ---> x'65' e
      How can we convert to unaccented characters ?
      Thanks Th er ese

Results of updates with uvhd are not perfect, because we converted the 1st byte of UTF-8 codes to space (as we see in 'Th er ese').

We could have removed 1st byte by converting to null (v99 x'C3','') but should not do with uvhd, because it updates in place & rewriting would shift the record terminating LineFeed x'0A' left 1, thus creating a space prior to the next record.

But, we can remove the 1st byte x'C3's by using uvcp because we will copy the file while removing x'C3's & converting the accents to equivalents.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A6. converting accented characters to unaccented equivalents

converting accented characters with 'uvcp'


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #0c. cp dat1/accents2.bak dat1/accents2  <-- restore file if re-testing
      =================================

 #1. rm -f tmp/*     <-- remove any old files from tmp/...
     ===========         (uvcp will write output file to tmp/...)

 #2. uvcp "fili1=dat1/accents2,rcs=256,typ=LSTt,filo1=tmp/accents2,rep=0(256):0xC3:\
                        ,rep=0(256):0xA0:0x61,rep=0(256):0xA9:0x65,rep=0(256):0xA8:0x65"
     ===================================================================================
      alternative --->  ,rep=0(256):0xA0:a,rep=0(256):0xA9:e,rep=0(256):0xA8:e"

 #3. cat tmp/accents2   <-- confirm output results as desired
     ================
      Dear Owen: Re: French Accents (dat1/accents2)
      1. accent grave a ---> a ---> x'C3A0' ---> x'61' a
      2. accent acute e ---> e ---> x'C3A9' ---> x'65' e
      3. accent grave e ---> e ---> x'C3A8' ---> x'65' e
      How can we convert to unaccented characters ?
      Thanks Therese
Note
  • We have provided the uvcp command as a script in demo/sf/fixaccents2
  • you can inspect the script & execute as follows:

 #4. cat sf/fixaccents2     <-- inspect uvcp script to remove accents
     ==================       - script same as manually entered uvcp command above
                                but with #comment explanations added
      #!/bin/ksh
      # fixaccents2 - uvcp script to convert French accents to unaccented equivalents
      #             - see demo in www.uvsoftware.ca/uvdemos.htm#10A1
      uvcp "fili1=?dat1/accents2,rcs=256,typ=LSTt,filo1=?tmp/accents2,rep=0(256):0xC3:\
                           ,rep=0(256):0xA0:0x61,rep=0(256):0xA9:0x65,rep=0(256):0xA8:0x65"
      #====================================================================================
      #   alternative ---> ,rep=0(256):0xA0:a,rep=0(256):0xA9:e,rep=0(256):0xA8:e"
      #
      # - could have coded the replacements as characters vs hex codes as shown above

 #5. sf/fixaccents2     <-- execute uvcp script to remove accents
     ==============

The easiest way to convert accented characters to unaccented equivalents would be to use 'uvfix1' the pre-programmed uvcopy job, that prompts for instructions (& saves writing the uvcopy job framework). We will run uvfix1 for dat1/accents1 which has all 13 French accents (vs accents2 demo with only 3).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A7. converting accented characters to unaccented equivalents

converting accented characters with 'uvfix1'


 #0a. login userxx --> /home/userxx
 #0b. cd demo      --> /home/usrxx/demo

 #1. uvfix1 dat1/accents1    <-- call uvfix1, arg1 specifies the file to be modified
     ====================      - console log with prompts shown below:
      uvfix1 - copy a text file & modify via uvcopy instruction keyins
             - input rec in area a has already been copied to output rec in b
      uop=q1c0n99r8192 - user option defaults
            c1         - pass thru any "#" col1 #comment records
              n99      - output record count limit (high# copies to EOF)
                 r8192 - recsize ONLY for Fixed-Length OUTPUT records if desired
                       - for Text files: accept default recsize & default typ=LSTt
      User OPtion (uop) defaults  = q1c0n999999999r8192
 #1a. -->null to accept or enter/override -->  <-- enter null to accept default options
      typ=LSTt default, null accept or enter new typ -->   <-- null accept default typ
      mvc  b27(25),a0  <-- sample instrns, dup cols 1-25 to 27-51
                         - moves from op2 area a to op1 area b
      enter a uvcopy instruction (period "." terminates entries)

 #1b. ---> rep b0(256),x'C3',''   <--- ENTER rep to remove 1st byte of UTF-8 pairs
           ====================
 #1c. ---> repm b0(256),x'87A9A0A8B9ABAFBCA2AAAEB4BB','ceaeueiuaeiou'
           =========================================================
            - ENTER repm to REPlace Multiple bytes with alternates
            - specifying searches in hexadecimal & replacements as characters
 #1d. ---> .                      <--- ENTER '.' to continue
           190225:140942:uvfix1: EOF fili01 rds=17 size=834: dat1/accents1
           190225:140942:uvfix1: EOF filo01 wrts=17 size=819: tmp/accents1
           check output in tmp/llsx, then move back to sf/llsx ?

 #2. cat tmp/accents1      <-- inspect output results
     ================
      Dear Owen: Re: French Accents (dat1/accents1)
      11. cedilla      c ---> c ---> x'C387' ---> x'63' c
      21. accent acute e ---> e ---> x'C3A9' ---> x'65' e
      31. accent grave a ---> a ---> x'C3A0' ---> x'61' a
      32.              e ---> e ---> x'C3A8' ---> x'65' e
      33.              u ---> u ---> x'C3B9' ---> x'75' u
      41. dieresis     e ---> e ---> x'C3AB' ---> x'65' e
      42.              i ---> i ---> x'C3AF' ---> x'69' i
      43.              u ---> u ---> x'C3BC' ---> x'75' u
      51. circumflex   a ---> a ---> x'C3A2' ---> x'61' a
      52.              e ---> e ---> x'C3AA' ---> x'65' e
      53.              i ---> i ---> x'C3AE' ---> x'69' i
      54.              o ---> o ---> x'C3B4' ---> x'6F' o
      55.              u ---> u ---> x'C3BB' ---> x'75' u
      UTF-8 2 byte codes (1st byte x'C3')
      How can we convert to unaccented characters ?
      Thanks Therese

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Permuted Index of Keywords from ** Headings **

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

1C2 $home: copy demo files to your $HOME/demo
3B2 $home: copy $UV/demo to your $HOME/demo/...
3B2 $uv: copy $UV/demo to your $HOME/demo/...

3E3 >: searching with > & < conditions
3E3 <: searching with > & < conditions

3I1 *arg: Notes Re selectfdd1 selections by *arg1 or arg2
3I2 *arg2: Notes selections by arg1 or *arg2 & options 's1t1'
3G1 *selectlfgfd1: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 *selectlfgfd2: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl

3C2 +2@canada: selects/dat1_nameadrs1_+1@Townsend+2@Canada-3@Owen

3B2 ...: copy $UV/demo to your $HOME/demo/...
3D4 ...: reports written to selects/... subdir
3F1 ...: 4 SQL "customer" scripts in sqls/...
3I1 ...: stats Report written to selects/...
3I2 ...: stats Report written to selects/...
4A1 ...: table3d report created in stats/...
1B2 .bash_profile: for userxx

10A1 Accented: dat1/accents1 - French UTF-8 accented characters
10A2 Accented: Investigating Accented Characters with 'uvhd'
10A3 Accented: Notes re UTF-8 French Accented Characters
10A4 Accented: converting accented characters to unaccented equivalents
10A6 Accented: converting accented characters with 'uvcp'
10A7 Accented: converting accented characters with 'uvfix1'
2C1 Accents: dat1/accents3 - UTF-8 accents, short test file
10A1 Accents1: dat1/accents1 - French UTF-8 accented characters
2C1 Accents3: dat1/accents3 - UTF-8 accents, short test file
1C1 Account: setup user account to run demos
3D3 Adm: executing sf/adm/testselectlf1
1B4 Advantages: Advantages of stub_profile,& common_profile
3A2 Advantages: SelectJob Advantages
3H1 And: selectlgfd1 - select JCL SYSINs with AND & OR conditions
3H2 And: selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)
10A1 Applications: high-bit characters may cause problems for old applications
2C1 Applications: high-bit characters may cause problems for old applications
3I2 Arg: Notes selections by arg1 or *arg2 & options 's1t1'
3D4 Arg1: Special Characters in arg1 patterns
3I1 Arg1: selectfdd1 Example#1 - multi-patterns on 1 line (arg1)
3I1 Arg1=pattern: select ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line
3I1 Arg2: Notes Re selectfdd1 selections by *arg1 or arg2
3I2 Arg2: selectfdd1 Example#2 - multi-patterns on ANY line (arg2)
3I2 Arg2=pattern: select ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line
4D1 Arguments: Notes re arguments & options
2D2 Ascii: Notes Re option 'a' translate EBCDIC to ASCII
2E1 Ascii: uvcp - translate to ASCII & insert LineFeeds at same time
2E1 Ascii: uvcp demo - translate to ASCII (& insert LineFeeds)
5B1 Audit: output audit file

4A1 Benefits: Notes, Benefits of table3d reports
3H2 Better: selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)
2C1 Brief: Brief Intro here, More demos later

3E1 Camps: dat1/camps - larger test file to demo Select Jobs
3E2 Camps: suggested patterns for searching dat1/camps
3E1 Canadian: sample selection from CAnadian Members of Parliament
3A2 Capabilitie: Example to Illustrate SelectJob Capabilities vs grep
2B4 Carriage: Fixing the problem - Removing Carriage Returns (CRs)
10A1 Cause: high-bit characters may cause problems for old applications
2C1 Cause: high-bit characters may cause problems for old applications
10A1 Characters: dat1/accents1 - French UTF-8 accented characters
10A1 Characters: high-bit characters may cause problems for old applications
10A2 Characters: Investigating Accented Characters with 'uvhd'
10A3 Characters: Notes re UTF-8 French Accented Characters
10A4 Characters: converting accented characters to unaccented equivalents
10A6 Characters: converting accented characters with 'uvcp'
10A7 Characters: converting accented characters with 'uvfix1'
2C1 Characters: high-bit characters may cause problems for old applications
3D4 Characters: Special Characters in arg1 patterns
4F4 Combine: Solution - combine SQL statements onto 1 line
4D1 Command: now execute the table3d command
3E3 Commands: Try some other commands such as
1B4 Common_profile: Advantages of stub_profile,& common_profile
1B3 Common_profile_abc: for VU users
3D1 Compare: dat1/nameadrs1 - compare demo outputs to these 8 records
3E3 Conditions: searching with > & < conditions
3H1 Conditions: selectlgfd1 - select JCL SYSINs with AND & OR conditions
3H2 Conditions: selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)
3B2 Copied: SelectJobs demo files copied to your homedir
1C2 Copy: demo files to your $HOME/demo
3B2 Copy: $UV/demo to your $HOME/demo/...
5B1 Copying: rep1d - search/replace 1 pattern copying all files to 2nd directory
5B2 Copying: rep2d search/replace multiple patterns copying all files to 2nd dir
4F5 Correct: now, use table3d to create correct table summary
4A1 Counts: TableJobs - create Table Summary Counts from Keywords/Targetwords
2B10 Crs: 4. Remove CRs with 'uvfix1' pre-programmed uvcopy job
2B11 Crs: 5. Remove CRs with 'uvfixA' pre-programmed uvcopy job
2B4 Crs: Fixing the problem - Removing Carriage Returns (CRs)
2B6 Crs: 1. Remove CRs using script 'removeCR'
2B6 Crs: Alternate solutions (vs uvhd) to Remove CRs from files
2B7 Crs: 2. Remove CRs using 'uvcp'
2B8 Crs: 3. Remove CRs using 'uvcopy'
3F1 Customer: 4 SQL "customer" scripts in sqls/...
3F1 Customer: select "customer" references from all scripts in directory

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

10A1 Dat1: dat1/accents1 - French UTF-8 accented characters
2C1 Dat1: dat1/accents3 - UTF-8 accents, short test file
3C1 Dat1: dat1/nameadrs1 - testdata file for 1st demo
3D1 Dat1: dat1/nameadrs1 - compare demo outputs to these 8 records
3E1 Dat1: dat1/camps - larger test file to demo Select Jobs
3E2 Dat1: suggested patterns for searching dat1/camps
3C2 Dat1_nameadrs1_+1@toselects/dat1_nameadrs1_+1@Townsend+2@Canada-3@Owen
1D1 Data: demo input files - DATA files
3D4 Data: Try selectlf1 with YOUR DATA files & PATTERNS
2B3 Default: Notes re uvhd (display above) with default options
1C2 Demo: copy demo files to your $HOME/demo
1C2 Demo: copy demo files to your $HOME/demo
1D1 Demo: input files - DATA files
1D3 Demo: input files - SQL scripts
1D4 Demo: input files - pf
1D4 Demo: input files - sf
1D5 Demo: input files - JCL
2B1 Demo: uvhd demo - investigating problem files
2D1 Demo: uvhd demo - investigating unknown files
2E1 Demo: uvcp demo - translate to ASCII (& insert LineFeeds)
3B2 Demo: SelectJobs demo files copied to your homedir
3B2 Demo: copy $UV/demo to your $HOME/demo/...
3B2 Demo: copy $UV/demo to your $HOME/demo/...
3C1 Demo: dat1/nameadrs1 - testdata file for 1st demo
3D1 Demo: dat1/nameadrs1 - compare demo outputs to these 8 records
3E1 Demo: dat1/camps - larger test file to demo Select Jobs
3H1 Demo: Notes re tts & selectlgfd1 demo
4E2 Demo: tableIP - simplified table summary demo
4E3 Demo: Notes re uvcopy demo job 'tableIP'
1C1 Demos: setup user account to run demos
2C1 Demos: Brief Intro here, More demos later
3D1 Demos: Test/Demos to illustrate Pattern Rules
3D3 Demos: script to run 10 selectlf1 test/demos
1C1 Dir: Alternative - setup subdir in your home dir
5B2 Dir: rep2d search/replace multiple patterns copying all files to 2nd dir
1C2 Directories: llr lists directories 1st
1C2 Directories: llr lists files within directories
3F1 Directory: select "customer" references from all scripts in directory
3F1 Directory: selectlfd1 - SELECT from all files in a directory
3I1 Directory: select ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line
3I2 Directory: select ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line
5A1 Directory: scan1d - scan all files in directory reporting matches to 1 pattern
5B1 Directory: rep1d - search/replace 1 pattern copying all files to 2nd directory
2B3 Display: Notes re uvhd (display above) with default options

2D2 Ebcdi: Notes Re option 'a' translate EBCDIC to ASCII
3I1 Entire: select ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line
3I2 Entire: select ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line
1B1 Env: 'profiles' provided in /home/uvadm/env
10A4 Equivalents: converting accented characters to unaccented equivalents
3A2 Example: Example to Illustrate SelectJob Capabilities vs grep
3C1 Example: selectlf1 - 1st Example for Pattern Rules following
3I1 Example: selectfdd1 Example#1 - multi-patterns on 1 line (arg1)
3I2 Example: selectfdd1 Example#2 - multi-patterns on ANY line (arg2)
4A1 Example: Example#1 - table summarize all IP#s in FTP parms
4C1 Example: Example#2 - option 'w#' TargetWord offset from KeyWord
4C2 Example: Example#3 - option 'y' no of words to table
4D1 Execute: now execute the table3d command
3D3 Executing: sf/adm/testselectlf1
3E2 Expected: Other searches showing only number of matches expected

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

2B4 Fixing: Fixing the problem - Removing Carriage Returns (CRs)
3C1 Following: selectlf1 - 1st Example for Pattern Rules following
3C2 Format: Report Format for SelectJobs
10A1 French: dat1/accents1 - French UTF-8 accented characters
10A3 French: Notes re UTF-8 French Accented Characters
4A1 Ftp: Example#1 - table summarize all IP#s in FTP parms

4D1 General: General Purpose Utilities
3A2 Grep: Example to Illustrate SelectJob Capabilities vs grep

2C1 Here: Brief Intro here, More demos later
10A1 High-bit: characters may cause problems for old applications
2C1 High-bit: characters may cause problems for old applications
1B1 Home: 'profiles' provided in /home/uvadm/env
1C1 Home: Alternative - setup subdir in your home dir
1A2 Homedir: Vancouver Utilities homedir contents
3B2 Homedir: SelectJobs demo files copied to your homedir

3A2 Illustrate: Example to Illustrate SelectJob Capabilities vs grep
3D1 Illustrate: Test/Demos to illustrate Pattern Rules
2E1 Insert: uvcp - translate to ASCII & insert LineFeeds at same time
2E1 Insert: uvcp demo - translate to ASCII (& insert LineFeeds)
1A3 Installing: Installing Vancouver Utilities - Summary
2C1 Intro: Brief Intro here, More demos later
3A1 Introduction: SelectJobs - Introduction & Overview
10A2 Investigating: Investigating Accented Characters with 'uvhd'
2B1 Investigating: uvhd demo - investigating problem files
2D1 Investigating: uvhd demo - investigating unknown files

4C1 Keyword: Example#2 - option 'w#' TargetWord offset from KeyWord
4A1 Keywords: TableJobs - create Table Summary Counts from Keywords/Targetwords

3E1 Larger: dat1/camps - larger test file to demo Select Jobs
2C1 Later: Brief Intro here, More demos later
3E2 Liberal: search CA MP file for Liberals in Vancouver
3G1 Line: selectlgfd1 - Multi-Condition-Series on any ONE line
3G2 Line: selectlgfd2 - Multi-Condition-Series on MULTI lines (vs 1 line)
3I1 Line: select ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line
3I1 Line: selectfdd1 Example#1 - multi-patterns on 1 line (arg1)
3I2 Line: select ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line
3I2 Line: selectfdd1 Example#2 - multi-patterns on ANY line (arg2)
4F4 Line: Solution - combine SQL statements onto 1 line
2E1 Linefeed: uvcp - translate to ASCII & insert LineFeeds at same time
2E1 Linefeeds: uvcp demo - translate to ASCII (& insert LineFeeds)
1C2 Lists: llr lists directories 1st
1C2 Lists: llr lists files within directories
1C2 Llr: lists directories 1st
1C2 Llr: lists files within directories
3B2 Locations: uvcopy job Locations

5A1 Matche: scan1d - scan all files in directory reporting matches to 1 pattern
3E2 Matches: Other searches showing only number of matches expected
10A1 May: high-bit characters may cause problems for old applications
2C1 May: high-bit characters may cause problems for old applications
3E1 Members: sample selection from CAnadian Members of Parliament
4F3 Missing: PROBLEM - some TABLE names missing ?
4C1 Most: useful options - option 'w#'
4C2 Most: useful options - option 'y#'
3E2 Mp: search CA MP file for Liberals in Vancouver
3G2 Multi: selectlgfd2 - Multi-Condition-Series on MULTI lines (vs 1 line)
3G1 Multi-condition-seriselectlgfd1 - Multi-Condition-Series on any ONE line
3G2 Multi-condition-seriselectlgfd2 - Multi-Condition-Series on MULTI lines (vs 1 line)
3I1 Multi-patterns: selectfdd1 Example#1 - multi-patterns on 1 line (arg1)
3I2 Multi-patterns: selectfdd1 Example#2 - multi-patterns on ANY line (arg2)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

3C1 Nameadrs1: dat1/nameadrs1 - testdata file for 1st demo
3D1 Nameadrs1: dat1/nameadrs1 - compare demo outputs to these 8 records
4D1 Now: execute the table3d command
4F5 Now: now, use table3d to create correct table summary
3E2 Number: Other searches showing only number of matches expected

4C2 Of: Example#3 - option 'y' no of words to table
4C1 Offset: Example#2 - option 'w#' TargetWord offset from KeyWord
10A1 Old: high-bit characters may cause problems for old applications
2C1 Old: high-bit characters may cause problems for old applications
3G1 One: selectlgfd1 - Multi-Condition-Series on any ONE line
4F4 Onto: Solution - combine SQL statements onto 1 line
2D2 Option: Notes Re option 'a' translate EBCDIC to ASCII
4C1 Option: Example#2 - option 'w#' TargetWord offset from KeyWord
4C1 Option: most useful options - option 'w#'
4C2 Option: Example#3 - option 'y' no of words to table
4C2 Option: most useful options - option 'y#'
2B3 Options: Notes re uvhd (display above) with default options
3I2 Options: Notes selections by arg1 or *arg2 & options 's1t1'
4B1 Options: table3d options
4C1 Options: most useful options - option 'w#'
4C2 Options: most useful options - option 'y#'
4D1 Options: Notes re arguments & options
3D1 Outputs: dat1/nameadrs1 - compare demo outputs to these 8 records
3A1 Overview: SelectJobs - Introduction & Overview

3E1 Parliament: sample selection from CAnadian Members of Parliament
4A1 Parms: Example#1 - table summarize all IP#s in FTP parms
3C1 Pattern: selectlf1 - 1st Example for Pattern Rules following
3D1 Pattern: Test/Demos to illustrate Pattern Rules
5A1 Pattern: scan1d - scan all files in directory reporting matches to 1 pattern
5B1 Pattern: rep1d - search/replace 1 pattern copying all files to 2nd directory
3D4 Patterns: Special Characters in arg1 patterns
3D4 Patterns: Try selectlf1 with YOUR DATA files & PATTERNS
3D4 Patterns: Try selectlf1 with YOUR PATTERNS
3E2 Patterns: suggested patterns for searching dat1/camps
5B2 Patterns: rep2d search/replace multiple patterns copying all files to 2nd dir
2B10 Pre-programmed: 4. Remove CRs with 'uvfix1' pre-programmed uvcopy job
2B11 Pre-programmed: 5. Remove CRs with 'uvfixA' pre-programmed uvcopy job
4E1 Pre-programmed: more about table3d, pre-programmed jobs vs specific uvcopy jobs
3G1 Prior: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 Prior: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl
2B1 Problem: uvhd demo - investigating problem files
2B4 Problem: Fixing the problem - Removing Carriage Returns (CRs)
3F1 Problem: PROBLEM Notes
4F1 Problem: PROBLEM in SQL scripts for table3d
4F3 Problem: PROBLEM - some TABLE names missing ?
10A1 Problems: high-bit characters may cause problems for old applications
2C1 Problems: high-bit characters may cause problems for old applications
1B1 Profiles: 'profiles' provided in /home/uvadm/env
4D1 Program: table3d - create summary table of programs in JCL
1B1 Provided: 'profiles' provided in /home/uvadm/env

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

3D1 Records: dat1/nameadrs1 - compare demo outputs to these 8 records
3F1 References: select "customer" references from all scripts in directory
3B1 Relevant: Vancouver Utilities - subdirs relevant to SelectJobs
2B10 Remove: 4. Remove CRs with 'uvfix1' pre-programmed uvcopy job
2B11 Remove: 5. Remove CRs with 'uvfixA' pre-programmed uvcopy job
2B6 Remove: 1. Remove CRs using script 'removeCR'
2B6 Remove: Alternate solutions (vs uvhd) to Remove CRs from files
2B7 Remove: 2. Remove CRs using 'uvcp'
2B8 Remove: 3. Remove CRs using 'uvcopy'
2B6 Removecr: 1. Remove CRs using script 'removeCR'
2B4 Removing: Fixing the problem - Removing Carriage Returns (CRs)
5B1 Rep1d: - search/replace 1 pattern copying all files to 2nd directory
5B2 Rep2d: search/replace multiple patterns copying all files to 2nd dir
5B1 Replace: rep1d - search/replace 1 pattern copying all files to 2nd directory
5B2 Replace: rep2d search/replace multiple patterns copying all files to 2nd dir
5A1 Reporting: scan1d - scan all files in directory reporting matches to 1 pattern
10A5 Results: Results Before & After conversion
2B4 Returns: Fixing the problem - Removing Carriage Returns (CRs)
3C1 Rules: selectlf1 - 1st Example for Pattern Rules following
3D1 Rules: Test/Demos to illustrate Pattern Rules
1C1 Run: setup user account to run demos
3D3 Run: script to run 10 selectlf1 test/demos

3I2 S1t1: Notes selections by arg1 or *arg2 & options 's1t1'
2E1 Same: uvcp - translate to ASCII & insert LineFeeds at same time
3E1 Sample: selection from CAnadian Members of Parliament
5A1 Scan: scan1d - scan all files in directory reporting matches to 1 pattern
5A1 Scan1d: - scan all files in directory reporting matches to 1 pattern
3E2 Search: CA MP file for Liberals in Vancouver
5B1 Search: rep1d - search/replace 1 pattern copying all files to 2nd directory
5B2 Search: rep2d search/replace multiple patterns copying all files to 2nd dir
3E2 Searches: Other searches showing only number of matches expected
3E2 Searching: suggested patterns for searching dat1/camps
3E3 Searching: with > & < conditions
3E1 Select: dat1/camps - larger test file to demo Select Jobs
3F1 Select: "customer" references from all scripts in directory
3F1 Select: selectlfd1 - SELECT from all files in a directory
3H1 Select: selectlgfd1 - select JCL SYSINs with AND & OR conditions
3H2 Select: selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)
3I1 Select: ENTIRE FILE to a 2nd directory - arg1=patterns on 1 line
3I2 Select: ENTIRE FILE to a 2nd directory - arg2=patterns on ANY line
3I1 Selectfdd1: Notes Re selectfdd1 selections by *arg1 or arg2
3I1 Selectfdd1: Example#1 - multi-patterns on 1 line (arg1)
3I2 Selectfdd1: Example#2 - multi-patterns on ANY line (arg2)
3E1 Selection: sample selection from CAnadian Members of Parliament
3I1 Selection: Notes Re selectfdd1 selections by *arg1 or arg2
3I2 Selections: Notes selections by arg1 or *arg2 & options 's1t1'
3A2 Selectjob: Example to Illustrate SelectJob Capabilities vs grep
3A2 Selectjob: SelectJob Advantages
3A1 Selectjobs: SelectJobs - Introduction & Overview
3B1 Selectjobs: Vancouver Utilities - subdirs relevant to SelectJobs
3B2 Selectjobs: SelectJobs demo files copied to your homedir
3C2 Selectjobs: Report Format for SelectJobs
3G1 Selectl: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 Selectl: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl
3C1 Selectlf1: - 1st Example for Pattern Rules following
3C2 Selectlf1: Output Report from selectlf1
3D3 Selectlf1: script to run 10 selectlf1 test/demos
3D4 Selectlf1: Try selectlf1 with YOUR DATA files & PATTERNS
3D4 Selectlf1: Try selectlf1 with YOUR PATTERNS
3G1 Selectlf1: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 Selectlf1: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl
3F1 Selectlfd1: - SELECT from all files in a directory
3H2 Selectlgfd: Notes re ttx vs tts & selectlgfd2 vs selectlgfd1
3G1 Selectlgfd1: - Multi-Condition-Series on any ONE line
3G2 Selectlgfd1: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl
3H1 Selectlgfd1: Notes re tts & selectlgfd1 demo
3H1 Selectlgfd1: - select JCL SYSINs with AND & OR conditions
3H2 Selectlgfd1: Notes re ttx vs tts & selectlgfd2 vs selectlgfd1
3G1 Selectlgfd2: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 Selectlgfd2: - Multi-Condition-Series on MULTI lines (vs 1 line)
3H2 Selectlgfd2: - select JCL SYSINs with AND & OR conditions (BETTER)
3C2 Selects: selects/dat1_nameadrs1_+1@Townsend+2@Canada-3@Owen
3D4 Selects: reports written to selects/... subdir
3I1 Selects: stats Report written to selects/...
3I2 Selects: stats Report written to selects/...
1C1 Setup: Alternative - setup subdir in your home dir
1C1 Setup: user account to run demos
2C1 Short: dat1/accents3 - UTF-8 accents, short test file
3E2 Showing: Other searches showing only number of matches expected
4E2 Simplified: tableIP - simplified table summary demo
4F4 Solution: Solution - combine SQL statements onto 1 line
2B6 Solutions: Alternate solutions (vs uvhd) to Remove CRs from files
3E3 Some: Try some other commands such as
4F3 Some: PROBLEM - some TABLE names missing ?
3D4 Special: Special Characters in arg1 patterns
4E1 Specific: more about table3d, pre-programmed jobs vs specific uvcopy jobs
1D3 Sql: demo input files - SQL scripts
3F1 Sql: 4 SQL "customer" scripts in sqls/...
4F1 Sql: PROBLEM in SQL scripts for table3d
4F1 Sql: table3d - create summaries of TABLEs in SQL scripts
4F4 Sql: Solution - combine SQL statements onto 1 line
3F1 Sqls: 4 SQL "customer" scripts in sqls/...
4F4 Statements: Solution - combine SQL statements onto 1 line
3I1 Stats: Report written to selects/...
3I2 Stats: Report written to selects/...
4A1 Stats: table3d report created in stats/...
1B4 Stub_profile: Advantages of stub_profile,& common_profile
1C1 Subdir: Alternative - setup subdir in your home dir
3D4 Subdir: reports written to selects/... subdir
3B1 Subdirs: Vancouver Utilities - subdirs relevant to SelectJobs
3E3 Such: Try some other commands such as
3E2 Suggested: patterns for searching dat1/camps
4F1 Summaries: table3d - create summaries of TABLEs in SQL scripts
4A1 Summarize: Example#1 - table summarize all IP#s in FTP parms
1A3 Summary: Installing Vancouver Utilities - Summary
4A1 Summary: TableJobs - create Table Summary Counts from Keywords/Targetwords
4D1 Summary: table3d - create summary table of programs in JCL
4E2 Summary: tableIP - simplified table summary demo
4F5 Summary: now, use table3d to create correct table summary
3H1 Sysins: selectlgfd1 - select JCL SYSINs with AND & OR conditions
3H2 Sysins: selectlgfd2 - select JCL SYSINs with AND & OR conditions (BETTER)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

4A1 Table: Example#1 - table summarize all IP#s in FTP parms
4A1 Table: TableJobs - create Table Summary Counts from Keywords/Targetwords
4C2 Table: Example#3 - option 'y' no of words to table
4D1 Table: table3d - create summary table of programs in JCL
4E2 Table: tableIP - simplified table summary demo
4F1 Table: table3d - create summaries of TABLEs in SQL scripts
4F3 Table: PROBLEM - some TABLE names missing ?
4F5 Table: now, use table3d to create correct table summary
4A1 Table3d: Notes, Benefits of table3d reports
4A1 Table3d: report created in stats/...
4B1 Table3d: options
4D1 Table3d: now execute the table3d command
4D1 Table3d: - create summary table of programs in JCL
4E1 Table3d: more about table3d, pre-programmed jobs vs specific uvcopy jobs
4F1 Table3d: PROBLEM in SQL scripts for table3d
4F1 Table3d: - create summaries of TABLEs in SQL scripts
4F5 Table3d: now, use table3d to create correct table summary
4E2 Tableip: tableIP - simplified table summary demo
4E3 Tableip: Notes re uvcopy demo job 'tableIP'
4A1 Tablejobs: TableJobs - create Table Summary Counts from Keywords/Targetwords
4C1 Targetword: Example#2 - option 'w#' TargetWord offset from KeyWord
4A1 Targetwords: TableJobs - create Table Summary Counts from Keywords/Targetwords
2C1 Test: dat1/accents3 - UTF-8 accents, short test file
3D1 Test: Test/Demos to illustrate Pattern Rules
3D3 Test: script to run 10 selectlf1 test/demos
3E1 Test: dat1/camps - larger test file to demo Select Jobs
3C1 Testdata: dat1/nameadrs1 - testdata file for 1st demo
3D3 Testselectlf1: executing sf/adm/testselectlf1
2B6 To: Alternate solutions (vs uvhd) to Remove CRs from files
2D2 Translate: Notes Re option 'a' translate EBCDIC to ASCII
2E1 Translate: uvcp - translate to ASCII & insert LineFeeds at same time
2E1 Translate: uvcp demo - translate to ASCII (& insert LineFeeds)
3D4 Try: Try selectlf1 with YOUR DATA files & PATTERNS
3D4 Try: Try selectlf1 with YOUR PATTERNS
3E3 Try: Try some other commands such as
3H2 Tt: Notes re ttx vs tts & selectlgfd2 vs selectlgfd1
3H1 Tts: Notes re tts & selectlgfd1 demo
3H2 Tts: Notes re ttx vs tts & selectlgfd2 vs selectlgfd1

10A4 Unaccented: converting accented characters to unaccented equivalents
2D1 Unknown: uvhd demo - investigating unknown files
2B5 Updating: Notes re Updating with uvhd
4C1 Useful: most useful options - option 'w#'
4C2 Useful: most useful options - option 'y#'
1C1 User: setup user account to run demos
1B3 Users: common_profile_abc for VU users
1B2 Userxx: .bash_profile for userxx
10A1 Utf-8: dat1/accents1 - French UTF-8 accented characters
10A3 Utf-8: Notes re UTF-8 French Accented Characters
2C1 Utf-8: dat1/accents3 - UTF-8 accents, short test file
1A2 Utilities: Vancouver Utilities homedir contents
1A3 Utilities: Installing Vancouver Utilities - Summary
3B1 Utilities: Vancouver Utilities - subdirs relevant to SelectJobs
4D1 Utilities: General Purpose Utilities
1B1 Uvadm: 'profiles' provided in /home/uvadm/env
2A1 Uvcopy: 6. uvcopy
2B10 Uvcopy: 4. Remove CRs with 'uvfix1' pre-programmed uvcopy job
2B11 Uvcopy: 5. Remove CRs with 'uvfixA' pre-programmed uvcopy job
2B8 Uvcopy: 3. Remove CRs using 'uvcopy'
2B9 Uvcopy: Notes re uvcopy jobs
3B2 Uvcopy: job Locations
4E1 Uvcopy: more about table3d, pre-programmed jobs vs specific uvcopy jobs
4E3 Uvcopy: Notes re uvcopy demo job 'tableIP'
10A6 Uvcp: converting accented characters with 'uvcp'
2A1 Uvcp:
2B7 Uvcp: 2. Remove CRs using 'uvcp'
2E1 Uvcp: - translate to ASCII & insert LineFeeds at same time
2E1 Uvcp: demo - translate to ASCII (& insert LineFeeds)
10A7 Uvfix1: converting accented characters with 'uvfix1'
2B10 Uvfix1: 4. Remove CRs with 'uvfix1' pre-programmed uvcopy job
2B11 Uvfixa: 5. Remove CRs with 'uvfixA' pre-programmed uvcopy job
10A2 Uvhd: Investigating Accented Characters with 'uvhd'
2A1 Uvhd:
2B1 Uvhd: demo - investigating problem files
2B3 Uvhd: Notes re uvhd (display above) with default options
2B5 Uvhd: Notes re Updating with uvhd
2B6 Uvhd: Alternate solutions (vs uvhd) to Remove CRs from files
2D1 Uvhd: demo - investigating unknown files
2A1 Uvlist:
2A1 Uvsort:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

1A2 Vancouver: Vancouver Utilities homedir contents
1A3 Vancouver: Installing Vancouver Utilities - Summary
3B1 Vancouver: Vancouver Utilities - subdirs relevant to SelectJobs
3E2 Vancouver: search CA MP file for Liberals in Vancouver
3G1 Vs: Notes Re *selectlfgfd1 (& selectlgfd2) vs prior selectlf1/selectl
3G2 Vs: Notes Re *selectlfgfd2 (& selectlgfd1) vs prior selectlf1/selectl
3G2 Vs: selectlgfd2 - Multi-Condition-Series on MULTI lines (vs 1 line)

1C2 Within: llr lists files within directories
4C2 Word: Example#3 - option 'y' no of words to table
3D4 Written: reports written to selects/... subdir
3I1 Written: stats Report written to selects/...
3I2 Written: stats Report written to selects/...

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Visitor Counters for ThisYear and LastYear