UVscripts - useful for any Unix/Linux site

UVdemos Part5 documents some of the most useful shell scripts used at UV Software. We will illustrate how useful these scripts can be to anybody who works with files & directories on the command line.

We will demo just a few of the most useful scripts here. You can see many more listed in scripts1.htm and if the Vancouver Utilities are installed at your site, you can see hundreds more in $UV/sf/... 4 subdirs (adm,demo,util,IBM).

And if you have the Utiltiies installed you can run the scripts with the demo files provided & check your results to the expected results listed.

UVdemos Part5 Contents


5A1. Initial Example (cfd) to get you interested in the cfd... scripts.

5B1. Setups & Checklist required to run UVscripts.
- Vancouver Utilities installed
- user login setup with stub profile calling common_profile in $UV/env/
- demo files copied to demo/ subdir in your homedir

5C1. Setup check-list - Ready to run UVscripts test/demos ?
See UVdemos.htm#Part_1 for complete setup procedures, but here is a short
list of the steps required before you can execute these UVscript test/demos.

5D1. 'cfd' scripts for Counting Files & Directories (& KiloBytes).
Test/Demos you can execute & compare your results to our results.

9 cfd... scripts as follows

      1. cfl   - Count Lines in 1 File
      2. cfd   - Count Files & kiloBytes in a Directory
      3. cfdt  - Totals-Only version of Count Files,Lines,& KiloBytes in a Directory
      4. cfdpf - Count Files in a Directory with a Pattern [or not] in filenames
      5. cfdpl - Count Files in a Directory with a Pattern [or not] on any line in any file
      6. cfdd  - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory
      7. cfddt - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory Totals-Only
      8. cfddf - Count Files,Lines,KB in ALL Sub-Dirs in a Super-Dir + 1st few files
      9. cfdmm - List Directory: File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#, Dir/Filename

5E1. cfshelp - Help screen to remind you of the 9 cfd scripts,
- just enter 'cfshelp' at your prompt

5E2. Vancouver Utility 'uvhelp' main menu & 'v12' sub-menu also include the help
menus for the cfd... scripts to count Lines & Files.

5F1. cfstest - script to run all 9 cfd scripts in batch mode
- reports collected in subdir rpts/... of current working directory

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

UVdemos Part5 Contents - continued


5G1. Recommendations using cfd... scripts on your own directories

5H1. Listings of cfd... scripts. If interested to see listings of all 9 scripts
please see https://uvsoftware.ca/scripts1.htm#Part_5.

5H2. Listing here of 1 script 'cfd' to count Files,lines,KB in 1 directory

5I1. Directory listing scripts based on 'ls -l' with various options for sorting
sequence by date/time, filesize, etc - llm,lla,lld,llr,llt,llu,etc.

5J1. 'rename' scripts - can save you a lot of manual keying when importing a lot
 of long awkward filenames from other systems (such as mainframes).
 Here are just a few of the 27 scripts available:
renameL - rename all filenames in a directory to lower case
renameU - rename all filenames in a directory to UPPER case
renameX - rename all files, changing 1 specified extension to an alternate
rename-X - remove specified extension from all files in directory
rename+X - add a specified extension to all files in directory

5K1. Remove/Insert Carriage Returns (removeCR & insertCR)
- often a problem when importing/exporting files from/to other systems.
 Special problem receiving scripts from other systems or by email
- the invisible CRs prevent scripts from executing on Unix/Linux systems

5L1. 'alldiff' scripts report differences between all files in directories
 Invaluable when making major changes to your software & datafiles
- save your programs & files before changes & reconversions
- run alldiff2 to confirm changes as intended & no unintended changes.
 'alldiff3' allows you to ignore differences due to date/time stamps, etc
- may add any unique pattern to a table of differences to be ignored.

5M1. 'chmod' scripts to correct file permissions in directory trees.
- automatically set permissions as desired on directories & files
  Essential for directories shared by multiple users
- directories must be 775 & files must be 664

5N1. lsold/mvold/rmold scripts to list/move/remove files older than x days.
 lsnew/mvnew/rmnew scripts to list/move/remove files newer than x days.
- can find files anywhere in your directories & list/move/remove as desired

5O1. 'dtree' script to draw directory tree illustrations.
- very useful for documentation (note usage in this documentation).

UVdemos.htm#Part_5 was duplicated to a separate document 'UVscripts.doc' because it is 1 of the most useful parts of UVdemos.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A1. UVscripts - Introduction & Overview

Initial Example to get you interested


 cfdmm dat1 - report Files,Lines,KB, Min/Max RecSize & RecNumbers in directory dat1/...
 ==========
 cfdmm dat1 - list File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#,Dir/Filename
  File#   Lines      KB MinRsiz MaxRsiz MinRnum MaxRnum  Directory/FileName
      1     335      32      95      95       1     335  dat1/CanadaMPs
      2      13       4      14      34      13       5  dat1/CanadaProvinces
      3       8       4      70      75       6       7  dat1/nameadrs1
      4     305      28      75      82       2       1  dat1/UScities
      5     539      40      71      74       2     422  dat1/UScongress
      6      50       4      13      23      15      40  dat1/USstates
      6    1250     112      13      95                  *Totals* in dat1
                            Report left in --> /home/uvadm/demo/rpts/dat1_cfdmm

UVscripts Benefits to Programmers & Analysts

  1. The 9 cfd... scripts automatically create their reports in subdir rpts/... & the reports are automatically named from the directory & script name. Any pattern arguments will be included in the rpts/filename, For example the report created by 'cfdp dat1 Canada*' will be 'rpts/dat1_cfdp_Canada_'.

  2. The reports include all relevant information - command line used, input directory, any patterns, date/time, full path-name of report created.

  3. The above conventions are convenient, you can run many reports without overwriting previous reports, and then use them to research problems or to assess the impacts of planned changes.

  4. The main objective of these demos is to give you the knowledge to run the scripts on your own directories to achieve your own objectives.

    Note - UVscripts separate & in UVdemos

UVscripts.htm is the same as UVdemos.htm#Part_5. UVscripts was duplicated as a separate document because it is 1 of the most useful parts of UVdemos.

Because UVscripts.doc is also a separate document from UVdemos.doc, the next 2 pages will duplicate the setup procedure for users to copy the demo files to a demo/ subdir within their homedir (rather than working in the $UV homedir.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B1. Vancouver Utility Programming Aids - UVscripts

Vancouver Utilities - subdirs relevant to UVscripts

Here is the Vancouver Utilities homedir showing only the most relevant subdirs required for the UVscript demos documented in this section of UVdemos.htm.

You could run the demos logged in as 'uvadm', but if you are sharing the system with other uvadm users, it would be better to copy the demo/subdirs/files to your own homedir. See the copy instructions below & an expanded view of the demo/... files on the next page.

 /home/uvadm             <-- $UV
 :-----bin                 - UV binaries (uvcopy,uvsort,uvcp,uvhd,uvlist,etc)
 :-----dat1                - test data files to demo various Vancouver Utilities
 :     :-----...         <-- 200 testdata files for all uvcopy job demos
 :     :                   - UVdemos testdata files copied to demo/dat1/...
 :-----doc                 - Vancouver Utilities documentation (text)
 :     :-----UVdemos.doc   - this documentation (1 of 150 files)
 :     :-----UVscripts.doc - Part5 of UVdemos dulicated as a separate document
 :-----dochtml             - documentation in HTML (uploaded to www.uvsoftware.ca)
 :-***-demo              <-- demo files for TestDemo tutorials, copy to your homedir -->
 :     :                   - copy /home/uvadm/demo/* to your /home/userxx/demo/...
 :     :-----dat1          - data files for UVscript demos
 :     :--. ..etc...       - see details on the next page
 :-----pf               <-- Parameter Files for uvcopy
 :     :-----adm               - uvcopy jobs for uvadm admin
 :     :-----demo              - uvcopy jobs for various demos
 :     :-----util              - uvcopy jobs for users
 :     :-----IBM               - uvcopy jobs for Mainframe conversions
 :-----sf                <-- Script Files (bash or Korn shell)
 :     :-----adm               - scripts for uvadm admin
 :     :-----demo              - demo scripts
 :     :-----util     <---***--- includes scripts (cfd,cfdd,etc) in this UVscripts.doc
 :     :-----IBM               - scripts for mainframe conversions
 :-----src               <-- Vancouver Utilities C source code (uvcopy,uvsort,etc)
 :-----tf                  - test files for various demos (similar to dat1/...)
 :-----tmp                 - tmp subdir (test/demo outputs)
 :-----tmp1                - misc, outdir for some utility scripts (like listall1)

Please copy the demo/... files to your homedir to avoid conflict with other users of Vancouver Utilities & to avoid losing your files when new versions of Vancouver Utilities are installed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B2. Vancouver Utility Programming Aids - UVscripts

UVscripts demo files copied to your homedir

Here is an expanded view of the most relevant subdirs/files required for the UVscript demos (omitting subdirs/files used for other parts of UVdemos.doc).

 /home/userxx/demo/             <-- your homedir demo directory
 :-----archive
 :     :---...misc...
 :-----backup                   <-- backup ParentDir
 :     :-----dat0                 - files in subdirs same as below
 :     :-----dat1
 :     :-----dat2
 :-----dat0                     <-- originals converted to dat1 fixed formats
 :     :-----CanadaMPs.csv
 :     :-----UScities.tab
 :     :-----UScongress.csv
 :     :-----USstates.csv
 :-----dat1                     <-- PRIMARY DEMO FILES for UVscripts
 :     :-----CanadaMPs
 :     :-----CanadaProvinces
 :     :-----nameadrs1
 :     :-----UScities
 :     :-----UScongress
 :     :-----USstates
 :-----dat2                     <-- N/A to UVscripts
 :     :-----accents1             - used for other parts of UVdemos
 :     :-----ar.custmas
 :     :-----ar.custmas.tellist
 :     :-----                     - several files omitted
 :     :-----nameadrs2
 :     :-----sales3
 :     :-----vendormas2
 :-----ftps
 :-----jcl0
 :-----jcl2   - N/A to UVscripts
 :-----jcl3
 :-----pf
 :     :-----CanadaMPscsv2txt   <-- uvcopy to convert .csv to fixed fields
 :     :-----UScitiescsv2txt
 :     :-----UScongresscsv2txt
 :     :-----USstatescsv2txt
 :-----rpts
 :     :-----dat1_cfd           <-- Reports created by UVscripts
 :     :----- ...                 - named from directory + scriptname
 :     :----- ...
 :-----sf    - scripts for other UVdemos
 :     :----- ... UVscripts are in $UV/sf/util/... (see prior page)
 :-----sql0  - N/A to UVscripts
 :-----sqls  - for other UVdemos
 :-----tmp

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5C1. Vancouver Utility Programming Aids - TableJobs

Setup check-list - Ready to run UVscripts test/demos ?

We will present a 'setup check-list' here, since you may be reading this in the separate UVscripts.doc vs UVdemos.doc which includes UVscripts as Part5. See UVdemos.htm#Part_1 for complete setup procedures, but here is a check-list of the steps required before you can execute these UVscript test/demos.

  1. Install Vancouver Utilities. See complete documenation at install.htm or the install summary in UVdemos.htm#1A3.

  2. Setup your user login account if not already done. See UVdemos.htm#1C1. Setup your user profile (.bash_profile or .profile for ksh). User profile in homedir calling common_profile in /home/appsadm/env/...

  3. Setup subdir demo/ in your homedir & copy $UV/demo/* to your $HOME/demo/... See instructions below. See the demo file directories on the previous page.

    copy $UV/demo to your $HOME/demo/...


 #1. Login userxx --> /home/userxx
 #2. mkdir demo   <-- make directory for copy of demo files in your homedir

 #3. cp -r /home/uvadm/demo/* demo/   <-- copy demo files to your homedir
     ==============================

 #4. cd demo    <-- change into demo/ directory
     =======      - see demo/... subdirs & files on prior paage
 :-----dat1     <-- PRIMARY FILES for UVscripts demos
 :     :-----CanadaMPs
 :     :-----CanadaProvinces
 :     :-----nameadrs1
 :     :-----UScities
 :     :-----UScongress
 :     :-----USstates

 #5. vi dat1/*   <-- look at demo file contents if you wish
     =========

Most of the cfd* scripts are not concerned with the file contents, but rather with counting the files, lines,& KB of the files. Some scripts such as 'cfdpf' selects files based on a pattern in the filenames. Only 1 script (cfdpl) selects files based on a pattern in the contents.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D1. Counting Files,Lines,& KB in directories

Run test/demos for 9 cfd* scripts


 #0a. Login userxx ---> /home/userxx
 #0b. cd demo      <--- change into the demo/directory

1. cfl - Count Lines & KB in 1 File


 #1. cfl $UV/src/uvcopy.c  <-- count lines in 1 file (uvcopy source program)
     ====================
      cfl /home/userxx/src/uvcopy.c - count lines in 1 file
       24400 lines,   1072 KB in /home/userxx/src/uvcopy.c
      Report left in --> /home/userxx/demo/rpts/_home_userxx_src_uvcopy.c_cfl

2. cfd - Count Files,lines,KB in 1 directory


 #2. cfd dat1   <-- Count Files,Lines,& KiloBytes in the dat1/ directory
     ========
      cfd dat1 - Count Files,Lines,KB in Directory
      File#    Lines      KB  Directory/Filename 20190726:1027
           1     335      32  dat1/CanadaMPs
           2      13       4  dat1/CanadaProvinces
           3       8       4  dat1/nameadrs1
           4     305      28  dat1/UScities
           5     539      40  dat1/UScongress
           6      50       4  dat1/USstates
      ******    1250     112  *Totals* in Directory /home/userxx/demo/dat1
                               Report left in --> /home/userxx/demo/rpts/dat1_cfd

3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)


 #3. cfdt dat1   <-- Count Files,Lines,& KiloBytes in dat1/... Totals-Only
     =========
      cfdt dat1 - Count Files,Lines,& KB in Directory - Totals-Only
       Files   Lines      KB  Directory/Filename 20190726:1027
           6    1250     112  *Totals-Only* in Directory /home/userxx/demo/dat1
                               Report left in --> /home/userxx/demo/rpts/dat1_cfdt

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D2. Counting Files,Lines,& KB in directories

4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filenames


 #4. cfdpf dat1 'Canada*'   <-- Count Files,Lines,KB for filenames begining with 'Canada'
     ====================
      cfdpf dat1 Canada* - Count Files,Lines,KB in Directory with Pattern [not] in filenames
      File#    Lines      KB  Directory/Filename 20190726:1027
           1     335      32  dat1/CanadaMPs
           2      13       4  dat1/CanadaProvinces
      ******     348      36  *Totals* in dat1 with pattern "Canada*"
                               Report left in --> /home/userxx/demo/rpts/dat1_cfdpf_Canada_

** 5. cfdpl - Count Files,Lines,KB with/without Pattern any line any file in dir **


 #5. cfdpl dat1 'Washington'   <-- Count Files,Lines,KB with 'Washington' on any line in any file
     =======================
      cfdpl dat1 Washington - Count Files,Lines,KB in Directory with Pattern [not] on any line in file
      File#    Lines      KB  Directory/Filename 20190726:1027
           1     305      28  dat1/UScities
           2      50       4  dat1/USstates
      ******     355      32  *Totals* in /home/userxx/demo/dat1 with  pattern "Washington"
                               Report left in --> /home/userxx/demo/rpts/dat1_cfdpl_Washington

6. cfdd - count Files,Lines,& KB in all subdirs of a superdir


 #6. cfdd $UV/sf    <-- Count Files,Lines,& KB in all Sub-Dirs of $UV/sf/...
     ===========
      cfdd  /home/userxx/sf - Count Files,Lines,& KB in SubDirs of SuperDir
        Dir#   Files   Lines      KB  SubDir/ParentDirectory  20190726:1027
           1     227    6502     944  /home/userxx/sf/adm
           2      86    2501     352  /home/userxx/sf/demo
           3     308   17102    1436  /home/userxx/sf/IBM
           4     541   21323    2248  /home/userxx/sf/util
      ******    1162   47428    4980  *Totals* for SubDirs of SuperDir /home/userxx/sf/...
                                       Report left in --> /home/userxx/demo/rpts/_home_userxx_sf_cfdd

7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-Only)


 #7. cfddt $UV/sf   <-- Count Files,Lines,& KB in all Sub-Dirs of $UV/sf/... Totals-Only
     ============
      cfddt /home/userxx/sf - Count Files,Lines,KB in SubDirs of Super-Dir - Totals-Only
        Dirs    Files   Lines      KB  SubDir/ParentDir 20190726:1027
           4     1162   47428    4980  *Totals* for SubDirs of SuperDir /home/userxx/sf/...
                                        Report left in --> /home/userxx/demo/rpts/_home_userxx_sf_cfddt

8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D3. Counting Files,Lines,& KB in directories


 #8. cfddf backup 2   <-- cfddf - for all Sub-Dirs in backup Dir + 1st 2 files of each subddir
     ==============
      cfddf backup 2 - Count Files,Lines,KB in SubDirs of SuperDir + 1st few files
        Dir#   Files   Lines      KB  SubDir/ParentDirectory  20190726:1027
           1       8    1299     268  backup/archive
                   1    1299      28  backup/archive/CanadaMPs.csv
                   2    1299       4  backup/archive/ftpsall
           2       6    1250     116  backup/dat1
                   1    1250      32  backup/dat1/CanadaMPs
                   2    1250       4  backup/dat1/CAprovinces
           3      24     211     108  backup/dat2
                   1     211       4  backup/dat2/accents1
                   2     211       4  backup/dat2/accents2
      ******      38    2760     492  *Totals* for SubDirs of SuperDir /home/userxx/demo/backup/...
                                       Report left in --> /home/userxx/demo/rpts/backup_cfddf

9. cfdmm - list directory: File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#


 #9. cfdmm dat1   <-- report Files,Lines,KB, Min/Max RecSize & RecNumbers in directory dat1/...
     ==========
      cfdmm dat1 - list File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#,Dir/Filename
       File#   Lines      KB MinRsiz MaxRsiz MinRnum MaxRnum  Directory/FileName
           1     335      32      95      95       1     335  dat1/CanadaMPs
           2      13       4      14      34      13       5  dat1/CanadaProvinces
           3       8       4      70      75       6       7  dat1/nameadrs1
           4     305      28      75      82       2       1  dat1/UScities
           5     539      40      71      74       2     422  dat1/UScongress
           6      50       4      13      23      15      40  dat1/USstates
           6    1250     112      13      95                  *Totals* in dat1
                                Report left in --> /home/userxx/demo/rpts/dat1_cfdmm

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E1. Counting Files,Lines,& KB in directories

cfshelp - Help screen for cfd... scripts

If you need a reminder of the 9 cfd scripts, just enter 'cfshelp' at yuour prompt.


 cfshelp    <-- display help screen reminders of cfd... scripts
 =======      - display as follows:
           ------- cfd... scripts & demo/dirs to count Files,Lines,KB -------
 1. cfl   $UV/src/uvcopy.c  # Count Lines & KB in the uvcopy utility program
 2. cfd   dat1              # Count Files,lines,KB in 1 directory (dat1)
 3. cfdt  dat1              # Totals-Only of cfd dat1 (6 files counts combined onto 1 line)
 4. cfdpf dat1 'Canada*'    # Counts for dat1/... filenames begining with 'Canada'
 5. cfdpl dat1 'Washington' # Counts for dat1/... files with 'Washington' on any line of the file
 6. cfdd  $UV/sf            # count Files,Lines,& KB for the 4 subdirs (adm,demo,util,IBM) of /home/userxx/sf
 7. cfddt $UV/sf            # Totals-Only of cfdd /home/userxx/sf (4 subdir counts combined onto 1 line)
 8. cfddf backup 2          # Counts for backup/... (archive,dat1,dat2) + 1st 2 files of each subdir
 9. cfdmm                   # Directory-Stats: files,lines,Minsize,Maxsize,Minrec#,Maxrec#
           ----- add 'export UVEXPERT=y' to your profile to inhibit these displays -----

In fact, you will get a prompt after you run a cfd... script, asking if you want to see this help screen - reply 'y' to display or null to ignore.

And when you know them & don't need them, you can inhibit them by adding 'export UVEXPERT=y' to your .bash_profile to inhibit future displays.

Help screens for each script (vs all 9 above)

You can see a help screen for each script, by entering the script-name only, without required arguments (omit directory). For example, if we enter just 'cfl'


 cfl     <-- enter script name without arguments (directory)
 ===       - displays ERROR msg, Correct Format,& sample report as follows:
 cfl - count lines in 1 file
 cfl filename                  <-- Usage ERROR, arg1 MUST be a FILE
 ============
 cfl /home/userxx/src/uvcopy.c  <-- Correct Format & Sample Report
 ============================
  24400 lines,   1072 KB in src/uvcopy.c
                         Report--> /home/userxx/rpts/src_uvcopy.c_cfl

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E2. Counting Files,Lines,& KB in directories

v12 - help menu for counting scripts

The 'v12' help menu is mentioned here because it also includes help menus for the cfd... scripts to count Lines & Files. v12 is just 1 of many help menus displayed by the 'uvhelp' main menu.

'help' screens are stored at $UV/help/... & it is in the profile PATH, so you can display the help/menu screen anytime anywhere, for example:


 uvhelp   <-- display main menu of help screens
 ======     - determine desired sub-menu code 'v12' in this case,
            - then quit the uvhelp main menu, & enter the sub-menu code 'v12'

 v12      <-- display sub-menu of 'counting lines & files' (shown below)
 ====
Note

v12 - counting Lines,Files,& KB in Directories

  1. cfl - Count Lines in 1 File
  2. cfd - Count Files & kiloBytes in a Directory
  3. cfdt - Totals-Only version of Count Files,Lines,& KiloBytes in a Directory
  4. cfdpf - Count Files in a Directory with a Pattern [or not] in filenames
  5. cfdpl - Count Files in a Directory with a Pattern [or not] on any line in any file
  6. cfdd - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory
  7. cfddt - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory Totals-Only
  8. cfddf - Count Files,Lines,KB in ALL Sub-Dirs in a Super-Dir + 1st few files
  9. cfdmm - List Directory: File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#, Dir/Filename

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5F1. Counting Files,Lines,& KB in directories

cfstest - run all 9 cfd scripts in batch mode

'cfstest' is a script to run all 9 cfd scripts in batch mode. All reports will be collected in subdir rpts/... of current working directory. You will be prompted to remove existing reports (reply to remove, null to keep).

 #!/bin/ksh
 # cfstest - run the 9 scripts to count Files,Lines,KB in directories
 #         - by Owen Townsend, UV Software, July 2019
 # cp -r $UV/demo $HOME  <-- assumes you have copied demo dir to your homedir
 # cd $HOME/demo         <-- ensure you are in the demo subdir
 export UVEXPERT=y   # inhibit prompt for cfshelp at end each cfs... script
 echo "cfstest - run 9 scripts to count Files,Lines,KB in directories"
 echo "cp -r $UV/demo $HOME  <-- assumes you copied demo dir to your homedir"
 echo "cd $HOME/demo         <-- ensure you are in the demo subdir"
 reply=x
 until [[ "$reply" == "y" || "$reply" == "n" ]]
   do echo " - reports will be collected in rpts/..., OK to remove all old rpts/* ? y/n"
      read reply; done
 if [[ "$reply" == "y" ]]; then rm -f rpts/*; fi
 cfl   $UV/src/uvcopy.c
 cfd   dat1
 cfdt  dat1
 cfdpl dat1 'Washington'
 cfdd  $UV/sf
 cfddt $UV/sf
 cfdpf dat1 'Canada*'
 cfddf backup 2
 cfdmm dat1
 echo "reports collected in rpts/* listed as folows:"
 cfd rpts    #<-- list reports with line counts
 echo " "
 echo "enter to see report contents via more, then spacebar for each report"; read reply
 more rpts/*
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5G1. Counting Files,Lines,& KB in directories

Recommendations using cfd scripts on your own directories

  1. Try the cfd... scripts on your own directories. Try cfd,cfdt,cfd[f,cfdpl,cfdmm on directories of files only & cfdd,cfddt,cfddf on super-directories that contain multiple sub-directories.

  2. The cfd reports are automatically created in subdir rpts/... of your current working directory, so you must stay above the directories you are reporting.

  3. The cfd scripts will create subdir 'rpts' if it does not already exist in your current working directory & you would not rpts to be created among your files. IE - it is best if your current working directory contains only directories & all your files are within their appropriate directories.

  4. The same argument applies to the './tmp' subdir that is automatically created by many of the Vancouver Utility scripts. We are not using the system '/tmp'. Many scripts create a tmp/ subdir in the current working directory if it does not exist - to create an 'ls' list of files to ensure processing alphabetically.

  5. Reports are automatically named from the directory & script name that created it. For example the report created by 'cfd dat1' will be 'rpts/dat1_cfd'.

  6. Any pattern arguments will be included in the rpts/filename, For example the report created by 'cfdp dat1 Canada*' will be 'rpts/dat1_cfdp_Canada_'. Any characters not allowed in filenames (such as '*') are converted to '_'.

  7. Actually, the last line of the report gives you the full pathname of the report. For example '/home/userxx/demo/rpts/dat1_cfdpf_Canada_'

  8. The above conventions are convenient, you can run many reports without overwriting previous reports, and then use them to research problems or to assess the impacts of planned changes.

  9. The reports include all relevant information - command line used, input directory, any patterns, date/time, full path-name of report created.

  10. Note that the main objective of these demos is to give you the knowledge to run these scripts on your own directories to achieve your objectives.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5H1. Counting Files,Lines,& KB in directories

Listings of cfd... scripts

If you are interested to see listings of the cfd... scripts, please see https://uvsoftware.ca/scripts1.htm#Part_5. You can click on the following links, but be warned that the "+" links will take you to separate document scripts1.htm.

To avoid losing your place in this UVdemos.doc, it is best to 'right-click' on the desired links & open in 'new-tab'. Then you will not get lost in scripts1.doc without an easy way back.


5B1+ cfl - Count Lines in 1 File

5C1+ cfd - Count Files & kiloBytes in a Directory

5D1+ cfdt - Totals-Only version of Count Files,Lines,& KiloBytes in a Directory

5E1+ cfdpf - Count Files in a Directory with a Pattern [or not] in filenames

5F1+ cfdpl - Count Files in a Directory with a Pattern [or not] on any line in any file

5G1+ cfdd - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory

5H1+ cfddt - Count Files,Lines,& KB in a ALL Sub-Dirs in a Super-Directory Totals-Only

5I1+ cfddf - Count Files,Lines,KB in ALL Sub-Dirs in a Super-Dir + 1st few files

5J1+ cfdmm - List Directory: File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#, Dir/Filename

In case you are curious, we will list just 1 of the 9 scripts on the following page:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5H2. scripts to manipulate text files

cfd - Count Files,lines,KB in 1 directory

 #!/bin/ksh
 # cfd - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # cfd - Count Files & kiloBytes in a Directory
 #     - by Owen Townsend, UV Software, updated July 2019
 echo "cfd - Count Files,Lines,& KiloBytes in a Directory"
 d1="$1";
 if [[ ! -d "$d1" ]]; then
    echo "cfd directory   <-- Usage ERROR, arg1 MUST be a DIRECTORY"
    echo "============="
    sleep 2
    echo "cfd dat1        <-- Correct Format & Sample Report"
    echo "========"
    echo "cfd - Count Files,Lines,& KiloBytes in a Directory"
    echo "File#    Lines      KB  Directory/Filename 20190722:0819"
    echo "     1     335      32  dat1/CanadaMPs"
    echo "     2      13       4  dat1/CAprovinces"
    echo "     3       8       4  dat1/nameadrs2"
    echo "     4     305      28  dat1/UScities"
    echo "     5     539      40  dat1/UScongress"
    echo "     6      50       4  dat1/USstates"
    echo "******    1250     112  *Totals* in Directory /home/userxx/demo/dat1"
    exit 1; fi
 #
 typeset -R6 files lines tlines fkb fkbt
 test -d rpts || mkdir rpts
 d2=$(echo $d1 | tr '/' '_')
 echo "File#    Lines      KB  Directory/Filename $(date +%Y%m%d:%H%M)" >rpts/${d2}_cfd
 for d1f in $d1/*
   { ((files+=1))
     lines=$(wc -l <$d1f)   # get line count
     ((tlines+=lines))      # count Total lines
     fdus=$(\du -s $d1f | tr '\t' ' '); fkb=${fdus%% *};
     if [[ -d $d1f ]]; then
        echo "$files  $lines  $fkb  ${d1f}/" >>rpts/${d2}_cfd
        else echo "$files  $lines  $fkb  $d1f" >>rpts/${d2}_cfd; fi
     ((fkbt+=fkb))
   }
 if [[ "$d1" == /* ]]; then fd1=$d1; else fd1=$PWD/$d1; fi
 echo "******  $tlines  $fkbt  *Totals* in Directory $fd1" >>rpts/${d2}_cfd
 echo "    - report left in --> $PWD/rpts/${d2}_cfd"
 more rpts/${d2}_cfd
 sleep 1
 cfshelp  # display help screen summary of 8 scripts to count Files,Lines,KB in directories"
 #--------# export UVEXPERT=n to inhibit
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5I1. Vancouver Utility scripts to list files in directories

List files using unix 'ls' options & more

l
  • 'l' (alias l='ls -l') saves keystrokes for this commonly used command
llm
  • List Long files in Directory & pipe to more (sorted by filename)
  • like 'ls -l | more', but allows argument directory (default current)
  • saves keystrokes & adds functionality to this often repeated command
lla
  • same as llm, but lists All files including hidden .files
llc
  • list all filenames in directory with file & line counts
lld
  • list directories only (omit files)
llr
  • list files & subdirs Recursively (all the way down)
llt
  • list files sorted by last modification/creation date/time
lltr
  • same as llt, but with Reverse sort (oldest 1st)
lls
  • list files sorted by Size, largest 1st
llsr
  • same as lls, but with Reverse sort, smallest 1st
llu
  • list files Unsorted, as stored in directory
spreadA
  • list filenames 4-up across
  • great for high volume scripts, programs, copybooks, etc

sample output for 'llc'


 llc $UV/sf/util           <-- list Vancouver Utility script filenames
 ===============             - with File & Line counts (25 files/screen)
 File#   Lines Directory/Filename
     1      43 sf/util//acum1
     2      22 sf/util//all2lower
     3      22 sf/util//all2upper
   454      52 sf/util//wrtbig
   454 files,   16680 total lines in  files in directory

more info on UV scripts

You can see a help screen for each script, by entering the script-name only, without its required arguments (omit the directory). See these scripts listed on pages http://uvsoftware.ca/scripts1.htm#3A1 - 3H1.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5I2. Vancouver Utility scripts to list files in directories

llt - sample of 20 scripts to list files

Here is just 1 of the scripts to list files in directories (based on unix 'ls' with various options). See more via the '+' links on the following page.

 #!/bin/ksh
 # llt - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # llt  - list directory long in date sequence with Newest first
 #      - pipe to more (show 1 screen & wait for null entry)
 #
 #usage with '*' wildcard patterns
 # llt dir/abc*      <-- does not work (will match only 1st occurrence)
 # llt 'dir/abc*'    <-- does work (enclose patterns in single quotes)
 #
 echo "llt - list directory in date sequence with Newest files 1ST"
 if (($# > 1));then
    echo "usage: llt directory"
    echo "       ============="
    echo "example: llt 'jcls/map*'"
    echo "         ==============="
    echo "- enclose any *wildcard patterns in single quotes"
    exit 1; fi
 #
 if [[ -z "$1" ]] ; then  d="."; else d=$1; fi
 ls -lt  $d | more
 #================
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5I3. Vancouver Utility scripts to list files in directories

All scripts to list files in directories

You can see All scripts at the "+" links in separate document scripts1.htm. To avoid losing your place in this UVdemos.doc, it is best to 'right-click' on the desired links & open in 'new-tab'. Then you will not get lost in scripts1.doc without an easy way back.


3A1+ llc - list files with File & Line Counts

3B1+ llm - list long & pipe to more
- similar to alias llm='ls -l | more' (but a script vs an alias)
- saves keystrokes for this often repeated command

3B2+ lla - same as llm, but lists All files including hidden .files

3C1+ llt - list files, with the latest First

3C2+ lls - list files in Size order

3C3+ llu - list files Unsorted

3D1+ lld - list directories only (omit files)

3E1+ llr - list files & subdirs Recursively (all the way down)

3F1+ lslp - list files & print immediately
- prints using 'uvlp12' - laser printing utility

3G1+ spreadA - list filenames 4-up across
- great for high volume scripts, programs, copybooks, etc

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5J1. Vancouver Utility scripts to Rename all files in directory

Rename scripts

'rename' scripts - can save you a lot of manual keying when you want to make mass chnages to filenames in a directory, often the case when you import a lot of long awkward filenames from other systems (such as mainframes).

We have an example in our demo files - demo/sql0/... files are from a mainframe and we modified them as shown in demo/sqls/... with the rename scripts. Here are the 1st 2 of the 11 filenames before & after renaming

         'SELECT.CUSTOMERS.ALL'
         'SELECT.CUSTOMERS.BASIC'
   --- 1st 2 before & after renaming ---
         select_customers_all.sql
         select_customers_basic.sql
  1. convert UPPER-case to lower-case
  2. remove the single quotes
  3. convert '.' period separators to '_' underscore separators
  4. add extension 'sql'

    Operating Procedures

Note that the 'rename' scripts rename the files within the existing directory. Since we wanted to preserve the originals so we could repeat the demos, we 1st copied the files from sql0/... to sql1/... before we renamed the scripts.


 #0a. Login userxx --> /home/userxx/
 #0b. cd demo -------> /home/userxx/demo/

 #0c. cp sql0/* sqls/    <-- copy files to 2nd dir (if originals must be preserved)
      ===============

 #1. renameL sqls        <-- rename all UPPER-case to lower-case
     ============

 #2. rename-QQ sqls      <-- remove single opening & closing quotes
     ==============

 #3. renameCC sqls       <-- convert separators '.' periods to '_' underscores
     =============

 #4. rename+X sqls .sql  <-- add extension '.sql' to all files
     ==================

alternate method to rename files

Also see 'uvfix1' at UVdemos.htm#7D1 which creates a script to do the same renaming as shown here by the 'rename...' scripts. uvfix1 requires some knowledge of the uvcopy instruciton set (vs the easy to use rename scripts), but could be useful for really complex situations.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5J2. Vancouver Utility scripts to Rename all files in directory

RenameL - 1 example of 37 in sf/util/...

Here is just 1 example of the 37 rename... scripts in $UV/util/...

 #!/bin/ksh
 # renameL - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # renameL - rename an entire directory of filenames to lower case
 #
 echo "rename all filenames in subdir to lower case"
 if [ -d "$1" ]; then :
    else echo "usage: renameL directory"
         echo "       ================="
         echo " - arg1 must be a directory"
         exit 1; fi
 #
 echo "will rename all files in $1 to lower case OK ? y/n"; read reply
 x=0; y=0
 for i in $1/*
 do let x=x+1
    f=${i##*/}
    typeset -l g=$f
    if [[ $g != $f ]]; then
       mv -i $1/$f $1/$g
       #================
       let y=y+1
       echo "file# $y (of $x) $1/$f - renamed to: $1/$g"
    fi
 done
 echo "total $y files in ${1}, $x renamed to lower case"
 exit 0

How to know there are 37 rename... scripts

I determined there were 37 rename... scripts by using the 'cfdpf' script documented at UVdemos.htm#5D2 as follows:


 cfdpf $UV/sf/util 'rename*'   <-- count files begining with 'rename'
 ===========================
 File#    Lines      KB  Directory/Filename 20190728:0955
      1      17       4  /home/uvadm/sf/util/rename1R
                            --- omitting 36 lines ---
     37      32       4  /home/uvadm/sf/util/rename-XX3
 ******    1322     152  *Totals* in /home/uvadm/sf/util with pattern "rename*"

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5J3. Vancouver Utility scripts to Rename all files in directory

Rename Script Listings - ALL

You can see All scripts at the "+" links in separate document scripts1.htm. To avoid losing your place in this UVdemos.doc, it is best to 'right-click' on the desired links & open in 'new-tab'. Then you will not get lost in scripts1.doc without an easy way back.


1B1+ renameL - translate filenames to lower case
1B2+ renameU - translate filenames to UPPER case
1B3+ renameLsd - translate filenames to lower case in all subdirs of a superdir

1C1+ renameP - replace existing prefix with alternate
1C2+ rename-P - remove any specified prefix
1C3+ rename-P. - remove specified prefix up to 1st '.'
1C4+ rename+P - add any specified prefix

1D1+ renameX - change any specified extension to any specified alternate
1D2+ rename.X - change specified .extension to any specified .alternate
1D3+ rename-X - remove any specified extension
1D4+ rename-.X - remove any specified extension back to last '.' (default)
- may specify an alternate extension (vs '.' default)
1D5+ rename+X - add any specified extension

1E1+ rename-AA - remove Any pattern Anywhere in filename
1E2+ renameAA - replace Any pattern with any Alternate

1F1+ renameB2U - change any embedded blanks to underscores
1F2+ renameD2U - change any '$' dollar signs to '_' underscores

1G1+ rename-QQ - remove any 'Quotes'
1G2+ rename-PP - remove any (Parenthesis)
1G3+ rename2PP - rename to whatever is in (Parenthesis)

1H1+ rename-TS - remove any timestamp _HHMMSS ('_' + 6 digits)
- preserves anything following _###### (.dat, .txt, etc)

1I1+ renameGDG - change mainframe GDG suffixes to Vancouver Utility gdg suffixes
- change '.G####V00' to '_00####'
1I2+ renameGDG1 - change '.G####V00' to '_000001' (keep only latest as 000001)

1J1+ renameCC - convert all occurrences of 1 character to a 2nd character
- example - convert all '.'s in filenames to '_'s

You can see a help screen for each script, by entering the script-name only, without its required arguments (omit the directory).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5K1. Vancouver Utility scripts

removeCR - remove Carriage Returns from text files

Carriage Returns x'0D' can be a problem on unix/linux systems especially for scripts that will not work with carriage returns. You might have this problem when receiving scripts from other systems or via email. The error-message is "bash: srriptname: /bin/bash^M: bad interpreter"

Note that the 'CR's are invisible using text editors, but you can see them with the 'uvhd' utility - as was demonstrated on pages '2B1' - 2B11. Here is just the 1st line of a script shown with uvhd.


 #5. uvhd sf/llsx t  <-- rerun with option 't' separate lines on LineFeeds x'0A'
     ==============
     rec#=1 rsize=13 fptr=0 fsize=244 records=7
                      10
 r#        1 0123456789012
           0 #!/bin/bash..
             2226662667600    <-- CR x'0D' causes problems
             31F29EF2138DA

removeCR - remove Carriage Returns from text files

 #!/bin/ksh
 # removeCR - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # removeCR - remove CR's Carriage Returns (x'0D')
 #          - by copy to tmp/... & prompt to copy back
 #          - by Owen Townsend, UV Software, Feb 2014
 #
 echo "removeCR  - copy a file to tmp/... removing CRs & nulls, prompt to copy back"
 echo "removeCR1 - alternate saves file in tmp/ before copy back removing CRs"
 df="$1";
 if [[ -f "$df" && -d tmp ]]; then :
 else echo "usage: removeCR subdir/file"
      echo "       ===================="
      echo " - arg1 must be a text file (with LFs & CRs)"
      echo " - current working directory must also contain a tmp/ subdir"
      echo " - will copy to tmp/ removing CRs & prompt to copy back"
      echo " - also see insertCR to insert CR's (create CR/LF)"
      exit 1; fi
 #
 f=$(basename $df)
 tr -d '\r' <$df >tmp/$f
 #==========================
 linesin=$(wc -l $df); sizein=$(du -bs $df);
 linesout=$(wc -l tmp/$f); sizeout=$(du -b tmp/$f);
 echo "input  lines & bytes:    $linesin $sizein"
 echo "output lines & bytes:    $linesout $sizeout"
 until [[ "$reply" = "y" || "$reply" = "n" ]]
    do echo "copy back overwriting original y/n ?"; read reply; done
 if [[ "$reply" = "y" ]]; then
    cp tmp/$f $df; echo "copied back to $df"; ls -l $df
 else echo "copy back aborted"; fi
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L1. Vancouver Utility scripts

alldiff2 & alldiff3 diff between all files in 2 directories

'alldiff2' repeats the unix system 'diff' utility for each pair of files found in two directories. This is great for confirming results of mass changes to directories of scripts, programs, or any kind of text files.

Before you make mass changes to directories of text files, it is a good idea to 1st save the original directory as '.old', perform the changes,& then run alldiff2 on the 2 directories.

This is really important when modifying utility programs that make the mass changes to hundreds or thousands of files (such scripts, programs, etc). alldiff2 & alldiff3 can make the difference between your project's success or failure, since they can confirm that you have made only the intended changes & no unintended changes.

We will illustrate alldiff2 using the 'jcl3' & jcl3.old' directories in $UV/demo/... Assume that we have modified the JCL converter & are ready to run the new version to convert all JCLs in jcl2/... to JCL/scripts in jcl3/... We proceed as follows:

Note
  • You can not run #1, #2, #3, But you can run #4 & #5
  • #3 jclxx51 (JCL conversion script) can not run in the demo directory
    because it requires other files not supplied in teh demo directory.

 #1. mv jcl3 jcl3.old   <-- change name of existing converter outputs
     ================

 #2. mkdir jcl3         <-- make new empty directory for new outputs
     ==========

 #3. jclxx51 jcl2 jcl3  <-- re-convert JCL with modified version of JCL converter
     =================

 #4. alldiff2 jcl3.old jcl3  <-- run alldiff2 to create tmp/jcl3.dif
     ======================

 #5. vi tmp/jcl3.dif         <-- investigate the dif report
     ===============

See tmp/jcl3.dif listed on the enxt page:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L2. alldiff2 & alldiff3 for all files in 2 directories

alldiff2 report - tmp/jcl3.dif

 39c39
 < exit $JCC #jclunix51 version=20190119, converted 20190728:145940 sitename=UV Software
 ---
 > exit $JCC #jclunix51 version=20190119, converted 20190728:155658 sitename=UV Software
 diff file# 1 - jcl3.old/... vs jcl3/jar100.ksh
 56c56
 < exportfile  SALELST data1/ar.sales.list #(,CATLG,DELETE)
 ---
 > exportfile  SALELST data1/ar.sales.list200 #(,CATLG,DELETE)
 68c68
 < exit $JCC #jclunix51 version=20190119, converted 20190728:145940 sitename=UV Software
 ---
 > exit $JCC #jclunix51 version=20190119, converted 20190728:155658 sitename=UV Software
 diff file# 2 - jcl3.old/... vs jcl3/jar200.ksh
 46c46
 < exit $JCC #jclunix51 version=20190119, converted 20190728:145940 sitename=UV Software
 ---
 > exit $JCC #jclunix51 version=20190119, converted 20190728:155658 sitename=UV Software
 diff file# 3 - jcl3.old/... vs jcl3/jgl100.ksh
       ------ omitting reports for file#s 3,4,5,6 (similar to all except #2) ------
 70c70
 < exit $JCC #jclunix51 version=20190119, converted 20190728:145940 sitename=UV Software
 ---
 > exit $JCC #jclunix51 version=20190119, converted 20190728:155658 sitename=UV Software
 diff file# 7 - jcl3.old/... vs jcl3/jgl320.ksh
 7 different of 7 files compared jcl3.old to jcl3

only #2 differences are significant

You can see that the only significant differences are in file #2. The differences in the other files are due only to the date/time stamps inserted by the JCL converter.

See alternative script 'alldiff3' which adds another step to alldiff2 to eliminate such differences using a table of keywords such as 'version'. When you run alldiff3 you will be prompted for other keywords to cause elimination, &/or you can add your keywords to the table for future alldiff3s.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L3. alldiff2 & alldiff3 for all files in 2 directories

run alldiff3 to drop insignicant differences

The operating procedures are similar to the above - just change the script name from 'alldiff2' to 'alldiff3'.


 #4. alldiff3 jcl3.old jcl3  <-- run alldiff3 to create tmp1/jcl3.dif
     ======================

You will get more console messages that for alldiff2, such as the prompt for other keywords to cause difference eliminations (not shown here). Note that alldiff3 creates the same report as alldiff2 in tmp/... And then it creates a 2nd report in tmp1/... with the differences eliminated.


 #5. vi tmp1/jcl3.dif   <-- investigate the alldiff3 report
     ================     - with insignificant differences eliminated

alldiff3 report - tmp/jcl3.dif

56c56 < exportfile SALELST data1/ar.sales.list #(,CATLG,DELETE) > exportfile SALELST data1/ar.sales.list200 #(,CATLG,DELETE) diff file# 2 - jcl3.old/... vs jcl3/jar200.ksh 1 significant difference in 7 files compared jcl3.old to jcl3

alldiff3 listed on next page

We will list only 'alldiff3' on the following page. It is similar to alldiff2 with the addition of a uvcopy job 'diffdrop2' which eliminates the insignifcant differences caused by date/time stamps, versions, etc.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L4. alldiff2 & alldiff3 for all files in 2 directories

alldiff3 script listing

 #!/bin/ksh
 # alldiff3 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # alldiff3 - script to compare all text files in 1 directory to a 2nd directory
 #          - redirects output to tmp/dir2.dif & prompts to view/print/etc
 #Jul11/17 - add diffdrop2 script to drop insignificant lines on JCL/script diffs
 #         - ver:, UVSW, diff file# line, empty diffs after pattern drops
 #
 d1="$1"; d2="$2";
 echo "alldiff3 - compare all files in 2 subdirs & write all diffs to tmp/dir2.dif"
 echo " - then drop diffs due to time stamps & table of patterns (prompt to add)"
 if [[ -d "$1" && -d $2 ]]; then :
 else echo "usage: alldiff3 dir1 dir2"
      echo "       =================="
      echo "example: alldiff3 cbls.old cbls"
      echo "         ======================"
      exit 1; fi
 #
 d2b=$(basename $d2)   # get basename of dir2 (drop any preceding /path/...)
 log=tmp/$d2b.dif      # make name for output log file
 log2=tmp1/$d2b.dif    # make name for diffdrop2 outputfile
 >$log                 #init logfile in tmp subdir w same name as dir2 + .dif
 x=0; y=0;
 for i in $d1/*
 do
   let x=x+1
   typ=$(file $i)
   if [[  $typ == *text* || $typ == *script* || $typ == *data* ]]
      then f=${i##*/}
           diff -a -bBw $d1/$f $d2/$f >>$log
           #=================================
           if [[ $? -gt 0 ]]; then
              echo "diff file# $x - $d1/... vs $d2/$f" >>$log
              echo " " >>$log
              let y=y+1
           fi
      else echo " file# $x $typ - NOT text/script file" >>$log
   fi
 done
 lines=$(wc -l $log)      # capture line count
 echo "$y different of $x files compared $d1 to $d2" >>$log
 echo "$y diff $x files in $d1 & $d2, report: $log $lines lines"
 echo "--> enter additional drop patterns 'arg1=xxx,arg2=yyy' or null"
 read args
 uvcopy diffdrop2,fili1=$log,filo1=$log2,$args
 #============================================
 echo "--> enter command to view (vi,cat,more,uvlp12,etc,or null)"
 read cmd
 if [[ ! "$cmd" = "" ]]; then $cmd $log2; fi
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5M1. system/application administration scripts

chmod1 - set perms on all levels - using 'find'

 #!/bin/ksh
 # chmod1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/...
 # chmod1 - change permissions on subdirs & files under a specified superdir
 #        - using 'find' to process all levels of directories & files
 #        - need to manually add 'x' perms on any bin/* & script/* dirs
 #        - by Owen Townsend, April 26/2006
 #
 # chmod1 directory dir-perms file-perms   <-- command format
 # ======================================
 # chmod1 directory 775 664                <-- recommended permissions
 # =========================
 #
 # - also see alternatives chmod2 & chmod3 (to this *chmod1)
 #*chmod1 - change perms on dirs & files, using 'find'
 # chmod2 - change perms on dirs & files, using 'recursion'
 # chmod3 - change perms on dirs & files, using 'recursion'
 #        - sets 'x' perm by testing for known names of bin/ & script/ subdirs
 #
 # After running chmod1 or chmod2, you must manually fix permissions on
 # executable programs & scripts via -->chmod 775 bin/*; chmod 775 scripts/*
 #
 # capture arguments & force perms integers
 dir="$1";
 typeset i dperm="$2"; typeset i fperm="$3";
 #
 # ensure arg1 is directory & length of perms are 3 digits
 dpl=${#dperm}; fpl=${#fperm};
 #
 if [[ -d "$dir" ]] && ((dpl==3 && fpl==3)); then :
    else echo "usage: chmod1 directory dir-perms file-perms"
         echo "       ====================================="
         echo "example: chmod1 dirxx 775 664"
         echo "         ===================="
         echo " - arg1 must be dir, args 2 & 3 must be 3 digits"
         exit 90; fi
 #
 echo -n "chmod1: set perms $dperm/$fperm from: $dir - enter to continue"
 read reply
 #
 find $dir -type d -exec chmod $dperm {} \;
 #=========================================
 #
 find $dir -type f -exec chmod $fperm {} \;
 #=========================================
 #
 echo "chmod1: perms set $dperm/$fperm for all subdirs & files within: $dir"
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5N1. lsold/mvold/rmold scripts

      lsold/mvold/rmold scripts to list/move/remove files older than x days.
      lsnew/mvnew/rmnew scripts to list/move/remove files newer than x days.
      - can find files anywhere in your directories & list/move/remove as desired

lsold1 - list files older than x days

 #!/bin/ksh
 # lsold1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # lsold1 - list files older than a specified number of days
 #
 if [[ -d "$1" && -n "$2" ]]; then :
    else echo "usage: lsold1 directory days"
         echo "       ====================="
         echo "example: lsold1 /u2/apps/data 30"
         echo "         ======================="
         echo "- list files in directory tree older than 30 days"
         echo " "
         echo "example2: lsold1 . 30  (use '.' if in directory)"
         echo "          ==========="
         echo "- Also see 'lsold1p' that allows a filename matching pattern"
         exit 1; fi
 days="+$2"
 #
 find $1 -mtime $days -type f -exec ls -l {} \;
 #=============================================
 # - list files older than a specified number of days
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

rmold1 - remove files older than x days

 #!/bin/ksh
 # rmold1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # rmold1 - remove files from a specified directory (& all subdirs),
 #          that are older(+) than a specified number of days
 #        - option to prompt y/n for each file to be removed
 #
 # 3 variations of this script as follows:
 #*rmold1 (this) - remove old files, option to prompt for each file to be removed
 # rmold2 - lists all old files, then prompts for each old file to be removed
 # rmold3 - lists all old files, then prompts once to remove all old files
 #
 # Also see:
 # lsold1 - list files older than specified number of days
 #        - might run this script before 'rmold1' (removes files)
 #          to prove files to be removed are as intended
 # mvold1 - might run this script instead of 'rmold1' (removes files)
 #          so you could recover files from the move to dir if required
 #
 if [[ -d "$1" && "$2" == +* && ("$3" == "-i" || "$3" == "-f") ]]; then :
    else echo "usage: rmold1 directory +days option"
         echo "       ============================="
         echo "example rmold1 history +730 -i"
         echo "        ======================"
         echo " - remove files over 2 yrs old, prompting for confirmation"
         echo " - arg1 must be a directory"
         echo " - arg2 must start with + (older than)"
         echo " - arg3 must be '-i' for confirmation, '-f' for none"
         exit 1; fi
 #
 find $1 -mtime $2 -exec rm $3 {} \;
 #==================================
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5O1. 'dtree' script to draw directory trees

 'dtree' script to draw directory tree illustrations.
 - very useful for documentation (note usage in this documentation).

dtree - create directory tree illustration

 #!/bin/ksh
 # dtree - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # dtree - list a directory tree
 #       - contributed by Howard Lobsinger (Peacock Engineering, Montreal)
 #usage: dtree directory
 #       ===============
 D=${1:-`pwd`}
 (cd $D; pwd)
 find $D -type d -print | sort |
 sed -e "s,^$D,,"\
     -e "/^$/d"\
     -e "s,[^/]*/\([^/]*\)$,\:-----\1,"\
     -e "s,[^/]*/,:     ,g" | more
 exit 0

dtree example


 #1. Login as yourself (or anybody)

 #2. dtree /home/uvadm/demo
     ======================
 /home/uvadm/demo
 :-----archive
 :     :-----bakup_dat1
 :     :-----bakup_dat2
 :-----dat1
 :-----dat2
 :-----ftps
 :-----jcl0
 :-----jcl2
 :-----jcl3
 :-----pf
 :-----rpts
 :-----selects
 :-----sf
 :-----sql0
 :-----sqls
 :-----stats
 :-----tf
 :-----tmp

We illustrate using the demo directory - a small directory in /home/uvadm.


 #3. dtree /home/uvadm   <-- would create about 600 lines
     =================     - not shown here, but you can run & observe

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Permuted Index of Keywords from ** Headings **

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

5C1 $home: copy $UV/demo to your $HOME/demo/...
5C1 $uv: copy $UV/demo to your $HOME/demo/...

5C1 ...: copy $UV/demo to your $HOME/demo/...
5J2 ...: RenameL - 1 example of 37 in sf/util/...

5D2 A: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5L1 Alldiff2: & alldiff3 diff between all files in 2 directories
5L2 Alldiff2: report - tmp/jcl3.dif
5L1 Alldiff3: alldiff2 & alldiff3 diff between all files in 2 directories
5L3 Alldiff3: listed on next page
5L3 Alldiff3: report - tmp/jcl3.dif
5L3 Alldiff3: run alldiff3 to drop insignicant differences
5L4 Alldiff3: script listing
5A1 Analysts: UVscripts Benefits to Programmers & Analysts
5J2 Are: How to know there are 37 rename... scripts
5L2 Are: only #2 differences are significant

5F1 Batch: cfstest - run all 9 cfd scripts in batch mode
5A1 Benefits: UVscripts Benefits to Programmers & Analysts

5K1 Carriage: removeCR - remove Carriage Returns from text files
5K1 Carriage: removeCR - remove Carriage Returns from text files
5D1 Cfd: 2. cfd - Count Files,lines,KB in 1 directory
5F1 Cfd: cfstest - run all 9 cfd scripts in batch mode
5G1 Cfd: Recommendations using cfd scripts on your own directories
5H2 Cfd: - Count Files,lines,KB in 1 directory
5D1 Cfd*: Run test/demos for 9 cfd* scripts
5E1 Cfd...: cfshelp - Help screen for cfd... scripts
5H1 Cfd...: Listings of cfd... scripts
5D2 Cfdd: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5D2 Cfddf: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5D2 Cfddt: 7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-On
5D3 Cfdmm: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5D2 Cfdpf: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5D1 Cfdt: 3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)
5D1 Cfl: 1. cfl - Count Lines & KB in 1 File
5E1 Cfshelp: - Help screen for cfd... scripts
5F1 Cfstest: - run all 9 cfd scripts in batch mode
5C1 Check-list: Setup check-list - Ready to run UVscripts test/demos ?
5M1 Chmod1: - set perms on all levels - using 'find'
5B2 Copied: UVscripts demo files copied to your homedir
5C1 Copy: $UV/demo to your $HOME/demo/...
5D1 Count: 1. cfl - Count Lines & KB in 1 File
5D1 Count: 2. cfd - Count Files,lines,KB in 1 directory
5D1 Count: 3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)
5D2 Count: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5D2 Count: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5D2 Count: 7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-On
5D2 Count: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5H2 Count: cfd - Count Files,lines,KB in 1 directory
5E2 Counting: v12 - counting Lines,Files,& KB in Directories
5E2 Counting: v12 - help menu for counting scripts

5N1 Days: lsold1 - list files older than x days
5B2 Demo: UVscripts demo files copied to your homedir
5C1 Demo: copy $UV/demo to your $HOME/demo/...
5C1 Demo: copy $UV/demo to your $HOME/demo/...
5C1 Demos: Setup check-list - Ready to run UVscripts test/demos ?
5D1 Demos: Run test/demos for 9 cfd* scripts
5L1 Diff: alldiff2 & alldiff3 diff between all files in 2 directories
5L2 Differences: only #2 differences are significant
5L3 Differences: run alldiff3 to drop insignicant differences
5D2 Dir: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5E2 Directories: v12 - counting Lines,Files,& KB in Directories
5G1 Directories: Recommendations using cfd scripts on your own directories
5I3 Directories: All scripts to list files in directories
5L1 Directories: alldiff2 & alldiff3 diff between all files in 2 directories
5D1 Directory: 2. cfd - Count Files,lines,KB in 1 directory
5D1 Directory: 3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)
5D3 Directory: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5H2 Directory: cfd - Count Files,lines,KB in 1 directory
5O1 Directory: dtree - create directory tree illustration
5L3 Drop: run alldiff3 to drop insignicant differences
5O1 Dtree: - create directory tree illustration
5O1 Dtree: example

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

5A1 Example: Initial Example to get you interested
5J2 Example: RenameL - 1 example of 37 in sf/util/...
5O1 Example: dtree example

5D2 Few: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5D2 Filename: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5M1 Find: chmod1 - set perms on all levels - using 'find'

5A1 Get: Initial Example to get you interested

5E1 Help: Help screens for each script (vs all 9 above)
5E1 Help: cfshelp - Help screen for cfd... scripts
5E2 Help: v12 - help menu for counting scripts
5B2 Homedir: UVscripts demo files copied to your homedir
5J2 How: How to know there are 37 rename... scripts

5O1 Illustration: dtree - create directory tree illustration
5D1 In: 1. cfl - Count Lines & KB in 1 File
5D1 In: 2. cfd - Count Files,lines,KB in 1 directory
5D1 In: 3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)
5D2 In: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5D2 In: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5D2 In: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5E2 In: v12 - counting Lines,Files,& KB in Directories
5H2 In: cfd - Count Files,lines,KB in 1 directory
5I1 Info: more info on UV scripts
5L3 Insignicant: run alldiff3 to drop insignicant differences
5A1 Interested: Initial Example to get you interested

5L2 Jcl3.dif: alldiff2 report - tmp/jcl3.dif
5L3 Jcl3.dif: alldiff3 report - tmp/jcl3.dif

5J2 Know: How to know there are 37 rename... scripts

5M1 Levels: chmod1 - set perms on all levels - using 'find'
5D3 List: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5I1 List: List files using unix 'ls' options & more
5I2 List: llt - sample of 20 scripts to list files
5I3 List: All scripts to list files in directories
5N1 List: lsold1 - list files older than x days
5L3 Listed: alldiff3 listed on next page
5L4 Listing: alldiff3 script listing
5H1 Listings: Listings of cfd... scripts
5J3 Listings: Rename Script Listings - ALL
5I1 Llc: sample output for 'llc'
5I2 Llt: - sample of 20 scripts to list files
5N1 Lsold1: - list files older than x days

5D3 Maxrec: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5D3 Maxsize: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5E2 Menu: v12 - help menu for counting scripts
5J1 Method: alternate method to rename files
5D3 Minrec: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5D3 Minsize: 9. cfdmm - list directory File#,Lines,Minsize,Maxsize,Minrec#,Maxrec#
5F1 Mode: cfstest - run all 9 cfd scripts in batch mode

5N1 Older: lsold1 - list files older than x days
5J1 Operating: Operating Procedures
5I1 Options: List files using unix 'ls' options & more
5G1 Own: Recommendations using cfd scripts on your own directories

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

5L3 Page: alldiff3 listed on next page
5D2 Patter: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename
5M1 Perms: chmod1 - set perms on all levels - using 'find'
5J1 Procedures: Operating Procedures
5A1 Programmers: UVscripts Benefits to Programmers & Analysts

5G1 Recommendations: Recommendations using cfd scripts on your own directories
5B1 Relevant: Vancouver Utilities - subdirs relevant to UVscripts
5K1 Remove: removeCR - remove Carriage Returns from text files
5K1 Remove: removeCR - remove Carriage Returns from text files
5K1 Removecr: removeCR - remove Carriage Returns from text files
5K1 Removecr: removeCR - remove Carriage Returns from text files
5J1 Rename: Rename scripts
5J1 Rename: alternate method to rename files
5J3 Rename: Rename Script Listings - ALL
5J2 Rename...: How to know there are 37 rename... scripts
5J2 Renamel: RenameL - 1 example of 37 in sf/util/...
5K1 Returns: removeCR - remove Carriage Returns from text files
5K1 Returns: removeCR - remove Carriage Returns from text files
5C1 Run: Setup check-list - Ready to run UVscripts test/demos ?
5D1 Run: Run test/demos for 9 cfd* scripts
5F1 Run: cfstest - run all 9 cfd scripts in batch mode
5L3 Run: alldiff3 to drop insignicant differences

5I1 Sample: output for 'llc'
5I2 Sample: llt - sample of 20 scripts to list files
5E1 Screen: cfshelp - Help screen for cfd... scripts
5E1 Screens: Help screens for each script (vs all 9 above)
5A1 Separate: Note - UVscripts separate & in UVdemos
5M1 Set: chmod1 - set perms on all levels - using 'find'
5C1 Setup: Setup check-list - Ready to run UVscripts test/demos ?
5L2 Significant: only #2 differences are significant
5D2 Subdir: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5D2 Subdir: 7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-On
5D2 Subdir: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5B1 Subdirs: Vancouver Utilities - subdirs relevant to UVscripts
5D2 Super-dir: 8. cfddf - Count Files,Lines,KB in SubDirs of Super-Dir + 1st few files
5D2 Superdir: 6. cfdd - count Files,Lines,& KB in all subdirs of a superdir
5D2 Superdir: 7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-On

5C1 Test: Setup check-list - Ready to run UVscripts test/demos ?
5D1 Test: Run test/demos for 9 cfd* scripts
5K1 Text: removeCR - remove Carriage Returns from text files
5K1 Text: removeCR - remove Carriage Returns from text files
5J2 There: How to know there are 37 rename... scripts
5L2 Tmp: alldiff2 report - tmp/jcl3.dif
5L3 Tmp: alldiff3 report - tmp/jcl3.dif
5D2 Totals-on: 7. cfddt - count Files,Lines,& KB all subdirs of superdir (Totals-On
5D1 Totals-only: 3. cfdt - Count Files,lines,KB in 1 directory (Totals-Only)
5O1 Tree: dtree - create directory tree illustration

5I1 Unix: List files using unix 'ls' options & more
5J2 Util: RenameL - 1 example of 37 in sf/util/...
5B1 Utilities: Vancouver Utilities - subdirs relevant to UVscripts
5I1 Uv: more info on UV scripts
5A1 Uvdemos: Note - UVscripts separate & in UVdemos
5A1 Uvscripts: Note - UVscripts separate & in UVdemos
5A1 Uvscripts: UVscripts Benefits to Programmers & Analysts
5B1 Uvscripts: Vancouver Utilities - subdirs relevant to UVscripts
5B2 Uvscripts: UVscripts demo files copied to your homedir
5C1 Uvscripts: Setup check-list - Ready to run UVscripts test/demos ?

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

5E2 V12: - counting Lines,Files,& KB in Directories
5E2 V12: - help menu for counting scripts
5B1 Vancouver: Vancouver Utilities - subdirs relevant to UVscripts

5D2 Without: 4. cfdpf - Count Files,Lines,KB in Dir with/without Pattern in filename

5A1 You: Initial Example to get you interested

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Visitor Counters for ThisYear and LastYear