Pages

SyntaxHighlighter

Tuesday, August 11, 2020

find text in a file

 

Over time we tend to collect a large number of program files.  It never fails that after much time has past that you need to access some code in a file but can't recall which program the keyword was in.

I work on a Linux operating system so I can use the GREP command to handle the task but I never remember the syntax options and do not like the way the data is returned.  That is why I wrote a macro named %findtext() that handles this for you in SAS via the use of a PIPE command.

If want to work for the keyword 'bitwise' in all SAS programs in a folder and any sub-folders that GREP command will be as follows:

> grep -nRi 'updatestagingdetails' '/em_data1/prod/macro' --include=*.sas

/em_data1/prod/macro/etl.sas:524:       %updatestagingdetails(env_out=&env_out.);

/em_data1/prod/macro/updatestagingdetails.sas:2:*     Program: updatestagingdetails.sas

/em_data1/prod/macro/updatestagingdetails.sas:9:*       Usage: %updatestagingdetails(dsn=work.stagingdetails,env_out=%env())

/em_data1/prod/macro/updatestagingdetails.sas:21:%macro updatestagingdetails( dsn  = work.stagingdetails, env_out = %env() ) ;

/em_data1/prod/macro/updatestagingdetails.sas:40:/*EOF: updatestagingdetails.sas */

Notice in the above output that the values are separated by a colon (:) and it is overall hard to read in my opinion.  This is where the SAS SCAN and FIND functions come into play to create a cleaner easier to read result.

The SAS macro call will be as follows (only the first two parameters are required):

%findtext(

  path      = /em_data1/prod/macro

 , text     = updatestagingdetails 

  , ignorecase   = Y

  , exactmatch   = N

  , extension    = sas

  , outdsn       = findtextresults

  , printresults = Y

);


 

/*******************************************************************************

     Program: findtext.sas

      Author: Tom Bellmer

Responsible: Tom Bellmer

     Created: 08/10/2020 @ 3:00:45 PM

SAS Version: SAS 9.4 (TS1M6)

          OS: LIN X64

     Purpose: read the contents of files searching for the string value

       Usage: %findtext(path = /folder, string = sql);

       Notes: colon is used as output separator.  Scan() function had issues

              with third column that contained colons.

  Parameters: path = (folder path to be searched)

              text = (text to be searched inside each file)

              ignorecase = Y/N to ignore the case of the &string.

              exactmatch = N/Y to match whole &string value

              extension = sas (file extension, use * for all files)

              outdsn = findtextresults (output data set name)

              printresults = Y (proc print the results? Y/N)

 

                             Modifications in descending order

FL-YYYYMMDD                             Description

----------- --------------------------------------------------------------------

 

         1    1    2    2    3    3    4    4    5    5    6    6    7    7    8

....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0

*******************************************************************************/

 

%macro findtext(

    path         =

  , text         =

  , ignorecase   = Y

  , exactmatch   = N

  , extension    = sas

  , outdsn       = findtextresults

  , printresults = Y

);

 

  %local options i;

  %do i = 1 %to 100;

    %local pathname&i;

  %end;

 

  %if %isblank(&path) %then %do;

    %put %str(E)RROR: Must pass in a value for path.;

    %return;

  %end;

 

  %if %isblank(&text) %then %do;

    %put %str(E)RROR: Must pass in a value for text.;

    %return;

  %end;

 

  /*   i = ignores case, n = show matched line number

     , R = recursively,  w = match whole word  */

  %let options = -nR;

  %if %upcase(&ignorecase) = Y %then %let options = &options.i;

  %if %upcase(&exactmatch) = Y %then %let options = &options.w;

 

  filename search pipe "grep &options '&text' '&path' --include='*.&extension'";

  data &outdsn(drop = fullfilename findpos);

    attrib

      Path         length = $256 label = 'Path'

      Filename     length = $128 label = 'Filename'

      Lineno       length = 8    label = 'Line No'

      Code         length = $256 label = 'Code'

      Fullfilename length = $256 label = 'Full Filename'

    ;

 

    infile search;

    input;

    fullfilename = scan(_infile_, 1':');

    filename     = scan(fullfilename, -1"/");

    path         = substr(_infile_, 1, lengthn(fullfilename) - lengthn(filename));

    lineno       = input(scan(_infile_, 2':'), 8.);

    findpos      = find(_infile_, ":");

    findpos      = find(_infile_, ":", findpos + 1);

    code         = substr(_infile_, findpos + 1);

  run;

  filename search clear;

 

  %if %getattr(dsn = &outdsn, attr = nlobs) = 0 %then %do;

    proc sql;

      insert into &outdsn

        set path     = "&path"

          , filename = "N/A"

          , lineno   = 0

          , code     = "No results were found using '&text'"

      ;

    quit;

  %end;

 

  %if %upcase(%substr(&printresults, 11)) = Y %then %do;

    proc sql noprint;

      select   distinct path

      into     :pathname1 -

      from     &outdsn;

    quit;

   

    title "Search results for: &text";

    %if &sqlobs = 1 %then %do;

      title2 "In Path: &pathname1";

    %end;

    proc print data = &outdsn;

      %if &sqlobs = 1 %then %do;

         var filename lineno code;

      %end;

    run;

    title;

  %end;

%mend;

 

/*EOF: findtext.sas */

Thursday, May 21, 2020

Future Proofing

It does not happen often, but it does happen. An event such as a server upgrade requires related changes to settings used by programs. Of course, alterations are not merely related to equipment changes but things like name or address changes do happen over time. This blog post reveals one technique to data drive those inevitable changes.

The approach used here was influenced by an equipment upgrade as well as SAS software upgrade from version 9.4m3 to 9.4m6. Instead of writing conditional macro code such as %IF-%THEN/%ELSE, the below code uses a macro function to return the name/value pair result directly inline. This technique is not only much more compact and flexible but it also retains changes over time.

The first thing needed is a permanent SAS data set to retain the values. Below is the structure that I used to support changes by SAS version that has been truncated to the first 9 characters so that incremental hot fixes do not impact the program. Slowly changing dimensions (SCD) are used via the effective begin and end dates to retain the history. The actual code is shown below.

data mylib.lookup;
  infile datalines dsd dlm = '~';
  attrib
    pk             length = 5
    version        length = $16
    eff_begin_date length = 4 format = date9. informat = date9.
    eff_end_date   length = 4 format = date9. informat = date9.
    name           length = $128
    value          length = $256
    createdby      length = $8
    createdate     length = 4 format = date9. informat = date9.
  ;
  input (pk -- createdate) (:);
  datalines;
1~9.04.01M3~01jan2017~30dec9999~WEB~oldserver@myco.org:8343~myid~21may2020
2~9.04.01M6~01may2020~30dec9999~WEB~newserver@myco.org:8343~myid~21may2020
3~9.04.01M3~01jan2017~30dec9999~METAPORT~8562~myid~21may2020
4~9.04.01M6~01may2020~30dec9999~METAPORT~8561~myid~21may2020
run;

/******************************************************************************* Program: lookup.sas Author: Tom Bellmer Created: 05/21/2020 @ 1:46:19 PM SAS Version: SAS 9.4 (TS1M3) OS: LIN X64 Purpose: Macro function to return value from a lookup name. Uses slowly changing dimension to support changes within a version Usage: %let port = %lookup(metaport); Notes: version is truncated to first nine characters to avoid hotfix changes Modifications in descending order FL-YYYYMMDD Description ----------- -------------------------------------------------------------------- 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 ....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0 *******************************************************************************/ %macro lookup(name); %local dsid rc retval; %let dsid = %sysfunc( open( mylib.lookup( where = ( "%substr(&sysvlong, 1, 9)" = version and %sysfunc(date()) >= eff_begin_date and %sysfunc(date()) <= eff_end_date and "%upcase(&name)" = upcase(name) ) ) ) ); %if &dsid. = 0 %then %put %sysfunc(sysmsg()); %else %do; %if %sysfunc(fetch(&dsid)) = 0 %then %let retval = %sysfunc(getvarc(&dsid, %sysfunc(varnum(&dsid., value)))); %else %put %str(E)RROR: no entry found for name = &name..; %let dsid = %sysfunc(close(&dsid)); %end; &retval. %mend; /*EOF: lookup.sas */

Saturday, April 25, 2020

Lock Release

Sometimes a SAS data set can get locked such that you can read it but no longer update it. Of course, it is best to attempt to find out who is holding the lock and on simple systems you can use things like the UNIX fuser command to find the process id and then kill that process. However, things are not that easy on a SAS grid environment with many possible servers.

One way to free up the file is to move the data set (and its index file) to another folder then move it back to release the pointer. The below macro makes it very easy to accomplish those actions. The only required argument is the data set name (dsn). The libref is used to obtain the path via the pathname function. A unique sub-folder is created by using the UUIDGEN() function.

The advantage of this technique versus an X command is that it does not rely on the XCMD setting and provides a return code to verify things behaved as expected. You can read thru the below code to see how this is all handled. The %getattr() macro is simply a wrapper of the ATTRN() function. The source code for the %movefile(), %copyfile() and deletefile() macros can be found here.


/*******************************************************************************
     Program: lockrelease.sas
      Author: Tom Bellmer
     Created: 04/25/2020 @ 8:17:58 PM
 SAS Version: SAS 9.4 (TS1M3)
          OS: LIN X64
     Purpose: Moves data set (and index) to sub-folder then back.  This is 
              needed as data sets get locked and this releases it
       Usage: %let x = %lockrelease(dsn = mylib.mymem);
       Notes: Better to see if anyone has file locked but if no other option
              then use this.  It works by changing the file pointer.  

                             Modifications in descending order
FL-YYYYMMDD                             Description
----------- --------------------------------------------------------------------

         1    1    2    2    3    3    4    4    5    5    6    6    7    7    8
....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0
*******************************************************************************/

%macro lockrelease(dsn = );
  %local
    libref 
    path
    memname
    guid
    starttime
    temppath
    indexed
    rc
  ;
  %let starttime = %sysfunc(datetime());

  %if not %sysfunc(exist(&dsn)) %then %do;
    %put %str(E)RROR: The dsn: &dsn does not exist;
    %return;
  %end;

  %if not %index(&dsn, .) %then %do;
    %put %str(E)RROR: This only supports two level (permanent) data sets;
    %return;
  %end;

  %let libref   = %scan(&dsn, 1, .);
  %let memname  = %scan(&dsn, 2, .);
  %let guid     = %sysfunc(uuidgen());
  %let path     = %sysfunc(pathname(&libref));
  %let temppath = %sysfunc(dcreate(&guid, &path));

  %if %isblank(&temppath) %then %do;
    %put %str(E)RROR: Unable to create sub-folder.;
    %return;
  %end;
  %else %do;
    %let indexed = %getattr(dsn = &dsn, attr = isindex);

    /********   Move dataset (and index) to sub-folder   *********************/
    %let rc = %movefile(infile = &path/&memname..sas7bdat
                 , outfile = &temppath/&memname..sas7bdat);
    %if &indexed and &rc %then %do;
      %let rc = %movefile(infile = &path/&memname..sas7bndx
                 , outfile = &temppath/&memname..sas7bndx);
    %end; 

    /*****   Move dataset (and index) from sub-folder back to original   *****/
    %if &rc %then %do;
      %let rc = %movefile(infile = &temppath/&memname..sas7bdat
                          , outfile = &path/&memname..sas7bdat);
      %if &indexed and &rc %then %do;
        %let rc = %movefile(infile = &temppath/&memname..sas7bndx
                    , outfile = &path/&memname..sas7bndx);
      %end; 
    %end;

    %if &rc %then %let rc = %deletefile(&temppath);    
  %end;

  %put %str(N)OTE Elapsed Time: %left(%sysfunc(putn(%sysevalf(%sysfunc(
     datetime()) - &starttime), mmss12.3)));

  &rc
%mend;

/*EOF: lockrelease.sas */

Wednesday, January 8, 2020

Importing a delimited file

As it turns out, this is my 100th blog post since I started doing this back in November 2011. I create these posts as a service to other SAS users that might find some value in things that I have encountered as an issue and found a solution or just pointing out another way to do things.

This post is about a very clever and efficient way to import a delimited file. Credit for this idea goes out to my sagacious co-worker, Dave Devoll.

Let's assume you have a file of comma separated values (CSV) that should be imported into a data set with the same structure as the sashelp.class data set. The use of the IF 0 THEN SET dataset command compiles but never executes so it creates the program data vector (PDV) values. The input (_all_) references the PDV while the (:) modifier works as a separator between the values.

Contents of the cls.csv file
Name,Sex,Age,Height,Weight
Alfred,M,14,69,112.5
Alice,F,13,56.5,84
Barbara,F,13,65.3,98
Carol,F,14,62.8,102.5
Henry,M,14,63.5,102.5
data temp;
  infile '/temp/cls.csv' dsd dlm = ',' firstobs = 2;
  if 0 then set sashelp.class;  /* prime the program data vector */
  input (_all_) (:);            /* read in all the columns from the PDV */
run;

Now if you want to add the data to an existing data set, use proc append and you are done..

Saturday, April 27, 2019

X out the X command - use SYSTASK instead

This is an easy one when it comes to best practices. There are times when you may need to shell out to the operating system in order to perform a task that SAS does not otherwise handle.

The easy way to do this is via the X command, but is it the best technique to use? The same can be said for the use of %SYSEXEC and CALL SYSTEM commands. Look at the comparison table below.

Feature X Command SYSTASK
Asynchronous Processing? No Yes
Return Code? No Yes

Be sure to review the SAS documentation on SYSTASK, here is the syntax in a nutshell:

SYSTASK COMMAND "os command" <WAIT|NOWAIT> <TASKNAME=taskname>
    <STATUS=statusvar> <SHELL<='shell-command'>> <CLEANUP>;

  • WAIT | NOWAIT determines if the request will be handled asynchronously (NOWAIT the default) or synchronously (WAIT)
  • TASKNAME = taskname uniquely identifies the task - use this with the WAITFOR command
  • STATUS = statusvar is a unique macro name that stores the status of the task
  • SHELL this specifies that the command should be executed by the OS shell command. You can specify a shell name else the default shell is used.
  • CLEANUP specifies that the task should be removed from the LISTTASK output when the task completes. This allows you to reuse the taskname. NOTE: This option is not available under the Windows operating system.

SYSTASK LIST <_ALL_ | taskname> <STATE> <STATVAR>;

SYSTASK KILL taskname <taskname>;

The two biggest reasons to use SYSTASK are the ability to run processes in parallel (asynchronously) and to get back a status code so you can handle any issues. As noted the X command does not support either of these features. However, SYSTASK can be run asynchronously using NOWAIT or synchronously using the WAIT option. Here is an example of copying three files at once:

/* If using Windows or not using the CLEANUP option then do this first */
/* SYSTASK KILL t1 t2 t3; */

SYSTASK COMMAND "cp &in/f1.txt &out/f1.txt" taskname=t1 status=s1 shell cleanup;
SYSTASK COMMAND "cp &in/f1.txt &out/f2.txt" taskname=t2 status=s2 shell cleanup;
SYSTASK COMMAND "cp &in/nofile.txt &out/nofile.txt" taskname=t3 status=s3 shell cleanup;

/* wait on task 1, 2 and 3 to complete before going to line of code that follow it */
WAITFOR _all_ t1 t2 t3;  

data _null_;
   if &s1 ne 0 then putlog "ERR" "OR: issue copying f1.txt";
   if &s2 ne 0 then putlog "ERR" "OR: issue copying f2.txt";
   if &s3 ne 0 then putlog "ERR" "OR: issue copying nofile.txt";
run;

ERROR: issue copying nofile.txt

UPDATE: After some initial feedback it is important to note that yes the XCMD option must be on for X command, FILENAMEE pipe, %SYSEXEC, CALL SYSTEM, SYSTASK and other commands. I used a simple example of the UNIX cp (copy) command but this could have been handled with the built in SAS function FCOPY(). I have covered the use of FCOPY() in this previous blog post.

Thursday, April 11, 2019

Binary Permutations

Would you like to store more than one value in a single variable or column? Doing so can save considerable space resulting in less network traffic and faster throughput. Inspiration and credit for this post came from the SAS Global Forum paper "Deciphering PROC COMPARE Codes: The Use of the bAND Function" by Hinson and Coughlin.

To keep things simple, consider that we have four books A, B, C and D and need to track every combination of those four. That is someone may have no books or only B and C. To make this all work, assign values using the power of 2 from 0 forward as follows:

  • 1 as 20 = 1
  • 2 as 21 = 2
  • 3 as 22 = 4
  • 4 as 23 = 8
  • 5 as 24 = 16
  • 6 as 25 = 32
  • 7 as 26 = 64
  • 8 as 27 = 128

Using permutation with replacement of 4 objects taken 2 at a time (have it or not) there are 42 or 16 possible combinations. This can be stored in a single column via the use of bitwise operators such as SAS's band function (the bitwise logical AND of two arguments). The following code will help better illustrate how this works.

data x(drop = value);
  value = 15;
  do num = 0 to 15;
    binary = put(num, binary4.);
    if num > 0 then binval = 2 ** (num - 1);
    binsum + binval;
    if num > 0 then match = band(binval, value);
    format binval binsum comma8.;
    output;
  end;
run;

The binval column contains the value associated with the num column. So if you want books B and C that is the sum of 2 + 4 or 6. If all books were desired, the value is 15 in the binsum column or the cumulative sum of the binval column. You can look at the binary column to see these values in binary format reading back from right to left every combination is covered.

Below is the sample books data set followed by the actual macro function code. The use of low level functions are used to read the data and return the matched column values. This does assume that the data set is in the correct sorted order and only has the same number of rows as needed and no more. The limit to this process is 15 combination which is one less than 215 or 32,767

data books;
  length i 3 book $6;
  do i = 1 to 4;
    book = cat("Book ", byte(64 + i));
    output;
  end;
run;

%macro band_permutations(
     dsn       =
   , column    =
   , value     = 
   , seperator = %str(,)
);
  %local dsid position type i binval retval;
    
  %let dsid = %sysfunc(open(&dsn, i));  /* open data set */
  %if &dsid %then %do;
    %let position = %sysfunc(varnum(&dsid, &column)); /* find position of column */
    %let type = %sysfunc(vartype(&dsid, &position));  /* is it 'C'har or 'N'um */
    %do i = 1 %to %sysfunc(attrn(&dsid, nlobs));      /* how many rows in data set */
      %let binval = %eval(2 ** %eval(&i - 1);         /* Calc the binary value */
   
      %if %sysfunc(band(&binval, &value)) > 0 %then %do;  /* Did value match */
        %if %sysfunc(fetchobs(&dsid, &i)) = 0 %then %do;  /* Retrive the row */
          %if %length(&retval) = 0 %then 
            %let retval = %sysfunc(getvar&type(&dsid, &position)); /* read column value */
 
            %else %let retval=&retval &seperator %sysfunc(getvar&type(&dsid, &position));
          %end;
          %else %sysfunc(sysmsg());  /* Write out message */
        %end;
      %end;  
      %let dsid = %sysfunc(close(&dsid)); /* close data set */
    %end; 
  %else %put %sysfunc(sysmsg());  /* unable to open the data set */
   
  /* return the value */
  &retval
%mend;

/* below resolves to result = Book B, Book C */
%put result = %band_permutations(dsn=books, column = book, value = 6);