TS-224 21 Tips for Increasing Memory under the SAS(R) System 6.03/6.04 for PC/MS-DOS. These tips and techniques may or may not work in your individual environment. All sizes are approximate. SAS Institute assumes no responsibility as to any of these tips and techniques in fact working on your individual machine. Please understand that this paper addresses MEMORY issues not PERFORMANCE issues. Some tips presented may reduce your overall system performance while increasing available memory. This is a tradeoff only you can decide is worth while. The number of different personal computer types currently in operation is extremely large and therefore the number of different configurations staggers the mind. With that said let us explore how you may, emphasis on MAY, increase the amount of CONVENTIONAL RAM that is accessible by the SAS System under PC-DOS or MS-DOS. 1. Use EXPANDED memory. If you use the -EMS ALL option in your CONFIG.SAS file, the SAS System will use no more than 2 MB of LIM 3.2 or later expanded memory. The SAS System can use more by setting the number of 16 KB pages manually with the -EMS option. The installation manual recommends using LIM 3.0 or higher. This is incorrect and should read LIM 3.2 or higher. We recommend using LIM 4.0. The SAS System will load EMS loadable modules into ex- panded memory thus freeing up as much of the conventional memory as possible for use by data and SAS System data step programs. If your computer is an Intel(R) 8088/8086 or 80286 then an expanded memory board must be installed. If it is an Intel 80386SX, 80386DX, 80486SX or 80486 then you may convert the extended memory on the motherboard to expanded memory using an expanded memory manager. In many cases you will be required to use expanded memory or your program will not run due to lack of conventional memory. Approximate increase in conventional memory - 200-250,000 bytes. 2. It does NOT help to set a length for numeric variables shorter than 8 bytes to save RAM. The SAS System will expand them to 8 bytes when moved into RAM. This is to ensure that the actual number itself is set correctly so it may be moved between different operating systems thus maintaining the correct number representation. Further discussion about this may be referenced in your "SAS(R) Language Guide for Personal Computers, Release 6.03", SAS Publications Catalog Number #C5856. Look under the LENGTH statement section, with the subheading of "Numeric Data" on page 196. External memory, i.e., disk space, may be saved with the LENGTH statement. 3. Change your version of DOS to 3.3 or 5.0, if possible, as the DOS kernel for those versions are, on the average, 10-47,000 bytes smaller. If you have a 32 MB or larger partitioned disk you may still want to use a 3.3 version of DOS. There is third party software that allows the partitioning of the hard drive beyond 32 MB. Compaq(R) DOS 3.31 allows 32 MB or greater hard disk partitions. There are also third party hard drive partition extenders available such as OnTrack's DISK MANAGER(R) that may be helpful. DOS 4.0 and 5.0 allow the partitioning of the hard disk beyond 32 MB. Please understand that SAS Institute cannot and will not support any problems that arise from the utilization of these third party products. The chart shown below will give you an idea of how the different versions of DOS affect conventional memory concerning how much memory remains after DOS loads. These are typical machines and the amount may vary depending on the manufacturer of both the computer and DOS. These figures represent a system configured with a VGA adapter and with no memory resident software loaded. DOS 3.3 DOS 4.01 DOS 5.0 IBM PC-XT 591 KB 580 KB 581 KB IBM PC-AT 586 KB 576 KB 623 KB Clone 386/33 585 KB 575 KB 621 KB Approximate increase in conventional memory - 10-47,000 bytes. 4. Run your program in noninteractive (batch) mode. This mode is explained on page 311 in the "SAS(R) Language Guide for Personal Computers, Release 6.03", SAS Publications Catalog Number #C5856. Approximate increase in conventional memory - 100,000 bytes. 5. Comment out the FILEBUFFER statement from your CONFIG.SAS file. Each FILEBUFFER takes 512 bytes for each file buffer that you have open. See page 480 of the "SAS(R) Language Guide for Personal Computers, Release 6.03" for further information about this statement. This statement helps improve your system performance by reducing the number of read and write disk accesses when you access a SAS System data set. This is also true with files and the INFILE, FILE or %INCLUDE statements. This becomes a performance issue vs. a memory issue. Only you can decide if this helps or hinders you. If you have a disk caching program already loaded in memory you would, as a rule, be better off without this option. Approximate increase in conventional memory - 2,560 bytes. 6. Comment out the FILECACHE statement in your CONFIG.SAS. This will incur a small performance hit again but if your program references only a few files and you are already using a disk caching program this may not affect you. Since this will save such a small amount you may want to keep it set in your CONFIG.SAS. Filecaching uses 16 bytes for each entry in the table. See page 481 of the "SAS(R) Language Guide for Personal Computers, Release 6.03" for further information about this statement. If you are using DOS 4.00 or 4.01 we recommend that you comment these statements out as they have been known to cause some problems with the operation of the SAS System. Approximate increase in conventional memory - 528 bytes. 7. Remove as many Terminate and Stay Resident (TSR) programs as possible. Examples of TSR's that you could remove or remark out from your CONFIG.SYS are: 7.1. Network drivers, if the network is not needed to run the SAS System. Remove these programs or you can use expanded or extended memory versions of these programs if available. Novell ships XMSNET3.COM, XMSNET4.COM, XMSNET5.COM, EMSNET3.COM, EMSNET4.COM, and EMSNET5.COM free with the Novell Netware(R) product. They allow the movement of the drivers into extended or expanded memory thus freeing conventional memory for use by the SAS System and still allow connection to the network. Similar third party products are available. Talk to your network administrator about moving the network drivers into expanded or extended memory. Approximate increase in conventional memory - 2-20,000 bytes. 7.2. Keyboard enhancers such as KEYDO, DOSKEY or CED. Approximate increase in conventional memory - 16,000 bytes. 7.3. Mouse drivers, since the SAS System 6.04 will not use a mouse. A mouse can be used with SAS/ASSIST(R) if desired but it is not required. Approximate increase in conventional memory - 12-30,000 bytes. 7.4. ANSI.SYS driver, if you KNOW you will not "X" out of the SAS System under DOS 4.00 or 5.0. There is no need for it with DOS 3.3. If you will be "X'ing" out of the SAS System, under DOS 4.0 or 5.0, this will be required to be in your CONFIG.SYS. Approximate increase in conventional memory - 5,000 bytes. If you have an Intel 80386 or 80486 based machine you might want to use an expanded memory manager such as Quarter-deck's QEMM386(R) version 5.13 or Qualtas' 386MAX(R) version 5.1. For the IBM PS/2(R) machines use Qualtas' BlueMAX(R) version 5.1. These programs allow you to load some TSR's into HIGH memory thus freeing CONVENTIONAL memory for use by The SAS System. DOS 5.0 also includes a LOADHIGH program that allows the loading of pro- grams into expanded. Also DOS 5.0 can load itself into HIGH memory (Upper Memory Blocks). When using these programs you want to insure that you do not interfere with the EMS page frame. SAS Institute assumes no responsibility for supporting or resolving any conflicts these programs may introduce into the operation of the SAS System. 8. Reduce the FILE HANDLES as low as possible as each file handle set in your CONFIG.SYS via the FILES= statement takes 60 bytes. For the SAS System to operate at all you need FILES=50 as a minimum. If increased to FILES=75 you are using 1,500 extra bytes of conventional memory. Since the SAS System cannot address more than 75 open files do not set the FILES= statement beyond 75. This will require some experimentation on your behalf to find what works and what does not. This is a separate issue discussed later. Approximate increase in conventional memory - 1,500 bytes. 9. Set your DOS environment size as low as possible with the CONFIG.SYS SHELL=C:\COMMAND.COM /E:XXX /P statement, where XXX is the number of bytes. The DOS system default is 160 bytes. The environment is a small amount of conventional memory that DOS sets aside to hold the drive and subdirectory path information for later use. If this is set to a number greater than 160 you may want to experiment and lower it. Approximate increase in conventional memory - ? 10. If you have an EGA/VGA video card and Quarterdeck's QEMM386 or QRAM (used with 286 based machines) you may want to try the VIDRAM program that comes free with QEMM386 or QRAM. By invoking it you can increase usable conventional RAM space by approximately 64,000 bytes under DOS 3.3 and 4.0. DOS 5.0 will increase about 97,000 bytes, if you utilize the expanded memory manager, EMM386.EXE, that comes with DOS 5.0. This may not work with IBM PC-DOS 5.0. If it does not work, try using the Quarterdeck QEMM386 memory manager. This will require some experimentation and may not always work. This is at the expense of EGA/VGA video RAM. If you do not plan to use SAS/GRAPH(R) or other high resolution graphic programs this is an excellent way of increasing conventional memory. Since the SAS System does not use high resolution graphics for BASE SAS itself, the only thing you will notice is that the SAS System screens refresh a bit slower. The above figures are specific to one machine here at SAS Institute and may not reflect your savings. Approximate increase in conventional memory - 64-97,000 bytes. 11. Add STACKS=0,0 to your CONFIG.SYS. DOS itself sets this to a default of STACKS=9,128. By setting it to 0,0 you will increase your available memory. Approximate increase in conventional memory - 3,200 bytes. 12. Remove the FASTOPEN command from your CONFIG.SYS. This is the same as the SAS System filecache program. Do not use both at the same time. We recommend you use the -filecache option in the CONFIG.SAS if you desire filecaching. FASTOPEN allows DOS 3.3 or higher to speed up file access by keeping in memory a list of locations of subdirectories and files that have were used during the current session. DOS checks this list before searching the disk. The form of the command is: fastopen = /x The files parameter is the number of files and subdirectory locations DOS will keep in memory. For each file specified 50 bytes is required. The /x switch is available for DOS 4.00 and 5.00 only and allows fastopen to be moved into expanded memory instead of conventional memory. The FASTOPEN command should as a rule be deleted when using the SAS System as it is known to cause problems when used with DOS 4.00 or 4.01. Those versions of DOS have problems and the SAS System occasionally will present a "DOS CRITICAL ERROR" when trying to read or write from a disk that have been referenced by the FASTOPEN command. The safest option is to remove this command completely. Approximate increase in conventional memory - 500-1,000 bytes. 13. If you are using DOS 3.3 or 5.0, the BUFFERS= command in your CONFIG.SYS may be written as follows: BUFFERS=number where num- ber is the number of buffers set aside in conventional memory for look ahead reading of the disk. The number of buffers may be lowered or the statement removed altogether if desired, if you have a disk caching program installed. Microsoft recommends lowering this figure to 10 when using the SMARTDRV.SYS disk caching program. Normal setting for the SAS System is: BUFFERS=17 Each buffer requires 532 bytes. A number greater than 17 may be a waste of memory. Initially set it to 17. If you feel a need to increase this number do so in groups of 17, i.e - 17, 34, 51. This is a performance vs. memory issue, slower performance for more memory. 14. If you are using DOS 4.00 or 4.01 you can place the /X option on your buffers= statement in your CONFIG.SYS. This allows the buffer option to be placed in expanded memory. This is not available with any version of DOS 3.3 and below or DOS 5.0. The settings for the buffers command under DOS 4.0 and higher is: buffers=number,advance /x. The number is the number of 512 bytes (one disk sector) DOS reads from disk. The advance number is the number of 512 bytes that DOS will look ahead following the sector currently in use. As always there is a price to be paid for this option. Each regular buffer takes 532 bytes and each advance (look ahead buffer) takes 512 bytes. A typical setting might be: buffers=17,2 /x Approximate increase in conventional memory - 9,010 bytes. 15. If you use the DOS shell under 4.0 or 5.0 you may want to consider dropping it. The shell is not needed for the operation of the SAS System and takes up large amounts of memory. Approximate increase in conventional memory - 30,000 bytes At this point our discussion will end concerning tasks that can be done before starting the SAS System itself. As you can see, there are some big and small things that you may want to do that could in- crease the usable conventional memory. Use a combination of these items to fit your individual situation. Please remember that the figures used are approximate and may or may not reflect your savings. These savings are based on the type of computer you use, the type of DOS you have loaded on your computer and your individual computer configuration. SAS Institute has recently published a book entitled "SAS Program- ming Tips, A Guide to Efficient SAS Processing", SAS Publication Catalog Number #C56150. This guide will help in reducing the conventional memory requirements when coding SAS System programs. Below are suggestions on how you could increase available conventional memory with the Guides' examples. 16. Process only the variables you need using the DROP= or KEEP= data set option. This saves memory by setting up an internal ram buffer called a program data vector (PDV), for the needed variables, not all the variables in the SAS System dataset. Example: data temp; set sasdataset(keep= var1 var22 var31); 17. Use a binary search, not a null merge. Use the POINT= option in the SET statement to locate a set of observations in a master data set that correspond to observations having a common variable in the transaction data set. 18. Use the WHERE condition in procedures and when referencing SAS System data sets. 19. Mimic large sorts with other techniques. When a SAS System data set is too large to sort on your system, use a DATA step with multiple data set names and OUTPUT statements to divide the original data set into smaller data sets, each of which contains observations with one value of the BY variable. 20. Do not use the SYMGET function. This function will use a lot of memory (hundreds of bytes at a time) and not release it when done. Certainly do not use it in a DO loop. You may see this function while using macros. 21. When sorting use the undocumented SORTSIZE=XXX option of PROC SORT. Find the number XXX as follows. Total the size of each variable that you have in your data set. All numeric variables must be counted as 8 bytes even if they are smaller. Take this number and add 10 to it. Multiply the resulting number by 2. Take this result and add 100 to it for the final number to place in the SORTSIZE= option. The following is an example of how to compute this. Example: DATA SET TEMP has: var1 - Num - length 8 var2 - Char - length 20 var3 - Char - length 15. The total sum of the lengths = 43. Add 10 to 43 for 53. Multiply this by 2 for 106. Add 100 for a result of 206. I.E. ((((8+20+15)+10)*2)+100)=206 Proc Sort data=temp sortsize=206; Remember that this algorithm produces the minimum amount of conventional memory needed to sort the dataset, not the optimum amount. Sometimes you may not have a choice in using this option otherwise your program may not run. This again becomes a memory vs. performance issue and only you can decide if it is worthwhile to incorporate in your programming. The following illustrates a real world example of how a 386SX clone using Compaq DOS 3.31, 4.01 and MS-DOS 5.0 would benefit from the above hints and advice. Minimum configuration CONFIG.SYS and AUTOEXEC.BAT settings needed to get the maximum amount of memory. DOS 3.3 and 4.0 CONFIG.SYS DEVICE=C:\QEMM\QEMM386.SYS BUFFERS=17 FILES=50 MS-DOS 5.0 CONFIG.SYS DEVICE=C:HIMEM.SYS DOS=HIGH,UMB DEVICE=C:EMM386.EXE ON 3000 BUFFERS=17 FILES=50 AUTOEXEC.BAT (for both config's) prompt $p$g /* not needed but nice to have */ CONFIG.SAS -ems all /* Notice the following options are commented out */ /*-filebuffer 5 512 */ /*-filecache !sasroot\sasexe\base 15 */ /*-filecache !sasroot\sasexe\sasmsg 15 */ If the system is started with the above settings, you will see results similar to those in the table below. An undocumented Program Editor command "CDE" was used to figure out how much conventional ram the SAS System sees for it's use. The CDE command will present in the LOG window the largest memory block in conventional memory left for use. doing a 'CDE' in program editor COMPAQ 3.31 COMPAQ 4.01 MS-DOS 5.00 Using above settings 451,776 441,376 487,312 Add /X to Buffers= 451,776* 450,386 487,312* Add Stacks=0,0 454,992 453,664 489,184 Invoking VIDRAM ON 519,280 517,952 586,176 * Not available as an option NOTE: All figures are for conventional ram memory. With the above settings you can see that a large amount of conventional memory can be recovered. However you may not want to have your CONFIG.SYS, CONFIG.SAS and AUTOEXEC.BAT continually in that state. Below is a real world CONFIG.SYS and AUTOEXEC.BAT you might use under Compaq DOS 3.31. The following programs from Quarterdeck are used to increase memory: LOADHI.COM used to load programs/tsr's into high memory LOADHI.SYS used to load device drivers into high memory VIDRAM ON used to gain video ram as conventional memory When using these options you can gain close to the same amount of conventional usable ram as the minimal setting presented above. CONFIG.SYS shell=c:command.com /e:200 /p device=c:\qemm\qemm386.sys ram device=c:\qemm\loadhi.sys r:1 c:\dos\smartdrv.sys 1024 256 device=c:\qemm\loadhi.sys r:2 c:\dos\ansi.sys buffers=17 files=50 stacks=0,0 AUTOEXEC.BAT set comspec=c:command.com path=c:;c:\dos;c:\qemm;c:\sas prompt $p$g c:\qemm\loadhi /r:1 c:\dos\mouse.com c:\qemm\loadhi /r:2 c:\utils\keydo.com c:\qemm\vidram on With the above settings and performing a 'CDE' command on the program editor command line in SAS System 6.04 under Compaq DOS 3.31 - 518,800 bytes were free for use by the SAS System. The difference between the real world example and the minimal approach which gave us 519,280 bytes is 480 bytes. Please note that Quarterdeck's LOADHI.COM, LOADHI.SYS and VIDRAM programs did most of the savings. The loadhigh command available under DOS 5.0 does the same thing as loadhi with QEMM386. It would be worth purchasing QEMM386 just to get the VIDRAM program when operating under MS-DOS 5.0 as you could increase memory by approximately 97,000 bytes. Other memory managers may or may not give similar results. How much conventional memory saved depends on what you are doing in the DATA step and the amount will differ between each user, again based on the hardware and soft- ware installed. In some case the use of any expanded memory manager may result in a smaller than expected amount of expanded memory that the SAS System recognizes. SAS Institute will not guarantee that any of the expanded memory programs will work in your system configuration. The discussion above is how one might approach an "Insufficient Memory" or "Out of memory" error messages. As an alternative we recommend that you switch to the OS/2(R) operating system that unlike DOS, does not limit the amount of memory you can use to just 640 KB. The SAS System 6.06 uses the OS/2 operating system and technical support has never (to date) had a user complain about running out of memory. OS/2 can address 16 MB of extended memory and over 512 MB of virtual memory. SUMMARY: Conventional Conventional Performance Performance USE Memory Increase Memory Decrease Increase Decrease OS/2 X X VIDRAM ON X X Stacks=0,0 X Sortsize= X X DOS 3.3 X DOS 5.0 X Filebuffer X X Filecache X X Batch Mode X X Remove TSR X ? Disk Cache X X Files=50 X X Files=75 X X Fastopen X X Fastopen /x X X Buffers= X X Buffers= /x X X PLEASE NOTE: As can be seen above there are numerous factors that can be taken into consideration. One or more of these may be used to fit your particular need. Please understand that SAS Institute Technical Support cannot support every configuration that may be in use by you the user. As you can imagine there are too many factors to take into account and there are a number of non-standard (clone) ROM BIOS's that may be in use by your computer. The suggestions offered here are just that - suggestions. They may not solve your particular situation. If you try some or all of these suggestions and they do not work please do not call Technical Support as we will not support another companies product i.e. DOS or Quarterdecks. We will support the minimum configuration. OTHER ISSUES 1. An error occasionally seen is the "OUT OF FILE HANDLES" message. This will be seen when you use the default FILES=50 statement in your CONFIG.SYS. If you get this message you will need to reduce as much as possible the number of handles open and increase the number DOS and the SAS System can access. An open handle is equivalent to a pointer to a file. SAS itself needs a minimum of 50 to operate. This number can be increased to 75. The SAS System will not address more than 75. To address this problem do the following: 1.1 Change CONFIG.SYS to read files=75. Note: This reduces conventional memory available. 1.2 Reduce the number of open files in your SAS System program. Every INFILE, %INCLUDE, FILE, PUT (to a file), SET and MERGE statement is an open file handle. Try to merge multiple single datasets into one main dataset for processing. This error is hard to address and there are no clear cut answers or solutions. Some experimentation on your behalf will be required. 2. If you get a "STACKS OVERFLOW" error message, please be aware that this is a DOS error and may be cleared up by placing STACKS=0,0 in your CONFIG.SYS file. This may or may not work depending on what caused the error. 3. When using PROC SORT you may get an error that says "DISK FULL ON DEVICE X:" where X: is the device that the SAS System is trying to write to. This is referencing the floppy drive or hard drive disk space. You will need space free on your drive dependent on the following: 3.1 If you use: data temp; infile rawdatafile ...; input varname ...; . proc sort data=temp; by varname; . Disk space equal to 1.5 times the size of the rawdatafile must be free where the your SASWORK subdirectory is located. 3.2 If you use: data temp; set sasdataset; . proc sort data=temp; by varname; . Disk space equal to the size of sasdataset must be free on the drive where sasdataset is located. 3.3 If you use: proc sort data=twolevel.dsname; Room equal to the size of the SAS System dataset being sorted must be free where the unsorted twolevel.dsname is located. 3.4 If you use: data temp; infile rawdatafile ...; input varname ...; proc sort data=dsname out=onelevelname; Disk space equal to 2.5 times the SAS System dataset size must be free where your SASWORK subdirectory is located. 3.5 If you use: proc sort data=dsname out=two.levelname; 1.5 times the SAS System dataset size must be free where the out= libref points to. As you can see the amount of free disk space and where it must be free depends on what you are doing. The "DISK FULL ON DEVICE X:" statement will tell you where you must have free space. If you do not have the space available use the TAGSORT= option that is discussed in your "SAS Procedures Guide, Release 6.03" SAS Publications Catalog Number #C5843. Further reading: Brumitt, David - SAS Institute (1989), "Configuring your PC to Run the SAS System", Proceedings of the Fourteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 513-519. Pabst, Mark, W. - Bureau of Reclamation (1990), "SAS System Performance on 80386 Class PC's", Proceedings of the Fifteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 317-322. Deese, Donald - Computer Management Sciences, Inc. & Ebner, John - Litton Computer Services (1990), "SAS Software Performance in the Personal Computer Environment", Proceedings of the Fifteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 323-332. Klusman, Eric - Federal Reserve Bank of Chicago (1990), "Aspects of LAN-Based Decision Support Systems", Proceedings of the Fifteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 979-982. Mandel, Stephen - PSC, Inc (1991), "The Advantages of SAS 6.06 Under OS/2", Proceedings of the Sixteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 472-475. Hockman, Joe - E. I. du Pont Nemours and Company (Inc) (1991), "Using Experimental Design to Optimize SAS Performance on the Personal Computer", Proceedings of the Sixteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 481-487. Pabst, Mark W. - Bureau of Reclamation (1991), "PC Performance- How to Upgrade Existing Equipment, What Kind of Machine to Buy", Proceedings of the Sixteenth SUGI Conference, Cary, NC: SAS Institute, Inc., pp. 906-912.