Autoload

Introduction
How Autoload Works
The Autoload Directories
Autoload Data Directory (Drop Zone)
Autoload Scripts Directory
Timing of Autoload
How to Start Autoload
How to Stop Autoload
How to Add an Implementation
Introduction
Instructions: UNIX
Instructions: Windows
Additional Considerations
Reference
Logs and Process IDs for Autoload
Metadata Server Connection
Configuration Files for Autoload
User-Defined Formats for Autoload
Library-Level Attributes for Autoload
Processing of Delimited Files and Spreadsheets

Introduction

You can use autoload to keep a set of source tables in memory. Users or processes place source tables in a specified host location (a drop zone). Corresponding in-memory data is periodically updated to reflect the contents of the drop zone.
Benefits of autoload include the following:
  • You do not have to start the server. If a SAS LASR Analytic Server stops, the next run of autoload starts the server and loads data from the drop zone.
  • You do not have to register the source tables in metadata.
  • Browser-based constraints on the size of locally imported files do not apply to autoload.
For limitations of autoload, see Additional Considerations.
For a concise summary of basic information, see Autoload for Beginners in SAS Visual Analytics: Getting Started with Data on Windows.

How Autoload Works

Here is a summary of how autoload works:
  1. Autoload periodically scans the contents of a designated host directory, which is referred to as the autoload data directory or drop zone.
  2. After each scan, autoload synchronizes in-memory data against source tables in the autoload data directory as follows:
    • For each delimited file and spreadsheet, a corresponding source table (SAS data set) is created. For a delimited file or spreadsheet that already has a newer corresponding source table, this step is omitted.
    • Source tables that are not already in memory are loaded.
    • Source tables that are newer than their corresponding in-memory tables are refreshed (unloaded and then reloaded).
    • Source tables that are in the Unload subdirectory and in memory when a run of autoload begins are unloaded in that run.
    • Source tables that are in the Append subdirectory and newer than their corresponding in-memory tables are appended to their corresponding in-memory tables. If a table in the Append subdirectory has no corresponding in-memory table, it is loaded as a new table.
      • Each Append table is also appended to its corresponding table in the autoload data directory. If no corresponding table exists, a new table is added to the autoload data directory.
      • To prevent redundant append actions, data in the Append subdirectory is compared to corresponding data in the autoload data directory. The append action is performed on only data in the Append subdirectory that is newer than its corresponding data in the autoload data directory.
    Note: To ensure that refresh and append actions occur for only source tables that are newer than their corresponding in-memory tables, autoload compares file timestamps of source tables to load timestamps of corresponding in-memory tables.

The Autoload Directories

Autoload Data Directory (Drop Zone)

In the standard configuration, autoload data directories are in the AppData branch of the SAS configuration directory: 
/AppData/SASVisualAnalytics/VisualAnalyticsAdministrator/AutoLoad
Each autoload data directory has four required subdirectories (Append, Formats, Logs, and Unload). The term drop zone refers to the autoload data directory and its subdirectories.
Note: The scheduler account and anyone who places tables in these directories must have Read and Write access to these directories.

Autoload Scripts Directory

In the standard configuration, autoload scripts directories are in the Applications branch of the SAS configuration directory: 
/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator/
Note: The scheduler account must have Read and Write access to the autoload scripts directory and its contents.

Timing of Autoload

Autoload runs as a periodic scheduled task. In the standard configuration, a new run of autoload is started every 15 minutes. The timing is controlled by a setting in schedule.sh (on UNIX) or schedule.bat (on Windows) in the autoload scripts directory.
Here are additional details:
  • A new run of autoload starts only after the previous run is complete.
  • Starting the associated SAS LASR Analytic Server does not trigger an immediate run of autoload.
  • Stopping the associated SAS LASR Analytic Server does not stop autoload activity. If the server is down when a run of autoload begins, autoload starts the server.
UNIX Specifics: The interval clock starts on the hour. For example, if the interval is 15 minutes, then autoload runs on the hour and at 15, 30, and 45 minutes after the hour.
Windows Specifics: The interval clock starts when autoload is scheduled. For example, if the interval is 15 minutes, then autoload runs 15 minutes after the schedule script is invoked, and every 15 minutes thereafter.

How to Start Autoload

To start scheduled runs for an implementation of autoload:
  1. On the machine that hosts the implementation, identify or create a scheduler account.
    • Give the account the host-layer privileges that are required to start the associated SAS LASR Analytic Server and load data. See Host Account Privileges.
    • On UNIX, enable the account to run cron jobs.
    • In the SAS configuration directory, give the account Read and Write access to the autoload directories and their contents. For the public implementation of autoload, the locations are as follows:
      Data:
      /AppData/SASVisualAnalytics/VisualAnalyticsAdministrator/AutoLoad
      Scripts:
      /Applications/SASVisualAnalytics/VisualAnalyticsAdministrator
      Note: For the public implementation, access to subdirectories for other implementations (for example, EVDMLA and VALIBLA) is not required.
  2. In the metadata, create a corresponding individual metadata identity. (For the public implementation, the new identity does not need any explicit group memberships.) See How to Add a User.
    Note: This requirement reflects the standard configuration. See Metadata Server Connection.
    Make sure the scheduler account’s metadata identity has the required metadata-layer permissions on the target server, library, and folder.
    For the public implementation, all registered users have sufficient access, so no adjustments are required. Here are the details:
    Server:
    Public LASR Analytic Server
    RM, WM, A
    Library:
    Visual Analytics Public LASR
    RM, R, WM, A
    Folder:
    /Shared Data/SAS Visual Analytics/Public/LASR
    RM, R, WMM, W
  3. Log on to the host as the scheduler account, navigate to the implementation’s scripts directory, and invoke schedule.sh (on UNIX) or schedule.bat (on Windows).
    Tip
    You can change the schedule interval by editing the schedule script. For validation, an interval of 2 minutes is suggested.
  4. Verify that the scheduled task is running.
    Windows Specifics: Access the Task Scheduler (for example, select Startthen selectControl Panelthen selectAdministrative Toolsthen selectTask Scheduler). Locate the task in the Task Scheduler Library (for example, Visual Analyt Hi-Perf Cfg - Auto Load Scheduler).
    UNIX Specifics: Run the command: crontab -l
  5. If necessary, edit the schedule script to adjust the interval. The standard interval is 15 minutes.
  6. (Optional) Verify that tables that are placed in the autoload data directory are processed as described in How Autoload Works.
    For example, place a CSV file or a SAS data set in the autoload data directory. After 15 minutes, use the LASR Tables tab to verify that the data is loaded. See Get Table Information.

How to Stop Autoload

To stop the scheduled task, use the scheduler account to invoke unschedule.sh (on UNIX) or unschedule.bat (on Windows). Stopping autoload does not stop the associated SAS LASR Analytic Server.

How to Add an Implementation

Introduction

Each LASR library that supports autoload must have its own implementation of autoload. For details about the predefined implementations, see Predefined LASR Libraries.
This section creates an implementation of autoload for sales data. Here is a depiction of the host directories that this example creates:
Example: VASALES Implementation of Autoload
autoload directories
Note: In the preceding image, the new autoload data directory is above the new autoload scripts directory. Directories that are not essential to this example are omitted from the image.

Instructions: UNIX

  1. Create a new autoload data directory as follows:
    autoload-data-branch/VASALES
autoload-data-branch/VASALES/Append
autoload-data-branch/VASALES/Formats
autoload-data-branch/VASALES/Logs
autoload-data-branch/VASALES/Unload
  2. Create a new autoload scripts directory.
    1. Make a sibling copy of an existing autoload scripts directory. For this example, copy autoload-scripts-branch/VALIBLA (or your equivalent of that predefined existing scripts directory) to a new directory named autoload-scripts-branch/VASALES.
      Note: Creating the new scripts directory beneath the existing autoload scripts branch facilitates migration.
    2. In the new autoload-scripts-branch/VASALES/Logs directory, delete any copied files.
  3. In the new autoload-scripts-branch/VASALES directory, edit the copied files as follows:
    AutoLoad.sas
    Change the %LET AL_META_LASRLIB= value to the metadata name of the new implementation’s LASR library. For example:
    %LET AL_META_LASRLIB=SalesAutoload;
    runsas.sh
    Edit the AUTOLOAD_ROOT= value to reference the new autoload scripts directory. For example:
    AUTOLOAD_ROOT="autoload-scripts-branch/VASALES"
    Verify that the appropriate configuration files are referenced. See Configuration Files for Autoload.
    schedule.sh and unschedule.sh
    Edit the RUNSAS_PATH= value to reference the new implementation’s autoload scripts directory. For example: 
    RUNSAS_PATH="autoload-scripts-branch/VASALES/runsas.sh"
  4. In SAS Management Console or SAS Environment Manager, identify or create a metadata folder for generated LASR table objects (in this example, /Shared Data/SAS Visual Analytics/Autoload/SALES).
  5. Configure a LASR library to support autoload.
    • If you do not want to modify an existing library, add a LASR library. If you do not want to assign the new library to an existing server, add a SAS LASR Analytic Server.
    • The library’s name must exactly match the value that you entered in the AutoLoad.sas file in step 3 (in this example, SalesAutoload).
    • The library must be in a metadata folder that has appropriate permission settings (in this example, /Shared Data/SAS Visual Analytics/Autoload/SALES).
    • Set the library’s extended attributes as follows:
      VA.AutoLoad.Location
      autoload-data-branch/VASALES
      VA.Default.MetadataFolder
      /Shared Data/SAS Visual Analytics/Autoload/SALES
      VA.AutoLoad.AutoStart
      Yes
      VA.AutoLoad.Enabled
      Yes
      VA.AutoLoad.Sync.Action1
      Yes
      VA.AutoLoad.Compress.Enabled
      No (or, to enable compression, Yes)
      VA.AutoLoad.Debug.Enabled
      No
      VA.AutoLoad.ExpandChars.Enabled
      No
      VA.AutoLoad.Import.Delimiter.TXT
      TAB
      VA.AutoLoad.Import.RowsToScan
      500
      1Set all 6 of the Sync attributes (Enabled, Import, Load, Refresh, Append, Unload) to Yes.
      Tip
      If a new library’s extended attributes are not initially visible, save and then reopen the library.
  6. Start the new scheduled task by logging on to the host as the scheduler account and invoking schedule.sh.

Instructions: Windows

  1. Create a new autoload data directory as follows:
    autoload-data-branch\VASALES
autoload-data-branch\VASALES\Append
autoload-data-branch\VASALES\Formats
autoload-data-branch\VASALES\Logs
autoload-data-branch\VASALES\Unload
  2. Create a new autoload scripts directory.
    1. Make a sibling copy of an existing autoload scripts directory. For this example, copy autoload-scripts-branch\VALIBLA (or your equivalent of that predefined existing scripts directory) to a new directory named autoload-scripts-branch\VASALES.
      Note: Creating the new scripts directory beneath the existing autoload scripts branch facilitates migration.
    2. In the new autoload-scripts-branch\VASALES\Logs directory, delete any copied files.
  3. In the new autoload-scripts-branch\VASALES directory, edit the copied files as follows:
    AutoLoad.sas
    Change the %LET AL_META_LASRLIB= value to the metadata name of the new implementation’s LASR library. For example:
    %LET AL_META_LASRLIB=SalesAutoload;
    runsas.bat
    Edit the AUTOLOAD_ROOT= value to reference the new autoload scripts directory. For example:
    AUTOLOAD_ROOT="autoload-scripts-branch\VASALES"
    Verify that the appropriate configuration files are referenced. See Configuration Files for Autoload.
    schedule.bat and unschedule.bat
    Edit the RUNSAS_PATH= value to reference the new implementation’s autoload scripts directory. For example: 
    RUNSAS_PATH="autoload-scripts-branch\VASALES\runsas.bat"
    In the schedule.bat and unschedule.bat files, change the name of the scheduled task. For example, if you began by copying scripts from the public implementation of autoload, the task name in the copied files is initially Visual Analyt Hi-Perf Cfg - Auto Load Scheduler. Change that name to any different value, such as Private Autoload Scheduler\". (The name change is necessary because the Windows Task Scheduler requires that each task name is unique.)
  4. In SAS Management Console or SAS Environment Manager, identify or create a metadata folder for generated LASR table objects (in this example, /Shared Data/SAS Visual Analytics/Autoload/SALES).
  5. Configure a LASR library to support autoload.
    • If you do not want to modify an existing library, add a LASR library. If you do not want to assign the new library to an existing server, add a SAS LASR Analytic Server.
    • The library’s name must exactly match the value that you entered in the AutoLoad.sas file in step 3 (in this example, SalesAutoload).
    • The library must be in a metadata folder that has appropriate permission settings (in this example, /Shared Data/SAS Visual Analytics/Autoload/SALES).
    • Set the library’s extended attributes as follows:
      VA.AutoLoad.Location
      autoload-data-branch\VASALES
      VA.Default.MetadataFolder
      /Shared Data/SAS Visual Analytics/Autoload/SALES
      VA.AutoLoad.AutoStart
      Yes
      VA.AutoLoad.Enabled
      Yes
      VA.AutoLoad.Sync.Action1
      Yes
      VA.AutoLoad.Compress.Enabled
      No (or, to enable compression, Yes)
      VA.AutoLoad.Debug.Enabled
      No
      VA.AutoLoad.ExpandChars.Enabled
      No
      VA.AutoLoad.Import.Delimiter.TXT
      TAB
      VA.AutoLoad.Import.RowsToScan
      500
      1Set all 6 of the Sync attributes (Enabled, Import, Load, Refresh, Append, Unload) to Yes.
      Tip
      If a new library’s extended attributes are not initially visible, save and then reopen the library.
  6. Start the new scheduled task by logging on to the host as the scheduler account and invoking schedule.bat.

Additional Considerations

  • Not all tables can be autoloaded. See Load Methods.
  • Autoload is supported for both distributed and non-distributed servers. However, you cannot autoload data from co-located storage.
  • A new log file is generated for each run. The autoload-scripts/Logs directory must be periodically emptied.
  • You cannot interactively reload an autoloaded table. You can instead interactively unload the table, and then wait for the next run of the autoload scheduled task, which refreshes (unloads and then reloads) the table.
  • You cannot autoload multiple tables that have the same base name. For example, if the files abc.xls and abc.xlsx are placed in an autoload data directory, only one data set (abc.sas7bdat) is loaded.
  • In a multi-machine deployment, autoload-related files are on the machine that hosts the workspace server.
  • If you move a delimited file or spreadsheet from the autoload data directory to the Unload subdirectory, remember to also delete the file’s corresponding SAS data set (from the autoload data directory and, if applicable, from the Append subdirectory).
  • If a table exists in both the autoload data directory and the Unload subdirectory, the table is repeatedly loaded and unloaded by alternating runs of autoload.
  • If the metadata name of a LASR library that supports autoload includes UTF-8 characters, the corresponding AutoLoad.sas program must be saved in UTF-8 encoding. (In the AutoLoad.sas program, the %LET AL_META_LASRLIB= parameter specifies the library’s metadata name.)
  • All synchronization actions create and update corresponding LASR table objects as needed. However, autoload does not delete LASR table objects.
  • Autoload runs a SAS session directly from SAS Foundation. To modify session behavior for autoload, set SAS options (such as MEMSIZE) in an appropriate location. See Configuration Files for Autoload.

Reference

Logs and Process IDs for Autoload

Comprehensive logs and any list output are written to the autoload-scripts/Logs directory. Each run of autoload generates a separate log, with a filename in the format AutoLoad_date-and-time-stamp.
Additional logs and any debug output are written to the autoload-data/Logs directory. Each run of autoload generates a new log (in both data set and text format) that overwrites the previous log.
Autoload process ID (PID) text files are written to the va.monitoringPath/PIDs directory in the format autoload_library-name.pid (for example, autoload_VisualAnalyticsPublicLASR.pid). See va.monitoringPath.
UNIX Specifics: An additional PID file (autoload.pid) is written to the autoload scripts directory. This additional PID file is used to prevent the runsas script from starting again if it is already running.

Metadata Server Connection

In the standard configuration, no metadata connection options are specified in the AutoLoad.sas program. Connection information is obtained as follows:
  • The metadata repository name is obtained from the associated sasv9.cfg file. See Configuration Files for Autoload.
  • The metadata server’s machine name and port are obtained from the file that the sasv9.cfg file references in its METAPROFILE setting. This is the preferred approach, because it supports both clustered and unclustered metadata servers.
  • The account that schedules autoload also runs autoload and connects to the metadata server. This is the preferred approach, because it does not require specifying credentials in any host file.
Note: For information about metadata server connection options, see SAS Language Interfaces to Metadata.

Configuration Files for Autoload

Although autoload does not run in a SAS Application Server, autoload can borrow settings from server configuration files. This borrowing can reduce the need to set the same option in multiple locations. Each implementation of autoload has its own list of references to configuration files.
Windows Specifics: The list is in the AutoLoad.cfg file in the implementation’s autoload scripts directory.
UNIX Specifics: The list is in the SASCFGPATH= variable in the implementation’s runsas script.
The standard list references the following files in the following order:
  1. The sasv9.cfg file for the SAS Application Server that is designated in the implementation’s runsas script (for example, SERVER_CONTEXT= SASApp). The designated SAS Application Server and the autoload implementation must be on the same machine.
  2. The sasv9_usermods.cfg file for the designated SAS Application Server.
  3. The implementation’s AutoLoad.cfg file.
  4. The implementation’s AutoLoad_usermods.cfg file.
The preceding list is in reverse precedence order. If an option is set in multiple configuration files, the setting in the last-listed file has precedence. For example, settings in an AutoLoad_usermods.cfg file override any conflicting settings in other configuration files.
You can add, remove, or adjust options in the referenced configuration files as needed. See Reference: Configuration Files for SAS Servers in the SAS Intelligence Platform: System Administration Guide.

User-Defined Formats for Autoload

For general information, see Supporting User-Defined Formats.
Any format catalogs that are made available through a referenced configuration file are available to autoload.
If you want to make certain user-defined formats available exclusively to a particular implementation of autoload, place format catalogs in that implementation’s autoload-data-branch/Formats directory. Catalogs in that directory have precedence over same-named catalogs that are available to autoload through configuration files.

Library-Level Attributes for Autoload

The following attributes support autoload:
VA.AutoLoad.Location
sets the autoload data directory. If you change the location, make sure you create the required subdirectories. For a new library, the suggested value is autoload-data-branch/LIBNAME.
VA.Default.MetadataFolder
sets the metadata location for the LASR table objects that autoload generates. For a new library, the initial value is your equivalent of /Shared Data/SAS Visual Analytics/Autoload.
VA.AutoLoad.Enabled
specifies whether the library supports any autoload features. For a new library, the initial value is No.
Note: Setting this attribute to Yes does not disable interactive loading. You can interactively load data to a library that supports autoload.
VA.AutoLoad.Sync.Enabled
specifies whether synchronization actions are enabled. This is a parent setting (and a prerequisite) for other *.Sync.* attributes. For a new library, the initial value is No.
To preview synchronization actions, set this value to No, run autoload, and then examine the autoload log file.
VA.AutoLoad.Sync.Import
specifies whether the import action is enabled. For a new library, the initial value is No.
VA.AutoLoad.Sync.Load
specifies whether the load action is enabled. For a new library, the initial value is No.
VA.AutoLoad.Sync.Refresh
specifies whether the refresh action is enabled. For a new library, the initial value is No.
VA.AutoLoad.Sync.Append
specifies whether the append action is enabled. For a new library, the initial value is No.
VA.AutoLoad.Sync.Unload
specifies whether the unload action is enabled. For a new library, the initial value is No.
VA.AutoLoad.Compress.Enabled
specifies whether compression is used when data is autoloaded. The default value is No. (For the administrative reporting library, EVDMLA, the initial value is Yes.)
VA.AutoLoad.Debug.Enabled
specifies whether debugging is enabled for autoload. The default value is No.
VA.AutoLoad.ExpandChars.Enabled
specifies whether autoload supports expansion of character variable lengths. The default value is No. To enable character expansion, set the value to Yes.
Note: Character expansion occurs when a SAS data set that is not UTF-8 encoded is autoloaded to a server that uses UTF-8 encoding. For more information, see Avoiding Character Data Truncation By Using the CVP Engine in the SAS National Language Support (NLS): Reference Guide.
CAUTION:
Format widths are not expanded with character variable lengths. If you enable character expansion, in-memory data might appear to be truncated.
In the designer and explorer, you can adjust formats as needed. To minimize the potential impact, enable character expansion in a separate LASR library that contains only tables that require character expansion. For more information, see the technical paper "Processing Multilingual Data with the SAS 9.2 Unicode Server".
VA.AutoLoad.Import.Delimiter.TXT
specifies the delimiter to use when autoload imports TXT files. The default value is TAB, which specifies to use the Tab character as the delimiter. You can specify a single character (for example, |, !, or &), SPACE (to use a space delimiter), or a hexadecimal code (for example, ‘09’x).
VA.AutoLoad.Import.RowsToScan
specifies the number of rows to scan to determine the data type and length for each column in an imported table. You can specify a positive integer or the value ALL. The default value is 500. (For the administrative reporting library, EVDMLA, the initial value is ALL.)
Tip
The header row counts. For example, to scan one row of data, specify 2 as the value.
The following attribute is used by autoload (but is not exclusive to autoload):
VA.AutoLoad.AutoStart
specifies whether the associated SAS LASR Analytic Server starts on demand for load requests against this library. For a new library, the initial value is No.
Note: Except where otherwise specified, the supported values are No and Yes.
Note: Changes take effect on the next run of autoload. For information about how tables that are already loaded are affected, see How Autoload Works.
To set library-level extended attributes, access a LASR library’s metadata definition in SAS Management Console or SAS Environment Manager.

Processing of Delimited Files and Spreadsheets

In general, autoload processes delimited files and spreadsheets in the same way that these files are processed during a self-service import. For information about supported file types, requirements, missing values, and valid names, see the SAS Visual Analytics: User’s Guide.
The following details are specific to autoload:
  • The file size limitation for interactive import is not applicable to autoload.
  • You cannot autoload a ZIP file.
  • Autoload always reads column names from the first row and begins data import on the second row.
  • When you autoload a spreadsheet that has multiple worksheets, only the first worksheet is loaded.
  • For append actions, column data types and lengths in both files must match.
  • Autoload of XLSB and XLSM files is supported only on Windows. The 64-bit version of Microsoft Access Database Engine (formerly known as Microsoft Office Access Connectivity Engine, or ACE) is required.
  • To autoload files that use a delimiter (other than a comma or the Tab character), use the TXT file extension and specify the delimiter in the VA.AutoLoad.Import.Delimiter.TXT extended attribute.
Copyright © SAS Institute Inc. All Rights Reserved.
Last updated: December 18, 2018