Problem Note 67943: A performance issue might occur when you try to insert a BigQuery table into another BigQuery table
The Google BigQuery database does not allow you to insert a DATETIME variable into a TIMESTAMP variable. An error results when you try to do this action. If you use explicit pass-through to insert one BigQuery table into another BigQuery table with the data-type mismatch, an error similar to the following might occur:
ERROR: CLI execute error: Error: googleapi: Error 400: Query column 1 has type DATETIME which cannot be inserted into column dt, which has type TIMESTAMP at [1:35], invalidQuery
If you use the LIBNAME engine to insert a BigQuery table into another BigQuery table and the DBIDIRECTEXEC system option is enabled, SAS® attempts to pass the insert to BigQuery, which results in the error. SAS reads the input table into SAS and then issues inserts to load the data into the target BigQuery table rather than stopping and failing when the error occurs. This behavior allows the step to run. However, instead of seeing the error regarding the insert failing, you might notice that the insert takes longer than expected. This performance issue is due to the time it takes for the transfer of the data back and forth between the database and SAS.
SAS uses a data type of DATETIME when issuing the CREATE TABLE that is passed to BigQuery. For more information about this behavior, see SAS Note 67420, SAS/ACCESS Interface to Google BigQuery changes the default data type used from TIMESTAMP to DATETIME while creating Google BigQuery tables." If you try to use the table to insert into an existing BigQuery table with timestamp variables, you will encounter this issue.
The workaround is to use explicit pass-through to pass the INSERT statement directly to the database. Here is a simple example:
PROC SQL;
CONNECT TO BIGQUERY( ...);
EXECUTE(insert into <master table name> select name, timestamp(dob), gender from <input table name>) BY BIGQUERY;
DISCONNECT FROM BIGQUERY;
QUIT:
Operating System and Release Information
SAS System | SAS Data Connector to Google BigQuery | Linux for x64 | Viya | |
SAS System | SAS/ACCESS Interface to Google BigQuery (on SAS Viya) | Linux for x64 | Viya | |
SAS System | SAS/ACCESS Interface to Google BigQuery | Linux for x64 | | |
*
For software releases that are not yet generally available, the Fixed
Release is the software release in which the problem is planned to be
fixed.
Type: | Problem Note |
Priority: | high |
Date Modified: | 2021-05-26 11:25:52 |
Date Created: | 2021-05-21 09:15:26 |