Problem Note 55199: An upgrade of SAS® 9.4 software might fail or an out-of-memory condition might occur and result in the corruption of metadata
 |  |  |  |  |
SAS® Metadata Server repositories might become corrupted under the following conditions:
- when you perform an upgrade of SAS 9.4 software
- when you experience an out-of-memory condition
Specifically, the IDGROUP records for IDGROUP and METATYPE Foundation repository data sets might become damaged and prevent normal operation of the metadata server. The error messages that indicate the damage are detailed in the sections below.
SAS® 9.4 Software Upgrade
During an upgrade of SAS 9.4 software from a lower maintenance release to a higher maintenance release, the following error messages might appear in the metadata-server log file:
2014-10-12T23:28:59,393 INFO [00411091] :sas - Server stopped 2014-10-12T23:28:59,393 INFO [00000004] :sas - SAH019001I SAS Metadata Server (8561), State, stopping 2014-10-12T23:28:59,394 INFO [00411091] :sas - Processor has been stopped. 2014-10-12T23:28:59,412 WARN [00411091] :sas - DEBUG INTERNAL ERROR: Used IDs (0) don't match expected (-307905775) during reclaim. 2014-10-12T23:28:59,413 WARN [00411091] :sas - DEBUG INTERNAL ERROR: Used IDs (0) don't match expected (-198575167) during reclaim. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - Record not added to data set, I0000002.METATYPE. It has ID variable, A4JWIQRS.A2000004, which is a Duplicate ID. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - The SAS Metadata Server I/O subsystem has returned error message 'Duplicate keys are not allowed in this index.'. Contact the SAS Metadata Server administrator for assistance. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - Partial workunit commit. container=10 A4JWIQRS.metatype queue=1 ADD queue item=0. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - The error which caused the commit to fail is: 807fe934 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - The object cannot be added to the metatype container in the Foundation repository. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - Emergency situation requires we discard all in memory data which may have been affected. 2014-10-16T14:42:04,738 ERROR [00412727] 16187:sasadm@saspw - *** FOMSworkunitCommit: Close and unload container #1 A4JWIQRS.tree *** 2014-10-16T14:42:04,742 ERROR [00412727] 16187:sasadm@saspw - *** FOMSworkunitCommit: Close and unload container #2 A4JWIQRS.jfjob ***
Out-of-memory Condition SAS® Metadata Server
Importing a SAS package might result in an out-of-memory condition. This condition might not cause an immediate failure of the metadata server. However, errors such as the following might occur later:
2014-12-11T22:55:33,706 ERROR [01561] :sas - Record not added to data set, I0000032.IDGROUP. It has ID variable, A4YH32VX.A100000P, which is a Duplicate ID. 2014-12-11T22:55:33,706 ERROR [01561] :sas - The SAS Metadata Server I/O subsystem has returned error message 'Duplicate keys are not allowed in this index.'. Contact the SAS Metadata Server administrator for assistance. 2014-12-11T22:55:33,706 ERROR [01561] :sas - Record not added to data set, I0000032.IDGROUP. It has ID variable, A4YH32VX.A100000Q, which is a Duplicate ID. 2014-12-11T22:55:33,706 ERROR [01561] :sas - The SAS Metadata Server I/O subsystem has returned error message 'Duplicate keys are not allowed in this index.'. Contact the SAS Metadata Server administrator for assistance. 2014-12-11T22:55:33,706 ERROR [01561] :sas - Partial workunit commit. container=25 A4YH32VX.idgroup queue=1 ADD queue item=0. 2014-12-11T22:55:33,706 ERROR [01561] :sas - The error which caused the commit to fail is: 807fe934 2014-12-11T22:55:33,706 ERROR [01561] :sas - The object cannot be added to the idgroup container in the Foundation repository. 2014-12-11T22:55:33,706 ERROR [01561] :sas - Emergency situation requires we discard all in memory data which may have been affected. 2014-12-11T22:55:33,706 ERROR [01561] :sas - *** FOMSworkunitCommit: Close and unload container #1 A4YH32VX.tree ***
Sometimes, you might not know the cause of the original out-of-memory condition, but the metadata server might log messages similar to these when you attempt to back up metadata:
2014-11-03T01:00:00,091 INFO [01043906] :SYSTEM@NODE1 - A Backup has been started by META:Scheduler using CONTEXT 0009638F. 2014-11-03T01:00:00,137 INFO [01043906] :SYSTEM@NODE1 - Closed journal file. 2014-11-03T01:00:00,137 INFO [01043906] :SYSTEM@NODE1 - Initializing journal file: Journal/2014-11-03T01_00_00-05_00_Journal.dat 2014-11-03T01:00:00,137 INFO [01043906] :SYSTEM@NODE1 - Creating new metadata journal file: Journal/2014-11-03T01_00_00-05_00_Journal.dat 2014-11-03T01:00:00,137 INFO [01043906] :SYSTEM@NODE1 - Using the TKR compression algorithm for the journal file. 2014-11-03T01:00:00,137 INFO [01043906] :SYSTEM@NODE1 - Opened journal file Journal/2014-11-03T01_00_00-05_00_Journal.dat. 2014-11-03T01:00:00,137 WARN [01043906] :SYSTEM@NODE1 - DEBUG INTERNAL ERROR: Used IDs (39522) don't match expected (38522) during reclaim. 2014-11-03T01:00:00,153 INFO [01043906] :SYSTEM@NODE1 - NOTE: This transaction has been journalled but will not be applied to the permanent repository until the in progress backup is completed and journal reload is performed. Journal entry number 2721. 2014-11-03T01:00:00,184 INFO [00000015] :SYSTEM@NODE1 - User libref, TMP000, deassigned from path, C:\SAS\Config\Lev1\SASMeta\MetadataServer\MetadataRepositories\Foundation, and server libref, I0000050. 2014-11-03T01:00:02,321 INFO [00000003] :SYSTEM@NODE1 - AVL Index, ID, for dataset, I0000050.ASSCTNPR, was used 1 times with an average of 1 selections. The index was rejected 0 times with an average of 0 selections. The index was early rejected 0 times with an average more than 0 selections. 2014-11-03T01:00:02,385 INFO [01043906] :SYSTEM@NODE1 - CloseRepository Id=A43T7ETG, Name=Foundation. 2014-11-03T01:00:02,399 INFO [00000015] :SYSTEM@NODE1 - User libref, RPOSMGR, deassigned from path, C:\SAS\Config\Lev1\SASMeta\MetadataServer\rposmgr, and server libref, I0000049. 2014-11-03T01:00:02,571 INFO [00000003] :SYSTEM@NODE1 - AVL Index, ID, for dataset, I0000049.RPOSCTRL, was used 0 times with an average of 0 selections. The index was rejected 415 times with an average of 1 selections. The index was early rejected 0 times with an average more than 0 selections. 2014-11-03T01:00:02,602 INFO [01043906] :SYSTEM@NODE1 - CloseRepository Id=A0000001, Name=REPOSMGR. 2014-11-03T01:00:02,602 INFO [00000003] :SYSTEM@NODE1 - NOTE: Unable to open I0000075.METAIMDB for update. Index usage statistics will not be updated. 2014-11-03T01:00:02,602 INFO [00000015] :SYSTEM@NODE1 - User libref, RPOSMGR, assigned to path, C:\SAS\Config\Lev1\SASMeta\MetadataServer\rposmgr, with server libref, I0000075. 2014-11-03T01:00:02,618 INFO [00000003] :SYSTEM@NODE1 - NOTE: Unable to open I0000076.METAIMDB for update. Index usage statistics will not be updated. 2014-11-03T01:00:02,618 INFO [00000015] :SYSTEM@NODE1 - User libref, TMP000, assigned to path, C:\SAS\Config\Lev1\SASMeta\MetadataServer\MetadataRepositories\Foundation, with server libref, I0000076. 2014-11-03T01:00:02,665 INFO [01043906] :SYSTEM@NODE1 - OpenRepository Id=A43T7ETG, Name=Foundation, PauseState=READONLY, RepositoryFormat=15, Model=15.02. 2014-11-03T01:00:02,852 ERROR [01043906] :SYSTEM@NODE1 - The mdassoc container in the Foundation repository has an error in its ID registry entry, because the highest container ID is A3000UHU which exceeds the ID registry value of A3000UFN. An administrator should analyze and repair this repository.
Hot-Fix Details
Click the Hot Fix tab in this note to access the hot fix for this issue.
The hot fix prevents damage from occurring in the IDGROUP records for IDGROUP and METATYPE. In addition, the hot fix adds functionality to the Metadata Analyze and Repair Tools, which you can use to repair any damage that has already occurred.
Operating System and Release Information
| Product Family | Product | System | Product Release | SAS Release | ||
| Reported | Fixed* | Reported | Fixed* | |||
| SAS System | SAS Metadata Server | Microsoft® Windows® for x64 | 9.4 | 9.4 | 9.4 TS1M3 | |
| Microsoft Windows 8 Enterprise 32-bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8 Enterprise x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8 Pro 32-bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8 Pro x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8.1 Enterprise 32-bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8.1 Enterprise x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8.1 Pro | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows 8.1 Pro 32-bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2008 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2008 R2 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2008 for x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2012 Datacenter | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2012 R2 Datacenter | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2012 R2 Std | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Microsoft Windows Server 2012 Std | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Enterprise 32 bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Enterprise x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Home Premium 32 bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Home Premium x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Professional 32 bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Professional x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Ultimate 32 bit | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Windows 7 Ultimate x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| 64-bit Enabled AIX | 9.4 | 9.4 | 9.4 TS1M3 | |||
| 64-bit Enabled Solaris | 9.4 | 9.4 | 9.4 TS1M3 | |||
| HP-UX IPF | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Linux for x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
| Solaris for x64 | 9.4 | 9.4 | 9.4 TS1M3 | |||
A fix for this issue for Base SAS 9.4_M2 is available at:
https://tshf.sas.com/techsup/download/hotfix/HF2/R19.html#55199A fix for this issue for Base SAS 9.4_M1 is available at:
https://tshf.sas.com/techsup/download/hotfix/HF2/M88.html#55199
When you perform an upgrade of SAS 9.4 software, the task might fail and result in a corruption of metadata. An out-of-memory condition might also occur and result in a corruption of metadata. If either of these failures occur, the result is an unusable metadata server.
| Type: | Problem Note |
| Priority: | medium |
| Date Modified: | 2015-06-03 14:06:59 |
| Date Created: | 2015-02-11 10:35:55 |