Problem Note 67986: A "NoSuchMethodError" message appears when you access SAS® SPDE data sets through Hive in Cloudera Hadoop 7.1
 |  |  |  |  |
When you access SAS® Scalable Performance Data Engine (SPDE) data sets through Apache Hive in Cloudera Hadoop 7.1, a “NoSuchMethodError” message might occur.
Here is an example of the error message:
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1612553652178_0007_1_00, diagnostics=[Vertex vertex_1612553652178_0007_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: misc11 initializer failed, vertex=vertex_1612553652178_0007_1_00 [Map 1], java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.plan.MapWork.getPathToPartitionInfo()Ljava/util/LinkedHashMap;
at com.sas.hadoop.serde.spde.hive.SPDInputFormat.setObsLength(SPDInputFormat.java:92)
at com.sas.hadoop.serde.spde.hive.SPDInputFormat.getSplits(SPDInputFormat.java:65)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:541)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:830)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:249)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:280)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:271)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:255)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]
at com.sas.hadoop.serde.spde.hive.SPDInputFormat.setObsLength(SPDInputFormat.java:92)
at com.sas.hadoop.serde.spde.hive.SPDInputFormat.getSplits(SPDInputFormat.java:65)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:541)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:830)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:249)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:280)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:271)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:255)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]
This error occurs because of a change that was made to the getPathToPartitionInfo() Hive function in Cloudera Hadoop 7.
Click the Hot Fix tab in this note to access the hot fix for this issue.
Operating System and Release Information
| Product Family | Product | System | Product Release | SAS Release | ||
| Reported | Fixed* | Reported | Fixed* | |||
| SAS System | Base SAS | z/OS | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 |
| z/OS 64-bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft® Windows® for x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8 Enterprise 32-bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8 Enterprise x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8 Pro 32-bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8 Pro x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8.1 Enterprise 32-bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8.1 Enterprise x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8.1 Pro 32-bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 8.1 Pro x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows 10 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2008 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2008 R2 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2008 for x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2012 Datacenter | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2012 R2 Datacenter | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2012 R2 Std | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2012 Std | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2016 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Microsoft Windows Server 2019 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Enterprise 32 bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Enterprise x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Home Premium 32 bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Home Premium x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Professional 32 bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Professional x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Ultimate 32 bit | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Windows 7 Ultimate x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| 64-bit Enabled AIX | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| 64-bit Enabled Solaris | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| HP-UX IPF | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Linux for x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
| Solaris for x64 | 9.4_M7 | 9.4_M7 | 9.4 TS1M7 | 9.4 TS1M7 | ||
A fix for this issue for SAS Hive SerDe for SPDE Data 9.44 is available at:
https://tshf.sas.com/techsup/download/hotfix/HF2/K4D.html#67986| Type: | Problem Note |
| Priority: | medium |
| Date Modified: | 2021-06-08 12:35:29 |
| Date Created: | 2021-06-01 17:41:57 |