FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

[Mich Talebzadeh]

Hive version 3

An ORC partitioned table

0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
generation failed with exception: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)

Something is missing here! Is this specific to ORC tables?

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

[Prasanth Jayachandran]

Hi
What hadoop version are you using? I am guessing this is from old hadoop version. Can you try with hadoop-3.1?

Thanks
Prasanth



On Mon, Jun 25, 2018 at 9:55 AM -0700, "Mich Talebzadeh" <***@gmail.com<mailto:***@gmail.com>> wrote:

Hive version 3

An ORC partitioned table

0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I (state=08S01,code=1)

Something is missing here! Is this specific to ORC tables?


Dr Mich Talebzadeh



LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

[Mich Talebzadeh]

Thanks

This is Hadoop 3.0.3

hadoop version
Hadoop 3.0.3
Source code repository
https://***@git-wip-us.apache.org/repos/asf/hadoop.git -r
37fd7d752db73d984dc31e0cdfd590d252f5e075
Compiled by yzhang on 2018-05-31T17:12Z
Compiled with protoc 2.5.0
From source with checksum 736cdcefa911261ad56d2d120bf1fa
This command was run using
/home/hduser/hadoop-3.0.3/share/hadoop/common/hadoop-common-3.0.3.jar

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 25 Jun 2018 at 18:05, Prasanth Jayachandran <

Post by Prasanth Jayachandran
Hi
What hadoop version are you using? I am guessing this is from old hadoop
version. Can you try with hadoop-3.1?
Thanks
Prasanth
On Mon, Jun 25, 2018 at 9:55 AM -0700, "Mich Talebzadeh" <
Hive version 3

Post by Mich Talebzadeh
An ORC partitioned table
0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)
Something is missing here! Is this specific to ORC tables?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for
any loss, damage or destruction of data or any other property which may
arise from relying on this email's technical content is explicitly
disclaimed. The author will in no case be liable for any monetary damages
arising from such loss, damage or destruction.

[Gopal Vijayaraghavan]

Post by Mich Talebzadeh
This is Hadoop 3.0.3
java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I (state=08S01,code=1)
Something is missing here! Is this specific to ORC tables?

No, it is a Hadoop BUG.

https://issues.apache.org/jira/browse/HADOOP-1468

Fixed in Hadoop-2.8.2+ & most likely this is happening because you have built something against Hadoop-2.7.3 (Parquet and ORC default to the 2.7.3, so you need to rebuild them over for Hadoop3).

Cheers,
Gopal

[Mich Talebzadeh]

Thanks

I assume two things

Create these ORC tables from scratch and populate them. These are older
tables from 2.7

Do I need to upgrade to Hadoop 3.1 as suggested as well? Or I can keep the
current 3.0.3 of Hadoop and just redo these ORC tables again

Sounds like I need to upgrade Hadoop to 3.1?


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Post by Mich Talebzadeh

Post by Mich Talebzadeh
This is Hadoop 3.0.3

org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)

Post by Mich Talebzadeh
Something is missing here! Is this specific to ORC tables?

No, it is a Hadoop BUG.
https://issues.apache.org/jira/browse/HADOOP-1468
Fixed in Hadoop-2.8.2+ & most likely this is happening because you have
built something against Hadoop-2.7.3 (Parquet and ORC default to the 2.7.3,
so you need to rebuild them over for Hadoop3).
Cheers,
Gopal

[Mich Talebzadeh]

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



With Hadoop 3.0.3 I had the following

0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
generation failed with exception: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)


I upgraded Hadoop from 3.0.3 to 3.1.0and the same operation on that ORC
table works now


0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
+---------+
| _c0 |
+---------+
| 917359 |
+---------+

Post by Prasanth Jayachandran
Thanks
I assume two things
Create these ORC tables from scratch and populate them. These are older
tables from 2.7
Do I need to upgrade to Hadoop 3.1 as suggested as well? Or I can keep the
current 3.0.3 of Hadoop and just redo these ORC tables again
Sounds like I need to upgrade Hadoop to 3.1?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Post by Mich Talebzadeh

Post by Mich Talebzadeh
This is Hadoop 3.0.3

org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)

Post by Mich Talebzadeh
Something is missing here! Is this specific to ORC tables?

No, it is a Hadoop BUG.
https://issues.apache.org/jira/browse/HADOOP-1468
Fixed in Hadoop-2.8.2+ & most likely this is happening because you have
built something against Hadoop-2.7.3 (Parquet and ORC default to the 2.7.3,
so you need to rebuild them over for Hadoop3).
Cheers,
Gopal