HIVE_CURSOR_ERROR when Querying S3 Inventory With Athena - Is size column correct?

I'm attempting to do some analysis on one of our S3 buckets using Athena and I'm getting some errors that I can't explain or find solutions for anywhere I look.

The guide I'm following is .

I created my S3 inventory yesterday and have now received the first report in S3. The format is Apache ORC, the last export shows as yesterday and the additional fields stored are Size, Last modified, Storage class, Encryption.

I can see the data stored under s3://{my-inventory-bucket}/{my-bucket}/{my-inventory} so I know there is data there.

The default encryption on the inventory bucket and inventory configuration both have SSE-S3 encryption enabled.

To create the table, I am using the following query:

CREATE EXTERNAL TABLE my_table ( `bucket` string, key string, version_id string, is_latest boolean, is_delete_marker boolean, size bigint ) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat' LOCATION 's3://{my-inventory-bucket}/{my-bucket}/{my-inventory}/hive/'; 

Once the table has been created, I load the data using:

MSCK REPAIR TABLE my_table; 

The results from loading the data show that data has been loaded:

Partitions not in metastore: my_table=2021-07-17-00-00 Repair: Added partition to metastore my_table=2021-07-17-00-00 

Once that's loaded, I verify the data is available using:

SELECT DISTINCT dt FROM my_table ORDER BY 1 DESC limit 10; 

Which outputs:

1 2021-07-17-00-00 

Now if I run something like the below, everything runs fine and I get the expected results:

SELECT key FROM my_table ORDER BY 1 DESC limit 10; 

But as soon as I include the size column, I receive an error:

SELECT key, size FROM my_table ORDER BY 1 DESC limit 10; Your query has the following error(s): HIVE_CURSOR_ERROR: Failed to read ORC file: s3://{my-inventory-bucket}/{my-bucket}/{my-inventory}/data/{UUID}.orc This query ran against the "my_table" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: {UUID}. 

I feel like I've got something wrong with my size column. Can anyone help figure this out?

1 Answer

So frustrating. Think I found the answer here:

IsLatest – Set to True if the object is the current version of the object. (This field is not included if the list is only for the current version of objects.) 

Removing that column fixed the problem.