Handle erroneously empty rfile block reads

[ddanielr]

Describe the bug
When HDFS returns empty results for a rfile block that actually has data, Accumulo does not error or attempt additional validation of the rfile block. This can cause issues with metadata table reads as specific tablet contents fail to return.

Versions (OS, Maven, Java, and others, as appropriate):

• Affected version(s) of this project: 1.10.3

Expected behavior
Accumulo should be able to determine if a rfile content location is actually empty or if data should have been returned.

[dlmarion]

So many questions:

You are saying that HDFS is returning a zero byte file block and the RFile Reader does not fail?
Does the DataNode log any error in this case?
What does fsck say about the file?
Is this a transient issue?
Are other copies of the block correct?

[ddanielr]

You are saying that HDFS is returning a zero byte file block and the RFile Reader does not fail?

The RFile Reader does not fail and the HDFS contents are currently speculated to be empty.
Dmesg reports IO errors for the disk that hosted the rfile block.

Does the DataNode log any error in this case?

The DataNode log does report Slow flushOrSync operations for the drive that contained the requested rfile block.

What does fsck say about the file?

Currently unknown

Is this a transient issue?

Yes, on subsequent reads, the rfile contents are returned correctly.

Are other copies of the block correct?

Unknown, but is speculated that HDFS did not attempt to query the other block locations.

---

This issue is an attempt to chase down a transient bug with the AccumuloGC.
For this situation, the metadata table is separated into different locality groups groupA: loc,~tab, groupB: file,log,srv,future.

The tserver that hosted the groupB locality group reported disk errors.

The AccumuloGC queried the metadata table to find file references for deletion candidates.
The metadata table scan returned ~tab:~pr entries but failed to return any srv or file entries.

Because of this, the GC concluded that no file references existed for a collection of deletion candidates and then removed files from HDFS that had valid references on the metadata table.

[ctubbsii]

The AccumuloGC queried the metadata table to find file references for deletion candidates. The metadata table scan returned ~tab:~pr entries but failed to return any srv or file entries.

Because of this, the GC concluded that no file references existed for a collection of deletion candidates and then removed files from HDFS that had valid references on the metadata table.

This implies that one measure to try to prevent this from happening again could be to remove the locality groups in the metadata, so the file entries are in the same locality group as the file entries. However, this comes with its own problems: First, you'd have to force a full major compaction on the metadata table to force this to take effect (and that could be costly... and the IO could stress whatever underlying filesystem or disk errors are causing the problem in the first place). Second, this could cause slower metadata performance. Third, there may still be weird behavior where file entries and prev row entries are in different RFile blocks or different HDFS blocks, and there could still be hidden errors when one block is read and not when the other is read... it's not clear to me how that would actually manifest itself, though (because it's not clear to me why HDFS errors are failing to propagate up into our code to begin with).

[keith-turner]

I was reviewing some of the 2.1 Rfile code today and I realized this issue happened in 1.10. That made me diff the 1.10 and 2.1 code and I found that the fixes (#3622 and #3623) for #3617 are not in 1.10. Not sure if that matters, but just recording the finding.