incr.comp.: Don't encode Fingerprint values with leb128.

[michaelwoerister]

Since Fingerprint values have roughly random distribution, most of them will not profit from being stored in a variable-length encoding:

• encoding to leb128 takes additional time, and
• in the common case, the leb128 representation of the two 64 bit numbers in a Fingerprint will take up around 160 bits, so we are even wasting space.

We should not do that. UseSpecializedEncodable and UseSpecializedDecodable might be the way to circumvent the standard encoding methods of the opaque::Encoder.

[Xanewok]

I’d like to take that on, since it seems self-contained and at least I know the Fingerprint type :laugh:
Would you mind posting more (specific) mentoring instructions on how would you like to see it implemented?

[michaelwoerister]

@Xanewok Sure, let's see:

• Our libserialize defines the UseSpecializedEncodable and UseSpecializedDecodable traits. If you implement this trait for a type, you can provide encoder- and decoder-specific implementations for encoding and decoding. This way you can have implementations that have access to things that are not part of the general Encoder and Decoder interface. When implementing the UseSpecializedXxx traits, you have to provide a fallback implementation that only relies on the regular Encoder and Decoder, like here:
  

  rust/src/libsyntax/ast.rs

  Lines 240 to 250 in d5ff0e6

  	impl serialize::UseSpecializedEncodable for NodeId {
  	fn default_encode<S: Encoder>(&self, s: &mut S) -> Result<(), S::Error> {
  	s.emit_u32(self.0)
  	}
  	}
  	impl serialize::UseSpecializedDecodable for NodeId {
  	fn default_decode<D: Decoder>(d: &mut D) -> Result<NodeId, D::Error> {
  	d.read_u32().map(NodeId)
  	}
  	}

• We can use the above functionality to do two things:
  • Extend serialize::opaque::Encoder and serialize::opaque::Decoder with methods that allow saving raw bytes.
  • Provide serialization methods for Fingerprint that are specialized to serialize::opaque::Encoder and serialize::opaque::Decoder which use these methods directly. An example how such as specialized implementation can be provided can be found here:

    rust/src/librustc_metadata/decoder.rs

    Lines 289 to 298 in d5ff0e6

    	impl<'a, 'tcx> SpecializedDecoder<CrateNum> for DecodeContext<'a, 'tcx> {
    	fn specialized_decode(&mut self) -> Result<CrateNum, Self::Error> {
    	let cnum = CrateNum::from_u32(u32::decode(self)?);
    	if cnum == LOCAL_CRATE {
    	Ok(self.cdata().cnum)
    	} else {
    	Ok(self.cdata().cnum_map.borrow()[cnum])
    	}
    	}
    	}

    In this example, you would replace DecodeContext with serialize::opaque::Decoder.

Let me know if you need more info to get started.

[michaelwoerister]

It would also be great to get some numbers on how this affects the size of the dep-graph.bin file in the incr. comp. session directory.

[wesleywiser]

Hi @Xanewok, are you still working on this?

[Xanewok]

Sorry, couldn't find more time to work on this. I'm afraid it's not going to change until the holidays, so @wesleywiser it's all yours if you want to work on this!

[michaelwoerister]

@wesleywiser, how is it going? Did you make any progress?

[wesleywiser]

@michaelwoerister I made some progress over the weekend. I've added methods to the Encoder and Decoder types for manipulating raw bytes using the same kind of technique as the code that handles str values (emit usize length prefix and then raw bytes). This got me thinking, by using these functions to handle the Fingerprint type, it will use one usize plus a [u8; 16] consuming 192 bits on a 64 bit platform. Was there a reason we wanted to do the raw bytes approach instead of just calling {emit,read}_u64 twice? It looks like there is also a u128 variant we could use if we changed Fingerprint to be a tuple struct of one u128 instead of two u64 values (or, I suppose we could just combine them during serialization). Thoughts?

[michaelwoerister]

This got me thinking, by using these functions to handle the Fingerprint type, it will use one usize plus a [u8; 16] consuming 192 bits on a 64 bit platform.

For these methods we assume that we already know how many bytes we want to read or write, so you don't have to encode the length. I suggest making the parameter for the write method &[u8] and the parameter for the read method &mut [u8]. In both cases the length of the slice tells you how many bytes you are dealing with. Both methods would return a Result<(), _> and for the read method we assume that the requested bytes are in the buffer if the method returned Ok(()).

Was there a reason we wanted to do the raw bytes approach instead of just calling {emit,read}_u64 twice?

These methods will use the leb128 encoding, which is what we want to avoid (because we know that fingerprints are random values and thus won't profit from a variable-length encoding).

[wesleywiser]

Thanks @michaelwoerister, that's really helpful! I've implemented UseSpecialized{Encoder,Decoder} and Specialized{Encoder,Decoder} but it doesn't seem to be working. I've stubbed out the function bodies with panic!()s but running the resulting compiler doesn't panic. Any ideas?

(Code is here and I'm invoking the compiler with CARGO_INCREMENTAL=1 cargo build)