Segfault in LuaTable::set

[jonas-schievink]

After a long debugging session, I've noticed that cargo test --release fails with a segfault on the current nightly.

This might as well be a regression of the compiler, I'm not yet sure.

[jonas-schievink]

Valgrind stacktrace (take this with a grain of salt, it's optimized after all):

==11084== Invalid read of size 8
==11084==    at 0x14F0E6: lua_touserdata (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x1256DA: rlua::util::error_guard::call_impl (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x601C9CF: ???
==11084==    by 0x162641: luaD_callnoyield (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x160351: luaD_rawrunprotected (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x162DDE: luaD_pcall (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x152354: lua_pcallk (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x127CA2: rlua::lua::LuaTable::set (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)
==11084==    by 0x144779: rlua::tests::test_error (in /home/jonas/dev/rlua/target/release/rlua-c35ad98123863f87)

==11084== by 0x601C9CF: ??? looks pretty suspicious.

Instruction at fault:

Dump of assembler code for function lua_touserdata:
   0x000055555559b0e0 <+0>:	sub    rsp,0x8
   0x000055555559b0e4 <+4>:	test   esi,esi
=> 0x000055555559b0e6 <+6>:	mov    r8,QWORD PTR [rdi+0x20]

rdi is the first argument, ie. the pointer to the Lua state, and has a value of 0x40 (64), which is obviously not a valid pointer.

I suspect that luaD_callnoyield calls a bad function pointer, causing this mess.

[kyren]

hmmmm, that's concerning, it's ONLY on the current nightly compiler? Do you have a reproduction I could look at?

[jonas-schievink]

It's also at least on yesterday's nightly, but I neither have the bandwidth nor the disk space to bisect. The reproduction is just to run cargo test --release (I'm referring to rluas own tests here). Maybe it won't reproduce on other platforms, though.

EDIT: I'm on x64 Linux, for reference

[jonas-schievink]

The easiest way to cause this:

let lua = Lua::new();
lua.globals().set("bla", LuaNil).unwrap();

It happens with any value, not just LuaNil.

[kyren]

It doesn't seem to be failing anymore, but it might be because I have "fixed" #19. I say "fixed" because I'm half convinced nothing will ever be fixed again, and that the Lua API is designed to drive me insane.

[jonas-schievink]

@kyren I can still reproduce this

[kyren]

It's not failing on travis, maybe I was incorrectly assuming travis was x86_64 linux? Let me actually boot up a vm and try it.

[kyren]

I completely can't reproduce this, all tests pass on my linux x86_64 vm on the latest nightly.

I realize that's not super helpful.

[jonas-schievink]

Odd. Meanwhile, I've also reproduced this on ARM (also Linux). My exact rustc version, for the record:

rustc 1.20.0-nightly (ae98ebfcb 2017-07-20)
binary: rustc
commit-hash: ae98ebfcb9ad5a5384fd229a6ee91315b02ca969
commit-date: 2017-07-20
host: x86_64-unknown-linux-gnu
release: 1.20.0-nightly
LLVM version: 4.0

Also reproduces on rustc 1.20.0-nightly (8d22af87d 2017-07-22) (the current one).

Maybe it's a difference between C compilers that's exposing UB? I'm using gcc version 7.1.1 20170630, but I can reproduce with clang 4.0.1, too.

[kyren]

I should be more clear, I said:

It doesn't seem to be failing anymore

But what I should have said is I actually have not seen this fail at all so far.

[kyren]

I have tried gcc version 6.3, but I can try 7.1.1. I've also tried whatever the latest apple clang is, I guess Apple LLVM version 8.1.0 (clang-802.0.42)

[kyren]

Actually can you just tell me what OS you're using, that might be faster than trying to set up a specific compiler version

[jonas-schievink]

Oh, right. Arch Linux.

Currently bisecting by the way, so no need to start that if you manage to reproduce.