Is it possible to get (or compile) .pdb file for sharp.node for specfic sharp version?

[TomaterID]

Question about an existing feature

What are you trying to achieve?

I am currently using sharp in my photo management application Tonfotos. Library does great job, thank you guys!

However, it crushes from time to time at customer side. I am collecting crash dumps, but unfortunately I cannot get much of useful information from it as I don't have .pdb file corresponding to the binary that is being shipped with npm module. Best stack trace I can get looks like that:

(... skipped windows internals...)
KERNELBASE!RaiseException+0x69
sharp_win32_x64+0x294cc
sharp_win32_x64+0xbac6
sharp_win32_x64+0x421e2
sharp_win32_x64+0x2d680
sharp_win32_x64+0x2c5dd
(... more skipped ....)

Obviously somehting is happened inside sharp, but we will never know what exactly and where until we have corresponding .pdb file. I am using sharp 0.29.3, and I would love to update, if I would be able to check if that bug was indeed fixed in later versions. But this amount of information is far from enought to find similar issues here. Having name of the failing function will be a great help. I would be able to see the code changes history and make educated decision about version to chose.

Last resort would be to upgrade to latest and pray, but that would be a lottery. This will not guarantee that this issue will go, but would give non-zero probability of adding new issues, too. (No offense, it is just the way life is.)

When you searched for similar issues, what did you find that might be related?

Unfortunately, I was not able to find relevant discussions. Everything with 'pdb' keyword looks to be installation issue.

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Hmm... not sure this is relevant to my question.

Please provide sample image(s) that help explain this question

I have no idea what causes crash, is it connected to any images, etc. It is happening at customer side and all I get is crash dump from crashpad.

So I would be greateful for any help:

1. Instructions how do I build sharp.pdb for my version of sharp
2. Or, mabe there is already prebuilt one somewhere
3. Or maybe you know for sure this bug was already fixed and can point me to the right version to upgrade to
4. Or any other suggestions.

Thank you very much!

[lovell]

Rather than the sharp binary, I suspect this will be coming from one of libvips dependencies. This means you'll need to build your own debug binaries for the specific version of libvips you're using with https://github.com/libvips/build-win64-mxe and its --with-debug option.

I'm 80% certain this will relate to font discovery/rendering. The next v0.32.0 release of sharp will include prebuilt binaries that include the latest cairo, pango and fontconfig, all of which have seen a lot of font-related fixes and improvements in the last year, so waiting might be easier.

[TomaterID]

Thank you for your reply! My next question was about libvips, since I have crashes with this one too, and now I will try building it myself and saving PDB file for future. However, my original question was indeed about sharp, as it does crash. Any ideas how to debug this?

[lovell]

Please can you provide a full stack trace of a crash that you believe is caused directly by sharp (as opposed to a crash from one of libvips dependencies where a call to sharp happens to appear in the call stack).

[TomaterID]

Sure. This archive includes sample .dmp file as well as stack trace decoded by cdb.exe
sharp_win32_x64+0x294cc.zip

[lovell]

Thanks, the salient part appears to be:

0000007a`4a7f9f80 00007ff7`1c4f0d37     : 00000000`00000002 00007ff8`00000000 0000007a`c00000bb 00000000`00000000 : KERNELBASE!SleepEx+0x9e
0000007a`4a7fa020 00007ff8`d83a0207     : 00000000`00000000 00007ff7`1c4f0c70 00000000`00000000 aaaa0064`6c6f6873 : tonfotos!crashpad::`anonymous namespace'::UnhandledExceptionHandler+0xc7
0000007a`4a7fa1a0 00007ff8`da655530     : 0000007a`4a7fa3e0 00007ff8`da6f7658 00000000`00000000 0000007a`4a7fa378 : KERNELBASE!UnhandledExceptionFilter+0x1e7
0000007a`4a7fa2c0 00007ff8`da63c876     : 00007ff8`da724a24 00007ff8`da5b0000 0000007a`4a7fa3e0 00007ff8`da5e0e7b : ntdll!memset+0x13b0
0000007a`4a7fa300 00007ff8`da65241f     : 00000000`00000000 0000007a`4a7fa8e0 0000007a`4a7fb340 00000000`00000000 : ntdll!_C_specific_handler+0x96
0000007a`4a7fa370 00007ff8`da6014a4     : 00000000`00000000 0000007a`4a7fa8e0 0000007a`4a7fb340 00000000`00000001 : ntdll!_chkstk+0x11f
0000007a`4a7fa3a0 00007ff8`da6011f5     : 00000000`00000000 0000007a`4a7fb1e0 00000000`00000000 0000007a`4a7faaf0 : ntdll!RtlRaiseException+0x434
0000007a`4a7faab0 00007ff8`d82bcd29     : 0000007a`4a7fb388 00007ff8`b33e7020 0000007a`4a7fb4a0 0000007a`4a7fb4b8 : ntdll!RtlRaiseException+0x185
*** WARNING: Unable to verify checksum for sharp-win32-x64.node
0000007a`4a7fb320 00007ff8`b33b94cc     : 00006832`00e3df80 00000000`00000000 0000007a`4a7fc370 00007ff8`b3392154 : KERNELBASE!RaiseException+0x69
0000007a`4a7fb400 00007ff8`b339bac6     : 0000007a`4a7fd960 0000007a`4a7fd7b0 0000007a`4a7fd960 0000007a`4a7fd7b0 : sharp_win32_x64+0x294cc
0000007a`4a7fb460 00007ff8`b33d21e2     : 0000007a`4a7fc370 00007ff8`b33ba0a5 0000007a`4a7fb658 0000007a`4a7fd7b0 : sharp_win32_x64+0xbac6
0000007a`4a7fb560 00007ff8`b33bd680     : 00007ff8`b33d21cc 0000007a`4a7fda60 0000007a`4a7fda60 aaaaaaaa`aaaaaaaa : sharp_win32_x64+0x421e2
0000007a`4a7fb5a0 00007ff8`b33bc5dd     : 00007ff8`b33d21cc 0000007a`4a7fc4d8 0000034f`00000100 00007ff7`1a5a311c : sharp_win32_x64+0x2d680
0000007a`4a7fb5d0 00007ff8`da6517a6     : 00000000`00000000 00000000`00000002 0912df01`00000000 0000007a`4a7fd7b0 : sharp_win32_x64+0x2c5dd
0000007a`4a7fb6b0 00007ff8`b33a01eb     : aaaaaaaa`aaaaaaaa 0000024d`85289300 00006832`00e3df80 0000034f`cfc6a7d1 : ntdll!RtlCaptureContext2+0x4a6 (TrapFrame @ 0000007a`4a7fba38)
0000007a`4a7fda60 00007ff8`b339cb94     : 00006832`0061fff0 00007ff7`1a541d49 0000034f`cfc6a721 00006832`01a6bd28 : sharp_win32_x64+0x101eb
0000007a`4a7fdb10 00007ff7`194955bc     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa : sharp_win32_x64+0xcb94
(Inline Function) --------`--------     : --------`-------- --------`-------- --------`-------- --------`-------- : tonfotos!`anonymous namespace'::uvimpl::Work::AfterThreadPoolWork::<lambda_1>::operator
()+0x3e (Inline Function @ 00007ff7`194955bc)
(Inline Function) --------`--------     : --------`-------- --------`-------- --------`-------- --------`-------- : tonfotos!napi_env__::CallIntoModule+0x4c (Inline Function @ 00007ff7`194955bc)
0000007a`4a7fdbb0 00007ff7`195d0388     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa 0000007a`4a7fdcb0 00007ff7`1f5a02e0 : tonfotos!`anonymous namespace'::uvimpl::Work::AfterThreadPoolWork+0xdc
0000007a`4a7fdc90 00007ff7`18fabcbc     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa : tonfotos!uv__work_done+0xc8
(Inline Function) --------`--------     : --------`-------- --------`-------- --------`-------- --------`-------- : tonfotos!uv_process_reqs+0x16f (Inline Function @ 00007ff7`18fabcbc)
0000007a`4a7fdcf0 00007ff7`18f7872b     : 00000000`00000000 00006832`002d1080 00006832`0060e900 00006832`00708000 : tonfotos!uv_run+0x1ec
0000007a`4a7fed90 00007ff7`18f78b22     : 00000000`000007bc 00007ff8`00000000 00000000`00000000 00007ff7`1aa63023 : tonfotos!node::Environment::CleanupHandles+0x14b
0000007a`4a7fee00 00007ff7`18f4502c     : 00007ff7`1f4f6260 00007ff7`1f4f6260 00006832`00708380 00007ff7`18f785c5 : tonfotos!node::Environment::RunCleanup+0xc2
0000007a`4a7ff080 00007ff7`1798c4f4     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa 00007ff7`1f4f6260 aaaaaaaa`aaaaaaaa : tonfotos!node::FreeEnvironment+0x6c
0000007a`4a7ff0e0 00007ff7`1797d723     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa 00000000`00000030 0000007a`4a7ff248 : tonfotos!electron::NodeEnvironment::~NodeEnvironment+0x14
(Inline Function) --------`--------     : --------`-------- --------`-------- --------`-------- --------`-------- : tonfotos!std::__1::default_delete<electron::NodeEnvironment>::operator()+0x8 (Inline Fu
nction @ 00007ff7`1797d723)
(Inline Function) --------`--------     : --------`-------- --------`-------- --------`-------- --------`-------- : tonfotos!std::__1::unique_ptr<electron::NodeEnvironment,std::__1::default_delete<electr
on::NodeEnvironment> >::reset+0x19 (Inline Function @ 00007ff7`1797d723)
0000007a`4a7ff110 00007ff7`1846ca9a     : 0000007a`4a7ff270 0000007a`4a7ff538 00000000`00000000 0000007a`4a7ff268 : tonfotos!electron::ElectronBrowserMainParts::PostMainMessageLoopRun+0xc3
0000007a`4a7ff1c0 00007ff7`1846e2ed     : 00006832`002bc8a0 0000007a`4a7ff538 00000000`00000000 aaaaaaaa`aaaaaaaa : tonfotos!content::BrowserMainLoop::ShutdownThreadsAndCleanUp+0x1ca
0000007a`4a7ff2d0 00007ff7`18469bb8     : 0000007a`4a7ff4a0 00007ff7`1aed54ee 00006832`00291970 00006832`002408c0 : tonfotos!content::BrowserMainRunnerImpl::Shutdown+0xad

This looks like it might be an attempt to shutdown libuv whilst threads from its worker pool are still processing data. If you've not seen it, Electron provides an API to handle graceful shutdowns - https://www.electronjs.org/docs/latest/api/app#event-before-quit

The stack dump mentions Electron v14.2.6, which is EOL and uses an out-dated Node.js, so I'd recommend upgrading that bit first.

[TomaterID]

Sorry, I am not sure I understand. Yes, there is 'before quit' event in electron and I use it. Is there something I should also do with sharp library too in that call in order to avoid such crashes?

Updating Electron is huge effort as it contains lots of breaking changes. And it works just fine. In any case, it does not look like it is who is causing crashes, so I don't think this is the priority right now.

[TomaterID]

By the way, I provided you with just one crash dump, but the thing is, this is what happens regularly for different users. And the stack is always the same, it always has this sharp_win32_x64+0x294cc line.

[lovell]

Did you see sharp.counters() and the sharp.queue event emitter? These will allow you to check for any in-flight processing. You should prevent shutdown until these are all complete, i.e. counters are all zero.

https://sharp.pixelplumbing.com/api-utility#counters
https://sharp.pixelplumbing.com/api-utility#queue

[lovell]

Upgrading Electron to a version with Node.js 16 would bring in nodejs/node#35021 that might fix this.

[kleisauke]

Building a PDB locally for sharp v0.29.3 using this patch:

--- a/node_modules/sharp/binding.gyp
+++ b/node_modules/sharp/binding.gyp
@@ -205,7 +205,8 @@
               'VCCLCompilerTool': {
                 'ExceptionHandling': 1,
                 'Optimization': 1,
-                'WholeProgramOptimization': 'true'
+                'WholeProgramOptimization': 'true',
+                'DebugInformationFormat': 3 # Generate a PDB
               },
               'VCLibrarianTool': {
                 'AdditionalOptions': [

Produces this stack trace in WinDbg (the .symopt +0x40 option was used to load mismatched PDBs):

00 00007ff8`b33b94cc     : 00006832`00e3df80 00000000`00000000 0000007a`4a7fc370 00007ff8`b3392154 : KERNELBASE!RaiseException+0x69
01 00006832`00e3df80     : 00000000`00000000 0000007a`4a7fc370 00007ff8`b3392154 00007ff8`b3390000 : sharp_win32_x64!_invalid_parameter+0x9c [minkernel\crts\ucrt\src\appcrt\misc\invalid_parameter.cpp @ 112] 
02 00000000`00000000     : 0000007a`4a7fc370 00007ff8`b3392154 00007ff8`b3390000 00000000`19930520 : 0x00006832`00e3df80

So, it looks like you ran into #2999, which was fixed in sharp v0.30.0.

[kleisauke]

So, it looks like you ran into #2999, which was fixed in sharp v0.30.0.

Actually, the stack trace of that issue was:

Details

ucrtbase!invoke_watson+0x18
ucrtbase!_invalid_parameter_internal+0x39260
ucrtbase!invalid_parameter+0x2c
ucrtbase!invalid_parameter_noinfo+0x9
ucrtbase!_get_osfhandle+0x3bf1f
libvips_42!vips__open+0x51
libvips_42!vips_tracked_open+0xa
libvips_42!vips_target_write_amp+0x655
libvips_42!vips_object_build+0x19
libvips_42!vips_target_new_to_file+0xae
libvips_42!vips_jpegsave_mime+0x996
libvips_42!vips_object_build+0x19
libvips_42!vips_cache_operation_buildp+0x43
libvips_cpp!vips::VImage::call_option_string+0x80 [node_modules\sharp\src\libvips\cplusplus\VImage.cpp @ 536] 
libvips_cpp!vips::VImage::call+0x11 [node_modules\sharp\src\libvips\cplusplus\VImage.cpp @ 562] 
libvips_cpp!vips::VImage::jpegsave+0xea [node_modules\sharp\src\libvips\cplusplus\vips-operators.cpp @ 1777] 
sharp_win32_x64!PipelineWorker::Execute+0x5cb9 [node_modules\sharp\src\pipeline.cc @ 975] 
sharp_win32_x64!Napi::AsyncWorker::OnExecute+0x1e [node_modules\node-addon-api\napi-inl.h @ 4890] 
node!v8::base::SharedMutex::SharedMutex+0x8e8
node!uv_queue_work+0x28e
node!uv_poll_stop+0xed
node!inflateValidate+0x24c30
KERNEL32!BaseThreadInitThunk+0x1d
ntdll!RtlUserThreadStart+0x28

So, it's probably not related to that.

[TomaterID]

@lovell thank you for clarification. Actually, I have lots of other native stuff going on in background of my application, but I don't need to do any magic in 'before-quit' to wait for it to finish, as electron (the node core) just waits for all running tasks to finish before killing the process. I have pretty heavy computations for face recognition, and node waits for it to finish without issue. Therefore my quess is that in sharp case it will behave the same way, unless there is bug in sharp. My point is, rather than building strange workarounds it would make sense to figure out what exactly is wrong and maybe fix it.

As I can see from discussion (thank you guys!), there is no clarity on what exaclty this bug is. Therefore my plan probably should remain as it was:

• Generate .pdb and new sharp.node (AFAIK you can't build compatible PDB for old code, the only way is to build one together with code)
• Release new version, collect new dumps
• Analyze the problem with confidence and then decide what to do - fix/upgrade/make workaround in 'before-quit'/etc.

Please let me know what you think. In the mean time I will try to figure out how to compile sharp.node when it is installed as npm library.

[lovell]

Were you able to add a call to sharp.counters() in the before-quit event handler to verify the queue length? If so, have you been able to confirm that this always zero when Electron shuts down all threads?

[TomaterID]

I don't believe this will give us anything regarding this issue for multiple reasons:

• In electron 'will-quit' is called when app is only intending to quit, not right before it will be killed. At that moment sharp.counters() could legitimately be >0, but that does not mean anything. Right in response to this call I will initiate shutdown of background processing. And sometimes several seconds can pass between this call and actual moment program gets shut down (i.e. all threads finished their execution). I guess you would like to know counters at that final moment, but this is not where 'will-quit' can help us.
• Moreover, even though this bug occurs from time to time, it only occurs in production, at user site. I cannot reproduce it in my test environment. I believe it would be helpful to know counters value right before program crushes, but I have no idea how to pass that value to me together with crash log.