Fix recursion in QLoggingRegistry

QLoggingRegistry relies on any Qt code it uses not having category
logging in it, because it causes a global variable to be recursively
initialized. This has been the case so far, but with CTF tracing
enabled, there is Qt code that uses category logging if the first
tracing event originates from a logging category, causing the tracing
library to be loaded.

It is undefined behavior to recurse into the initialization.

Here is the backtrace of the situation:

QGlobalStatic<QtGlobalStatic::Holder<()::Q_QGS_qtLoggingRegistry> >::instance
QGlobalStatic<QtGlobalStatic::Holder<()::Q_QGS_qtLoggingRegistry> >::operator()
QLoggingRegistry::instance
QLoggingCategoryWithEnvironmentOverride::registerOverride
QLoggingCategoryWithEnvironmentOverride::QLoggingCategoryWithEnvironmentOverride
QtPrivateLogging::qt_lcDebugPlugins
locatePlugin
QPluginLoader::setFileName
QPluginLoader::QPluginLoader
loadPlugin
initialize
_tracepoint_enabled
QtPrivate::trace_QObject_ctor
QObject::QObject
 ...
QFile::QFile
loadRulesFromFile
QLoggingRegistry::initializeRules
QLoggingRegistry::QLoggingRegistry
(anonymous namespace)::Q_QGS_qtLoggingRegistry::innerFunction
QtGlobalStatic::Holder<::Q_QGS_qtLoggingRegistry>::Holder
QGlobalStatic<QtGlobalStatic::Holder<::Q_QGS_qtLoggingRegistry> >::instance
QGlobalStatic<QtGlobalStatic::Holder<::Q_QGS_qtLoggingRegistry> >::operator()
QLoggingRegistry::instance
QLoggingCategory::setFilterRules
main

Add a recursion guard in the QLoggingRegistry that causes
QLoggingRegistry::instance to return null if recursion occurs.
The internal code appears to handle nullptr well, while external code
should always receive a non-null pointer.

Fixes: QTBUG-135326
Pick-to: 6.8 6.9
Change-Id: I91f1ff341b096031930004cca8421f2ee0bab9b1
Reviewed-by: Thiago Macieira <[email protected]>

Author: anttimaa

src/corelib/io/qloggingregistry.cpp

@@ -11,6 +11,7 @@
 #include <QtCore/qtextstream.h>
 #include <QtCore/qdir.h>
 #include <QtCore/qcoreapplication.h>
+#include <QtCore/qscopedvaluerollback.h>
 
 #if QT_CONFIG(settings)
 #include <QtCore/qsettings.h>
@@ -429,6 +430,10 @@ QLoggingRegistry::installFilter(QLoggingCategory::CategoryFilter filter)
 
 QLoggingRegistry *QLoggingRegistry::instance()
 {
+    Q_CONSTINIT thread_local bool recursionGuard = false;
+    if (recursionGuard)
+        return nullptr;
+    QScopedValueRollback<bool> rollback(recursionGuard, true);
     return qtLoggingRegistry();
 }