Skip to content

Commit 9dec5c9

Browse files
dwertheimerdwertheimer
committed
Merge branch 'templating-tr-trigger' into dynamicDialog-requestHandlers
2 parents 1d00733 + d9b8fd0 commit 9dec5c9

16 files changed

Lines changed: 1109 additions & 89 deletions

‎EMOJI_CORRUPTION_BUG_REPORT.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Emoji Encoding Corruption Bug Report
2+
3+
**Date**: 2026-01-06
4+
**Reported by**: @dwertheimer
5+
**Issue**: Emoji characters (specifically 🧩) are being corrupted when data is transmitted from the plugin's JavaScript context to the React WebView's JavaScript context via `HTMLView.runJavaScript`.
6+
7+
## Summary
8+
9+
When a note title contains an emoji (e.g., "Dashboard Plugin 🧩"), the emoji is correctly preserved throughout the plugin's JavaScript context but becomes corrupted (e.g., "Dashboard Plugin 🧩") when the data is transmitted to the React WebView via `HTMLView.runJavaScript` and `postMessage`.
10+
11+
## Evidence
12+
13+
### Correct Encoding (Plugin Context)
14+
- **Location**: `sendToHTMLWindow :: JavaScript code to execute`
15+
- **Title**: "Dashboard Plugin 🧩" (length=19, charCodes=68,97,115,104,98,111,97,114,100,32,80,108,117,103,105,110,32,55358,56809)
16+
- **Status**: ✅ Correct UTF-16 surrogate pair (55358,56809 = U+1F9E9)
17+
18+
### Corrupted Encoding (WebView Context)
19+
- **Location**: `[ENCODING DEBUG] In WebView JS - payloadData BEFORE postMessage`
20+
- **Title**: "Dashboard Plugin 🧩" (length=21, charCodes=68,97,115,104,98,111,97,114,100,32,80,108,117,103,105,110,32,240,376,167,169)
21+
- **Status**: ❌ Corrupted - UTF-8 bytes (240,376,167,169) interpreted as Latin-1 characters
22+
23+
## Root Cause Analysis
24+
25+
The corruption occurs **between**:
26+
1. Creating the JavaScript code string in the plugin context (correct emoji)
27+
2. Executing that JavaScript code in the WebView context (corrupted emoji)
28+
29+
**Critical Finding**: The corruption happens **BEFORE** `postMessage` is called in the WebView. This means the issue is in how `HTMLView.runJavaScript` transmits/executes the JavaScript string from the plugin's JavaScript context to the WebView's JavaScript context.
30+
31+
## Technical Details
32+
33+
### Current Implementation
34+
```javascript
35+
// In helpers/HTMLView.js
36+
const stringifiedPayload = JSON.stringify(dataWithUpdated) // Correct emoji here
37+
const jsCodeToExecute = `
38+
(function() {
39+
const payloadDataString = ${JSON.stringify(stringifiedPayload)};
40+
const payloadData = JSON.parse(payloadDataString); // Corrupted emoji here
41+
window.postMessage({ type: '${actionType}', payload: payloadData }, '*');
42+
})();
43+
`
44+
await HTMLView.runJavaScript(jsCodeToExecute, windowIdToSend)
45+
```
46+
47+
### The Problem
48+
When `HTMLView.runJavaScript` executes the JavaScript string:
49+
- The JavaScript string itself contains the correct emoji when created
50+
- But when the string is transmitted/executed in the WebView, the emoji is already corrupted
51+
- This suggests the issue is in how NotePlan's `HTMLView.runJavaScript` handles Unicode characters when transmitting JavaScript strings between contexts
52+
53+
## Logging Evidence
54+
55+
We've added extensive logging throughout the data flow:
56+
57+
1. ✅ **Plugin Context** - All logs show correct emoji (55358,56809)
58+
- `makeDashboardParas`
59+
- `createSectionOpenItemsFromParas`
60+
- `getTodaySectionData`
61+
- `getSomeSectionsData`
62+
- `sendToHTMLWindow :: BEFORE JSON.stringify`
63+
- `sendToHTMLWindow :: AFTER JSON.stringify`
64+
- `sendToHTMLWindow :: JavaScript code to execute`
65+
66+
2. ❌ **WebView Context** - All logs show corrupted emoji (240,376,167,169)
67+
- `[ENCODING DEBUG] In WebView JS - payloadDataString BEFORE JSON.parse` (if logged)
68+
- `[ENCODING DEBUG] In WebView JS - payloadData BEFORE postMessage`
69+
- `Root/onMessageReceived :: Raw event.data (stringified)`
70+
- `Root/onMessageReceived :: Payload BEFORE processing`
71+
72+
## Request for Investigation
73+
74+
**Eduard**, could you please investigate:
75+
76+
1. **How does `HTMLView.runJavaScript` transmit JavaScript strings?**
77+
- Does it use a specific encoding (UTF-8, UTF-16, etc.)?
78+
- Is there any string conversion/encoding happening during transmission?
79+
- Could there be a bug in how Unicode characters are handled?
80+
81+
2. **Is there a known issue with Unicode/emoji in `HTMLView.runJavaScript`?**
82+
- Have other plugins reported similar issues?
83+
- Is there a workaround or best practice for sending Unicode data?
84+
85+
3. **Potential Solutions**:
86+
- Should we use a different method to send data to the WebView?
87+
- Should we base64-encode the JSON string before embedding it?
88+
- Is there a way to ensure UTF-8 encoding is preserved?
89+
90+
## Workaround Attempts (Unsuccessful)
91+
92+
We've tried:
93+
- ✅ Using `JSON.stringify()` to escape the JSON string before embedding (still corrupted)
94+
- ✅ Using `JSON.parse()` in the WebView instead of direct embedding (still corrupted)
95+
- ❌ The corruption happens during `HTMLView.runJavaScript` execution, not in our code
96+
97+
## Impact
98+
99+
- **Severity**: Medium - Affects display of note titles with emojis
100+
- **Scope**: Any plugin using `HTMLView.runJavaScript` to send Unicode data to React WebViews
101+
- **User Impact**: Note titles with emojis display incorrectly (e.g., "🧩" becomes "🧩")
102+
103+
## Next Steps
104+
105+
1. Wait for Eduard's investigation of `HTMLView.runJavaScript` Unicode handling
106+
2. Consider alternative data transmission methods if `HTMLView.runJavaScript` has a known limitation
107+
3. Document this limitation if it's a known issue with NotePlan's JavaScript bridge
108+
109+
---
110+
111+
**Note**: All temporary logging code has been documented in `ENCODING_DEBUG_LOGGING_TO_REMOVE.md` for cleanup once the issue is resolved.
112+

‎EMOJI_CORRUPTION_ISSUE_DESCRIPTION.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Emoji Corruption Issue - Root Cause Analysis
2+
3+
## Summary
4+
5+
The emoji corruption issue (`🧩` → `🧩`) occurs during the transmission of data from the plugin's JavaScript context to the WebView's JavaScript context via `HTMLView.runJavaScript`. Once corrupted data is stored in the WebView, it propagates through all subsequent data operations, creating a self-perpetuating cycle of corruption.
6+
7+
## Evidence from Logs
8+
9+
### 1. Data Generation is Correct ✅
10+
11+
All logs from the plugin's data generation pipeline show the emoji is **correct**:
12+
- `makeDashboardParas`: `"Dashboard Plugin 🧩"` (charCodes=55358,56809) ✅
13+
- `createSectionOpenItemsFromParas`: `"Dashboard Plugin 🧩"` (charCodes=55358,56809) ✅
14+
- `getTodaySectionData`: `"Dashboard Plugin 🧩"` (charCodes=55358,56809) ✅
15+
- `getSomeSectionsData`: `"Dashboard Plugin 🧩"` (charCodes=55358,56809) ✅
16+
17+
**Conclusion**: The plugin's JavaScript context correctly handles Unicode emojis as UTF-16 surrogate pairs.
18+
19+
### 2. Corruption Occurs During Transmission ❌
20+
21+
The first appearance of corrupted data is in the WebView's JavaScript context:
22+
- `Root/onMessageReceived`: `"Dashboard Plugin 🧩"` (charCodes=240,376,167,169) ❌
23+
24+
**Timeline**:
25+
1. Plugin generates correct data: `🧩` (UTF-16: 55358,56809)
26+
2. Plugin calls `sendToHTMLWindow` → `HTMLView.runJavaScript`
27+
3. JavaScript string is transmitted from plugin's JavaScriptCore to WebView's JavaScriptCore
28+
4. **Corruption occurs during this transmission**
29+
5. WebView receives corrupted data: `🧩` (Latin-1: 240,376,167,169)
30+
31+
**Conclusion**: The corruption happens in the `HTMLView.runJavaScript` bridge mechanism itself.
32+
33+
### 3. Corruption Propagates Through System 🔄
34+
35+
Once corrupted data is stored in the WebView's `globalSharedData`:
36+
- Every subsequent `getGlobalSharedData` call returns corrupted data
37+
- `setPluginData` reads corrupted data from WebView and merges it with new correct data
38+
- The merge operation (`mergeSections`) combines corrupted existing data with correct new data
39+
- The corrupted data "wins" because it's already in the WebView's storage
40+
- This creates a self-perpetuating cycle: corrupted data → stored in WebView → read back → merged with new data → sent again → corrupted again
41+
42+
**Conclusion**: The corruption is not just a one-time issue, but a systemic problem that propagates through all data operations.
43+
44+
## Root Cause
45+
46+
The corruption occurs in the **`HTMLView.runJavaScript` bridge** when transmitting JavaScript strings from the plugin's JavaScriptCore environment to the WebView's JavaScriptCore environment.
47+
48+
### Technical Details
49+
50+
1. **Correct State (Plugin JavaScript Context)**:
51+
- Emoji: `🧩`
52+
- UTF-16 surrogate pair: `55358, 56809` (0xD83E, 0xDDE9)
53+
- JavaScript string representation: Correct UTF-16
54+
55+
2. **Transmission (HTMLView.runJavaScript)**:
56+
- The JavaScript string containing the JSON payload is transmitted
57+
- During transmission, the UTF-16 surrogate pair is incorrectly converted
58+
- The emoji bytes are misinterpreted as Latin-1 characters
59+
60+
3. **Corrupted State (WebView JavaScript Context)**:
61+
- Emoji: `🧩`
62+
- Latin-1 bytes: `240, 376, 167, 169` (0xF0, 0xF8, 0xA7, 0xA9)
63+
- These are the UTF-8 bytes `240, 159, 167, 169` (0xF0, 0x9F, 0xA7, 0xA9) interpreted as Latin-1
64+
65+
### Why This Happens
66+
67+
The `HTMLView.runJavaScript` mechanism appears to:
68+
1. Convert the JavaScript string to a byte sequence (likely UTF-8)
69+
2. Transmit those bytes to the WebView
70+
3. Reconstruct the string in the WebView's JavaScript context
71+
4. **But somewhere in this process, the byte encoding is misinterpreted**
72+
73+
The UTF-8 bytes for the emoji (`240, 159, 167, 169`) are being interpreted as Latin-1 characters (`240, 376, 167, 169`), where:
74+
- `159` (0x9F) becomes `376` (0xF8) - this is the key corruption
75+
- The other bytes remain the same but are now interpreted as Latin-1
76+
77+
## Impact
78+
79+
1. **Initial Corruption**: First `sendToHTMLWindow` call corrupts the emoji during transmission
80+
2. **Storage**: Corrupted data is stored in WebView's `globalSharedData`
81+
3. **Propagation**: All subsequent operations read corrupted data and merge it with new correct data
82+
4. **Persistence**: The corruption persists across all data refresh cycles
83+
84+
## Solution Required
85+
86+
The fix must be implemented in the **`HTMLView.runJavaScript` bridge mechanism** (likely in Swift/Objective-C code that we don't have access to). The bridge needs to:
87+
88+
1. Properly handle Unicode characters when transmitting JavaScript strings
89+
2. Ensure UTF-8 encoding is correctly interpreted on both sides
90+
3. Preserve UTF-16 surrogate pairs through the transmission
91+
92+
## Workaround Attempts (Unsuccessful)
93+
94+
We attempted several workarounds in JavaScript:
95+
- Manual Unicode escaping
96+
- Double JSON.stringify
97+
- Manual emoji encoding/decoding
98+
99+
None of these worked because the corruption occurs **during the transmission itself**, before the JavaScript code in the WebView even executes.
100+
101+
## Next Steps
102+
103+
1. **Report to Eduard**: This is a bug in the `HTMLView.runJavaScript` mechanism that requires a fix in the native Swift/Objective-C code
104+
2. **Temporary Mitigation**: Consider avoiding emojis in note titles or implementing a workaround at the data source level (not recommended, as it limits functionality)
105+
3. **Investigation**: Eduard needs to investigate how `HTMLView.runJavaScript` handles Unicode strings when bridging between JavaScriptCore environments
106+
107+
## Files Involved
108+
109+
- `helpers/HTMLView.js` - Contains `sendToHTMLWindow` which calls `HTMLView.runJavaScript`
110+
- `np.Shared/src/react/Root.jsx` - Receives the corrupted data via `postMessage`
111+
- `jgclark.Dashboard/src/dashboardHelpers.js` - `setPluginData` and `mergeSections` propagate the corruption
112+
- `jgclark.Dashboard/src/refreshClickHandlers.js` - `refreshSomeSections` merges corrupted and correct data
113+
114+
## Logging Evidence
115+
116+
All logging shows:
117+
- ✅ Plugin context: Correct emoji (charCodes=55358,56809)
118+
- ❌ WebView context: Corrupted emoji (charCodes=240,376,167,169)
119+
- 🔄 Propagation: Corrupted data from WebView merges with correct new data
120+
121+
The corruption is **definitively** occurring in the `HTMLView.runJavaScript` transmission bridge.
122+
123+
---
124+
125+
## Succinct Version for Discord
126+
127+
**Emoji Corruption Bug in `HTMLView.runJavaScript`**
128+
129+
Emojis in note titles (e.g., `🧩`) are being corrupted (`🧩`) when transmitted from plugin JS to WebView JS via `HTMLView.runJavaScript`.
130+
131+
**Evidence:**
132+
- Plugin side: Emoji is correct `🧩` (UTF-16: 55358,56809)
133+
- After `HTMLView.runJavaScript`: Corrupted `🧩` (Latin-1: 240,376,167,169)
134+
- The UTF-8 bytes `240,159,167,169` are being misinterpreted as Latin-1 `240,376,167,169` (byte `159` → `376`)
135+
136+
**Root Cause:**
137+
The corruption occurs in the native bridge code during string transmission between JavaScriptCore environments. The UTF-8 encoding is being incorrectly interpreted as Latin-1.
138+
139+
**Impact:**
140+
Once corrupted, the data propagates through all operations because it's stored in WebView's `globalSharedData` and merged with new correct data.
141+
142+
**Fix Required:**
143+
The `HTMLView.runJavaScript` bridge needs to properly handle Unicode/UTF-8 encoding when transmitting JavaScript strings. This requires a fix in the native Swift/Objective-C code.
144+
145+
**Logs show the corruption happens between:**
146+
- Plugin: `sendToHTMLWindow` → `HTMLView.runJavaScript` (correct)
147+
- WebView: `Root/onMessageReceived` (corrupted)
148+
149+
JavaScript workarounds don't work because the corruption occurs during the native bridge transmission itself.

‎ENCODING_DEBUG_LOGGING_TO_REMOVE.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Temporary Encoding Debug Logging - TO REMOVE
2+
3+
This file tracks all temporary logging code added to debug emoji encoding corruption. Remove all of these once the issue is fixed.
4+
5+
## Files with Temporary Logging:
6+
7+
### 1. `helpers/HTMLView.js`
8+
- **Function**: `sendToHTMLWindow`
9+
- **Lines**: ~723-770
10+
- **What to remove**:
11+
- Logging block BEFORE JSON.stringify (checks data object)
12+
- Logging block AFTER JSON.stringify (checks stringified payload)
13+
14+
### 2. `helpers/HTMLView.js`
15+
- **Function**: `getGlobalSharedData`
16+
- **Lines**: ~772-788
17+
- **What to remove**: Logging block that checks for emoji/corruption in data read from React
18+
19+
### 3. `jgclark.Dashboard/src/dashboardHelpers.js`
20+
- **Function**: `makeDashboardParas`
21+
- **Lines**: ~268-272
22+
- **What to remove**: Logging block that checks for emoji/corruption in noteTitle
23+
24+
### 4. `jgclark.Dashboard/src/dashboardHelpers.js`
25+
- **Function**: `setPluginData`
26+
- **Lines**: ~823-836
27+
- **What to remove**: Logging block that checks for emoji/corruption before sendToHTMLWindow
28+
29+
### 5. `jgclark.Dashboard/src/dashboardHelpers.js`
30+
- **Function**: `mergeSections`
31+
- **Lines**: ~850-875
32+
- **What to remove**: Logging blocks before and after merge
33+
34+
### 6. `jgclark.Dashboard/src/refreshClickHandlers.js`
35+
- **Function**: `refreshSomeSections`
36+
- **Lines**: ~151-152, ~176-195
37+
- **What to remove**: All `[ENCODING DEBUG]` logging statements
38+
39+
### 7. `np.Shared/src/requestHandlers/noteHelpers.js`
40+
- **Function**: `convertNoteToOption`
41+
- **Lines**: ~44-50
42+
- **What to remove**: Logging block that checks for emoji/corruption in note title
43+
44+
### 8. `jgclark.Dashboard/src/react/components/ItemNoteLink.jsx`
45+
- **Function**: `ItemNoteLink` component
46+
- **Lines**: ~36-42
47+
- **What to remove**: Logging block that checks for emoji/corruption when React receives title
48+
49+
### 9. `helpers/HTMLView.js`
50+
- **Function**: `getGlobalSharedData`
51+
- **Lines**: ~770-780
52+
- **What to remove**: Logging block that checks for emoji/corruption when data is read back from React
53+
54+
### 10. `np.Shared/src/react/Root.jsx`
55+
- **Function**: `onMessageReceived` (at the start, before parsing)
56+
- **Lines**: ~270-290
57+
- **What to remove**: Logging blocks that check raw event.data (stringified) before parsing
58+
59+
### 11. `np.Shared/src/react/Root.jsx`
60+
- **Function**: `onMessageReceived` (UPDATE_DATA case)
61+
- **Lines**: ~300-320
62+
- **What to remove**: Logging blocks that check payload before processing, and when data is received and stored
63+
64+
### 12. `jgclark.Dashboard/src/dashboardHelpers.js`
65+
- **Function**: `createSectionOpenItemsFromParas`
66+
- **Lines**: ~936-950, ~950-960
67+
- **What to remove**: Logging blocks before and after creating sectionItem objects
68+
69+
### 12. `jgclark.Dashboard/src/dataGenerationDays.js`
70+
- **Function**: `getTodaySectionData`
71+
- **Lines**: ~175-180, ~254-262, ~310-318
72+
- **What to remove**: Logging blocks before creating section, after pushing section, and before return
73+
74+
### 13. `jgclark.Dashboard/src/dataGeneration.js`
75+
- **Function**: `getSomeSectionsData`
76+
- **Lines**: ~88-110, ~145-155
77+
- **What to remove**: Logging blocks after getTodaySectionData returns, after pushing today sections, and before return
78+
79+
### 14. `helpers/HTMLView.js`
80+
- **Function**: `sendToHTMLWindow` (JavaScript code execution)
81+
- **Lines**: ~786-796, ~798-809, ~814-822
82+
- **What to remove**: Logging blocks in the JavaScript code that executes in WebView (payloadDataString BEFORE JSON.parse, payloadData AFTER JSON.parse, messageObj BEFORE postMessage)
83+
84+
## Search Pattern to Find All:
85+
Search for: `[ENCODING DEBUG]`
86+

0 commit comments

Comments
 (0)