Restructure the repository

[mokou]

• Don't bundle all the games in one patch. Split them up per game, per translation/function.
• Don't download each file separately. Bundle them up in a zip.
• Namespacing.
• While we are at it, change .js and .jsdiff extensions to .json.
  • For jsdiff, either document the existing markup better or change it to something like markdown.
• There is probably something I forgot about, I'll add it later if so.

Not bundling all games into a single patch.
enhancement
Bundling up the translations

It appears to me that thpatch is on the path to becoming much like their static equivalent now. This is bad because the original focus of the patch was to do three things. One, allow the fans to easily contribute to the translations. Two, speedy upon release patching as was shown for Touhou 14, fans no longer have to wait months for the patchwork to be developed. And three, treating the fans to something better. Having them want to expect a better playing experience. Becoming like the static patches sort of ruins the image as this "new alien technology" as it was on occasion referred to, the ability to not only have translations edited on the fly, in many languages at once and all bundled in a single patch, but also the chance for it being a fully modded toolkit, as fans could swap out the text and images to create their own little Touhou adventure. In short, disbanding the games into different parts as well as making it so files are bundled up in a zip therefore on the fly translations no longer are possible if edits were made really makes me sad. At least, that's what it appears to me.

[mokou]

I think you misunderstand my intentions, so I'll explain a bit better.
The reasoning behind the restructuring of the repository is to cut down on HTTP overhead and not download data that will never be used. What currently happens is that you select the English patch and it downloads the translation for EVERY game supported; this download takes ~2 minutes on my connection. If the user only owns one game both time and bandwidth are wasted downloading ~19MB of data that is useless to the user. This change has less to do with static patches and more to do with not wasting people's time and resources. Also, zip files can be edited.

In doing so you remove the real time aspect. You must edit the zip files yourself, which means that the approach no longer makes them instantly modifiable. Now, there are ways to keep this new version of resource preservation you wish to have. Three options. One asking if they are first time users and download all the necessary files once, second if they are an existing user and select which patches they want to use so it doesn't install everything again, or three, want to always download everything all the time. . Yeah, the engine no longer itself is self updating with this approach but it's a comprise, I'd think. Basically, make the patcher provide a feature for those with slower connections and keep it the same for the ones who opt for the bigger stuff. It's only text and images after all. Maybe compatibility for music themes makes this a much more concerning issue but one thing at a time.

[mokou]

I don't plan to make the user edit anything manually, I also plan to try to migrate the existing patches to whatever format we end up using.

[nmlgc]

I guess the main reason for this confusion is that it remains unclear from that short list how many of these features only apply to the backend and what will propagate to the frontend, and will thus be visible to users.

If patches now covered only one game and this extended to the frontend, would users then have to download them separately for all the games they have? How is this not going to turn into an usability nightmare? And what about data shared between games (of which there really ought to be more in the future)? Will that come in a separate patch, and thus add even more clutter to a typical patch stack?

As for real-time patching, we should first define what this actually means. Of course, if we don't lock write access to the files that make up a patch while the game is running (and we don't), it will always be possible to edit a patch. Thus, it all comes down to the points when thcrap makes the game read that modified data. These points are best referred to as the time after which a certain type of data can't be patched anymore:

• Not modifiable after game startup time
  Includes any patch data that needs to be processed before the game starts. This is mostly data where it wouldn't make any sense to reload changes later (such as the game name or update URLs) or where real-time patching would be very difficult and/or dangerous (binary hacks and breakpoints). Basically, everything in <game>.js, <game>.<build>.js and global.js (upon which the other two are stacked).

• Not modifiable after the game loaded the file containing the original data
  Includes any data that is modified by "classically" patching the original formats: Windows dialogs such as the resolution selection and custom.exe, in-game dialogue, ending text, and images.

  Note that in-game data is not reloaded if a run is restarted on the same stage. (i.e. in practice modes, full game runs restarted on Stage 1, or continuing in the same stage after a Game Over in TH09, TH10, TH11 and TH12)

• Not modifiable after the content was rendered to its associated texture
  Includes any custom JSON format introduced by thcrap. Currently, these are:

  • Hardcoded strings (stringlocs.js, stringdefs.js)
  • Music titles (themes.js) and Music Room comments (musiccmt.js)
  • Spell cards (spells.js) and comments (spellcomments.js)
  • Specifications for ZUN formats (formats.js)

  This category didn't exist before the implementation of hot-repatching in 2014-05-06. Previously, all these files were only resolved once at game startup time and could not be modified afterwards. I didn't want to reload all of these files from all of the active patches for every line rendered by the game.

  In contrast, hot-repatching introduces a separate thread to monitor the patch directories for file modification or creation events, and then reloads only the affected files after a short delay. For example, you could enter a game's Music Room, scroll down to a specific track, press the Shot button to load its comment, notice a typo, quickly open the comment file of your patch in a text editor, fix the typo, save, wait a second, go back to the game, press the Shot button again, and your typo will appear fixed. This is as "real-time" and efficient as automatic patching can sensibly get.

If patches were bundled in zip files on the client side as well, this would strongly reduce the efficiency of hot-repatching. First, we'd no longer be getting these dedicated events per-file from the file system, and we would have to keep the state of the old archive and parse the state of the new archive after every change. Also, zip files still introduce a huge inconvenience for those who want to edit their patches locally. The main reason for thcrap's zip file module was to unpack engine updates and to circumvent the 8.3 filename limitation under DOS, which is why it would have been a required feature for the DOS port. (Since DOS is a single-tasking operating system, hot-repatching wouldn't have made much sense there.)

Although I did initially consider to use it for Windows game patches as well, I quickly dropped the idea after hot-repatching got implemented. And as far as compression ratios are concerned, it's certainly not worth the trouble either - most of the space is taken up by PNGs, and these are better run through OptiPNG on the server side, rather than Deflate/gzip. It might be sort of understandable if your patch contains huge .anm files, which compress pretty well. But even then, why don't we just go for gzipping individual files and throw away that useless (and crappy) zip container?

Speaking of gzipping individual files. I think that using a better HTTP library and enabling gzip compression for text files on both the server and the client alone will reduce the download times to a pretty agreeable amount. Getting just anywhere close to the performance of Wine's implementation of wininet would be pretty great, and that doesn't seem to be that hard.

As for cutting out unnecessary files, I still prefer the fast bootstrapping solution. It's essentially nothing more than a client-only extension that works on top of the existing protocol, maintains backwards compatibility, needs little user interaction, and will address the problem of long downloads for the majority of users.

(In fact, I think I really should go forward and implement this right now.)

The updater already behaves like the second option Nazeo mentioned, which means that there's technically no difference between the initial download and any subsequent updates.

---

As for the markup, all the tab layout and alignment stuff was added because we will need it – the result screens in TH07 and TH08 need all of the existing features in order to look pretty with proportional fonts in any font size. The ability to define tabstop widths by using multiple reference strings, without having to parse the entire script in advance, is especially important, as it allows the implementation to come with a minimum of state and complexity.

Markdown, with its focus on offering adequate markup for short documents (and its often implied support for inline HTML), simply isn't the right tool for the job. It has no support for tables, multiple tabstops, or even simple alignment across a paragraph, and we have no use for the majority of features it offers (headings, lists, hyperlinks, images, blockquotes, etc.) Sure, if all we wanted to have is bold or italic markup, we should have definitely picked that syntax. However, these were only meant as nice bonus features.

---

Therefore, it's down to better documenting the markup. And as we don't have any clearly designated place where we document such things yet, (except for maybe these large comments I've been adding to the more recent parts of the source code, and I'm sure that's not what you were referring to), this brings us back to the question of how to approach documenting thcrap as a whole in the first place. (It certainly won't be a wiki, but what else should we use?)

Well, I don't have an answer for that, as I still prefer pointing people to the actual source code in question.

[mokou]

The main reason I have been pushing for zip is not for the compression, but to download multiple files bundled together. I have yet to decide if it will stay a zip file on disk or be extracted afterwards, but it seems there are advantages to not storing it as a a zip.

I guess fast bootstrapping is more or less the same thing I'm proposing, but instead of downloading all the files one by one, they are bundled together. I do agree that a better HTTP library should be used and some sort of compatible fast bootstrapping should be implemented ASAP. These features do not need to wait for a 2.0 release, but a 2.0 release should not happen without them.

Also, I'm not saying we need to change the existing markup (I think we should, it's pretty ugly), but it would be better to make breaking changes with other breaking changes. I also realize Markdown is a poor choice for this, so I agree we probably shouldn't use it.

[DankRank]

Speaking of breaking changes, we should also change the file checksums to SHA256 or something like that.