Discuss: Higher fidelity language highlighting (in general)

[Turnerj]

Taking over this issue to use it as the proposal for higher fidelity highlighting as a general idea and a plan to get there - if we should (I think we should).

---

[Editor] Original issue content:

The short version is I want to get this (the C# snippet from the highlight.js website):

Closer to this (same snippet but in the latest version of Visual Studio 2019):

Actual snippet of code:

using System.IO.Compression;

#pragma warning disable 414, 3021

namespace MyApplication
{
    [Obsolete("...")]
    class Program : IInterface
    {
        public static List<int> JustDoIt(int count)
        {
            Console.WriteLine($"Hello {Name}!");
            return new List<int>(new int[] { 1, 2, 3 })
        }
    }
}

My question isn't about writing the CSS to do that for a theme, its about having classes to target those styles in the first place. Where does the use of a specific highlight.js class start and end (eg. "hljs-function" for declarations but not calls?). Like, is there a spec somewhere for when something should be used?

I've read through some of the related issues like #1731 and #1378 so I do understand reluctance for adding classes that can target these types of cases as core functionality especially when not all languages do this. I guess I'm querying, more as a guide, if I were to implement (some of) this for C#, I don't want to step on toes about class name consistency.

[joshgoebel]

The snaps are a helpful "big picture" but I'd suggest you need to start with breaking out a specific well defined list of which things you want to highlight... and perhaps also which classes we might use to do so. That could be a basis for a discussion. There may be one or two things we're open to, but I think you'll find most of it is off the table [for the core library] for a few different reasons (that are probably also started elsewhere):

• Lack of theme support and the difficulty of updating 100s of existing themes.
• It may not even be technically possible.
• Lack of developer time to maintain a ruleset of the necessary complexity
• Consistency across languages - if you want to improve C# you have to improve them all.

If you provide a specific list we could talk about perhaps what we COULD improve in the core library. I for one wouldn't mind seeing function dispatch highlighted. IE coloring writeLine in the following.

Console.writeLine("...");

But even then the problem becomes which classes to use. function has always meant function declaration, not invocation.

---

Now, another angle is if you want to branch off on your own. Fork the C# grammar, provide a custom theme (with all new CSS classes), and publish a 3rd party grammar module. Then all possibilities are open to you (within the technical constraints of what the engine itself can support)... This likely will require time and ongoing maintenance of course...

Honestly I think the best outcome for many languages would be if the language community got super interested and maintained their own 3rd party grammars rather than all the grammars being first-party.

[joshgoebel]

That said the VS Code is all OSS. If you want syntax highlighting "that good" you might want to look at just using the same code Visual Studio uses to do the highlighting. But their syntax file for some languages is like 1mb while ours is like 10kb.... so that level of detail comes at a price.

[joshgoebel]

Also if you attached plain-text versions for anyone who wanted to play with this that'd be greatly appreciated.

[Turnerj]

That said the VS Code is all OSS. If you want syntax highlighting "that good" you might want to look at just using the same code Visual Studio uses to do the highlighting. But their syntax file for some languages is like 1mb while ours is like 10kb.... so that level of detail comes at a price.

Yeah, there are definitely trade-offs if I wanted to match my example images 100%. If I can bring them a little closer though that would be great. It was just funny seeing a code snippet of mine, with highlighting enabled, seem like nothing really was highlighted except for like 1 if-statement.

Of the different constructs, I think function calls are probably the easiest with a Regex similar to (?<functionCall>[a-zA-Z0-9_]+)\( (though it would capture this [Obsolete("...")] erroneously as a function call). I'm thinking maybe avoiding the re-use of hljs-function to keep better backwards compatibility and instead could have hljs-invocation as it is exactly what we are doing. It does mean other uses of that function wouldn't work (eg. highlighting of nameof(myFunctionName) would leave myFunctionName unhighlighted but this goes back to your point above).

For doing things like how return is a different colour - it might be simple enough to have just different categories of keywords. Could then use two classes of hljs-keyword and additionally hljs-keyword-N where N is the category/group index. That way this isn't coded too specifically to a language construct, just an extension of how the existing code base works. While saying this though, return might be the only case in C# like that which I can recall off the top of my head so this entirely might be overkill.

The other cases start to get a bit more difficult as they rely on greater context and are more language specific. Like its easy enough to have a Regex pick up "IInterface" as C# follows the convention of an "I" prefix on interfaces however many languages don't have interfaces. Then are we splitting declaration of an interface (public interface IInterface) from its use like a cast (IInterface)myObj? I could live without this being specifically detected.

One thought I had was also potentially separating out and/or allowing extension of "built_in". That is, while the built-in types might work well from something like JS, if we counted the entire System namespace in C# as built-in (which it effectively is), there would be hundreds of items which would be a huge bloat to the language definition. Being able to define or extend it to be more like "known types" as a separate package could allow flexibility both ways.

So after writing all this, I think really two specific pieces would be helpful and easy enough to approach:

• Function invocation
• Extending built-in/known types (initially I could just add super common ones like List or Math though ideally would still want some more exotic - but still built-in - types to be included)

[Turnerj]

Also if you attached plain-text versions for anyone who wanted to play with this that'd be greatly appreciated.

Not entirely sure what you mean here. Like I know what "plain text" is but are you referring to the actual code snippet from the earlier screenshots and just wanting a non-image copy of it? Happy to do that, just not sure if that's what you meant.

[joshgoebel]

but are you referring to the actual code snippet from the earlier screenshots

Yes.

[joshgoebel]

I wonder what theme you were using there since many of the "titles" are indeed highlighed.

[joshgoebel]

Extending built-in/known types (initially I could just add super common ones like List or Math though ideally would still want some more exotic - but still built-in - types to be included)

Definitely open to this. One of the easiest things, for sure.

Like its easy enough to have a Regex pick up "IInterface" as C# follows the convention of an "I" prefix on interfaces

Seems reasonable, except we already seem to highlight interfaces as titles from where I'm sitting.

and instead could have hljs-invocation as it is exactly what we are doing.

Naming sounds reasonable but I can't see adding it to core unless we had some automated way of updating all the existing themes... like perhaps copying another style or some sort of automated "adjustment" of styles... ie invocation is just like xyz, but 20% brighter, dimmer, etc...

There is no point in a new class (in core) if only 1% of the themes support it.

One thought I had was also potentially separating out and/or allowing extension of "built_in". That is, while the built-in types might work well from something like JS, if we counted the entire System namespace in C# as built-in (which it effectively is), there would be hundreds of items which would be a huge bloat to the language definition. Being able to define or extend it to be more like "known types" as a separate package could allow flexibility both ways.

I think this is best answered by someone forking C# and making a much better version (and maintaining it) - and then ours is removed or becomes the "basic" version. Though technically it's already possible to do this with requireLanguage. Look at Arduino for a simple example of how this can be done.

If the changes were significant though it'd be easy to start from scratch or fork though.

[joshgoebel]

#pragma warning disable 414, 3021

This might be simple too, but I'm not sure what the rules are... would numbers in a compiler directive always be highlighted or only for certain directives?

[Turnerj]

I wonder what theme you were using there since many of the "titles" are indeed highlighed.

That was the VS2015 theme.

May I ask what you used to generate that image with the class names displayed inline like that? Looks like it will totally help me in my own debugging.

I guess some of the titles are different to others - eg "namespace MyNamespace" vs "class MyClass" vs "interface IMyInterface". If a language parser could configure multiple class names where all of those are hljs-title's but the namespace might be hljs-level-1 with the class and interface definitions as hljs-level-2 or something. This way a theme can still style appropriately while not necessarily being aware of any specific language construct.

That said, while it is generic and would allow me to target what I want, it still feels a little bit of a kludge and probably not a good idea in the core.

Naming sounds reasonable but I can't see adding it to core unless we had some automated way of updating all the existing themes... like perhaps copying another style or some sort of automated "adjustment" of styles... ie invocation is just like xyz, but 20% brighter, dimmer, etc...

There is no point in a new class (in core) if only 1% of the themes support it.

I kinda see it more like - some of these themes were never designed with additional constructs/language pieces in mind. Actually applying a styling (even just brighter or dimmer) might be a bad thing whereas having a class available for a theme to use (if it was appropriate for the theme) might be a good thing.

---

I might have a mess around with the C# parser, at least adding some built-in types for C# if nothing else is easily approachable.

[joshgoebel]

May I ask what you used to generate that image with the class names displayed inline like that? Looks like it will totally help me in my own debugging.

tools/developer.html

If a language parser could configure multiple class names where all of those are hljs-title's but the namespace might be hljs-level-1 with the class and interface definitions as hljs-level-2 or something.

If we have an existing class that makes sense we might have room for some variety - there might be room to detect the titles differently, but I worry we may not.

I kinda see it more like - some of these themes were never designed with additional constructs/language pieces in mind.

That's why you'd use one of the existing styles (the closest match) and either copy it verbatim or have an algorithmic way of changing it subtly.

class available for a theme to use (if it was appropriate for the theme) might be a good thing.

3rd party language modules are free to innovate here, but I don't see a lot of room in core because I don't like the idea of "blessed" themes vs "old" themes.

To be clear: I'm not saying "never ever". Just if we're to add CSS classes to core it would require a reasonable plan to deal with old themes in some way other than saying "oh most of the themes just don't support those new classes"...

[joshgoebel]

One idea would be for a few new classes that add "nuance" to existing ones... or for grammars to add nuance that themes COULD support. title.interface, ala how TextMate grammars work. Then all themes would still highlight it as a title, but some might highlight it with more nuance.

That might be worth discussing.

It might be interesting to take a complex piece of C#/JS/TS code and spit out all the Textmate scopes involved and then see what that looks like.

[Turnerj]

Thanks for that debugging link!

3rd party language modules are free to innovate here, but I don't see a lot of room in core because I don't like the idea of "blessed" themes vs "old" themes.

That's totally fair! There are a ton of first-party supported themes which is a blessing and a curse in cases like this.

One idea would be for a few new classes that add "nuance" to existing ones... or for grammars to add nuance that themes COULD support. title.interface, ala how TextMate grammars work. Then all themes would still highlight it as a title, but some might highlight it with more nuance.

I'm relatively unfamiliar with TextMate however that sounds like what I'm thinking. I wouldn't want a grammar to not be supplying the common/existing class - it should 100% provide that (for backwards compatibility + simplicity for more "basic"/light themes) and an optional something else that can provide it a little more context.