Mathics scanner import

[rocky]

This time for sure!

[mmatera]

I see you have move to MathicsScanner more than the characters conversion table. OK, it is more ambitions that what I was envised, but it seems be useful. The only "issue" that I find on this is that if you want to modify the semantic of a symbol or add new operators, you would have to modify two projects, coordinately. However, I think that is not something that we have to do frequently, so I think this will work.
So, when the parser and mathics pass all the tests, I am OK to merge it.

[mmatera]

BTW, great work!

[rocky]

I see you have move to MathicsScanner more than the characters conversion table.

As it says in the README.rst for the mathics scanner, it would be good to use this in the pygments tokenizer.

OK, it is more ambitions that what I was envised, but it seems be useful. The only "issue" that I find on this is that if you want to modify the semantic of a symbol or add new operators, you would have to modify two projects, coordinately.

Yes, that is a downside. It also requires a little more sophisication of the developers. But I think/hope we can handle it.

I wanted to move out the parser too, however right now it is a little too tied right now to AST building, and Mathics objects like Number, and so on. So at least to my mind that's for later.

However, I think that is not something that we have to do frequently, so I think this will work.
So, when the parser and mathics pass all the tests, I am OK to merge it.

Great!

[rocky]

BTW, great work!

Actually, @GarkGarcia deserves the lion's share of the credit here.

[GarkGarcia]

The only "issue" that I find on this is that if you want to modify the semantic of a symbol or add new operators, you would have to modify two projects, coordinately. However, I think that is not something that we have to do frequently, so I think this will work.

I don't like this too. We could parameterize the functions of the tokenizer that use the table of operators. That way there's a single source of truth. I'm not sure if it's worth it though.

@rocky This LGTM, but we should import names_wildcard, base_name_pattern and full_names_pattern directly from mathics_scanner.tolenizer since mathics_scanner.letters and mathics_scanner.letterlikes aren't exported anymore since we merged Mathics3/mathics-scanner#3 (I took the time to transfer the code to there). I'll get this fixed and then we can merge this.

[rocky]

The only "issue" that I find on this is that if you want to modify the semantic of a symbol or add new operators, you would have to modify two projects, coordinately. However, I think that is not something that we have to do frequently, so I think this will work.

I don't like this too. We could parameterize the functions of the tokenizer that use the table of operators. That way there's a single source of truth. I'm not sure if it's worth it though.

@rocky This LGTM, but we should import names_wildcard, base_name_pattern and full_names_pattern directly from mathics_scanner.tolenizer since mathics_scanner.letters and mathics_scanner.letterlikes aren't exported anymore since we merged Mathics3/mathics-scanner#3 (I took the time to transfer the code to there). I'll get this fixed and then we can merge this.

Ok. Thanks.

I do think more will be added to the YAML file and that it is worth the effort. However at least for me, I need to spread the effort out over a longer period of time since need to limit the amount of time spent per week.

More about this some other time...

[GarkGarcia]

@rocky The PyYAML thing is really starting to annoy me, I had to basically add a line to each CI configuration file saying "install PyYAML" before we can do anything else.

A simpler way add this to make and use that. I have no doubt that down the line we can find something even better though.

[rocky]

Thanks