Escaped unicode characters are encoded incorrectly

[fsoikin]

In strings with espaced Unicode characters (i.e. "\uABCD"), characters in the range from D800 to DFFF always get encoded as FFFD (65533 decimal).

Here's from the F# Interactive:

    > "\uD500".[0] |> uint16 ;;
    val it : uint16 = 54528us
    > "\uD700".[0] |> uint16 ;;
    val it : uint16 = 55040us
    > "\uD800".[0] |> uint16 ;;
    val it : uint16 = 65533us
    > "\uD900".[0] |> uint16 ;;
    val it : uint16 = 65533us
    > "\uE000".[0] |> uint16 ;;
    val it : uint16 = 57344us

Here's an SO question that prompted me to fiddle with it: http://stackoverflow.com/questions/29359408/surrogate-pair-detection-fails

Here's another issue I've (mistakenly?) opened in the other repo: fsharp/fsharp#399

[forki]

[forki]

I can reproduce in db7374b and confirm it works in C#:

[forki]

[<Test>]
member this.``Can encode all unicode characters``()=
    for i in [1us..65533us] do
        let s = (i |> char).ToString()
        let r = s.[0] |> uint16
        Assert.AreEqual(i, r)

works.

So the bug is in lexer/parser and not conversion to uint16

[forki]

Assert.AreEqual(55296us, Lexhelp.unicodeGraphShort("D800"))

works too.

[forki]

I can also confirm that Lexhelp.unicodeGraphShort is called with the correct value and returns the correct value.

[forki]

seems I'm stuck for now. any ideas where to look further?

[dsyme]

The bug is related to the behaviour of the .NET library function System.Text.Encoding.Unicode.GetString, used in the "string finisher" code in lexhelp.fs.

This

System.Text.Encoding.Unicode.GetString([| 0uy; 216uy |],0,2).[0] |> int32

returns 65533.

Note that D800 to DFFF are not actually valid UTF-16 characters, see http://en.wikipedia.org/wiki/UTF-16. So it is not entirely unreasonable that this should give odd results. It actually seems like the F# compiler should give an error on this particular Unicode literal.

[fsoikin]

@dsyme but strings containing these characters can be constructed by using a surrogate pair.
For example:

> let s = "𠮷".Substring(0,1)
let len = s.Length
let code = s.[0] |> uint16
;;

val s : string = "�"
val len : int = 1
val code : uint16 = 55362us

Therefore, doesn't it make sense for the compiler to allow using them in literals directly? Otherwise, it becomes very difficult to work with such "invalid" strings, as the OP on SO discovered.

[fsoikin]

Though, after some more thinking, I guess an error message explaining why this is invalid would be almost as useful.