[Feature Request] Support for import locale-specific encoding strings in SPSS sav

[shun2wang]

Hi,
Thank you for providing such a great tool.

Usually ReadStat can successfully import UTF-encoded datasets, but when reading locale encoded .sav files, it will prompt "Unable to convert string to the requested encoding (invalid byte sequence)"

I provide two sample .sav datasets(remove its .zip suffix) here for testing:

• 165-locale-specific.sav dataset encoded by Chinese locale-specific encoding
• 165-UTF-encoding.sav Chinese character dataset encoded by UTF
  165-locale-specific.sav.zip
  165-UTF-encoding.sav.zip

In ReadStat and Haven, the first dataset could not be read with the above error，But GNU PSPP can successfully open it. I guess that when encountering a localized encoded file, try to convert it to UTF through some methods and then read it, but I haven't seen such a practice yet（maybe the practice of PSPP is a success ).
Could you consider implementing it.

Thank you!

[JorisGoosen]

As https://www.loc.gov/preservation/digital/formats/fdd/fdd000469.shtml pointed out to me:

The best way to determine the specific character encoding in use is to consult the character encoding record, if present, and failing that the character_code in the machine integer info record, (which, despite the name given to the record by the GNU PSPP team, has indicators for character and floating point encodings, not just for integer encoding).

And this led me to https://www.gnu.org/software/pspp/pspp-dev/html_node/System-File-Format.html
https://www.gnu.org/software/pspp/pspp-dev/html_node/Character-Encoding-Record.html

And that seems to explain why PSPP knows how to read it. Would be good to have this in ReadStat, maybe I can create a PR for this some time this year.

[evanmiller]

Hi,

The file you provided self-reports its encoding as Code Page 936. ReadStat is attempting to decode it as "ISO-IR-58", but this is likely incorrect behavior. I'll have a fix soon.