-
Notifications
You must be signed in to change notification settings - Fork 1k
fwrite(dec=',') should use it for timestamps with milliseconds for consistency #6446
Description
Okay, I just looked at the discussions of the standard, and it actually allows the comma separator (which is surprising).
My concern is data that potentially mixes separators, because even preferring , for decimal, software will likely output . for ISO timestamps (data.table itself is an example):
t <- fread(text="x\tDatetime\n1,1\t2023-10-12T06:53:53,123Z", sep="\t")
fwrite(t, file="", dec=",", sep=";")x;Datetime
1,1;2023-10-12T06:53:53.123Z
So this fix (which, if I understand correctly, is trying to reparse with the other separator if one fails) should be OK?
Here's a sample of my real data that tripped this issue, which was output by data.table's fwrite():
CoilID;AntennaID;Time;TagID;Pen;Side;Position;Location;Coil_Y;Coil_X
1;16403160;2023-10-12T10:30:55.270Z;DA2C6411;1;AKB;Litter central;middle;1;6
3;16403160;2023-10-12T10:30:55.270Z;DA459D86;1;AKB;Litter central;middle;1;4
15;16402963;2023-10-12T10:31:00.900Z;DA459D86;1;AKB;Litter central;right;2;3
14;16402963;2023-10-12T10:31:02.240Z;DA2C6411;1;AKB;Litter central;right;2;1
11;16403160;2023-10-12T10:31:02.650Z;DA2C6411;1;AKB;Litter central;middle;2;6
P.S. Datetimes are weird. Were you aware that ISO 8601 allows fractional values other than seconds? E.g. 10:20,5 is valid and means 10:20:30... Probably not a good idea to support this though. Something more sane like RFC 3339 draft (which fwrite seems to follow) is already supported and that's likely good enough.
Originally posted by @kav2k in #6445 (comment)