Python 3 compatibility

[ryanhiebert]

Fixes #57

This introduces real Python 3 support. Python 3's csv module is great, but it's not reasonable to attempt to write code that is compatible with Python 2 and 3 with unicodecsv. This fixes that.

Start by running the test suite for Python 3. This involved a lot of test changes to make them Python 3 compatible, as well as writing more code into the py3 module.

Then make sure that the example in the README works for Python 2 and 3, and also add that to the test runner, so that we're sure we don't screw it up again. I had to use _ to capture a couple variables, because their output is different on Python 2 than Python 3, because the csv module in each deals with and reports write counts in the native str type.

[ryanhiebert]

@jdunck: How are you feeling about this change? Think it's reasonable to get it merged?

[jdunck]

@ryanhiebert Is the idea that under py3, the file has been opened in binary mode? I suspect there's a lot of code that currently "just works" under py2 when handing text-opened files to the csv module.

What do you think about adding a test, either at each use of UnicodeReader, or in the constructor of UnicodeReader itself, to check whether the given stream is in binary mode? One approach is fairly fiddly:

('b' in getattr(f, 'mode', '')) or isinstance(f, BytesIO)

Another approach needs to wait until the first read to be certain:

def maybe_decode(bs_maybe):
   if hasattr(bs_maybe, 'decode'):
      return bs_maybe.decode(...)
  else:
      return bs_maybe

The latter approach is likely to be slower for large files.

Anyway, this branch is an improvement, sorry it's been so long for me to give feedback.

[ryanhiebert]

Yes, in Python 3 it must be opened in binary mode, or else the encoding argument to the constructor makes no sense. I don't see any reasonable way around that for user code to be Python 3 compatible.

[ryanhiebert]

It sounds like you'd want to add that bit just to the reader? I suppose that makes some sense, though it's no longer symmetrical.

I can add that to the pull request if you like, or it could be a follow up.

[jdunck]

Eh, since this 2+3 interop is a new feature, we can just make a rule: open your files in binary mode - and document that. Fair enough?

[ryanhiebert]

That's what I had in mind.

[ryanhiebert]

I'm rebasing to master to fix the merge conflict.

[ryanhiebert]

And the tests are failing now. Working on that.

[ryanhiebert]

I went ahead and fixed the failing test caused by #39.

I also removed testing for Python 2.6, added testing for 3.5 and pypy3, and switched Travis CI to use container-based builds.

As I mentioned in #39, I'd prefer if we did not keep the dict field order by using an OrderedDict, because 1) it's an extra point of asymmetry from the csv module, and 2) it requires overriding a core method of Python 3's DictReader. However, I've gone ahead and added a commit with that change so that it'll work whichever you decide.

[jdunck]

You're right about the OrderedDict change - I've reverted that in master.

[ryanhiebert]

Feel free to drop those two commits, or I'll do it later tonight.

[ryanhiebert]

OK, I've rebased to the latest master, and removed the OrderedDict stuff. I think it's mergeable now.

[ryanhiebert]

Rebased again, cleaned up some minor stuff, and added back Python 2.6 support, since we're no longer using OrderedDict.

[jdunck]

Merged, thank you for your code and persistence, @ryanhiebert

[ryanhiebert]

Woohoo!

[ryanhiebert]

@jdunck : I'm obviously antsy to get this released. What other issues are you wanting to get wrapped up before a release? I'll see what I can do to help get them wrapped up.

[jdunck]

Oh, I cut a release just after merging - 0.14.1 should have your work included, and therefore have 2/3 interop. Sorry I didn't make that clear.

[ryanhiebert]

Ah, OK. I saw that you bumped the version the day before you merged, so I thought you'd also created a release then. Sorry about the confusion.

[jdunck]

... Oh, I forgot to tag the last couple releases. Lemme fix that.