Discussion: What does it take to defragment an open source community?

[campagnola]

Current situation: several python projects already exist, partially competing and partially complementary. For the most part we agree this is wasteful, that we could potentially do much more by working together, but actually accomplishing that can be very difficult.

In this thread I'd like to collect thoughts about this process:

1. How to decide whether successfully defragmenting would actually be a net benefit to the community
2. How to decide whether a community is capable of defragmenting, and how much effort it will take
3. How to ensure the project has the resources and attention necessary to succeed
4. How / whether to ensure older projects are deprecated to avoid exacerbating the problem (https://xkcd.com/927/)
5. Pitfalls that have doomed projects like these in the past

[campagnola]

My favorite success story has to be NumPy, but I wasn't close enough to that to know what made it work (perhaps @njsmith has some ideas?).

@aquilesC mentioned the PyLab project via this discussion, which is a joint effort between the developers of lantz, pymeasure, and instrumentkit (maybe others). That seems to have had some success, but again I can't really tell you that story (I welcome comments from @hgrecco, @bilderbuchi, @scasagrande).

From my own experience working on vispy, I can say that project was not easy, but ultimately successful in some of its goals (importantly, we reduced the number of active projects and created a self-sustaining community project). The existence of large, prior codebases and strong, competing views about architecture were a major challenge. I and others have worked to avoid that by keeping the scope of this project minimal as a first step toward establishing a cooperative community.

I think one of the most important things we'll need is a core set of developers who are able to commit significant time to the project. Typically a project like this is driven by necessity--either someone is developing code because they need it for their own research, or because someone is paying them to do it. Many of the people in this group have already invested in their own acquisition systems and will not have the resources or incentive necessary to push the project through. So perhaps an early question to resolve is whether anyone in this group has the necessary resources and incentive, or what organizations would be willing to support the project.

[bilderbuchi]

Thank you for the invitation to contribute. I'll try to be brief to keep focus (a lot can be said about this topic), so if my comments appear curt that's probably because of that.
Also pinging @cjermain and @cgranade who might have something to contribute.

So, from my perspective the effort to unify you mentioned has not really worked. I invite you to look through pymeasure/pymeasure#53 (warning: long thread), which is what the effort to catalogue you mention above came out of. It shows basically the course of the effort to unify several hardware interaction projects, with a lot of discussion and reasoning from a variety of contributors.
I think the whole thing fizzled out primarily because of too little developer/maintainer availability and political/technical reasons (people not wanting to fold their project into another one).

[bilderbuchi]

One thing that made the unification effort hard is that there are so many ways to segment the space of interest (see the online sheet in the OP of LabPy/labpy-discussion#23 for a try to make structured sense of the landscape), so it's very easy for multiple groups to have different focus (and projects), instead of collaborating on one package.

Only having quickly looked from the outside, this also seems to have happened here - you seem to want to start a package focused around microscopy, but instead of using and contributing to an existing project (e.g. https://itom.bitbucket.io/ or https://github.com/python-microscopy/python-microscopy), you initiate a separate thing. Instead of trying to contribute to another effort to unify (e.g. https://github.com/LabPy/ which you might not have found before), you start a fresh community. This will probably lead to more fragmentation in the space, even though you cite the imo extremely relevant https://xkcd.com/927/.
Please know that I am not judging you for that! There are a multitude of reasons to do your own thing (not having known about others; it's more difficult to attract funding for contributing to another project instead of building your own, slightly different focus, license preferences, commercial considerations,...), as I also found in the previous discussions, so maybe this is tragically and inherent property of the space we are operating in.

[bilderbuchi]

Sadly, this exacerbates the one primary problem that I see OSS project having - lack of developer resources. Therefore, we all end up with N projects with slightly different scope and each with ~ 0.M time-starved maintainers, instead of one project with a more sustainable pool of N*0.M maintainers.
This is also very relevant with contributions from academia - often projects are started by grad students, postdocs or other temporary staff - when they finish their work the project often ends up abandoned (hello, bus factor). If there is no professor/permanent staff to focus efforts, or a whole group that uses a package, or several involved institutions, it's very hard to end up with a sustainable pool of developers - critical mass is extremely important IMO.

This is also how I read the situation from LabPy/labpy-discussion#23. Personally, from a candidate pool of InstrumentKit, Lantz and Pymeasure, I have drifted towards using (and trying to contribute to) Pymeasure mainly because there is at least some activity there, so there is a chance of getting your bugs fixed.

Nowadays I have come to regard developer activity as one of the primary rubrics when evaluating if I should use a project or package.

[bilderbuchi]

So, what is the solution? I would love to see a Numpy for data acquisition :-)

I think, considering the tendency of potential contributors to have different foci, it could be worthwhile to evaluate what is out there, and

• find a common set of overlaps, and try to build solutions for that (domain-specific things should probably be elsewhere)
• plus some kind of plugin infrastructure to make it easy for others to adapt your solution to their needs (pytest is a good reference for that I think)
• plus making it easy for people to contribute small things back

IMO, one of these kernels is device drivers for data acquisition, i.e. something on top of VISA, IVI, etc. that implements and tests(!) lots of ready-made interfaces for the many instruments out there! GUI and procedure automation would be different things and out of scope - there, people already have wildly differing needs/wants. Everybody likes pre-made instrument drivers e.g. for the oscilloscope in their lab, though. This is still a hard problem, and lots of different way to "skin the cat" (e.g. what a common API should be)!

I think InstrumentKit is a good example for that, especially the way to test against instrument protocols, but sadly there's very little activity. Lantz is basically in the same situation.
As mentioned, I am using Pymeasure's instrument drivers features for this (although Pymeasure's focus is larger than that). I try to give back as much as possible, but sadly I am also not in a situation where I can invest much time into it - a pool of contributors/maintainers would definitely help!

[bilderbuchi]

Oh man, this still grew too long, sorry for the wall of text. >.<
Also, keep in mind that this is primarily coloured by my own experience, I also don't have the wisdom of the ancients. There already seems to be a number of people collected here, so your opinions might contrast with my own.

I guess my tl,dr could be:

• We couldn't pull this off before, but please use what we found and learn from us
• Gather a large amount/pool of contributors
• Identify a common need
• Build something as generic as possible
• Make sure it's extensible (e.g. via plugins)
• Build specific things on top/elsewhere

[campagnola]

Many thanks for your comments, @bilderbuchi !

One note I want to throw in: although we have already started some work on a new project, I consider this just research in that it is intended to help us decide whether we can agree on the kind of API we need, and from there whether any of the existing systems are adequate, or can be adapted to our needs, or would be willing to deprecate their own code in favor of another.

[bilderbuchi]

Sure, I hope it helps!

I think in the frame of device drivers, my personally ideal package would have

• a driver definition style like pymeasure,
• unitful support like InstrumentKit or Lantz,
• a test harness like InstrumentKit (❤️),
• auto-GUI construction like Lantz (or Pymeasure),
• concurrency features and querying all properties like Lantz.

Ideally also then the

• unified users and
• driver collection (maybe wrapped or autoconverted, but that's probably a metaprogramming nightmare)

from all three to get nearer to critical mass. :dreams:
Of course, the resources or funding for such an endeavour are just not there, we can't even keep up with PR review for Pymeasure in our free time. :-/ At work, barring some funding (EU/research grant), actual research tasks take precedence.
Also, it's probable that the next person over will have different ideas or priorities, and that's when things start drifting apart or into discussions, again.

[bilderbuchi]

I think what is important for attracting a sizeable crowd is that what you are rallying people around is generic and not focused on a specific use case. For example, if the direction goes into "control for microscopy", or Quantum computing something, this puts me on a sceptical footing concerning the usefulness for my usecase (I am mostly controlling vacuum chambers and high-voltage power supplies), and I think other people will react similarly. If I think about the larger, well-established projects in this space (numpy, PyVISA, pyserial, matplotlib/pyqtgraph,...), they offer building blocks, not specialist tools.
I think making building blocks for "every"one, and building more specialised tooling on top, would be the more promising approach.

[bilderbuchi]

In any case, I recommend sitting down in a comfy chair, making a glass of your favourite concoction, and read through the long but hopefully illuminating pymeasure/pymeasure#53, I feel we have covered similar ground over there.

[cjermain]

@bilderbuchi, I think you summarized the situation very well, and definitely captured my feeling on this. That issue is a great display of multiple failed attempts to consolidate the space. :)

[berl]

@bilderbuchi very interesting read on pymeasure#53, especially considering the range of timestamps traversed by that discussion!

I think that there is a natural tension between a specialist group (could be single molecule microscopists, or slice electrophysiologists or whatever) trying to do this for their use cases and the desire you expressed to

Build something as generic as possible

And then the impedance mismatch between project scope and contributor hours is probably easiest adjusted by starting small and specialized and building up, and that path of least resistance has led to the hordes (?) of dead or zombie projects in this space staggering around github.

I still think that an organic approach based on specialist projects wanting to remove driver dependencies from their projects might have the right combination of risk-reward to gain community buy-in incrementally. It sounds like I'm not the first person to naively hope for this...

One more thing Iearned from heading down the amazing array of python hardware control rabbit holes (some from your comments I had no idea about!): There are wildly disparate use-cases represented here and wildly diverse sets of hardware represented in these labs. I think our initial ideas and discussions implicitly included only a tiny fraction of this diversity. Given the resource problems you spelled out above, it seems like carving out the scope of this project (like a list of hardware devices that we're considering and not considering) is critical to making it manageable to even have a discussion.

[bilderbuchi]

@berl thank you - basically, 👍 on what you said!
I agree that unifying the driver situation could be most promising approach. Drivers are where starting to use Python needs additional effort compared to, e.g. Labview, where drivers are often already implemented by the manufacturer. In addition, I feel like drivers are still a low-level enough unifying component to concern most users irrespective of background/application, so it should be easier to get critical mass.

I think the complex thing here would be to define a generic but extensible API and usage patterns fitting as many device types as possible, so that everybody wanting to contribute a device driver can find their slot. IVI instrument classes are one way to approach this/learn from, but I am not sure how well this translates to Python.
Then, people can put other stuff on top (classical GUI, notebook interfaces, multiprocessing|threading|concurrency, distributed solutions, scripting/automation)
The application-specific stuff (things for analysing black holes, microscopy, controlling lathes, or whatnot) can go on top, and everybody can profit from what is below.
But yeah, that's probably a hard/large undertaking...