Feature request: `spago init` should create a skeleton that separates `src` and `tests`

[m-bock]

Currently spago init creates this structure:

├── packages.dhall
├── spago.dhall
├── src
│   └── Main.purs
└── test
    └── Main.purs

and cat spago.dhall:

{-
Welcome to a Spago project!
You can edit this file as you like.
-}
{ name =
    "my-project"
, dependencies =
    [ "effect", "console", "psci-support" ]
, packages =
    ./packages.dhall
, sources =
    [ "src/**/*.purs", "test/**/*.purs" ]
}

I'd suggest that in any way there should be two spago.dhall files generated: One for the tests and the other one for the actual project

[f-f]

@thought2 what would be the use case for this?

[m-bock]

Isn't it the most common case that you'd like to have your test dependencies separated from your project's dependencies?

[JordanMartinez]

I'm not as familiar with Dhall as others, so I'm not sure how relevant this comment is. However, do we really need two separate files though?

It seems to me that the name and packages part of this file will be the same for both the src code, test code or even a hypothetical benchmark code. Most likely one will depend on another, but I'd imagine that it should still be flexible enough to use different package sets if needed.

Could it instead be:

{-
Welcome to a Spago project!
You can edit this file as you like.
-}
let packages = ./packages.dhall
let main = { packages = packages
           , dependencies = [ "effect", "console", "psci-support" ]
           , sources = [ "src/**/*.purs" ]
           }
let test = { packages = packages
           , dependencies = main.dependencies # [ "spec" ]
           , sources = main.sources # [ "test/**/*.purs" ]
           }
-- let otherTarget = --
in { name =
     "my-project"
   , targets =
     { main = main
     , test = test
--  , otherTarget = otherTarget
     }
   }

and then spago -x otherFile.dhall build would become spago -x otherTarget build.

[f-f]

@JordanMartinez I'd prefer your solution actually, because of two reasons:

• it allows us to enforce more nicely a single package set for the whole project
• it lets us have a decent implementation for spago sources (e.g. maybe the union of the sources of the targets? I'm not sure what behaviour we should default to, but what matters is that we have all the sources and deps in one place), which was my biggest doubt when it came to editor support for implementing this

[JordanMartinez]

Regarding the spago sources implementation. I think there are two main use cases, right?

1. it provides data that is consumed by other tools
2. it provides data that is consumed by humans.

If being consumed by tools, it would output JSON or something. If consumed by humans, it would output something more readable. For example, human readable output might look like this::

target: main
.spago/console/v4.2.0/src/**/*.purs
.spago/effect/v2.0.1/src/**/*.purs
.spago/newtype/v3.0.0/src/**/*.purs
.spago/partial/v2.0.1/src/**/*.purs
.spago/prelude/ps-0.14/src/**/*.purs
.spago/psci-support/v4.0.0/src/**/*.purs
.spago/safe-coerce/v0.0.2/src/**/*.purs
.spago/unsafe-coerce/v4.0.0/src/**/*.purs
src/**/*.purs

target: test
.spago/console/v4.2.0/src/**/*.purs
.spago/effect/v2.0.1/src/**/*.purs
.spago/newtype/v3.0.0/src/**/*.purs
.spago/partial/v2.0.1/src/**/*.purs
.spago/prelude/ps-0.14/src/**/*.purs
.spago/psci-support/v4.0.0/src/**/*.purs
.spago/safe-coerce/v0.0.2/src/**/*.purs
.spago/unsafe-coerce/v4.0.0/src/**/*.purs
.spago/spec/v4.0.1/src/**/*.purs
src/**/*.purs
test/**/*.purs

Not sure what the best way to format that would be for the human readable one. I think the JSON one should be pretty straight forward.

[f-f]

@JordanMartinez yeah, though right now the tools consume the newline-separated representation, and I think they should continue to do so for backwards compatibility.
If they commit to follow on a possible transition then we could switch in the least painful way, which is: add some flag (e.g. --next) to output the new format (possibly in json?), then wait for the tooling to use it, then switch to the new format by default, then remove the flag (all of this throughout the span of many versions)

AFAIK sources has never been about human-readable output though

[thomashoneyman]

I've been reading through this issue as I've been migrating more of the contributor libraries to Spago.

I wanted to share some cases in which splitting a project's sources / dependencies is a good idea, and to endorse this feature request.

To a degree, a project relies on its sources and that’s that. But there are reasonable cases for when you want to split them. Examples that come to mind:

1. A library that wants to be able to run tests / examples but not incur those dependencies on other projects which import the library. You shouldn’t have to install halogen-storybook to be able to use hooks, for example.
2. An application that wants to be able to run tests / examples but which should bundle without those sources. Or when using a tool like spago2nix, the distribution of the app shouldn’t have any of the testing code included, which would otherwise not be possible to disentangle. (It's still possible to do things like DCE and add rules to a bundler, I'll admit).

With that in mind I think it’s reasonable to require a primary build target which is used for distribution to other libraries, and to allow other build targets in the same config file. Maybe some kind of default can exist (spago build does the main target, spago test uses the test target, and otherwise you can do spago build --target examples or something).

I'm less certain how package sets should handle this. When installing a package I would expect only the 'main' target to be installed, but when a package is tested for package sets, do we only verify the 'main' target compiles? Or do we want the 'main' and 'test' (if it exists) target to build and tests to pass? Or do all targets need to build?

For the purposes of package sets I suspect we at most want 1) a package to be installed with its main target and 2) a package to be built with its main target and, possibly, tested with its test target (if it exists).

I haven't had problems with the dependencies / devDependencies used by NPM, Bower, etc. in the past, and if it's simpler to support just those two keys instead of arbritrary targets that's OK with me as well.

[hdgarrood]

Note that we do effectively have these two sets of dependencies separate as things stand now: the contents of spago.dhall in a library are only looked at when you’re working on that library, so they are effectively the devDependencies. The list of dependencies which matters when you’re pulling that library in as a dependency is the list of dependencies in the package set.

[thomashoneyman]

There is one other wrinkle, which is that the Bower file generated when publishing will include both test and non-test dependencies if they’re all in the same Spago file. This won’t be an issue when there’s no longer a need to publish via Bower, but it’s still an issue now.

One other thought: how do we keep the package set’s list of dependencies in sync with the library? For example, if I drop a dependency from my library and someone else bumps the version in package sets, there will be some drift between the package set version and what my library specifies. (I can ask this on package-sets alternately.) If the dependencies were pulled from the library’s actual spago.dhall file then there’s one source of truth.

[hdgarrood]

Right, that's a good point. I agree that ideally a single package manifest file stored in the library repo would be the source of truth for both production and development dependencies. I think it's probably best to tackle this as part of the ongoing registry work in the registry repo - in fact that may already be handled there, I'm not sure exactly where that's up to now.

[f-f]

Closing this one in favor of #681