Support incremental builds

[deemp]

Superseded by purescript/purescript#3995

Related to #700

Problem

Hi! spago seems to ignore already compiled files.

Steps to reproduce

1. Clone this repo at master or at a commit, e.g. bbe37b6.

2. Remove the output directory.

3. Run spago build.

   • you may see logs like this:

   Reading Spago workspace configuration...
   
   ✅ Selecting 6 packages to build:
       docs-search-client-halogen
       docs-search-common
       docs-search-index
       spago
       spago-bin
       spago-core
   
   Downloading dependencies...
   Building...
   [  11 of 1014] Compiling Record.Unsafe
   [  10 of 1014] Compiling Spago.Command.Script
   [   9 of 1014] Compiling Web.DOM.Internal.Types
   [   3 of 1014] Compiling Spago.Generated.BuildInfo
   [   7 of 1014] Compiling Type.Function
   [   2 of 1014] Compiling Type.Row
   [   1 of 1014] Compiling Unsafe.Coerce
   [   6 of 1014] Compiling Web.Event.Internal.Types
   [   4 of 1014] Compiling Type.Proxy
   [   8 of 1014] Compiling Type.Equality
   [   5 of 1014] Compiling Spago.Texts
   [  12 of 1014] Compiling Partial
   ...

4. Interrupt the build (Ctrl+C in my case) before it finishes.

5. Check the output directory (part of the logs is omitted).

   tree output
   output
   |-- Control.Semigroupoid
   |   |-- corefn.json
   |   |-- docs.json
   |   |-- externs.cbor
   |   |-- index.js
   |   `-- index.js.map
   |-- Data.ArrayBuffer.Types
   |   |-- corefn.json
   |   |-- docs.json
   |   |-- externs.cbor
   |   |-- index.js
   |   `-- index.js.map
   |-- Data.Boolean
   |   |-- corefn.json
   |   |-- docs.json
   |   |-- externs.cbor
   |   |-- index.js
   |   `-- index.js.map
   
   ...
   
   23 directories, 116 files

6. Record the statistics on output/Control.Semigroupoid/index.js.

   $ stat output/Control.Semigroupoid/index.js
   Access: 2024-01-25 01:19:46.011057433 +0300
   Modify: 2024-01-25 01:32:12.667933173 +0300
   Change: 2024-01-25 01:32:12.667933173 +0300
   Birth: 2024-01-25 01:19:46.011057433 +0300

7. Run spago build again.

   Reading Spago workspace configuration...
   
   ✅ Selecting 6 packages to build:
       docs-search-client-halogen
       docs-search-common
       docs-search-index
       spago
       spago-bin
       spago-core
   
   Downloading dependencies...
   Building...
   [   3 of 1014] Compiling Web.DOM.Internal.Types
   [   8 of 1014] Compiling Type.Function
   [   7 of 1014] Compiling Type.Proxy
   [   1 of 1014] Compiling Type.Equality
   [   5 of 1014] Compiling Type.Row
   [   6 of 1014] Compiling Web.Event.Internal.Types
   [   2 of 1014] Compiling Spago.Texts
   [   4 of 1014] Compiling Unsafe.Coerce
   [   9 of 1014] Compiling Spago.Generated.BuildInfo
   [  10 of 1014] Compiling Spago.Command.Script
   [  11 of 1014] Compiling Record.Unsafe
   [  12 of 1014] Compiling Partial
   [  13 of 1014] Compiling JS.Fetch.Referrer
   [  14 of 1014] Compiling Data.Void
   [  15 of 1014] Compiling Data.Unit
   [  16 of 1014] Compiling Data.Traversable.Accum
   [  17 of 1014] Compiling Data.String.Unsafe
   [  18 of 1014] Compiling Data.NaturalTransformation
   [  19 of 1014] Compiling Data.Int.Bits
   [  20 of 1014] Compiling Data.Boolean
   [  21 of 1014] Compiling Data.ArrayBuffer.Types
   [  22 of 1014] Compiling Control.Semigroupoid

8. Interrupt the build.

9. Record the statistics on output/Control.Semigroupoid/index.js.

   Access: 2024-01-25 01:19:46.011057433 +0300
   Modify: 2024-01-25 01:35:40.739062556 +0300
   Change: 2024-01-25 01:35:40.739062556 +0300
   Birth: 2024-01-25 01:19:46.011057433 +0300

Expected result

• No built file is modified during a new build
• spago build-s less files (now it's 1014) on subsequent runs

[f-f]

Is this a spago issue? Does it behave differently if you just call purs? (if you run spago with -v it will print the purs commands it's calling under the hood)

[deemp]

IDK whether purs or spago is responsible for tracking the built files.

spago build -v shows that purs is called once and for all .purs files.

[       9ms] CWD: /home/eyjafjallajokull/Desktop/gh/spago
[      13ms] Global cache: "/home/eyjafjallajokull/.cache/spago-nodejs"
[      13ms] Local cache: "/home/eyjafjallajokull/Desktop/gh/spago/.spago"
[      47ms] DB: Connecting to database at /home/eyjafjallajokull/.cache/spago-nodejs/spago.v1.sqlite
[      50ms] Reading Spago workspace configuration...
[      50ms] Reading config from spago.yaml
[     147ms] Found packages at these paths:
  bin/spago.yaml
  core/spago.yaml
  docs-search/client-halogen/spago.yaml
  docs-search/common/spago.yaml
  docs-search/index/spago.yaml
[     148ms] Reading config from bin/spago.yaml
[     152ms] Reading config from core/spago.yaml
[     154ms] Reading config from docs-search/client-halogen/spago.yaml
[     158ms] Reading config from docs-search/common/spago.yaml
[     160ms] Reading config from docs-search/index/spago.yaml
[     362ms] Lockfile is up to date, using it
[     363ms] Found lockfile, using the package set from there
[     364ms] 
✅ Selecting 6 packages to build:
    docs-search-client-halogen
    docs-search-common
    docs-search-index
    spago
    spago-bin
    spago-core

[     364ms] Requested to install these packages: []
[     372ms] Downloading dependencies...
[     393ms] Building...
[     400ms] Running command:
purs compile --codegen corefn,docs,js,sourcemaps --json-errors .spago/p/aff-7.1.0/src/**/*.purs .spago/p/aff-promise-4.0.0/src/**/*.purs

...

[f-f]

Spago doesn't track builds, that's entirely happening in the compiler. Spago only puts together the globs and processes the output.

[deemp]

I see the compiler repo has a similar issue. I updated the description of this issue and will close it now.

[MonoidMusician]

The compiler won't reuse artefacts from canceled builds. You have to let a full build complete before you can see it skip compiling modules.