Github action to automatically update precompile files

[aminya]

Automatic repository updating using Github actions for uploading the files generated by snoopCompile.jl script

Similar to https://github.com/julia-actions/julia-uploadcodecov or https://github.com/julia-actions/julia-docdeploy

[timholy]

Cool idea. These seem to require JavaScript, a language I don't speak, so I'd rely on external contributors for such bots.

[aminya]

Also, I found this push bot: https://github.com/ad-m/github-push-action which does exactly what we want. We just need to run the Julia script before this.

[aminya]

Probably @davidanthoff can help us with this

[aminya]

I made a workflow file that does what we want! 😃
https://github.com/juliamatlab/MatLang/blob/master/.github/workflows/SnoopCompile.yml

Runs the snoopCompile script and creates a pull request.

[davidanthoff]

Cool! Once you guys have tried it out for a while I'd be happy to create a proper github action and integrate it with https://github.com/davidanthoff/PkgButlerEngine.jl.

[aminya]

I get this error: https://github.com/aminya/AcuteML.jl/commit/141099db133d89c147f94779d06a20cdbed6be65/checks?check_suite_id=337167681#step:7:5
Do you know what the reason is?

Launching new julia process to run commands...
ERROR: LoadError: could not open file /home/runner/work/AcuteML.jl/AcuteML.jl/deps/SnoopCompile/precompile/precompile_AcuteML.jl
Stacktrace:
 [1] include at ./boot.jl:328 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1105
 [3] include at ./Base.jl:31 [inlined]
 [4] include(::String) at /home/runner/work/AcuteML.jl/AcuteML.jl/src/AcuteML.jl:1
 [5] top-level scope at /home/runner/work/AcuteML.jl/AcuteML.jl/src/AcuteML.jl:1055
 [6] include at ./boot.jl:328 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1105
 [8] include(::Module, ::String) at ./Base.jl:31
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:330 [inlined]
 [11] eval(::Expr) at ./client.jl:425
 [12] top-level scope at ./none:3
in expression starting at /home/runner/work/AcuteML.jl/AcuteML.jl/src/AcuteML.jl:1055
ERROR: Failed to precompile AcuteML [6b84a0a3-a11a-41fa-a417-e600528de108] to /home/runner/.julia/compiled/v1.3/AcuteML/DX2uo_b3BsC.ji.
Stacktrace:
done.
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
 [3] _require(::Base.PkgId) at ./loading.jl:1024
 [4] require(::Base.PkgId) at ./loading.jl:922
 [5] require(::Module, ::Symbol) at ./loading.jl:917
 [6] macro expansion at /home/runner/work/AcuteML.jl/AcuteML.jl/deps/SnoopCompile/snoopCompile.jl:13 [inlined]

[aminya]

Oh!! We cannot run SnoopCompile when the __precompile__ is added to the source file! So, it should remove those lines of code before running the tests

[aminya]

Consider we want to comment and uncomment this

include("../deps/SnoopCompile/precompile/precompile_AcuteML.jl")
_precompile_()

I have written this but it doesn't work, it appends the code to the end of file instead of replacing it

packageName = "AcuteML"

function precompileDeactivator(packageName)

    file = open(joinpath(pwd(),"src","$packageName.jl"),"r+")

    packageText = read(file, String)

    packageEdited = foldl(replace,
                 (
                  "include(\"../deps/SnoopCompile/precompile/precompile_$packageName.jl\")" => "#include(\"../deps/SnoopCompile/precompile/precompile_$packageName.jl\")",
                  "_precompile_()" => "#_precompile_()",
                 ),
                 init = packageText)
                 print(packageEdited)
     write(file, packageEdited)
     close(file)
end

function precompileActivator(packageName)

    file = open(joinpath(pwd(),"src","$packageName.jl"),"r+")
    packageText = read(file, String)

    packageEdited = foldl(replace,
                 (
                  "#include(\"../deps/SnoopCompile/precompile/precompile_$packageName.jl\")" => "include(\"../deps/SnoopCompile/precompile/precompile_$packageName.jl\")",
                  "#_precompile_()" => "_precompile_()",
                 ),
                 init = packageText)

     write(file, packageEdited)
     close(file)
end


################################################################
precompileDeactivator(packageName);

rootPath = pwd()
cd(@__DIR__)

### Log the compiles
# This only needs to be run once (to generate log file)

SnoopCompile.@snoopc "$(pwd())/Snoop.log" begin

    # Use runtests.jl or your exmaples that uses package:

    using AcuteML, Pkg

    include(joinpath(dirname(dirname(pathof(AcuteML))), "test","runtests.jl"))
    include(joinpath(dirname(dirname(pathof(AcuteML))), "examples","extractor.jl"))
    include(joinpath(dirname(dirname(pathof(AcuteML))), "examples","constructor.jl"))
    include(joinpath(dirname(dirname(pathof(AcuteML))), "examples","templating","templating.jl"))


end

################################################################

### Parse the compiles and generate precompilation scripts
# This can be run repeatedly to tweak the scripts

data = SnoopCompile.read("$(pwd())/Snoop.log")

pc = SnoopCompile.parcel(reverse!(data[2]))
SnoopCompile.write("$(pwd())/precompile", pc)

################################################################
precompileActivator(packageName)
cd(rootPath)

[aminya]

I fixed the problem by updating the script. It works on my package now! 😃
https://github.com/juliamatlab/MatLang/commit/4171d4630e618ff12487d65464e88b697b0abf0e/checks?check_suite_id=337510876

PR generated: juliamatlab/MatLang#16

I think we should create an action repository to make the workflow file and also user's snoopCompile.jl file simpler.

[KristofferC]

FWIW, I don't think generating precompile statements based on full test suites is a good idea in general, and should not be something that should be enabled by default on packages with e.g. PkgButler type helpers.

[timholy]

Yes, generally what makes sense for precompilation is quite different from what makes sense for testing. For example, for FixedPointNumbers and ColorTypes, I just wrote the precompile files by hand, with some guidance from SnoopCompile, because the tests exercise all kinds of weird types (e.g., N3f5) that are never used in practice.

[aminya]

One should add 3 things to a package to make the bot work:

• worflow file https://github.com/juliamatlab/MatLang/blob/master/.github/workflows/SnoopCompile.yml
• the examples that call the functions or Pkg.test() https://github.com/juliamatlab/MatLang/blob/master/deps/SnoopCompile/snoopCompile.jl
• Two lines of code that includes the precompile file https://github.com/juliamatlab/MatLang/blob/0206d062096cf3cbe3b7c496d058319b04d8fed9/src/MatLang.jl#L27

Now I didn't choose test as the default but that's a script with good coverage of API. For MatLang, I used the examples I wrote, but for Atom I had to use the tests (nothing else is available).