WebGL fails to build for a newly created unity project with FileNotFoundException: Could not find file "[...]/ExampleProjectName.data.gz.compressed"

[GabLeRoux]

Bug description
The v0.9 release added support for webgl 🎉, but building a newly created project fails for the webgl target fails with the following error:

Filename: gzip
Arguments: -9 --keep -S ".compressed" "/builds/gableroux/unity3d-gitlab-ci-example/Temp/StagingArea/Data/Output/Build/ExampleProjectName.data.gz"
index: -1
FileNotFoundException: Could not find file "/builds/gableroux/unity3d-gitlab-ci-example/Temp/StagingArea/Data/Output/Build/ExampleProjectName.data.gz.compressed"
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <9577ac7a62ef43179789031239ba8798>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <9577ac7a62ef43179789031239ba8798>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.OpenRead (System.String path) [0x00000] in <9577ac7a62ef43179789031239ba8798>:0 
  at System.IO.File.ReadAllBytes (System.String path) [0x00000] in <9577ac7a62ef43179789031239ba8798>:0 
  at UnityEditor.WebGL.WebGlBuildPostprocessor.SetGzipComment (System.String path, System.String comment) [0x00001] in /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:761 
  at UnityEditor.WebGL.WebGlBuildPostprocessor.CompressAndMarkGzip (System.String path) [0x000aa] in /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:815 
  at UnityEditor.WebGL.WebGlBuildPostprocessor.CompressBuild (UnityEditor.Modules.BuildPostProcessArgs args) [0x00058] in /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:851 
  at UnityEditor.WebGL.WebGlBuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args) [0x002f5] in /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:920 
  at UnityEditor.Modules.DefaultBuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args, UnityEditor.BuildProperties& outProperties) [0x00001] in /home/bokken/buildslave/unity/build/Editor/Mono/Modules/DefaultBuildPostprocessor.cs:27 
  at UnityEditor.PostprocessBuildPlayer.Postprocess (UnityEditor.BuildTargetGroup targetGroup, UnityEditor.BuildTarget target, System.String installPath, System.String companyName, System.String productName, System.Int32 width, System.Int32 height, UnityEditor.BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report) [0x000dc] in /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline/PostprocessBuildPlayer.cs:324 
UnityEditor.BuildPipeline:BuildPlayerInternalNoCheck(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[], Boolean)
UnityEditor.BuildPipeline:BuildPlayerInternal(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[]) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:399)
UnityEditor.BuildPipeline:BuildPlayer(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[]) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:298)
UnityEditor.BuildPipeline:BuildPlayer(BuildPlayerOptions) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:272)
UnityEditor.BuildPipeline:BuildPlayer(String[], String, BuildTarget, BuildOptions) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:267)
BuildCommand:PerformBuild() (at Assets/Scripts/Editor/BuildCommand.cs:181)

How to reproduce

1. Create a blank unity project
2. Build the project using webgl docker image at version 0.9

See example failing pipeline here:
https://gitlab.com/gableroux/unity3d-gitlab-ci-example/-/jobs/1021215197

Codebase reproducing the error:
https://gitlab.com/gableroux/unity3d-gitlab-ci-example/-/tree/e7d8079debbe567d0a417f419b111bc3b48fbc86

Expected behavior
Should succeed

Additional details
This was reproduced here:
https://gitlab.com/gableroux/unity3d-gitlab-ci-example/-/merge_requests/55

Workaround

Set Compression Format to Disabled instead of the default Gzip in Project Settings > Player > Settings for WebGL.

Or directly in ProjectSettings/ProjectSettings.asset:
🔥 Brotli = webGLCompressionFormat: 0 Confirmed failing here
🔥 Gzip = webGLCompressionFormat: 1 Confirmed failing here
✅ Disabled = webGLCompressionFormat: 2 Confirmed working here

[GabLeRoux]

Using Brotli compression for webgl is also failing with following error:

Failed running python "/opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/python/bro.py" -o "/builds/gableroux/unity3d-gitlab-ci-example/Temp/StagingArea/Data/Output/Build/ExampleProjectName.data.br.compressed" -i "/builds/gableroux/unity3d-gitlab-ci-example/Temp/StagingArea/Data/Output/Build/ExampleProjectName.data.br" --comment "UnityWeb Compressed Content (brotli)"
stdout:
stderr:Traceback (most recent call last):  File "/opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/python/bro.py", line 8, in <module>    import brotli  File "build/bdist.macosx-10.10-x86_64/egg/brotli.py", line 7, in <module>  File "build/bdist.macosx-10.10-x86_64/egg/brotli.py", line 3, in __bootstrap__ImportError: No module named pkg_resources
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:LogError(Object)
UnityEditor.WebGL.ProgramUtils:StartProgramChecked(ProcessStartInfo) (at /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/ProgramUtils.cs:46)
UnityEditor.WebGL.WebGlBuildPostprocessor:CompressAndMarkBrotli(String) (at /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:835)
UnityEditor.WebGL.WebGlBuildPostprocessor:CompressBuild(BuildPostProcessArgs) (at /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:848)
UnityEditor.WebGL.WebGlBuildPostprocessor:PostProcess(BuildPostProcessArgs) (at /Users/bokken/buildslave/unity/build/PlatformDependent/WebGL/Extensions/Unity.WebGL.extensions/BuildPostprocessor.cs:920)
UnityEditor.Modules.DefaultBuildPostprocessor:PostProcess(BuildPostProcessArgs, BuildProperties&) (at /home/bokken/buildslave/unity/build/Editor/Mono/Modules/DefaultBuildPostprocessor.cs:27)
UnityEditor.PostprocessBuildPlayer:Postprocess(BuildTargetGroup, BuildTarget, String, String, String, Int32, Int32, BuildOptions, RuntimeClassRegistry, BuildReport) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline/PostprocessBuildPlayer.cs:324)
UnityEditor.BuildPipeline:BuildPlayerInternalNoCheck(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[], Boolean)
UnityEditor.BuildPipeline:BuildPlayerInternal(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[]) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:399)
UnityEditor.BuildPipeline:BuildPlayer(String[], String, String, BuildTargetGroup, BuildTarget, BuildOptions, String[]) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:298)
UnityEditor.BuildPipeline:BuildPlayer(BuildPlayerOptions) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:272)
UnityEditor.BuildPipeline:BuildPlayer(String[], String, BuildTarget, BuildOptions) (at /home/bokken/buildslave/unity/build/Editor/Mono/BuildPipeline.bindings.cs:267)
BuildCommand:PerformBuild() (at Assets/Scripts/Editor/BuildCommand.cs:181)

Updated issue description accordingly.

[mob-sakai]

GZip (in Unity 2020.1 or later)

Filename: gzip
Arguments: -9 --keep -S ".compressed" "/builds/gableroux/unity3d-gitlab-ci-example/Temp/StagingArea/Data/Output/Build/ExampleProjectName.data.gz"

• The above command (i.e.gzip -9 --keep -S ".compressed" ".../ExampleProjectName.data.gz") will fail for the following reasons.
  • gzip: /data/logs/log01.log.gz already has .gz suffix -- unchanged
• Workaround: add -f as default option for gzip
  • export GZIP=-f

Brotli (in all versions)

• No module named pkg_resources
  • pkg_resources module is included in python-setuptools package.
  • Workaround: apt-get install python-setuptools
• __bootstrap__ImportError: /root/.cache/Python-Eggs/Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg-tmp/brotli.so: invalid ELF header
  • Refer Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg instead of Brotli-0.4.0-py2.7-linux-x86_64.egg.
  • This reference is hard-coded in UnityEditor.WebGL.Extensions.dll, so we will need to copy or link the file.
  • Workaround: cp /opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-linux-x86_64.egg /opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg

---

I'm testing the validity of these fixes.

ARG ediotrImage=unityci/editor:2020.1.11f1-webgl-0.9.0
FROM $ediotrImage

ENV GZIP=-f

RUN apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    python-setuptools \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  && cp \
    /opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-linux-x86_64.egg \
    /opt/unity/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg

[mob-sakai]

Test results

I ran the following commands for log:

find . -name "7_Build project (GZip).txt" | sort | xargs -I {} grep -l '\(Build failed!\|CompressAndMarkGzip\)' {} > gzip_fails.txt
echo "GZip: `cat gzip_fails.txt | wc -l` versions failed"
find . -name "9_Build project (Brotli).txt" | sort | xargs -I {} grep -l '\(Build failed!\|CompressAndMarkBrotli\)' {} > brotli_fails.txt
echo "Brotli: `cat brotli_fails.txt | wc -l` versions failed"

Before the modification (0.9.0, 142 Unity versions)

Results: (NOTE) In Unity 2018.4.15 or earlier, even if the brotli fails, the build will be marked as successful.
gzip_fails.txt
brotli_fails.txt

GZip:       20 versions failed
Brotli:     142 versions failed

• GZip failed in 2020.1.x, 2020.2.0 and 2020.2.1.
• Brotli failed in all versions.

After the modification (142 Unity versions)

Results

GZip:       0 versions failed
Brotli:     0 versions failed

Conclusions

I think the modification is valid. I am going to create a PR.

[webbertakken]

Great work!

I think the modification is valid. I am going to create a PR.

Yea sounds good. Could you please include the size difference for the image?

[mob-sakai]

REPOSITORY       TAG                   IMAGE ID       CREATED          SIZE
unityci/editor   dev                   ed64129acfe7   49 seconds ago   6.21GB
unityci/editor   2020.1.11f1-webgl-0   e10428a15532   5 days ago       6.2GB

It's small.