Github CI: Laplace Dirichlet 3D example occasionally runs into increasing GMRES residuals

[inducer]

Like this:

 -----------------------------------------------------------------------
RUNNING ./laplace-dirichlet-3d.py
-----------------------------------------------------------------------
IT        0 1.55434597e+01
IT        1 6.94270121e+00
IT        2 3.52489688e+00
IT        3 3.36265158e+00
IT        4 2.97770742e+00
IT        5 2.72544530e+00
IT        6 2.68395559e+00
IT        7 2.59318187e+00
IT        8 2.59154516e+00
IT        9 2.40918314e+00
IT       10 1.30052827e+02
/home/runner/work/pytential/pytential/pytential/symbolic/primitives.py:1571: UserWarning: specified the name of the direction vector
  return DirectionalSourceDerivative(
/home/runner/work/pytential/pytential/.miniforge3/envs/testing/lib/python3.8/site-packages/sumpy/kernel.py:1020: UserWarning: specified the name of the direction vector
  return type(kernel)(
/home/runner/work/pytential/pytential/.miniforge3/envs/testing/lib/python3.8/site-packages/loopy/check.py:251: LoopyWarning: in kernel p2p_from_csr: Found unused inames in kernel: frozenset({'idim'}) Unused inames during linearization will be prohibited in Loopy version 2021.X. (add 'unused_inames' to silenced_warnings kernel attribute to disable)
  warn_with_kernel(
/home/runner/work/pytential/pytential/.miniforge3/envs/testing/lib/python3.8/site-packages/loopy/check.py:251: LoopyWarning: in kernel p2qbxl_from_csr: Found unused inames in kernel: frozenset({'idim'}) Unused inames during linearization will be prohibited in Loopy version 2021.X. (add 'unused_inames' to silenced_warnings kernel attribute to disable)
  warn_with_kernel(
/home/runner/work/pytential/pytential/.miniforge3/envs/testing/lib/python3.8/site-packages/sumpy/kernel.py:1020: UserWarning: specified the name of the direction vector
  return type(kernel)(
Traceback (most recent call last):
  File "laplace-dirichlet-3d.py", line 179, in <module>
    main()
  File "laplace-dirichlet-3d.py", line 134, in main
    gmres_result = gmres(
  File "/home/runner/work/pytential/pytential/pytential/solve.py", line 293, in gmres
    result = _gmres(op, rhs, restart=restart, tol=tol, x0=x0,
  File "/home/runner/work/pytential/pytential/pytential/solve.py", line 170, in _gmres
    raise GMRESError(state)
pytential.solve.GMRESError: non-monotonic residuals

I've only observed this on Github, not Gitlab. This may indicate that it's less likely to happen when more cores are available, i.e. possibly some synchronization thing.

Here's a run that exhibits this: https://github.com/inducer/pytential/runs/1832875842?check_suite_focus=true

As far as I remember, this has been happening for a while. I think I have an email from Jan 16 where this occurred, potentially even earlier. Delinquently, I've often just rerun the CI job which typically resolved the failure, until it would reoccur on a subsequent run.

The recent failures under discussion in #56 are distinct IMO.

[inducer]

I just ran that test over and over on dunkel (with 5525398), while simulating a resource-constrained environment:

while true; do rm-vis; POCL_MAX_PTHREAD_COUNT=4 PYOPENCL_TEST=port numactl -N 0,1 python ./laplace-dirichlet-3d.py || break ; done 

It ran around 100 times, without ever encountering the issue.

[inducer]

Ran the same command on a conda installation (which should be the same as what the Github CIs run?). It ran for about seven hours without crashing once.

[inducer]

In #61, I'm trying to use the (very slick-looking, actually) tmate (actually, their Github Action) to try and get a shell on the runner, to see what's happening.

[inducer]

I'm now running

i=0; while true; do i=$((i+1)); echo "RUN NUMBER $i";rm -f *vtu *vts; PYOPENCL_TEST=port python ./laplace-dirichlet-3d.py || break ; done

interactively on the Azure machine behind Github actions, and the test has run successfully ~10 times so far. (which doesn't match the stats otherwise) Maybe this is somehow dependent on the processor type? This one is a Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz.

[inducer]

ci-support will now log the CPU type: inducer/ci-support@610dcc4

[inducer]

OK, so much for the theory of being system-dependent.

• My interactive run has now finished 70 times without any hiccup. (on an "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz")
• https://github.com/inducer/pytential/runs/1947518681?check_suite_focus=true shows a run (on an "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz", i.e. the same thing unless I'm misreading something) that fails right away.

[inducer]

I think I've found a way to reproduce the problem somewhat reliably. Run the examples build on #61 and then

bash # start a new shell so that the tmux window doesn't close when an error occurs
(export PYOPENCL_TEST=port:pthread; source .miniforge3/bin/activate testing ;  source ci-support.sh ; run_examples)

[inducer]

Interesting: Do the same thing multiple times, get different results:

diff --git a/examples/laplace-dirichlet-3d.py b/examples/laplace-dirichlet-3d.py
index 48a737a..865b1c3 100644
--- a/examples/laplace-dirichlet-3d.py
+++ b/examples/laplace-dirichlet-3d.py
@@ -130,9 +130,20 @@ def main(mesh_name="torus", visualize=False):
 
     bvp_rhs = bind(places, sqrt_w*sym.var("bc"))(actx, bc=bc)
 
+    spop = bound_op.scipy_op(actx, "sigma", dtype=np.float64)
+
+    from meshmode.dof_array import flat_norm
+    for i in range(10):
+        res = spop.matvec(bvp_rhs)
+        print(i, actx.np.linalg.norm(res))
+    1/0
+
+
+
+
     from pytential.solve import gmres
     gmres_result = gmres(
-            bound_op.scipy_op(actx, "sigma", dtype=np.float64),
+            spop,
             bvp_rhs, tol=1e-14, progress=True,
             stall_iterations=0,
             hard_failure=True)

gives

0 18271.189240367086
1 18266.885665550613
2 18284.471850638434
3 18289.173559412557
4 18289.142404135426
5 18284.471850638434
6 18284.471850638434
7 18306.65272297942
8 18306.65272297942
9 18290.04129860244

For context: this prints the norm of what should be one and the same vector.

[inducer]

This diff still yields different matvec results each time the FMM is run:

diff --git a/examples/laplace-dirichlet-3d.py b/examples/laplace-dirichlet-3d.py
index 48a737a..3148440 100644
--- a/examples/laplace-dirichlet-3d.py
+++ b/examples/laplace-dirichlet-3d.py
@@ -130,9 +130,20 @@ def main(mesh_name="torus", visualize=False):
 
     bvp_rhs = bind(places, sqrt_w*sym.var("bc"))(actx, bc=bc)
 
+    spop = bound_op.scipy_op(actx, "sigma", dtype=np.float64)
+
+    from meshmode.dof_array import flat_norm
+    for i in range(10):
+        res = spop.matvec(bvp_rhs)
+        print(i, actx.np.linalg.norm(bvp_rhs), actx.np.linalg.norm(res))
+    1/0
+
+
+
+
     from pytential.solve import gmres
     gmres_result = gmres(
-            bound_op.scipy_op(actx, "sigma", dtype=np.float64),
+            spop,
             bvp_rhs, tol=1e-14, progress=True,
             stall_iterations=0,
             hard_failure=True)
diff --git a/pytential/qbx/fmm.py b/pytential/qbx/fmm.py
index a58524b..4118a22 100644
--- a/pytential/qbx/fmm.py
+++ b/pytential/qbx/fmm.py
@@ -543,23 +543,23 @@ def drive_fmm(expansion_wrangler, src_weight_vecs, timing_data=None,
 
     recorder.add("translate_box_multipoles_to_qbx_local", timing_future)
 
-    qbx_expansions = qbx_expansions + local_result
+    qbx_expansions = local_result
 
     local_result, timing_future = (
             wrangler.translate_box_local_to_qbx_local(local_exps))
 
     recorder.add("translate_box_local_to_qbx_local", timing_future)
 
-    qbx_expansions = qbx_expansions + local_result
+    #qbx_expansions = local_result
 
     qbx_potentials, timing_future = wrangler.eval_qbx_expansions(qbx_expansions)
 
     recorder.add("eval_qbx_expansions", timing_future)
 
-    ts_result, timing_future = \
-        wrangler.eval_target_specific_qbx_locals(src_weight_vecs)
+    #ts_result, timing_future = \
+    #    wrangler.eval_target_specific_qbx_locals(src_weight_vecs)
 
-    qbx_potentials = qbx_potentials + ts_result
+    #qbx_potentials = qbx_potentials + ts_result
 
     recorder.add("eval_target_specific_qbx_locals", timing_future)

This suggests that m2qbxl may have a role to play in this. The results of the remainder of the FMM don't seem to "wobble".

[inducer]

It's the sumpy backend that has the issue. Both the single- and double-layer kernels exhibit the problem.