Expand range of sites used for testing

[hategan]

The tentative goal is that each of us bootstrap tests on a reasonable set of machines we have access to.

[jameshcorbett]

LLNL machines uniformly give me a self-signed certificate error:

(37B) [corbett8@lassen709:psi-j-python]$ make tests -- --upload-results
PYTHONPATH=/usr/WS2/corbett8/ExaWorks/psi-j-python/src:/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins1:/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins2: \
	python -m pytest -v --upload-results
2022-01-25 15:02:56 DEBUG Attempting to acquire lock 35184439441728 on /g/g12/corbett8/.psij/.key.lock
2022-01-25 15:02:56 DEBUG Lock 35184439441728 acquired on /g/g12/corbett8/.psij/.key.lock
2022-01-25 15:02:56 DEBUG Attempting to release lock 35184439441728 on /g/g12/corbett8/.psij/.key.lock
2022-01-25 15:02:56 DEBUG Lock 35184439441728 released on /g/g12/corbett8/.psij/.key.lock
2022-01-25 15:02:56 INFO Ignoring duplicate entry in sys.path: /collab/usr/gapps/python/build/spack-coralea.3/opt/spack/linux-rhel7-ppc64le/gcc-4.9.3/python-3.7.2-yx4gc64m743jidvhr5i3kklxgnm74mzk/lib/python3.7/site-packages/six-1.12.0-py3.7.egg
2022-01-25 15:02:56 DEBUG Attempting to load aprun_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:56 DEBUG Found module "aprun_descriptor" with classes [_Descriptor(name=aprun, path=None)]
2022-01-25 15:02:56 DEBUG Registering _Descriptor(name=aprun, path=None)
2022-01-25 15:02:56 DEBUG Attempting to load cobalt_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:56 DEBUG Found module "cobalt_descriptor" with classes [_Descriptor(name=cobalt, path=None)]
2022-01-25 15:02:56 DEBUG Registering _Descriptor(name=cobalt, path=None)
2022-01-25 15:02:57 DEBUG Attempting to load core_descriptors from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:57 DEBUG Found module "core_descriptors" with classes [_Descriptor(name=local, path=None), _Descriptor(name=batch-test, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=local, path=None)
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=batch-test, path=None)
2022-01-25 15:02:57 DEBUG Found module "core_descriptors" with classes [_Descriptor(name=single, path=None), _Descriptor(name=multiple, path=None), _Descriptor(name=mpirun, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=single, path=None)
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=multiple, path=None)
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=mpirun, path=None)
2022-01-25 15:02:57 DEBUG Attempting to load flux_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:57 DEBUG Found module "flux_descriptor" with classes [_Descriptor(name=flux, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=flux, path=None)
2022-01-25 15:02:57 DEBUG Attempting to load jsrun_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:57 DEBUG Found module "jsrun_descriptor" with classes [_Descriptor(name=jrun, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=jrun, path=None)
2022-01-25 15:02:57 DEBUG Attempting to load lsf_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:57 DEBUG Found module "lsf_descriptor" with classes [_Descriptor(name=lsf, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=lsf, path=None)
2022-01-25 15:02:57 DEBUG Attempting to load rp_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:02:57 DEBUG Found module "rp_descriptor" with classes [_Descriptor(name=rp, path=None)]
2022-01-25 15:02:57 DEBUG Registering _Descriptor(name=rp, path=None)
/usr/WS2/corbett8/37B/lib/python3.7/site-packages/radical/utils/atfork/stdlib_fixer.py:77: UserWarning: logging handlers already registered.
  warnings.warn('logging handlers already registered.')
2022-01-25 15:03:01 DEBUG Attempting to load saga_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:03:01 DEBUG Found module "saga_descriptor" with classes [_Descriptor(name=saga, path=None)]
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=saga, path=None)
2022-01-25 15:03:01 DEBUG Attempting to load slurm_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:03:01 DEBUG Found module "slurm_descriptor" with classes [_Descriptor(name=slurm, path=None)]
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=slurm, path=None)
2022-01-25 15:03:01 DEBUG Attempting to load srun_descriptor from /usr/WS2/corbett8/ExaWorks/psi-j-python/src
2022-01-25 15:03:01 DEBUG Found module "srun_descriptor" with classes [_Descriptor(name=srun, path=None)]
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=srun, path=None)
2022-01-25 15:03:01 DEBUG Attempting to load descriptors from /usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins1
2022-01-25 15:03:01 DEBUG Found module "descriptors" with classes [_Descriptor(name=p1-tp1, path=None), _Descriptor(name=p2-tp1, path=None), _Descriptor(name=p2-tp3, path=None), _Descriptor(name=_always_loads, path=None), _Descriptor(name=_never_loads, path=None)]
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=p1-tp1, path=None)
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=p2-tp1, path=None)
2022-01-25 15:03:01 INFO [Errno 2] No such file or directory: '/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins1/_test_plugins1/ex2.py'
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=p2-tp3, path=None)
2022-01-25 15:03:01 INFO [Errno 2] No such file or directory: '/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins1/_test_plugins3/ex3.py'
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=_always_loads, path=None)
2022-01-25 15:03:01 DEBUG Registering _Descriptor(name=_never_loads, path=None)
2022-01-25 15:03:01 INFO No module named 'does_not_exist'
2022-01-25 15:03:01 DEBUG Attempting to load dummy from /usr/WS2/corbett8/ExaWorks/psi-j-python/tests/plugins2
2022-01-25 15:03:01 INFO Ignoring duplicate entry in sys.path: /usr/WS2/corbett8/ExaWorks/psi-j-python
/usr/WS2/corbett8/37B/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_wrapper.py:11: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/WS2/corbett8/37B/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_wrapper.sh' mode='r' encoding='UTF-8'>
  _WRAPPER_SCRIPT = open (os.path.dirname(__file__) + '/shell_wrapper.sh').read ()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2022-01-25 15:03:05 DEBUG Output from ['ssh', '-oBatchMode=yes', '-oStrictHostKeyChecking=no', 'localhost', 'true']:          **WARNING**WARNING**WARNING**WARNING**WARNING** 
 This is a Department of Energy (DOE) computer system. DOE
 computer systems are provided for the processing of official
 U.S. Government information only. All data contained within
 DOE computer systems is owned by the DOE, and may be audited,
 intercepted, recorded, read, copied, or captured in any
 manner and disclosed in any manner, by authorized personnel.
 THERE IS NO RIGHT OF PRIVACY IN THIS SYSTEM. System personnel
 may disclose any potential evidence of crime found on DOE
 computer systems to appropriate authorities.
  
 USE OF THIS SYSTEM BY ANY USER, AUTHORIZED OR UNAUTHORIZED,
 CONSTITUTES CONSENT TO THIS AUDITING, INTERCEPTION, RECORDING,
 READING, COPYING, CAPTURING, AND DISCLOSURE OF COMPUTER ACTIVITY.

        **WARNING**WARNING**WARNING**WARNING**WARNING**

Permission denied (gssapi-keyex,gssapi-with-mic,password,keyboard-interactive).

2022-01-25 15:03:05 DEBUG Starting new HTTPS connection (1): testing.exaworks.org:443
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
INTERNALERROR>     chunked=chunked)
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
INTERNALERROR>     self._validate_conn(conn)
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
INTERNALERROR>     conn.connect()
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
INTERNALERROR>     ssl_context=context)
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 345, in ssl_wrap_socket
INTERNALERROR>     return context.wrap_socket(sock, server_hostname=server_hostname)
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 412, in wrap_socket
INTERNALERROR>     session=session
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 853, in _create
INTERNALERROR>     self.do_handshake()
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 1117, in do_handshake
INTERNALERROR>     self._sslobj.do_handshake()
INTERNALERROR> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
INTERNALERROR>     timeout=timeout
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
INTERNALERROR>     _stacktrace=sys.exc_info()[2])
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
INTERNALERROR>     raise MaxRetryError(_pool, url, error or ResponseError(cause))
INTERNALERROR> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='testing.exaworks.org', port=443): Max retries exceeded with url: /result (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/main.py", line 265, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/config/__init__.py", line 982, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 205, in pytest_configure
INTERNALERROR>     _save_or_upload(config, data)
INTERNALERROR>   File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 467, in _save_or_upload
INTERNALERROR>     _upload_report(config, data)
INTERNALERROR>   File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 483, in _upload_report
INTERNALERROR>     'key': config.option.key, 'data': data})
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/api.py", line 117, in post
INTERNALERROR>     return request('post', url, data=data, json=json, **kwargs)
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/api.py", line 61, in request
INTERNALERROR>     return session.request(method=method, url=url, **kwargs)
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
INTERNALERROR>     resp = self.send(prep, **send_kwargs)
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
INTERNALERROR>     r = adapter.send(request, **kwargs)
INTERNALERROR>   File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
INTERNALERROR>     raise SSLError(e, request=request)
INTERNALERROR> requests.exceptions.SSLError: HTTPSConnectionPool(host='testing.exaworks.org', port=443): Max retries exceeded with url: /result (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))
2022-01-25 15:03:06 DEBUG Starting new HTTPS connection (1): testing.exaworks.org:443
Traceback (most recent call last):
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 345, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='testing.exaworks.org', port=443): Max retries exceeded with url: /result (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163, in main
    config=config
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/main.py", line 311, in wrap_session
    config._ensure_unconfigure()
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/_pytest/config/__init__.py", line 987, in _ensure_unconfigure
    self.hook.pytest_unconfigure(config=self)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/WS2/corbett8/37B/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 230, in pytest_unconfigure
    _save_or_upload(config, data)
  File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 467, in _save_or_upload
    _upload_report(config, data)
  File "/usr/WS2/corbett8/ExaWorks/psi-j-python/tests/conftest.py", line 483, in _upload_report
    'key': config.option.key, 'data': data})
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='testing.exaworks.org', port=443): Max retries exceeded with url: /result (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))
make: *** [Makefile:14: tests] Error 1
(37B) [corbett8@lassen709:psi-j-python]$ 

[hategan]

@jameshcorbett and I spoke about this a bit earlier and, it appears that something in James' environment causes python/requests to not recognize letsencrypt.org as a CA.

[jameshcorbett]

Got it working thanks to help from some LC people. The fix is to export REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem. So we now have our first LLNL results.

I have the same problem with my laptop on LLNL VPN but the solution there is to not use VPN.

[hategan]

Thanks @jameshcorbett!

I wonder why the default doesn't work and whether it's reasonable to assume that it should. I would not personally have expected a widely used CA to not be seen by default, but things clearly don't work according to my expectations.

[jameshcorbett]

The explanation I got was:

Most of the time, but certainly not always, this is due to the Lab's MITM web proxy replacing the SSL certificate with its own, but the client not knowing about the Lab's CA certificates. What application are you using to attempt to upload the results? It is possible to exempt certain web sites from the MITM SSL replacement, which might end up being needed here...

[hategan]

!#$!@#

You know, my first suspicion was that there was a lab proxy replacing the page, and, when I asked you to check the certificate with curl and it returned a valid certificate, I didn't check the actual cert, so I dismissed that theory.

My gut wants to say a few things about "MITM SSL replacement", but I worry I have nothing nice to say about it, so I'll abstain.

[hategan]

Moving to 1.0.0