Skip to content

working local pytorch transform #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jperez999 merged 3 commits into from
Dec 5, 2022
Merged

working local pytorch transform #246

jperez999 merged 3 commits into from
Dec 5, 2022

Conversation

@jperez999
Copy link
Collaborator

@jperez999 jperez999 commented Dec 4, 2022

This PR adds the transform logic required to run an inference call on a trained pytorch model without triton in an ensemble graph.

@jperez999 jperez999 added enhancement New feature or request feature labels Dec 4, 2022
@jperez999 jperez999 self-assigned this Dec 4, 2022
@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Dec 4, 2022

Click to view CI Results
GitHub pull request #246 of commit 58c804b4b3c3ba5f582a50d9b83279836238262e, no merge conflicts.
Running as SYSTEM
Setting status of 58c804b4b3c3ba5f582a50d9b83279836238262e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/563/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/246/*:refs/remotes/origin/pr/246/* # timeout=10
 > git rev-parse 58c804b4b3c3ba5f582a50d9b83279836238262e^{commit} # timeout=10
Checking out Revision 58c804b4b3c3ba5f582a50d9b83279836238262e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 58c804b4b3c3ba5f582a50d9b83279836238262e # timeout=10
Commit message: "working local pytorch transform"
 > git rev-list --no-walk 5e0b0e6ae51e2163a6af6212335e3b9391d6200e # timeout=10
[workspace] $ /bin/bash /tmp/jenkins341391798509168076.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+25.g58c804b.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+25.g58c804b,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2848438397'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-b3v6ygwr
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-b3v6ygwr
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=8d4609f7607966999b7ef226f4182615856f138bb46fc364440be7eaf841cfa0
  Stored in directory: /tmp/pip-ephem-wheel-cache-0et3gxhn/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+25.g58c804b) (2.4.0)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+25.g58c804b) (0.9.0+14.g4f73ff5)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+25.g58c804b) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+25.g58c804b) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+25.g58c804b) (1.1.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+25.g58c804b) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+25.g58c804b) (1.8.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+25.g58c804b) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+25.g58c804b-py3-none-any.whl size=139981 sha256=19891c20dfdc49752d0a056d128904c5ea3accf1a096e8d7c0e776b4d26d0ae5
  Stored in directory: /tmp/pip-ephem-wheel-cache-8f8tiof6/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+25.g58c804b
    Uninstalling merlin-systems-0.7.0+25.g58c804b:
      Successfully uninstalled merlin-systems-0.7.0+25.g58c804b
Successfully installed merlin-systems-0.7.0+25.g58c804b
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 108 items / 1 error

==================================== ERRORS ====================================

_ ERROR collecting tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py __

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

E   ModuleNotFoundError: No module named 'ops.torch'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     74    39%

merlin/systems/dag/ensemble.py                            46     26    43%

merlin/systems/dag/node.py                                23     13    43%

merlin/systems/dag/op_runner.py                           26     19    27%

merlin/systems/dag/ops/init.py                        11     10     9%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     51    34%

merlin/systems/dag/ops/feast.py                          118     95    19%

merlin/systems/dag/ops/fil.py                            211    117    45%

merlin/systems/dag/ops/implicit.py                        72     48    33%

merlin/systems/dag/ops/operator.py                        79     37    53%

merlin/systems/dag/ops/pytorch.py                         43     26    40%

merlin/systems/dag/ops/session_filter.py                  45     28    38%

merlin/systems/dag/ops/softmax_sampling.py                51     34    33%

merlin/systems/dag/ops/tensorflow.py                      67     45    33%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        29     18    38%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     66    31%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      3    73%

merlin/systems/dag/runtimes/triton/ops/pytorch.py         62     37    40%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53     32    40%

merlin/systems/dag/runtimes/triton/ops/workflow.py        47     29    38%

merlin/systems/dag/runtimes/triton/runtime.py            157    106    32%

merlin/systems/model_registry.py                          16      8    50%

merlin/systems/triton/init.py                         49     34    31%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    240    10%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     54    25%

merlin/systems/workflow/init.py                       22     20     9%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2892   2073    28%


!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

======================== 38 warnings, 1 error in 4.29s =========================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins12283020270051982638.sh

jperez999
jperez999 commented Dec 4, 2022
View reviewed changes
merlin/systems/dag/ops/pytorch.py
def transform(self, col_selector: ColumnSelector, transformable: Transformable):
tensor_dict = {}
for column in transformable.columns:
tensor_dict[column] = torch.from_numpy(np.squeeze(transformable[column].values))
Copy link
Collaborator Author

@jperez999 jperez999 Dec 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be doing this in another place? Should dict array have APIs for transforming to tensors of specific frameworks?

Copy link
Contributor

@karlhigley karlhigley Dec 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should eventually (soon, even) but since we don't have those yet, this seems reasonable for now

jperez999
jperez999 commented Dec 4, 2022
View reviewed changes
merlin/systems/dag/ops/pytorch.py

result = self.model(tensor_dict)
output = {}
for idx, col in enumerate(self.output_schema.column_names):
Copy link
Collaborator Author

@jperez999 jperez999 Dec 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows for more than one output, but ordering will matter at this point. Is there a better way to go about handling a multi output where we can disregard order? Luckily dictionaries hold on to order...

Copy link
Contributor

@karlhigley karlhigley Dec 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, it seems like letting the model return a dictionary is okay, since we don't have the limitations that the Triton Torch backend imposes on us 🤔

karlhigley
karlhigley reviewed Dec 5, 2022
View reviewed changes
@github-actions
Copy link

github-actions bot commented Dec 5, 2022

Documentation preview

https://nvidia-merlin.github.io/systems/review/pr-246

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Dec 5, 2022

Click to view CI Results
GitHub pull request #246 of commit 3c5df6beb39c3c6683e45e797209612f5e1c547f, no merge conflicts.
Running as SYSTEM
Setting status of 3c5df6beb39c3c6683e45e797209612f5e1c547f to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/567/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/246/*:refs/remotes/origin/pr/246/* # timeout=10
 > git rev-parse 3c5df6beb39c3c6683e45e797209612f5e1c547f^{commit} # timeout=10
Checking out Revision 3c5df6beb39c3c6683e45e797209612f5e1c547f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3c5df6beb39c3c6683e45e797209612f5e1c547f # timeout=10
Commit message: "adding init files for test file recognition and removed config in torch test"
 > git rev-list --no-walk c717fd7ea4565ad9ee23cf092a43b0a7eab82b1f # timeout=10
[workspace] $ /bin/bash /tmp/jenkins3186216112477564273.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+26.g3c5df6b.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+26.g3c5df6b,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2997353750'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-dd5ot69v
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-dd5ot69v
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=bd86f49acd0ed69375c376aa078fb44cb36b013e3f4cc79e6e6ff6725a654f74
  Stored in directory: /tmp/pip-ephem-wheel-cache-l764r7nb/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+26.g3c5df6b) (2.4.0)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+26.g3c5df6b) (0.9.0+14.g4f73ff5)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+26.g3c5df6b) (1.1.1)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+26.g3c5df6b) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+26.g3c5df6b) (2.22.0)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+26.g3c5df6b) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+26.g3c5df6b) (1.8.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (21.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+26.g3c5df6b) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+26.g3c5df6b-py3-none-any.whl size=139979 sha256=5bec36315717745ce1dd4db1b5fa8f13dc6120de78677373711eb624cca3e363
  Stored in directory: /tmp/pip-ephem-wheel-cache-7hyfpb6x/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+26.g3c5df6b
    Uninstalling merlin-systems-0.7.0+26.g3c5df6b:
      Successfully uninstalled merlin-systems-0.7.0+26.g3c5df6b
Successfully installed merlin-systems-0.7.0+26.g3c5df6b
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 112 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [  9%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 11%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 15%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 16%]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py ....          [ 20%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 25%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 25%]

...                                                                      [ 28%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 30%]

..                                                                       [ 32%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 35%]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py ..........   [ 44%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_ensemble.py ..  [ 46%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_op.py ..        [ 48%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 49%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 54%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 55%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 58%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 82%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 84%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 92%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 96%]

tests/unit/systems/ops/tf/test_op.py ....                                [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-True-True]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime1-None-executor_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime1-None-executor_model-True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      1    98%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         43      0   100%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        29      4    86%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      1    91%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/pytorch.py         62      7    89%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/ops/workflow.py        47     15    68%

merlin/systems/dag/runtimes/triton/runtime.py            157     11    93%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2892   1226    58%


================ 112 passed, 75 warnings in 1967.31s (0:32:47) =================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins8573376523884868900.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Dec 5, 2022

Click to view CI Results
GitHub pull request #246 of commit 036ca69a3fe72c4c01f2926b3f56946ca274da81, no merge conflicts.
Running as SYSTEM
Setting status of 036ca69a3fe72c4c01f2926b3f56946ca274da81 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/568/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/246/*:refs/remotes/origin/pr/246/* # timeout=10
 > git rev-parse 036ca69a3fe72c4c01f2926b3f56946ca274da81^{commit} # timeout=10
Checking out Revision 036ca69a3fe72c4c01f2926b3f56946ca274da81 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 036ca69a3fe72c4c01f2926b3f56946ca274da81 # timeout=10
Commit message: "Merge branch 'main' into add-torch-local"
 > git rev-list --no-walk 3c5df6beb39c3c6683e45e797209612f5e1c547f # timeout=10
[workspace] $ /bin/bash /tmp/jenkins8332958746079252602.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+28.g036ca69.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+28.g036ca69,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1159285423'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-mni7v2vs
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-mni7v2vs
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=52963b27db9885677ab2d1f6dbb849e7e3e173848732da2246f20240d6d7fb88
  Stored in directory: /tmp/pip-ephem-wheel-cache-n0myb_jr/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (0.9.0+14.g4f73ff5)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+28.g036ca69) (2.22.0)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (1.1.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.20.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+28.g036ca69-py3-none-any.whl size=139976 sha256=b26c13e69094d2ef73223a912d6a07e98d0a9efb84a7a47b2ee45fdb2f7e8a2a
  Stored in directory: /tmp/pip-ephem-wheel-cache-fdgkp1wf/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+28.g036ca69
    Uninstalling merlin-systems-0.7.0+28.g036ca69:
      Successfully uninstalled merlin-systems-0.7.0+28.g036ca69
Successfully installed merlin-systems-0.7.0+28.g036ca69
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 112 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [  9%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 11%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 15%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 16%]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py ....          [ 20%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 25%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 25%]

...                                                                      [ 28%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 30%]

..                                                                       [ 32%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 35%]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py ..........   [ 44%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_ensemble.py ..  [ 46%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_op.py ..        [ 48%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 49%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 54%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 55%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 58%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 82%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 84%]

tests/unit/systems/ops/implicit/test_op.py ......F..                     [ 92%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 96%]

tests/unit/systems/ops/tf/test_op.py ....                                [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-runtime1] ________________


model_cls = <function AlternatingLeastSquares at 0x7f5ab9b720d0>

runtime = <merlin.systems.dag.runtimes.triton.runtime.TritonExecutorRuntime object at 0x7f59b02792b0>

tmpdir = local('/tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("runtime", [None, TritonExecutorRuntime()])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, runtime, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema)
    ensemble_config, _ = triton_ens.export(tmpdir, runtime=runtime)

    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/implicit/test_op.py:121:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
        "--cuda-memory-pool-byte-size=0:536870912",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:82: RuntimeError

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2401.87it/s]

I1205 15:04:47.114543 12876 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7efffe000000' with size 268435456

I1205 15:04:47.115276 12876 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1205 15:04:47.118927 12876 model_lifecycle.cc:459] loading: executor_model:1

1205 15:04:50.019078 12915 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1205 15:04:50.046321 12876 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


I1205 15:04:50.046589 12876 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1205 15:04:50.046746 12876 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1205 15:04:50.047002 12876 server.cc:631]

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model          | Version | Status                                                                                                                                                   |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| executor_model | 1       | UNAVAILABLE: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error                                                              |

|                |         |                                                                                                                                                          |

|                |         | At:                                                                                                                                                      |

|                |         |   /usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount                                                                           |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup                                                                     |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/init.py(5):                                                                                    |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18):  |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18):      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (961): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34):                                                 |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+


I1205 15:04:50.111883 12876 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1205 15:04:50.112748 12876 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-0/test_ensemble_AlternatingLeast1                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 536870912                                                                                                                                                                                    |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1205 15:04:50.112785 12876 server.cc:262] Waiting for in-flight requests to complete.

I1205 15:04:50.112792 12876 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1205 15:04:50.112799 12876 server.cc:293] All models are stopped, unloading models

I1205 15:04:50.112805 12876 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1205 15:04:51.134360 12876 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-True-True]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime1-None-executor_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime1-None-executor_model-True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      1    98%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         43      0   100%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        29      4    86%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      1    91%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/pytorch.py         62      7    89%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/ops/workflow.py        47     15    68%

merlin/systems/dag/runtimes/triton/runtime.py            157     11    93%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2892   1224    58%


=========== 1 failed, 111 passed, 75 warnings in 1869.14s (0:31:09) ============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins5001465356889326523.sh

@jperez999
Copy link
Collaborator Author

jperez999 commented Dec 5, 2022

rerun tests

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Dec 5, 2022

Click to view CI Results
GitHub pull request #246 of commit 036ca69a3fe72c4c01f2926b3f56946ca274da81, no merge conflicts.
GitHub pull request #246 of commit 036ca69a3fe72c4c01f2926b3f56946ca274da81, no merge conflicts.
Running as SYSTEM
Setting status of 036ca69a3fe72c4c01f2926b3f56946ca274da81 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/569/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/246/*:refs/remotes/origin/pr/246/* # timeout=10
 > git rev-parse 036ca69a3fe72c4c01f2926b3f56946ca274da81^{commit} # timeout=10
Checking out Revision 036ca69a3fe72c4c01f2926b3f56946ca274da81 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 036ca69a3fe72c4c01f2926b3f56946ca274da81 # timeout=10
Commit message: "Merge branch 'main' into add-torch-local"
 > git rev-list --no-walk 036ca69a3fe72c4c01f2926b3f56946ca274da81 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins17244249769465500975.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+28.g036ca69.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+28.g036ca69,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3249757706'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ti0zcqkd
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ti0zcqkd
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=a6013e73fc7706631e6a41bf2b27f5b786e58225ca8cb93a3d1f70caf576df44
  Stored in directory: /tmp/pip-ephem-wheel-cache-_qkykwwo/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (0.9.0+14.g4f73ff5)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (1.1.1)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+28.g036ca69) (2.22.0)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.20.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+28.g036ca69-py3-none-any.whl size=139976 sha256=91c9826955f5d831e5d67c2c427b529e807c14dc74319bf0084eafe41dc65f5b
  Stored in directory: /tmp/pip-ephem-wheel-cache-ws3y998d/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+28.g036ca69
    Uninstalling merlin-systems-0.7.0+28.g036ca69:
      Successfully uninstalled merlin-systems-0.7.0+28.g036ca69
Successfully installed merlin-systems-0.7.0+28.g036ca69
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 112 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [  9%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 11%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 15%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 16%]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py ....          [ 20%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 25%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 25%]

...                                                                      [ 28%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 30%]

..                                                                       [ 32%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 35%]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py ..........   [ 44%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_ensemble.py .F  [ 46%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_op.py ..        [ 48%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 49%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 54%]

tests/unit/systems/ops/fil/test_ensemble.py F                            [ 55%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 58%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 82%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 84%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 92%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 96%]

tests/unit/systems/ops/tf/test_op.py ....                                [100%]


=================================== FAILURES ===================================

____ test_workflow_op_serving_triton[runtime1-None-executor_model-parquet] _____


tmpdir = local('/tmp/pytest-of-jenkins/pytest-1/test_workflow_op_serving_trito1')

dataset = <merlin.io.dataset.Dataset object at 0x7f1830062700>

engine = 'parquet'

runtime = <merlin.systems.dag.runtimes.triton.runtime.TritonExecutorRuntime object at 0x7f18ba2756d0>

model_name = None, expected_model_name = 'executor_model'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize(
    ["runtime", "model_name", "expected_model_name"],
    [
        (TritonEnsembleRuntime(), None, "ensemble_model"),
        (TritonExecutorRuntime(), None, "executor_model"),
    ],
)
def test_workflow_op_serving_triton(
    tmpdir, dataset, engine, runtime, model_name, expected_model_name
):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir, runtime=runtime, name=model_name)

    assert ens_config.name == expected_model_name

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/dag/runtimes/triton/ops/workflow/test_ensemble.py:84:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-1/test_workflow_op_serving_trito1')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
        "--cuda-memory-pool-byte-size=0:536870912",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=-11)


merlin/systems/triton/utils.py:82: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1205 15:42:52.244977 24425 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f71a6000000' with size 268435456

I1205 15:42:52.245742 24425 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1205 15:42:52.250221 24425 model_lifecycle.cc:459] loading: 0_transformworkflowtriton:1

I1205 15:42:52.250267 24425 model_lifecycle.cc:459] loading: executor_model:1

--------------------------- Captured stderr teardown ---------------------------

1205 15:43:00.208804 24464 pb_stub.cc:1016] Non-graceful termination detected.

_____________________ test_faiss_in_triton_executor_model ______________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())





  response = run_ensemble_on_tritonserver(





        tmpdir,
        ensemble.input_schema,
        make_df(request_data._columns),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )



tests/unit/systems/ops/faiss/test_executor.py:81:




merlin/systems/triton/utils.py:139: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
        "--cuda-memory-pool-byte-size=0:536870912",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode
                            raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")

                        try:
                            ready = client.is_server_ready()
                        except tritonclient.utils.InferenceServerException:
                            ready = False

                        if ready:
                            yield client
                            return

                        time.sleep(1)
                    LOG.error("Failed to start tritonserver in %s seconds", seconds)





              raise RuntimeError("Timed out waiting for tritonserver to become ready")





E                   RuntimeError: Timed out waiting for tritonserver to become ready


merlin/systems/triton/utils.py:96: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1205 15:43:01.908983 24618 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fc72e000000' with size 268435456

I1205 15:43:01.909760 24618 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1205 15:43:01.914033 24618 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1205 15:43:01.914073 24618 model_lifecycle.cc:459] loading: executor_model:1

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 60 seconds

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 120 seconds

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 300 seconds

_____________________ test_workflow_with_forest_inference ______________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-1/test_workflow_with_forest_infe0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_workflow_with_forest_inference(tmpdir):
    rows = 200
    num_features = 16
    X, y = sklearn.datasets.make_regression(
        n_samples=rows,
        n_features=num_features,
        n_informative=num_features // 3,
        random_state=0,
    )
    feature_names = [str(i) for i in range(num_features)]
    df = pd.DataFrame(X, columns=feature_names, dtype=np.float32)
    dataset = Dataset(df)

    # Fit GBDT Model
    model = xgboost.XGBRegressor()
    model.fit(X, y)

    input_column_schemas = [ColumnSchema(col, dtype=np.float32) for col in feature_names]
    input_schema = Schema(input_column_schemas)
    selector = ColumnSelector(feature_names)

    workflow_ops = feature_names >> wf_ops.LogOp()
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    triton_chain = selector >> TransformWorkflow(workflow) >> PredictForest(model, input_schema)

    triton_ens = Ensemble(triton_chain, input_schema)

    request_df = df[:5]
    ensemble_config, _ = triton_ens.export(tmpdir)





  response = run_ensemble_on_tritonserver(





        str(tmpdir), input_schema, request_df, ["output__0"], ensemble_config.name
    )



tests/unit/systems/ops/fil/test_ensemble.py:77:




merlin/systems/triton/utils.py:139: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = '/tmp/pytest-of-jenkins/pytest-1/test_workflow_with_forest_infe0'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
        "--cuda-memory-pool-byte-size=0:536870912",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode
                            raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")

                        try:
                            ready = client.is_server_ready()
                        except tritonclient.utils.InferenceServerException:
                            ready = False

                        if ready:
                            yield client
                            return

                        time.sleep(1)
                    LOG.error("Failed to start tritonserver in %s seconds", seconds)





              raise RuntimeError("Timed out waiting for tritonserver to become ready")





E                   RuntimeError: Timed out waiting for tritonserver to become ready


merlin/systems/triton/utils.py:96: RuntimeError

----------------------------- Captured stderr call -----------------------------

1205 15:51:02.770353 24657 pb_stub.cc:1016] Non-graceful termination detected.

I1205 15:51:08.671993 25173 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fc154000000' with size 268435456

I1205 15:51:08.672723 25173 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1205 15:51:08.678013 25173 model_lifecycle.cc:459] loading: 0_transformworkflowtriton:1

I1205 15:51:08.678072 25173 model_lifecycle.cc:459] loading: 1_filtriton:1

I1205 15:51:08.678105 25173 model_lifecycle.cc:459] loading: 1_predictforesttriton:1

I1205 15:51:08.695696 25173 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1205 15:51:08.695722 25173 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1205 15:51:08.695728 25173 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1205 15:51:08.696282 25173 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_filtriton (version 1)

I1205 15:51:08.701011 25173 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_filtriton_0 (GPU device 0)

I1205 15:51:08.753143 25173 model_lifecycle.cc:693] successfully loaded '1_filtriton' version 1

------------------------------ Captured log call -------------------------------

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 60 seconds

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 120 seconds

ERROR    merlin-systems:utils.py:94 Failed to start tritonserver in 300 seconds

--------------------------- Captured stderr teardown ---------------------------

1205 15:59:09.292545 25228 pb_stub.cc:1016] Non-graceful termination detected.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-True-True]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime1-None-executor_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime1-None-executor_model-True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      1    98%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         43      0   100%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        29      4    86%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      1    91%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/pytorch.py         62      7    89%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/ops/workflow.py        47     15    68%

merlin/systems/dag/runtimes/triton/runtime.py            157     11    93%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     10    86%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2892   1222    58%


=========== 3 failed, 109 passed, 75 warnings in 2800.87s (0:46:40) ============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins6484647139832591109.sh

@karlhigley
Copy link
Contributor

karlhigley commented Dec 5, 2022

rerun tests

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Dec 5, 2022

Click to view CI Results
GitHub pull request #246 of commit 036ca69a3fe72c4c01f2926b3f56946ca274da81, no merge conflicts.
GitHub pull request #246 of commit 036ca69a3fe72c4c01f2926b3f56946ca274da81, no merge conflicts.
Running as SYSTEM
Setting status of 036ca69a3fe72c4c01f2926b3f56946ca274da81 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/570/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/246/*:refs/remotes/origin/pr/246/* # timeout=10
 > git rev-parse 036ca69a3fe72c4c01f2926b3f56946ca274da81^{commit} # timeout=10
Checking out Revision 036ca69a3fe72c4c01f2926b3f56946ca274da81 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 036ca69a3fe72c4c01f2926b3f56946ca274da81 # timeout=10
Commit message: "Merge branch 'main' into add-torch-local"
 > git rev-list --no-walk 036ca69a3fe72c4c01f2926b3f56946ca274da81 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins13905031480826170522.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+28.g036ca69.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+28.g036ca69,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2196407829'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-3wws1icz
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-3wws1icz
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=90ca8914178aaa254502ebcabe8dc43186da8b3747c34169841b1cf940da1cb7
  Stored in directory: /tmp/pip-ephem-wheel-cache-6bpxm32f/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (0.9.0+14.g4f73ff5)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+28.g036ca69) (2.22.0)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+28.g036ca69) (1.1.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+28.g036ca69) (1.8.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+28.g036ca69) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+28.g036ca69-py3-none-any.whl size=139976 sha256=a03e42ed5a7b8d7765bafd4bde27cefe78c638bca48c31d33a7979181a21bd24
  Stored in directory: /tmp/pip-ephem-wheel-cache-sl7rcmpi/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+28.g036ca69
    Uninstalling merlin-systems-0.7.0+28.g036ca69:
      Successfully uninstalled merlin-systems-0.7.0+28.g036ca69
Successfully installed merlin-systems-0.7.0+28.g036ca69
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 112 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [  9%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 11%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 15%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 16%]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py ....          [ 20%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 25%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 25%]

...                                                                      [ 28%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 30%]

..                                                                       [ 32%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 35%]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py ..........   [ 44%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_ensemble.py ..  [ 46%]

tests/unit/systems/dag/runtimes/triton/ops/workflow/test_op.py ..        [ 48%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 49%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 54%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 55%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 58%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 82%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 84%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 92%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 96%]

tests/unit/systems/ops/tf/test_op.py ....                                [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-True-True]

tests/unit/systems/dag/runtimes/local/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving[runtime1-None-executor_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime0-None-ensemble_model-True-True]

tests/unit/systems/dag/runtimes/triton/ops/torch/test_op.py::test_pytorch_op_serving_python[runtime1-None-executor_model-True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      1    98%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         43      0   100%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        29      4    86%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      1    91%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/pytorch.py         62      7    89%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/ops/workflow.py        47     15    68%

merlin/systems/dag/runtimes/triton/runtime.py            157     11    93%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2892   1226    58%


================ 112 passed, 75 warnings in 1986.34s (0:33:06) =================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins14358368164636186621.sh

@jperez999 jperez999 merged commit a422d24 into NVIDIA-Merlin:main Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@karlhigley karlhigley karlhigley approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@jperez999 jperez999

Labels

enhancement New feature or request feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jperez999 @nvidia-merlin-bot @karlhigley