Skip to content

Add in K8s jobs for some common data cleaning commands#2785

Merged
jayoshih merged 7 commits intolearningequality:hotfixesfrom
aronasorman:data-cleaning-commands
Jan 14, 2021
Merged

Add in K8s jobs for some common data cleaning commands#2785
jayoshih merged 7 commits intolearningequality:hotfixesfrom
aronasorman:data-cleaning-commands

Conversation

@aronasorman
Copy link
Copy Markdown
Collaborator

@aronasorman aronasorman commented Jan 14, 2021

Add in both set_storage_used and mark_incomplete as cronjobs.

set_storage_used has been set to run every midnight.

mark_incomplete has been set to run every 2 years. This cronjob is actually just a hack so we can spawn this job whenever we need. Creating this as a cronjob/job rather than a one-off task we run in bash means that this can be retried by k8s when it fails, rather than needing manual restart intervention.

@aronasorman aronasorman changed the base branch from develop to hotfixes January 14, 2021 20:40
@aronasorman aronasorman requested a review from jayoshih January 14, 2021 20:42
@codecov
Copy link
Copy Markdown

codecov bot commented Jan 14, 2021

Codecov Report

Merging #2785 (0310da5) into hotfixes (28587f8) will increase coverage by 4.04%.
The diff coverage is 88.73%.

Impacted file tree graph

@@             Coverage Diff              @@
##           hotfixes    #2785      +/-   ##
============================================
+ Coverage     80.80%   84.84%   +4.04%     
============================================
  Files           281      291      +10     
  Lines         12659    15263    +2604     
============================================
+ Hits          10229    12950    +2721     
+ Misses         2430     2313     -117     
Impacted Files Coverage Δ
contentcuration/contentcuration/decorators.py 56.60% <50.00%> (-37.60%) ⬇️
contentcuration/contentcuration/forms.py 69.06% <72.41%> (+20.61%) ⬆️
...ntentcuration/contentcuration/db/models/manager.py 93.49% <93.33%> (-6.51%) ⬇️
contentcuration/contentcuration/api.py 92.06% <100.00%> (+1.43%) ⬆️
...tentcuration/contentcuration/context_processors.py 100.00% <100.00%> (ø)
...tcuration/contentcuration/db/models/expressions.py 93.33% <100.00%> (-6.67%) ⬇️
...entcuration/contentcuration/db/models/functions.py 100.00% <100.00%> (ø)
...ation/contentcuration/tests/test_rest_framework.py 36.93% <0.00%> (-63.07%) ⬇️
contentcuration/contentcuration/utils/format.py 21.05% <0.00%> (-49.54%) ⬇️
contentcuration/contentcuration/utils/cache.py 31.70% <0.00%> (-31.93%) ⬇️
... and 172 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4da2b89...0310da5. Read the comment docs.

@jayoshih jayoshih merged commit 60a27b5 into learningequality:hotfixes Jan 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants