Improved API key handling (#3094)

* Allow users to reset their API key

Fix #1027

* Only return API key once after it has been generated

* Add new UI to reset and display API key

How API keys are reset and displayed has changed since the initial version of API keys: Users will be able to view an API key exactly once after it has been created/reset. This requires a slightly different user interface. We’re also planning a few more changes to API keys in the future, and these UI changes prepare for that.

* Refactor existing settings screen

The existing settings UI was a little cluttered and unstructured. We’re going to add new settings in this PR and in follow-up PRs, so I took the time to clean up the UI (both visually and implementation-wise).

* Ensure that toasts are always visible, even when scrolling

This is a hacky workaround, but a proper fix would require quite some refactoring. Considering that this hack is pretty isolated and not going to affect any other parts of the UI and that we will need to upgrade to Blueprint 5 at some point anyway, I’ve opted for the quick-and-dirty solution for now.

* Do not display password setting when password auth is disabled

* Use session tokens for authentication in API tests

In the future, roles won’t have an API key by default anymore. As an alternative, we generate session tokens explicitly.

* Do not generate API tokens for new roles

Most users do not need API access so there’s no reason to generate an API key for them by default.

* Handle users without an API key properly in the settings UI

Previously, an API was generate automatically for new users, i.e. every user had an API key. This has now changed, and the settings UI needs to properly handle situations where a user doesn’t yet have an API key.

As this increases the complexity of the UI state, I’ve refactored the component to make use of a local reducer.

* Update wording to clarify that API keys are secrets

* Rename "reset_api_key" to "generate_api_key"

This method is now also used to generate an initial key for users who do not yet have an API key.

* Send email notification when API key is (re-)generated

* Extract logic to regenerate API keys into separate module

While the logic initially was quite simply, there will be more business logic related to API keys, e.g. sending notifications ahead of and when an API key has expired.

* Let API keys expire after 90 days

* Extract `generate_api_key` method from role model

Initially, I added this to the role model as the model to be consistent with the model's `set_password` method. However, as the logic to generate an API token has become more complex, it is clear that it shouldn't live in the model.

* Send notification when API keys are about to expire/have expired

* Display API key expiration date in UI

* Add CLI command to reset API key expiration of non-expiring API keys

* Replace use of deprecated `utcnow` method

https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

* Remove unnecessary keys from API JSON response

Aleph represents both users and groups using the role model. However, some API keys (such as `has_password` or `has_api_key` are not relevant for groups).

* Add note to remind us to remove/update logic handling legacy API keys

* Send API key expiration notifications on a daily basis

* Fix Alembic migration order

We merged a different migration in the meantime (8adf50) and as a result Alembic wasn’t able to figure out how to upgrade the database unambiguously.

* Reauthenticate user in test to update session cache

452e46 changes the way we handle authentication in tests. Previously, we used API keys to authenticate users. Now, as we don’t generate API keys for users by default anymore, we use session tokens (the same mechanism the UI uses as well). In general, authentication using session tokens and API keys is quite similar: In both cases, the token is passed as a request header.

However, there’s one significant difference between session tokens and API keys: When using API keys, data about the user (including group memberships) is loaded from the database on every request. When using session tokens, this data is cached in Redis for the lifetime of the session. For end users, this means they have to sign out and in again after their group memberships have been updated.

When I originally implemented this change in our tests, it didn’t affect any of the tests as none of them did rely on updating group memberships. However, I’ve since implemented and merged additional tests in #3865, including one test that covers updating the user’s group memberships after the initial sign in. When I rebased this branch to include these new tests, this particular test failed.

The solution for this is the same as for end users: In the test case, we need to reauthenticate the test user after updating the group memberships.

* Update wording of API key notification emails based on feedback

* Use strict equality check

* Clarify that API keys expire when generating a new key

* Display different UI messages in case the API key has expired

* Fix Alembic migration order

We merged other migrations in the meantime and as a result Alembic wasn’t able to figure out how to upgrade the database unambiguously.

Author: tillprochaska

aleph/logic/api_keys.py

@@ -0,0 +1,113 @@
+import datetime
+
+import structlog
+from flask import render_template
+from sqlalchemy import and_, or_, func
+
+from aleph.core import db
+from aleph.model import Role
+from aleph.model.common import make_token
+from aleph.logic.mail import email_role
+from aleph.logic.roles import update_role
+from aleph.logic.util import ui_url
+
+# Number of days after which API keys expire
+API_KEY_EXPIRATION_DAYS = 90
+
+# Number of days before an API key expires
+API_KEY_EXPIRES_SOON_DAYS = 7
+
+log = structlog.get_logger(__name__)
+
+
+def generate_user_api_key(role):
+    event = "regenerated" if role.has_api_key else "generated"
+    params = {"role": role, "event": event}
+    plain = render_template("email/api_key_generated.txt", **params)
+    html = render_template("email/api_key_generated.html", **params)
+    subject = f"API key {event}"
+    email_role(role, subject, html=html, plain=plain)
+
+    now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
+    role.api_key = make_token()
+    role.api_key_expires_at = now + datetime.timedelta(days=API_KEY_EXPIRATION_DAYS)
+    role.api_key_expiration_notification_sent = None
+
+    db.session.add(role)
+    db.session.commit()
+    update_role(role)
+
+    return role.api_key
+
+
+def send_api_key_expiration_notifications():
+    _send_api_key_expiration_notification(
+        days=7,
+        subject="Your API key will expire in 7 days",
+        plain_template="email/api_key_expires_soon.txt",
+        html_template="email/api_key_expires_soon.html",
+    )
+
+    _send_api_key_expiration_notification(
+        days=0,
+        subject="Your API key has expired",
+        plain_template="email/api_key_expired.txt",
+        html_template="email/api_key_expired.html",
+    )
+
+
+def _send_api_key_expiration_notification(
+    days,
+    subject,
+    plain_template,
+    html_template,
+):
+    now = datetime.date.today()
+    threshold = now + datetime.timedelta(days=days)
+
+    query = Role.all_users()
+    query = query.yield_per(1000)
+    query = query.where(
+        and_(
+            and_(
+                Role.api_key != None,  # noqa: E711
+                func.date(Role.api_key_expires_at) <= threshold,
+            ),
+            or_(
+                Role.api_key_expiration_notification_sent == None,  # noqa: E711
+                Role.api_key_expiration_notification_sent > days,
+            ),
+        )
+    )
+
+    for role in query:
+        expires_at = role.api_key_expires_at
+        params = {
+            "role": role,
+            "expires_at": expires_at,
+            "settings_url": ui_url("settings"),
+        }
+        plain = render_template(plain_template, **params)
+        html = render_template(html_template, **params)
+        log.info(f"Sending API key expiration notification: {role} at {expires_at}")
+        email_role(role, subject, html=html, plain=plain)
+
+    query.update({Role.api_key_expiration_notification_sent: days})
+    db.session.commit()
+
+
+def reset_api_key_expiration():
+    now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
+    expires_at = now + datetime.timedelta(days=API_KEY_EXPIRATION_DAYS)
+
+    query = Role.all_users()
+    query = query.yield_per(500)
+    query = query.where(
+        and_(
+            Role.api_key != None,  # noqa: E711
+            Role.api_key_expires_at == None,  # noqa: E711
+        )
+    )
+
+    query.update({Role.api_key_expires_at: expires_at})
+    db.session.commit()

aleph/manage.py

@@ -20,6 +20,7 @@
 from aleph.queues import get_status, cancel_queue
 from aleph.queues import get_active_dataset_status
 from aleph.index.admin import delete_index
+from aleph.logic.api_keys import reset_api_key_expiration as _reset_api_key_expiration
 from aleph.index.entities import iter_proxies
 from aleph.index.util import AlephOperationalException
 from aleph.logic.collections import create_collection, update_collection
@@ -566,3 +567,9 @@ def evilshit():
     delete_index()
     destroy_db()
     upgrade()
+
+
+@cli.command()
+def reset_api_key_expiration():
+    """Reset the expiration date of all legacy, non-expiring API keys."""
+    _reset_api_key_expiration()

aleph/migrate/versions/131674bde902_add_primary_key_constraint_to_role_membership.py

@@ -1,14 +1,14 @@
 """add primary key constraint to role_membership table
 
 Revision ID: 131674bde902
-Revises: c52a1f469ac7
+Revises: 8adf50aadcb0
 Create Date: 2024-07-17 14:37:25.269913
 
 """
 
 # revision identifiers, used by Alembic.
 revision = "131674bde902"
-down_revision = "c52a1f469ac7"
+down_revision = "8adf50aadcb0"
 
 from alembic import op
 import sqlalchemy as sa

aleph/migrate/versions/d46fc882ec6b_api_key_expiration.py

@@ -0,0 +1,30 @@
+"""API key expiration
+
+Revision ID: d46fc882ec6b
+Revises: 131674bde902
+Create Date: 2024-05-02 11:43:50.993948
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+# revision identifiers, used by Alembic.
+revision = "d46fc882ec6b"
+down_revision = "131674bde902"
+
+
+def upgrade():
+    op.add_column("role", sa.Column("api_key_expires_at", sa.DateTime()))
+    op.add_column(
+        "role", sa.Column("api_key_expiration_notification_sent", sa.Integer())
+    )
+    op.create_index(
+        index_name="ix_role_api_key_expires_at",
+        table_name="role",
+        columns=["api_key_expires_at"],
+    )
+
+
+def downgrade():
+    op.drop_column("role", "api_key_expires_at")
+    op.drop_column("role", "api_key_expiration_notification_sent")

aleph/model/role.py

@@ -1,13 +1,13 @@
 import logging
-from datetime import datetime
+from datetime import datetime, timezone
 from normality import stringify
 from sqlalchemy import or_, not_, func
 from itsdangerous import URLSafeTimedSerializer
 from werkzeug.security import generate_password_hash, check_password_hash
 
 from aleph.core import db
 from aleph.settings import SETTINGS
-from aleph.model.common import SoftDeleteModel, IdModel, make_token, query_like
+from aleph.model.common import SoftDeleteModel, IdModel, query_like
 from aleph.util import anonymize_email
 
 log = logging.getLogger(__name__)
@@ -52,6 +52,8 @@ class Role(db.Model, IdModel, SoftDeleteModel):
     email = db.Column(db.Unicode, nullable=True)
     type = db.Column(db.Enum(*TYPES, name="role_type"), nullable=False)
     api_key = db.Column(db.Unicode, nullable=True)
+    api_key_expires_at = db.Column(db.DateTime, nullable=True)
+    api_key_expiration_notification_sent = db.Column(db.Integer, nullable=True)
     is_admin = db.Column(db.Boolean, nullable=False, default=False)
     is_muted = db.Column(db.Boolean, nullable=False, default=False)
     is_tester = db.Column(db.Boolean, nullable=False, default=False)
@@ -68,6 +70,10 @@ class Role(db.Model, IdModel, SoftDeleteModel):
     def has_password(self):
         return self.password_digest is not None
 
+    @property
+    def has_api_key(self):
+        return self.api_key is not None
+
     @property
     def is_public(self):
         return self.id in self.public_roles()
@@ -160,11 +166,12 @@ def to_dict(self):
                 "label": self.label,
                 "email": self.email,
                 "locale": self.locale,
-                "api_key": self.api_key,
                 "is_admin": self.is_admin,
                 "is_muted": self.is_muted,
                 "is_tester": self.is_tester,
                 "has_password": self.has_password,
+                "has_api_key": self.has_api_key,
+                "api_key_expires_at": self.api_key_expires_at,
                 # 'notified_at': self.notified_at
             }
         )
@@ -192,6 +199,17 @@ def by_api_key(cls, api_key):
             return None
         q = cls.all()
         q = q.filter_by(api_key=api_key)
+        utcnow = datetime.now(timezone.utc)
+
+        # TODO: Exclude API keys without expiration date after deadline
+        # See https://github.com/alephdata/aleph/issues/3729
+        q = q.filter(
+            or_(
+                cls.api_key_expires_at == None,  # noqa: E711
+                utcnow < cls.api_key_expires_at,
+            )
+        )
+
         q = q.filter(cls.type == cls.USER)
         q = q.filter(cls.is_blocked == False)  # noqa
         return q.first()
@@ -211,9 +229,6 @@ def load_or_create(cls, foreign_id, type_, name, email=None, is_admin=False):
             role.is_blocked = False
             role.notified_at = datetime.utcnow()
 
-        if role.api_key is None:
-            role.api_key = make_token()
-
         if email is not None:
             role.email = email
 

aleph/templates/email/api_key_expired.html

@@ -0,0 +1,15 @@
+{% extends "email/layout.html" %}
+
+{% block content -%}
+  <p>
+    {% trans expires_at=(expires_at | datetimeformat) -%}
+    Your Aleph API key has expired on {{expires_at}} UTC.
+    {%- endtrans %}
+  </p>
+
+  <p>
+    {% trans settings_url=settings_url -%}
+    If you do not use the Aleph API, or only use the API to access public data you can ignore this email. If you use the Aleph API to access data that is not publicly accessible then you’ll need to <a href="{{settings_url}}">regenerate your API key</a> to maintain access.
+    {%- endtrans %}
+  </p>
+{%- endblock %}

aleph/templates/email/api_key_expired.txt

@@ -0,0 +1,11 @@
+{% extends "email/layout.txt" %}
+
+{% block content -%}
+{% trans expires_at=(expires_at | datetimeformat) -%}
+Your Aleph API key has expired on {{expires_at}} UTC.
+{%- endtrans %}
+
+{% trans settings_url=settings_url -%}
+If you do not use the Aleph API, or only use the API to access public data you can ignore this email. If you use the Aleph API to access data that is not publicly accessible then you’ll need to regenerate your API key ({{settings_url}}) to maintain access.
+{%- endtrans %}
+{%- endblock %}

aleph/templates/email/api_key_expires_soon.html

@@ -0,0 +1,15 @@
+{% extends "email/layout.html" %}
+
+{% block content -%}
+  <p>
+    {% trans expires_at=(expires_at | datetimeformat) -%}
+    Your Aleph API key will expire in 7 days, on {{expires_at}} UTC.
+    {%- endtrans %}
+  </p>
+
+  <p>
+    {% trans settings_url=settings_url -%}
+    If you do not use the Aleph API, or only use the API to access public data you can ignore this email. If you use the Aleph API to access data that is not publicly accessible then you’ll need to <a href="{{settings_url}}">regenerate your API key</a> to maintain access.
+    {%- endtrans %}
+  </p>
+{%- endblock %}

aleph/templates/email/api_key_expires_soon.txt

@@ -0,0 +1,11 @@
+{% extends "email/layout.txt" %}
+
+{% block content -%}
+{% trans expires_at=(expires_at | datetimeformat) -%}
+Your Aleph API key will expire in 7 days, on {{expires_at}} UTC.
+{%- endtrans %}
+
+{% trans -%}
+If you do not use the Aleph API, or only use the API to access public data you can ignore this email. If you use the Aleph API to access data that is not publicly accessible then you’ll need to regenerate your API key ({{settings_url}}) to maintain access.
+{%- endtrans %}
+{%- endblock %}

aleph/templates/email/api_key_generated.html

@@ -0,0 +1,13 @@
+{% extends "email/layout.html" %}
+
+{% block content -%}
+{% if event == "regenerated" -%}
+{% trans -%}
+Your Aleph API key has been regenerated. If that wasn’t you, please contact an administrator.
+{%- endtrans %}
+{% else -%}
+{% trans -%}
+An Aleph API key has been generated for your account. If that wasn’t you, please contact an administrator.
+{%- endtrans %}
+{%- endif %}
+{%- endblock %}