Schema changes example for mlos_benchd service

[bpkroth]

Pull Request

Title

Schema changes for mlos_benchd service.

---

Description

Schema changes for mlos_benchd service.

Storage APIs to adjust these to come in a future PR.

• See mlos_bench_service #732

---

Type of Change

• ✨ New feature

---

Testing

Local, CI

---

Additional Notes (optional)

@eujing have a look at the commit history in the PR for a sense of what's going on.
This can probably be merged as is. Happy to discuss further though.

---

[bpkroth]

Rough idea here is in the comments in the schema.py file.

At a high level, imagine that we have 1+ VMs running mlos_benchd as a service, continually polling the backend storage to see if there are new Experiments submitted that are PENDING.

Note: Here a "Worker" runs mlos_benchd to spawn multiple mlos_bench processes, one per Experiment. Where as within each mlos_bench 1+ TrialRunners may run multiple Trials for that Experiment in parallel (wip - #380).

A Worker starts a transaction to lay claim to an Experiment and then forks off a new mlos_bench process to run it in the background.

# quick pseudo code
while true:
  sleep(1)
  with conn.begin() as conn:
     try: 
       experiment_row = conn.select(schema.experiments)
           .where(
               schema.experiments.status == Status.PENDING.name,
               schema.experiments.worker_name is None,
               schema.experiments.start_ts >= datetime.utcnow(),
           )
           .limit(1)
      if experiment_row:
         # try to grab
         result = conn.update(schema.experiments)
           .values({
              schema.experiments.worker_name: sys.hostname,
              schema.experiments.status: status.READY.name,
           })
           .where(
              schema.experiments.worker_name is None,
              schema.experiments.experiment_id == experiment_row.experiment_id,
            )
         if result:
           # succeeded, commit the transaction and return
           conn.commit()
           # return this to calling code to spawn a new `mlos_bench`
           # process to fork and execute this Experiment on this host
           # in the background
           return experiment_row.experiment_id
         else:
           # someone else probably grabbed it
           conn.rollback()
  except SqlException as e:
     # probably a conflict 
     conn.rollback()
# try again in a moment

[motus]

Approved, but let's have a short discussion before merging it in