Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/news/save_entry_case.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
**Fixed:**

* Fixed creating and saving entries containing upper case characters, which are converted to lowercase, to match the frictionless specifications.
3 changes: 3 additions & 0 deletions examples/from_csv/UpperCase.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
E,I
1,2
3,4
66 changes: 60 additions & 6 deletions unitpackage/entry.py
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,25 @@ def from_csv(cls, csvname, metadata=None, fields=None):
>>> entry.user
'Max Doe'

.. important::
Upper case filenames are converted to lower case entry identifiers!

A filename containing upper case characters::

>>> import os
>>> fields = [{'name':'E', 'unit': 'mV'}, {'name':'I', 'unit': 'A'}]
>>> entry = Entry.from_csv(csvname='examples/from_csv/UpperCase.csv', fields=fields)
>>> entry
Entry('uppercase')

Casing in the filename is preserved in the metadata::

>>> entry.resource # doctest: +NORMALIZE_WHITESPACE
{'name': 'uppercase',
'type': 'table',
'path': 'UpperCase.csv',
...

"""
from unitpackage.local import create_unitpackage

Expand Down Expand Up @@ -788,11 +807,23 @@ def from_df(cls, df, metadata=None, fields=None, outdir=None, *, basename):

>>> entry.save(outdir='./test/generated/from_df')

TESTS
.. important::
Basenames with upper case characters are stored with lower case characters!
To separate words use underscores.

The basename will always be converted to lowercase entry identifiers::

>>> import pandas as pd
>>> from unitpackage.entry import Entry
>>> df = pd.DataFrame({'x':[1,2,3], 'y':[2,3,4]})
>>> entry = Entry.from_df(df=df, basename='TEST_DF')
>>> entry
Entry('test_df')

TESTS:

Verify that all fields are properly created even when they are not specified as fields::

>>> import os
>>> fields = [{'name':'x', 'unit': 'm'}, {'name':'P', 'unit': 'um'}, {'name':'E', 'unit': 'V'}]
>>> metadata = {'user':'Max Doe'}
>>> entry = Entry.from_df(df=df, basename='test_df', metadata=metadata, fields=fields)
Expand Down Expand Up @@ -832,9 +863,13 @@ def save(self, *, outdir, basename=None):
True

When a ``basename`` is set, the files are named ``basename.csv`` and ``basename.json``.
Note that for a valid frictionless Data Package this base name
MUST be lower-case and contain only alphanumeric
characters along with ".", "_" or "-" characters'::

.. note::
For a valid frictionless Data Package the basename
MUST be lower-case and contain only alphanumeric
characters along with ``.``, ``_`` or ``-`` characters'
Comment on lines +867 to +870

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

afaik does frictionless enforce always lowercase. The validity of names is checked upon loading a datapackage by frictionless, so we do not run our test. I added the note here so people don't have to dig deep into the frictionless documentation. If the frictionless policy changes, we might have to rewind the change from this PR.

I think adopting a lowercase solution is the most simple one. Creating, for example, camel case strings is not trivial since the meaning of the string is often unclear. Assume your string is CVmeasurement then that should result in cvMeasurement, which can not be achieved unless you provide an appropriate dictionary.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to adopt camel case in our JSON files.

According to the getting started section from frictionless

Arguments conform to the following naming convention:
- for Python interfaces, they are snake_cased, e.g. `missing_values`
- within dictionaries or JSON objects, they are camelCased, e.g. `missingValues`
- in the command line, they use dashes, e.g. `--missing-values`


A valid basename::

>>> import os
>>> entry = Entry.create_examples()[0]
Expand All @@ -843,10 +878,29 @@ def save(self, *, outdir, basename=None):
>>> os.path.exists(f'test/generated/{basename}.json') and os.path.exists(f'test/generated/{basename}.csv')
True

Upper case characters are saved lower case::

>>> import os
>>> import pandas as pd
>>> from unitpackage.entry import Entry
>>> df = pd.DataFrame({'x':[1,2,3], 'y':[2,3,4]})
>>> basename = 'Upper_Case_Save'
>>> entry = Entry.from_df(df=df, basename=basename)
>>> entry.save(outdir='./test/generated')
>>> os.path.exists(f'test/generated/{basename.lower()}.json') and os.path.exists(f'test/generated/{basename.lower()}.csv')
True

>>> new_entry = Entry.from_local(f'test/generated/{basename.lower()}.json')
>>> new_entry.resource # doctest: +NORMALIZE_WHITESPACE
{'name': 'upper_case_save',
'type': 'table',
'path': 'upper_case_save.csv',
...

TESTS:

Save the entry as Data Package with metadata containing datetime format,
which is not natively supported by JSON.
which is not natively supported by JSON.::

>>> import os
>>> from datetime import datetime
Expand Down
2 changes: 1 addition & 1 deletion unitpackage/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def create_unitpackage(csvname, metadata=None, fields=None):
)
package.infer()

resource = package.get_resource(Path(csv_basename).stem)
resource = package.get_resource(Path(csv_basename).stem.lower())

resource.custom.setdefault("metadata", {})
resource.custom["metadata"].setdefault("echemdb", metadata)
Expand Down