mast_contributor_tools.filename_check package

Submodules

mast_contributor_tools.filename_check.fc_app module

mast_contributor_tools.filename_check.fc_app.check_filenames(hlsp_name: str, file_list: list[Path], dbFile: str, output_format: str = 'db') → None

Recursively check filenames in a directory tree of HLSP products

Parameters:

• hlsp_name (str) – Official identifier (abbreviation/acronym/initialism) for the HLSP collection

• file_list (list[str]) – List of files to check, typically output from get_file_paths()

• dbFile (str, optional) – Name of SQLite database file to contain results

• output_format (str, optional) – Alternate format to save results to: ‘csv’, ‘fits’, ‘html’, or ‘excel’. Default: “db”

mast_contributor_tools.filename_check.fc_app.check_single_filename(file_name: str, hlsp_name: str = '') → None

HLSP filename module CLI driver.

Parameters:

• file_name (str) – File name of an HLSP product to test: for example ‘hlsp_my-hlsp_readme.txt’. This is a string, and does not need to be a real file.

• hlsp_name (str, optional) – Name of example HLSP collection. For example, ‘my-hlsp’. If not supplied, the hlsp_name is inferred using the second field of the filename.

mast_contributor_tools.filename_check.fc_app.get_file_paths(hlsp_path: str, from_file: str = '', search_pattern: str = '*.*', exclude_pattern: str | None = None, max_n: int | None = None) → list[Path]

Build a list of filename Paths relative to the given directory.

Parameters:

• hlsp_path (str) – Head of directory containing HLSP collection files. The base directory defaults to the current working directory.

• from_file (str, optional) – Path to a text file containing a list of filenames to check, instead of scanning a directory

• search_pattern (str, optional) – Search pattern to limit files to test. For example, ‘.fits’ will only return the fits files. Default value is ‘.*’ for all files

• exclude_pattern (str, optional) – Search pattern to exclude files from testing. For example, ‘*.png’ will only skip all of the png files.

• max_n (int, optional) – Maximum number of files to check, for testing purposes. For example, max_n=10 will only check the first 10 files found.

Returns:

A list of filename Paths contained within the given directory

Return type:

list[Path]

mast_contributor_tools.filename_check.fc_db module

Create and manage an SQLite database for storing results of file checking.

class mast_contributor_tools.filename_check.fc_db.Hlsp_SQLiteDb(filename: str)

Bases: object

Create an SQLite DB to store results.

Parameters:

filename (str) – name of the SQLite DB file to be created

add_fields(elements: list[dict]) → None

Add metadata for each of a filename’s fields to the fields table

Add results of checking mmultiple filename fields to the fields table. Metadata for each field take the form of key:value pairs.

Parameters:

elements (list[dict]) – List of element attribute dictionaries.

add_filename(file_record: dict) → None

Add file metadata to the filename table

Add file metadata to the DB in the form of a list of key:value pairs.

Parameters:

file_record (dict) – File attributes

close_db() → None

create_db() → None

Create the database and construct the tables.

:raises sqlite3.Error : Error: Raised if the DB cannot be created or the tables fail to be created.

print_summary() → str

Returns a string detailing some summary information on how many files have passed validation

write_to_alternate_format(save_format: str) → list[str]

Write out the SQLite DB as an alternate format.

Parameters:

save_format (str) – Format to save output: ‘csv’, ‘excel’, ‘html’, or ‘fits’

Returns:

files_written

Return type:

list[str]: List of file names written out

mast_contributor_tools.filename_check.fc_db.color_formatter(value: str) → str

Color mapping for use in write_to_alternate_format(). Color-codes table cells based on verdict: green for “PASS”, red for “FAIL”, etc.

Parameters:

value (str)

mast_contributor_tools.filename_check.hlsp_filename module

The main logic module to check filename compliance

class mast_contributor_tools.filename_check.hlsp_filename.ExtensionField(value: str, field_indx: int = 8)

Bases: FilenameFieldAB

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.FieldRule

Bases: object

Rules for filename validation.

This class embodies rules for validating attributes of field names. The approach to validiting field values varies by field. The expressions that validate the version or target fields can be verified at https://regex101.com

capitalization() → str

Test the captilizaiton: the entire filename must be lowercase. Returns ‘pass’ or ‘fail’ based on results.

field_verdict() → str

Determine the final verdict for this field: ‘pass’, ‘needs review’ or ‘fail’, determined as the worst of the input scores.

length(max_length: int) → str

Test if the character count is non-zero and within the limit for that field. Returns ‘pass’ or ‘fail’ based on results.

match_choice(choice_list: list[str], score_level='lax') → str

Checks value against a list, typically from oif.yaml. Returns ‘pass’ or ‘needs review’ or ‘fail’ based on results. The optional ‘score_level’ argument determines if ‘fail’ or ‘needs review’ is returned (default lax)

match_multi_choice(choice_list: list[str], score_level='lax') → str

Checks multiple values against a list, typically from oif.yaml. Returns ‘pass’ or ‘needs review’ or ‘fail’ based on results. The optional ‘score_level’ argument determines if ‘fail’ or ‘needs review’ is returned (default lax)

match_pattern(regex_expr: Pattern) → str

Test that the field contains no forbidden characters. Returns ‘pass’ or ‘fail’ based on results.

nfields() → str

Tests that the field index is less than 9; Returns ‘pass’ or ‘fail’ based on results.

class mast_contributor_tools.filename_check.hlsp_filename.FilenameFieldAB(field_name: str, field_value: str, field_indx: int)

Bases: ABC

Template for Filename Field classes.

Each field of a filename in an HLSP collection will be evaluated for: length, capitalization, content, and often a match against valid values. Each evaluation results in a score, which is one of: - ‘pass’ for no detected problems - ‘fail’ for a detected problem that must be fixed - ‘needs review’ for a possible but non-fatal problem that requires review by MAST Staff

The final verdict of the set of evaluations is determined as the worst of the input scores.

Parameters:

• field_name (str) – Internal name for the field being created

• field_value (str) – Value of the field (i.e. text of the field in the filename)

abstractmethod evaluate()

Evaluate the field for each rule

get_scores()

Return final scores

class mast_contributor_tools.filename_check.hlsp_filename.FilterField(value: str, field_indx: int = 5)

Bases: FilenameFieldAB

A container for attributes of the filename Filtername field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.GenericField(value: str, id: int, field_indx: int)

Bases: FilenameFieldAB

Generic field concrete class.

Since some filename fields are optional, this class handles the case of fields that are not identifiable with the standard set. In this case the field will be validated for length and capitalization, but not for value.

evaluate() → None

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.HlspField(value: str, field_indx: int = 0)

Bases: FilenameFieldAB

A container for attributes of the literal ‘hlsp’ prefix field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.HlspFileName(filepath: Path, hlsp_name: str)

Bases: object

HLSP filename validation

Filenames are composed of fields separated by underscores, except that the last field is really composed of two fields separated by a period. The last part of the last field may also contain a period. Certain fields are further composed of elements, separated by hyphens.

Filenames must have at least 4 and as many as 9 fields to be valid. For valid filenames:

• The first two and the last two fields are required

• the third from last (N-2) is always required except when the value of N-1 is ‘readme’

Unless all 9 fields are present, or only 4 are present, it is not possible to determine robustly what the other fields (if present) contain.

Parameters:

• path (str) – Filesystem path relative to the root of the HLSP collection files

• filename (str) – Filename of a collection product

• hlsp_name (str) – Official abbreviation/acronym/initialism of this HLSP collection

Raises:

ValueError – If the number of fields falls outside the limits.

create_fields() → None

Create Field objects for each field in the filename.

evaluate_fields()

Evaluate attributes of each field

Returns:

List of result dictionaries for each field

evaluate_filename()

Evaluate attributes of the filename.

Note that the filename ‘status’ depends upon having evaluated the fields.

Returns:

dict[str, Any]

Dictionary of file name attributes

partition() → None

Partition the filepath into path+filename, and filename into fields

class mast_contributor_tools.filename_check.hlsp_filename.HlspNameField(value: str, ref_name: str, field_indx: int = 1)

Bases: FilenameFieldAB

A container for attributes of the HLSP name field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.InstrumentField(value: str, field_indx: int = 3)

Bases: FilenameFieldAB

A container for attributes of the filename Instrument field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.MissionField(value: str, field_indx: int = 2)

Bases: FilenameFieldAB

A container for attributes of the filename Mission (or observatory) field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.ProductField(value: str, field_indx: int = 7)

Bases: FilenameFieldAB

A container for attributes of the filename ProductType field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.TargetField(value: str, field_indx: int = 4)

Bases: FilenameFieldAB

A container for attributes of the filename TargetName field.

evaluate()

Evaluate the field for each rule

class mast_contributor_tools.filename_check.hlsp_filename.VersionField(value: str, field_indx: int = 6)

Bases: FilenameFieldAB

A container for attributes of the filename Version field.

evaluate()

Evaluate the field for each rule

Module contents