hipscat_import.catalog.arguments

Utility to hold all arguments required throughout partitioning

Module Contents

Classes

ImportArguments

Container class for holding partitioning arguments

Functions

check_healpix_order_range(order, field_name[, ...])

Helper method to check if the order is within the range determined by the

class ImportArguments

Bases: hipscat_import.runtime_arguments.RuntimeArguments

Container class for holding partitioning arguments

epoch: str = 'J2000'

astronomical epoch for the data. defaults to “J2000”

catalog_type: str = 'object'

level of catalog data, object (things in the sky) or source (detections)

input_path: hipscat.io.FilePointer | None

path to search for the input data

input_file_list: List[hipscat.io.FilePointer]

can be used instead of input_path to import only specified files

input_paths: List[hipscat.io.FilePointer]

resolved list of all files that will be used in the importer

input_storage_options: Dict[Any, Any] | None

optional dictionary of abstract filesystem credentials for the INPUT.

ra_column: str = 'ra'

column for right ascension

dec_column: str = 'dec'

column for declination

use_hipscat_index: bool = False

use an existing hipscat spatial index as the position, instead of ra/dec

sort_columns: str | None

column for survey identifier, or other sortable column. if sorting by multiple columns, they should be comma-separated. if add_hipscat_index=True, this sorting will be used to resolve the counter within the same higher-order pixel space

add_hipscat_index: bool = True

add the hipscat spatial index field alongside the data

use_schema_file: str | None

path to a parquet file with schema metadata. this will be used for column metadata when writing the files, if specified

constant_healpix_order: int

healpix order to use when mapping. if this is a positive number, this will be the order of all final pixels and we will not combine pixels according to the threshold

lowest_healpix_order: int = 0

the lowest possible healpix order that we will use for the final catalog partitioning. setting this higher than 0 will prevent creating partitions with a large area on the sky.

highest_healpix_order: int = 7

healpix order to use when mapping. this will not necessarily be the order used in the final catalog, as we may combine pixels that don’t meed the threshold

pixel_threshold: int = 1000000

maximum number of rows for a single resulting pixel. we may combine hierarchically until we near the pixel_threshold

mapping_healpix_order: int

healpix order to use when mapping. will be highest_healpix_order unless a positive value is provided for constant_healpix_order

debug_stats_only: bool = False

do not perform a map reduce and don’t create a new catalog. generate the partition info

file_reader: hipscat_import.catalog.file_readers.InputReader | str | None

instance of input reader that specifies arguments necessary for reading from your input files

resume_plan: hipscat_import.catalog.resume_plan.ResumePlan | None

container that handles read/write of log files for this pipeline

__post_init__()

_check_arguments()

Check existence and consistency of argument values

to_catalog_info(total_rows) → hipscat.catalog.catalog.CatalogInfo

Catalog-type-specific dataset info.

additional_runtime_provenance_info() → dict

Any additional runtime args to be included in provenance info from subclasses

check_healpix_order_range(order, field_name, lower_bound=0, upper_bound=hipscat_id.HIPSCAT_ID_HEALPIX_ORDER)

Helper method to check if the order is within the range determined by the lower_bound and upper_bound, inclusive.

Parameters:

• order (int) – healpix order to check

• field_name (str) – field name to use in the error message

• lower_bound (int) – lower bound of range

• upper_bound (int) – upper bound of range

Raises:

ValueError – if the order is outside the specified range, or bounds are unreasonable.