hipscat_import#
All modules for hipscat-import package
Subpackages#
Submodules#
Package Contents#
Classes#
Data class for holding runtime arguments |
- class RuntimeArguments[source]#
Data class for holding runtime arguments
- output_path: str = ''#
base path where new catalog should be output
- output_artifact_name: str = ''#
short, convenient name for the catalog
- output_storage_options: Dict[Any, Any] | None#
optional dictionary of abstract filesystem credentials for the OUTPUT.
- tmp_dir: str = ''#
path for storing intermediate files
- resume: bool = True#
If True, we try to read any existing intermediate files and continue to run the pipeline where we left off. If False, we start the import from scratch, overwriting any content of the output directory.
- progress_bar: bool = True#
if true, a tqdm progress bar will be displayed for user feedback of map reduce progress
- dask_tmp: str = ''#
directory for dask worker space. this should be local to the execution of the pipeline, for speed of reads and writes
- dask_n_workers: int = 1#
number of workers for the dask client
- dask_threads_per_worker: int = 1#
number of threads per dask worker
- resume_tmp: str = ''#
directory for intermediate resume files, when needed. see RTD for more info.
- completion_email_address: str = ''#
if provided, send an email to the indicated email address once the import pipeline has complete.
- catalog_path: hipscat.io.FilePointer | None#
constructed output path for the catalog that will be something like <output_path>/<output_artifact_name>
- tmp_path: hipscat.io.FilePointer | None#
constructed temp path - defaults to tmp_dir, then dask_tmp, but will create a new temp directory under catalog_path if no other options are provided