hipscat_import.catalog.run_import

Import a set of non-hipscat files using dask for parallelization

Methods in this file set up a dask pipeline using futures. The actual logic of the map reduce is in the map_reduce.py file.

Module Contents

Functions

_map_pixels(args, client)	Generate a raw histogram of object counts in each healpix pixel
_split_pixels(args, alignment_future, client)	Generate a raw histogram of object counts in each healpix pixel
_reduce_pixels(args, destination_pixel_map, client)	Loop over destination pixels and merge into parquet files
run(args, client)	Run catalog creation pipeline.

_map_pixels(args, client)

Generate a raw histogram of object counts in each healpix pixel

_split_pixels(args, alignment_future, client)

Generate a raw histogram of object counts in each healpix pixel

_reduce_pixels(args, destination_pixel_map, client)

Loop over destination pixels and merge into parquet files

run(args, client)

Run catalog creation pipeline.