hipscat_import.soap.resume_plan

Utility to hold the pipeline execution plan.

Module Contents

Classes

SoapPlan

Container class for holding the state of each file in the pipeline plan.

Functions

source_to_object_map(object_catalog, source_catalog)

Build a map of (source order/pixel) to the (object order/pixel)

class SoapPlan(args: hipscat_import.soap.arguments.SoapArguments)

Bases: hipscat_import.pipeline_resume_plan.PipelineResumePlan

Container class for holding the state of each file in the pipeline plan.

count_keys: List[Tuple[hipscat.pixel_math.healpix_pixel.HealpixPixel, List[hipscat.pixel_math.healpix_pixel.HealpixPixel], str]]

set of pixels (and job keys) that have yet to be counted

reduce_keys: List[Tuple[hipscat.pixel_math.healpix_pixel.HealpixPixel, str]]

set of object catalog pixels (and job keys) that have yet to be reduced/combined

source_pixel_map: List[Tuple[hipscat.pixel_math.healpix_pixel.HealpixPixel, List[hipscat.pixel_math.healpix_pixel.HealpixPixel], str]] | None

Map of object pixels to source pixels, with counting key.

object_catalog: hipscat.catalog.Catalog | None

COUNTING_STAGE = 'counting'

REDUCING_STAGE = 'reducing'

SOURCE_MAP_FILE = 'source_object_map.npz'

gather_plan(args)

Initialize the plan.

wait_for_counting(futures)

Wait for counting stage futures to complete.

is_counting_done() → bool

Are there sources left to count?

get_sources_to_count(source_pixel_map=None)

Fetch a triple for each source pixel to join and count.

Triple contains:

• source pixel

• object pixels (healpix pixel with both order and pixel, for aligning and neighboring object pixels)

• source key (string of source order+pixel)

classmethod reducing_key_done(tmp_path, reducing_key: str)

Mark a single reducing task as done

Parameters:

• tmp_path (str) – where to write intermediate resume files.

• reducing_key (str) – unique string for each reducing task (e.g. “3_57”)

get_objects_to_reduce()

Fetch a tuple for each object catalog pixel to reduce.

is_reducing_done() → bool

Are there partitions left to reduce?

wait_for_reducing(futures)

Wait for reducing stage futures to complete.

source_to_object_map(object_catalog, source_catalog)

Build a map of (source order/pixel) to the (object order/pixel) that are aligned, as well as neighboring object pixels.