Ctrl+K

HiPSCat Import

HiPSCat Import#

Utility for ingesting large survey data into HiPSCat structure.

Installation#

We recommend installing in a virtual environment, like venv or conda. You may need to install or upgrade versions of dependencies to work with hipscat-import.

pip install hipscat-import

Tip

Installing on Mac

healpy is a very necessary dependency for hipscat libraries at this time, but native prebuilt binaries for healpy on Apple Silicon Macs do not yet exist, so it’s recommended to install via conda before proceeding to hipscat-import.

>> conda config --append channels conda-forge
>> conda install healpy

Setting up a pipeline#

For each type of dataset the hipscat-import tool can generate, there is an argument container class that you will need to instantiate and populate with relevant arguments.

See dataset-specific notes on arguments:

Once you have created your arguments object, you pass it into the pipeline control, and then wait. Running within a main guard will potentially avoid some python threading issues with dask:

from dask.distributed import Client
from hipscat_import.pipeline import pipeline_with_client

def main():
    args = ...
    with Client(
        n_workers=10,
        threads_per_worker=1,
        ...
    ) as client:
        pipeline_with_client(args, client)

if __name__ == '__main__':
    main()

Acknowledgements#

This project is supported by Schmidt Sciences.

This project is based upon work supported by the National Science Foundation under Grant No. AST-2003196.

This project acknowledges support from the DIRAC Institute in the Department of Astronomy at the University of Washington. The DIRAC Institute is supported through generous gifts from the Charles and Lisa Simonyi Fund for Arts and Sciences, and the Washington Research Foundation.

Contents