lightcurvelynx.obstable.obs_table

The top-level module for survey related data, such as pointing and observation conditions. ObsTable class is a base class with specific implementations for different survey data, such as Rubin and ZTF.

Attributes

logger

Classes

ObsTable

A class that stores a table of information about the observations in a survey,

Module Contents

logger[source]

class ObsTable(table, *, colmap=None, detector_footprint=None, wcs=None, saturation_mags=None, **kwargs)[source]

A class that stores a table of information about the observations in a survey, such as pointing and observation conditions. ObsTables are specialized for different surveys and include information about that survey (e.g. default noise parameters, default filter characteristics, etc.). They also include helper functions for common computations.

Parameters:

table (dict or pandas.core.frame.DataFrame) – The table with all the survey information. Metadata can be included in the “lightcurvelynx_survey_data” entry of the attributes dictionary.
colmap (dict, optional) – A mapping of standard column names to a list of possible names in the input table. Each value in the dictionary can be a string or a list of strings. For example, in Rubin’s OpSim we might have the column “observationStartMJD” which maps to “time”. In that case we would have an entry with key=”time” and value=”observationStartMJD”.
detector_footprint (astropy.regions.SkyRegion, Astropy.regions.PixelRegion, or) – DetectorFootprint, optional The footprint object for the instrument’s detector. If None, no footprint filtering is done. Default is None.
wcs (astropy.wcs.WCS, optional) – The WCS for the footprint. Either this or pixel_scale must be provided if a footprint is provided as a Astropy region.
saturation_mags (dict, optional) – A dictionary mapping filter names to their saturation thresholds in magnitudes. The filters provided must match those in the table. If not provided, saturation effects will not be applied.
**kwargs (dict) – Additional keyword arguments to pass to the constructor. This can include overrides of any of the survey values.

survey_values[source]

A mapping for constant values for the survey used in various computations, such as readout noise and dark current.

Type:: dict, optional

filters[source]

The unique filters in the survey table (if provided).

Type:: np.ndarray

_table

The table with all the observation information mapped to standard column names.

Type:: pandas.core.frame.DataFrame

_colmap[source]

A mapping of standard column names to their names in the input table.

Type:: dict

_inv_colmap[source]

A dictionary mapping the custom column names back to the standard names.

Type:: dict

_spatial_data

A spatial_data structure of the survey pointings for fast spatial queries. We use the scipy kd-tree for most of the implementations instead of astropy’s functions so we can directly control caching.

Type:: scipy.spatial.KDTree or None

_detector_footprint

The footprint object for the instrument’s detector. If None, no footprint filtering is done. Default is None.

Type:: DetectorFootprint, optional

_saturation_mags[source]

The saturation thresholds in magnitudes for each filter. If unspecified, an instrument-specific default will be used, if available.

Type:: dict, optional

survey_values[source]

filters[source]

__len__()[source]

__getitem__(key)[source]: Access the underlying observation table by column or parameter name. This will return either a full column from the table or a survey parameter value.

__contains__(key)[source]: Check if a column exists in the survey table or a parameter in the parameter table.

copy()[source]: Create a copy of the ObsTable.

head(n=5)[source]: Return the first n rows of the observation table.

uses_footprint()[source]: Return whether the ObsTable uses a detector footprint for filtering.

clear_detector_footprint()[source]: Clear the detector footprint, so no footprint filtering is done.

set_detector_footprint(detector_footprint, wcs=None)[source]

Set the detector footprint, so footprint filtering is done.

Parameters:

detector_footprint (astropy.regions.SkyRegion, Astropy.regions.PixelRegion, or) – DetectorFootprint The footprint object for the instrument’s detector.
wcs (astropy.wcs.WCS, optional) – The WCS for the footprint. Either this or pixel_scale must be provided if a footprint is provided as a Astropy region.

get_value_per_row(key, *, indices=None, default=None)[source]

Get the values for each row from the table or survey values (defaults).

Parameters:

key (str) – The name of the column to retrieve.
indices (numpy.ndarray, optional) – The indices of the rows for which to retrieve values. If None, retrieve all rows. Default: None
default (any, optional) – The default value to use if the key is not found in the table or survey values. This can be None to indicate missing values. Default: None

Returns:

The values for each row in the table.

Return type:

numpy.ndarray

safe_get_survey_value(key)[source]

Get a survey value by key, checking that it is not None.

Parameters:: key (str) – The key of the survey value to retrieve.

property radius[source]: Return the radius if it exists.

property columns[source]: Get the column names.

classmethod from_db(filename, sql_query='SELECT * FROM observations', **kwargs)[source]

Create an ObsTable object from the data in an db file. Reads data matching what is produced by write_db (and matching the RubinOpsim table).

Parameters:

filename (str) – The name of the db file.
sql_query (str) – The SQL query to use when loading the table. Default: “SELECT * FROM observations”
kwargs (dict, optional) – Additional keyword arguments to pass to the Survey constructor.

Returns:

A table with all of the pointing data.

Return type:

ObsTable

Raises:

FileNotFoundError – if the file does not exist.
ValueError – if unable to load the table.

classmethod from_parquet(filename, **kwargs)[source]

Create an ObsTable object from a parquet file.

Parameters:

filename (str) – The name of the parquet file to read.
kwargs (dict, optional) – Additional keyword arguments to pass to the Survey constructor.

Returns:

A table with all of the pointing data.

Return type:

ObsTable

estimate_coverage(*, radius=None, max_depth=12, use_footprint=False)[source]

Estimate the sky coverage of the observations in the ObsTable. This is an approximate calculation based on a constructed MOC at a given depth.

Parameters:

radius (float, optional) – The radius to use for each image (in degrees). Only used if use_footprint is False. If None, the radius from the survey values will be used.
max_depth (int, optional) – The maximum depth of the MOC. Default is 12.
use_footprint (bool, optional) – Whether to use the detector footprint to build the MOC. If True, the footprint will be used to compute the MOC regions for each pointing. If False, a simple cone with the given radius will be used.

Returns:

coverage – The estimated sky coverage in square degrees.

Return type:

float

build_moc(*, duplicate_threshold=100.0 / 3600.0, max_depth=10, radius=None, use_footprint=False)[source]

Build a Multi-Order Coverage Map from the regions in the data set.

The MOCs can be either from simple cones around each pointing or from the detector footprint if available. The code does not currently support rotation when dealing with detector footprints.

Note

This can be very slow for large ObsTables or high max_depth values, especially when using detector footprints.

Parameters:

duplicate_threshold (float, optional) – The threshold to use for identifying duplicate pointings (in degrees). Default is 100.0 / 3600.0 degrees = 100 arcseconds.
max_depth (int, optional) – The maximum depth of the MOC. Default is 10.
radius (float, optional) – The radius to use for each image (in degrees). Only used if use_footprint is False. If None, the radius from the survey values will be used.
use_footprint (bool, optional) – Whether to use the detector footprint to build the MOC. If True, the footprint will be used to compute the MOC regions for each pointing. If False, a simple cone with the given radius will be used.

Returns:

The Multi-Order Coverage Map constructed from the data set.

Return type:

MOC

plot_footprint(*, depth=14, fig=None, ax=None, use_footprint=False, **kwargs)[source]

Plot the MOC footprint using matplotlib.

Parameters:

depth (int, optional) – The healpix depth to use for plotting. Default is 14.
fig (matplotlib.figure.Figure, optional) – An existing matplotlib figure to use. If None, a new figure is created.
ax (matplotlib.pyplot.Axes or None, optional) – The axes to use for the plot. If None, new axes will be created on the figure.
use_footprint (bool, optional) – Whether to use the detector footprint to build the MOC. If True, the detector footprint will be used. Default is False.
**kwargs (dict, optional) – Additional keyword arguments to pass to the plot_moc function.

Returns:

fig (matplotlib.figure.Figure) – The figure containing the plot.
ax (matplotlib.pyplot.Axes) – The axes containing the plot.

add_column(colname, values, *, overwrite=False)[source]

Add a column to the current data table.

Parameters:

colname (str) – The name of the new column.
values (int, float, str, list, or numpy.ndarray) – The value(s) to add.
overwrite (bool) – Overwrite the column is it already exists. Default: False

write_db(filename, *, tablename='observations', overwrite=False)[source]

Write out the observation table as a database to a given SQL table.

Parameters:

filename (str) – The name of the db file.
tablename (str) – The table to which to write. Default: “observations”
overwrite (bool) – Overwrite the existing DB file. Default: False

Raises:

FileExistsError – if the file already exists and overwrite is False.

write_parquet(filename, *, overwrite=False)[source]

Write out the observation table as a parquet file.

Parameters:

filename (str) – The name of the parquet file.
overwrite (bool) – Overwrite the existing parquet file. Default: False

Raises:

FileExistsError – if the file already exists and overwrite is False.

time_bounds()[source]

Returns the min and max times for all observations in the ObsTable.

Returns:: t_min, t_max – The min and max times for all observations in the ObsTable.
Return type:: float, float

filter_rows(rows)[source]

Filter the rows in the ObsTable to only include those indices that are provided in a list of row indices (integers) or marked True in a mask.

Parameters:: rows (numpy.ndarray) – Either a Boolean array of the same length as the table or list of integer row indices to keep.
Returns:: self – The filtered ObsTable object.
Return type:: ObsTable

is_observed(query_ra, query_dec, *, radius=None, t_min=None, t_max=None)[source]

Check if the query point(s) fall within the field of view of any pointing in the ObsTable.

Parameters:

query_ra (float or numpy.ndarray) – The query right ascension (in degrees).
query_dec (float or numpy.ndarray) – The query declination (in degrees).
radius (float or None, optional) – The angular radius of the observation (in degrees).
t_min (float or None, optional) – The minimum time (in MJD) for the observations to consider. If None, no time filtering is applied.
t_max (float or None, optional) – The maximum time (in MJD) for the observations to consider. If None, no time filtering is applied.

Returns:

seen – Depending on the input, this is either a single bool to indicate whether the query point is observed or a list of bools for an array of query points.

Return type:

bool or list[bool]

range_search(query_ra, query_dec, *, radius=None, t_min=None, t_max=None)[source]

Return the indices of the pointings that fall within the field of view of the query point(s).

Parameters:

query_ra (float or numpy.ndarray) – The query right ascension (in degrees).
query_dec (float or numpy.ndarray) – The query declination (in degrees).
radius (float or None, optional) – The angular radius of the observation (in degrees). If None uses the default radius for the ObsTable.
t_min (float, numpy.ndarray or None, optional) – The minimum time (in MJD) for the observations to consider. If None, no time filtering is applied.
t_max (float, numpy.ndarray or None, optional) – The maximum time (in MJD) for the observations to consider. If None, no time filtering is applied.

Returns:

inds – Depending on the input, this is either a list of indices for a single query point or a list of arrays (of indices) for an array of query points.

Return type:

list[int] or list[numpy.ndarray]

get_observations(query_ra, query_dec, *, radius=None, t_min=None, t_max=None, cols=None)[source]

Return the observation information when the query point falls within the field of view of a pointing in the ObsTable.

Parameters:

query_ra (float) – The query right ascension (in degrees).
query_dec (float) – The query declination (in degrees).
radius (float or None, optional) – The angular radius of the observation (in degrees). If None uses the default radius for the ObsTable.
t_min (float or None, optional) – The minimum time (in MJD) for the observations to consider. If None, no time filtering is applied.
t_max (float or None, optional) – The maximum time (in MJD) for the observations to consider. If None, no time filtering is applied.
cols (list or str) – A list of the names of columns to extract or a single column name. If None returns all the columns.

Returns:

results – A dictionary mapping the given column name to a numpy array of values.

Return type:

dict

compute_saturation(flux, flux_error, index)[source]

Apply the saturation limits to a given flux and flux error.

When a flux value exceeds the saturation limit, it is clipped to the limit and flagged as saturated. In these cases, the associated flux_error is increased to account for the offset introduced by clipping. The new error is computed as the quadrature sum of the original flux_error and the difference between the orginal flux and saturated flux:

saturated_flux_error = sqrt(flux_error**2 + (flux - saturated_flux)**2)

For unsaturated points, both flux and flux_error are returned unchanged.

Parameters:

flux (numpy.ndarray of float) – The bandflux in nJy. A size S x T array where S is the number of samples in the graph state and T is the number of time points.
flux_error (numpy.ndarray of float) – The bandflux error in nJy. A size S x T array where S is the number of samples in the graph state and T is the number of time points.
index (array_like of int) – The index of the observation in the ObsTable table.

Returns:

A tuple with three entries:

The saturated flux in nJy. A size S x T array where S is the number of samples in the graph state and T is the number of time points.
The saturated flux error in nJy. A size S x T array where S is the number of samples in the graph state and T is the number of time points.
A boolean array indicating which points are saturated. A size S x T array where S is the number of samples in the graph state and T is the number of time points.

Return type:

tuple of numpy.ndarray

make_resampled_table(times, *, ra=None, dec=None, filter=None, match_filter=False, seed=None, **kwargs)[source]

Create a new ObsTable object that is resampled to the given times (and optionally positions and filters). All other columns, including noise data, are sampled from the existing table.

By default the sampled rows are drawn from the entire table. However users can force the rows to be drawn from only those matching the given filter array by setting match_filter=True.

Parameters:

times (array_like of float) – The times (in MJD) for the new observations.
ra (float or array_like of float, optional) – The right ascension(s) (in degrees) for the new observations. If a single float is provided, it is used for all times. If None is provided, the existing RA values are kept. Default: None
dec (float or array_like of float, optional) – The declination(s) (in degrees) for the new observations. If a single float is provided, it is used for all times. If None is provided, the existing Dec values are kept. Default: None
filter (str or array_like of str, optional) – The filter(s) for the new observations. If a single string is provided, it is used for all times. If None is provided, the existing filter values are kept. Default: None
match_filter (bool, optional) – If True, when sampling from the existing table, only sample observations that match the provided filter(s). This is only relevant if filter is provided. Default: False
seed (int, optional) – A random seed for the resampling. If None, a random seed is used. Most users should use None here to get different results on each call. Default: None
kwargs (dict, optional) – Additional keyword arguments to pass to the ObsTable constructor for the new object. These can overwrite any of the values from the original object.

Returns:

A new ObsTable object with the resampled observations.

Return type:

ObsTable