Source Catalogs¶
Galaxy Catalog Format¶
The input catalog is a text file (or an equivalent FITS table) consisting of an initial header line and followed by one line of numbers per catalog object, for example a catalog with two objects might look like this:
galtileid ra dec redshift fluxnorm_bulge fluxnorm_disk fluxnorm_agn a_b a_d b_b b_d pa_bulge pa_disk u_ab g_ab r_ab i_ab z_ab y_ab
2200871446 0.418319702147 -0.000148399994941 0.496377289295 0.0 1.4144730572e-17 0.0 0.0 0.278649687767 0.0 0.221303001046 0.0 307.344329834 25.9418621063 25.129743576 23.9588813782 23.3607368469 23.0723800659 22.9095973969
2205921112 0.420028448104 -0.00100259995088 1.89508104324 0.0 1.91501907101e-18 0.0 0.0 0.358063697815 0.0 0.313674807549 0.0 137.791702271 25.848903656 25.867565155 25.9179477692 25.9851398468 25.8779563904 25.7642536163
The header line specifies a list of names, separated by white space, associated with each catalog parameter. Each catalog entry is represented by a a list of numbers, separated by white space, giving the corresponding parameter values.
The parameter names below are required by the simulate program, but other parameters may also be present in the file. Parameters can be listed in any order as long as the order of values matches the header line. The names in the table below are a bit idiosyncratic (e.g., mixing under_scores with CamelCase and using different schemes to denote bulge vs. disk) but are chosen to match the names used in the LSST DM galaxy catalog schema.
Name | Description |
---|---|
galtileid | Unique integer identifier for this object |
ra | Object centroid right ascension (degrees) |
dec | Object centroid declination (degrees) |
redshift | Object cosmological redshift |
fluxnorm_bulge | Multiplicative scaling factor to apply to the bulge SED |
fluxnorm_disk | Multiplicative scaling factor to apply to the disk SED |
fluxnorm_agn | Multiplicative scaling factor to apply to the AGN SED |
a_b | Semi-major axis of bulge 50% isophote (arcseconds) |
b_b | Semi-minor axis of bulge 50% isophote (arcseconds) |
a_d | Semi-major axis of disk 50% isophote (arcseconds) |
b_d | Semi-minor axis of disk 50% isophote (arcseconds) |
pa_bulge | Position angle of bulge (degrees) |
pa_disk | Position angle of disk (degrees) |
u_ab | Apparent AB magnitude in the LSST u-band, including extinction effects |
g_ab | Apparent AB magnitude in the LSST g-band, including extinction effects |
r_ab | Apparent AB magnitude in the LSST r-band, including extinction effects |
i_ab | Apparent AB magnitude in the LSST i-band, including extinction effects |
z_ab | Apparent AB magnitude in the LSST z-band, including extinction effects |
y_ab | Apparent AB magnitude in the LSST y-band, including extinction effects |
The catalog file is read using a astropy.io.ascii.Basic reader (created with default options) so can be embellished with comments and blank lines for readability, as supported by that class.
Create Galaxy Catalog From LSST Catalog Database¶
This section documents the process for creating an input catalog derived from the LSST Data Management (DM) simulation database, described in these SPIE proceedings and being documented here. This is not something you will normally need (or want) to do since the resulting catalog is already available to download. However, if you want to know exactly how the catalog was created or want to create your own, read on.
Warning
In case you are using the DM galaxy database directly, you should avoid using the DiskHalfLightRadius and BulgeHalfLightRadius columns since these do not have the usual definition where hlr == sqrt(a*b) and instead satisfy hlr == a. To avoid this confusion, use the a,b parameters directly and calculate hlr == sqrt(a*b).
The dbquery.py program automates the process of connecting to the database, extracting the relevant data, and writing a catalog file. The program uses the pymsql python interface to Microsoft SQL Server (which LSST DM uses), so you will need to install that and its dependencies (FreeTDS and Cython) in order to run dbquery.py.
The OneDegSq.dat catalog file was created using:
dbquery.py -o OneDegSq.dat --dec-min -0.5 --dec-max +0.5 --ra-min -0.5 --ra-max +0.5 --verbose
with FreeTDS v0.91, Cython v0.21, pymsql v2.1.1 under OS-X 10.10.1. The program takes about 2 minutes to run and saves 858502 rows to the 200 Mbyte output text file. The set of rows returned by a query is invariant, but they are returned in an arbitrary order so the output files obtained by running this program twice can be different (but only by a permutation of lines). Note that the “1 sq.deg.” here is defined by RA and DEC intervals of 1 degree centered on (RA,DEC) = (0,0), which is not exactly 1 sq.deg. of 2D imaging because of projection effects.
The query uses a custom stored procedure that tiles the full sky by repeating the same 4deg x 4deg patch of about 14M galaxies (the catalog actually contains a 4.5deg x 4.5deg patch, but only the smaller patch is tiled). The stored routine synthesizes the returned ra, dec, and galtileid values, where:
galtileid = TTTTGGGGGGGG
is a 64-bit integer that combines the tile number TTTT (0-4049) with the unique galaxy ID GGGGGGGG (0-17,428,283). The ID that dbquery writes to its output file is galtileid. For the OneDegSq.dat example above, all galaxies are unique and located in tileid = 22 or 4027.
Note that the LSST database uses standard port assignments for its Microsoft SQL Server. However, since these ports are frequently targets of network attacks, many organizations block outbound packets to these ports from internal IP addresses. In addition, the UW hosts of the database only allow inbound packets to their SQL server from IP address ranges included in their whitelist. Therefore, if you are unable to connect, the most likely reason is packet filtering on one or both ends of your connection. One option is to use a host that has already been configured for access to the LSST catalog database (on the NCSA cluster, for example).
You might find the following interactive python snippet useful for debugging connection problems:
import _mssql
conn = _mssql.connect(server='fatboy.npl.washington.edu', user='LSST-2', password='L$$TUser', database='LSST', port=1433)
print conn.tds_version
conn.execute_query("Select name from sysobjects where type like 'u'")
for row in conn: print row['name']
conn.execute_query("select COLUMN_NAME from INFORMATION_SCHEMA.COLUMNS where TABLE_NAME = 'galaxy'")
for row in conn: print row[0]
conn.execute_scalar("select count(*) from galaxy")
conn.close()
The second line will fail with a connection error after about 30 seconds if your packets are being filtered on either end:
MSSQLDatabaseException: (20009, 'DB-Lib error message 20009, severity 9:\nUnable to connect: Adaptive Server is unavailable or does not exist\nNet-Lib error during Operation now in progress (36)\n')
Convert ASCII Catalog to FITS Table¶
The catalog file can also be read as a FITS file containing a single table. A catalog in text format can be converted to a FITS file using, for example (this will not overwrite an existing output file):
import astropy.table
catalog = astropy.table.Table.read('OneDegSq.dat',format='ascii.basic')
catalog.write('OneDegSq.fits')
The resulting FITS file will be somewhat smaller (by about 30%) than the text original and significantly faster for the simulate program to read, but may be less convenient for other programs to read or generate.