Knowledgebase : Data Access / Herschel Science Archive

Due to an upgrade of the TLS protocol that the current version of HIPE cannot handle, the login to HSA from HIPE has suddenly stopped working properly.

There are three different simple solutions to resolve this. Users should apply the solution that best fits their usage of HIPE:

1. Adding a line, at the time of launching HIPE:

> hipe -Dhttps.protocols=TLSv1.2

2. Adding the following to the file .hcss/hipe.props:

https.protocols=TLSv1.2

3. Adding to your scripts:

from java.lang import System
System.setProperty("https.protocols","TLSv1.2");

To establish the SAMP connection between HSA and HIPE follow the steps given in the attached video:

1. Open the HSA Interface
2. Open the SAMP Hub
3. Open HIPE
4. Send data from HSA to HIPE

Note that HIPE and any SAMP connection opened by HIPE should be closed before starting the SAMP connection with the HSA interface. 


To find out about the steps to follow in reducing SPIRE spectroscopy and, in particular, how to carry background subtraction of spectra, there is a Herschel Science Centre short videotutorial that shows you the steps involved and also describes the data products that are available for which experts have carried out background reduction already. You can find the videotutorial here:

 https://www.youtube.com/watch?v=O_pqN4VDu9M

This videotutorial describes a useful interactive script for background reduction and shows a practical example of how to carry out background subtraction and the decisions involved in carrying out background subtraction interactively, identifying and eliminating interactively the outliers in background subtraction. It also gives you details of the available expert-reduced background subtracted spectra available as a Highly Processed Data Product.

As for SPIRE FTS spectroscopy, semi-extended sources provide the user with a very special set of problems, as they are sources of a size intermediate between point sources (i.e. unresolved) and extended sources (clearly resolved), for which neither point source reduction techniques, nor extended source techniques work well.

We assume that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

Or, at least, are familiar with the contents of this short videotutorial and its associated documentation. There is a Herschel Science Centre short videotutorial that will guide you through the process of processing and analyzing semi-extended sources. You can find the short videotutorial here:

https://www.youtube.com/watch?v=PNSuxM105UY

A pre-requisite for processing semi-extended sources is to know:

1. The surface brightness distribution.

and

2. The off-set of the source from the centre of the central spaxel.

If you do not have this information, you can still attempt to use these techniques (the short videotutorial will advise you on this), but may have larger errors in your calibration. 

You can find a package of useful support materials for this short videotutorial in the Herschel Explanatory Legacy library (HELL) in the Level 2, Data reduction, PACS spectroscopy section.

This short videotutorial you will be shown how to use the useful script that was developed to deal with semi-extended sources that extracts the spectrum of the central spaxel of a re-binned cube and applies a point source correction to it. It then compares the surface brightness of a point source with the known surface brightness distribution and creates an extended-to-point correction too that is used to generate the semi-extended source correction. It will show you how to apply the models and which data products to use for different observing modes. Finally, you will be shown some examples of PACS data processed with the script.

We recommend, as background information, that you consult the following Herschel Science Centre short videotutorials too:

Point Source Spectroscopy with PACS, https://www.youtube.com/watch?v=nYWTccEQz_M

PACS Cubes in Detail, https://www.youtube.com/watch?v=CxZF9-9D8UM

Or the following guide:

Dealing with extended sources observed with PACS spectroscopy

Extended sources provide the user with a particular set of problems that require special techniques.

We assume that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

You should also watch the Herschel Science Centre short videotutorial called PACS Cubes in Detail:

https://www.youtube.com/watch?v=CxZF9-9D8UM

Or, at least, are familiar with the contents of these short videotutorials and their associated documentation. There is a Herschel Science Centre short videotutorial that will guide you through the process of processing and analyzing extended sources. You can find the short videotutorial here:

https://www.youtube.com/watch?v=j270VI0Ah1g

A pre-requisite for processing extended sources is to be familiar with the different spectroscopy products: cubes and tables that are produced when processing PACS spectroscopy.

We assume here that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

And the Herschel Science Centre short videotutorial Point Source Spectroscopy with PACS:

https://www.youtube.com/watch?v=nYWTccEQz_M

Or, at least, are familiar with the contents of both short videotutorials and their associated documentation. If you have/are, you may find that using the PACS Pointing Off-set Correction Script will improve the quality of you results. There is a Herschel Science Centre short videotutorial that will show you how to use it and teach you the background information that you need to make effective use of it. You can find the short videotutorial here:

https://www.youtube.com/watch?v=f861D8CoYqc

The Pointing Off-set Correction Script will improve the spectra of point sources that suffer from pointing off-sets, or pointing jitter, provided that the off-set is no greater than 10 arcseconds. The script measures the average pointing off-set and its variation with time (the jitter). The script works best for:

1. Sources with a flux of about 10Jy or more.

2. Sources that are off-set by no more than 4 arcseconds.

For these objects you will get a much improved continuum shape and will be able to detect broad features much more reliably.

In contrast, the script will not work for:
  • A source located at the edge of the central 3x3 spaxel box.
  • Un-chopped spectroscopy.
And it will be unreliable for faint sources.

You can find the Pointing Off-set Correction Script in the HIPE Scripts menu and must be run from Level 0: i.e. with raw data. It is fully described in the PACS Data Reduction Guide. It will produce two spectra from the extractCentralSpectrum task - the so-called c9 and the c129 spectra - of which, the one with the higher flux is the one to use. You will see several examples of spectra of sources with off-sets from 2-6 arcseconds that have been processed with and without the Pointing Off-set Correction Script to illustrate the differences that can be found.


HIFI spectra can be of two kinds: ON source spectra, in which the keyword "isLine=True) and OFF source, reference position spectra, for which isLine=FALSE. The reference position though may, itself, be contaminated by background emission, compromising the reduced on-source spectrum. The best way to eliminate or, at least, to assess this effect is to examine the OFF source spectra. The Herschel Science Centre has produced a short videotutorial that explains how to examine the OFF spectra. You can find this videotutorial here:

https://www.youtube.com/watch?v=Tpte_3LTL-s

OFF spectra can be different things for different observing modes. The short videotutorial will show you what the OFF spectrum represents for each mode and will take you through a practical example of how to examine and plot the OFF spectrum in the observation context. It will also show you the effects of line contamination in an OFF spectrum when it is used for background subtraction.

Both point source photometry - from source fitting - and aperture photometry can be extracted from SPIRE maps.

If you prefer to obtain aperture photometry from SPIRE maps, you should check the appropriate Herschel Science Centre short videotutorial for this method instead (https://www.youtube.com/watch?v=fjzf-VN_X0s).

Point source photometry can be obtained from SPIRE maps either for a single source, or by using a source detection routine, such as Sussextractor to give you a list of detected sources in a field as input to extract photometry for multiple sources.

There Herschel Science Centre has produced a short videotutorial on point source photometry, which reviews the methods of source extraction for single and multiple sources in a field, how to use the source extraction as input for photometry and which are the best photometric methods to use with your data.

You can find this short videotutorial here:

https://www.youtube.com/watch?v=nbN6OWtWIR0

The Herschel Science Centre has produced a useful short videotutorial that shows you how to use a useful script in HIPE for SPIRE line fiiting (NB: it does not cover the fully interactive mode of line-fitting). You can find the short videotutorial here:

https://www.youtube.com/watch?v=OYmJzolPF00

The short videotutorial covers the Instrumental Line Shape (ILS) of the SPIRE spectrometer, what preliminary processing of spectra is required, the inputs and outputs of the line-fitting script, and how the script works. Examples are shown of fitting lines to real SPIRE Fourier Transform Spectrometer data, how to inspect the residuals of the fit and to add missing lines in the fitted spectrum. Finally, it will introduce the Spectral Feature Finder Catalogue and how to use it as input for the line fitting script, running a spectrum fit with Spectral feature catalogue.

We have prepared a short videotutorial on Inspecting Data Products in HIPE. This will explain to you the different types of data products that you can find and their content and applicability. You can find it here:

https://www.youtube.com/watch?v=Y7LJK0mE_3I


The HIFI Observation Context consists of six categories of products that are found in the Herschel Science Archive (data at various levels of processing, browse products. calibration products, auxiliary products, quality products and trend analysis products). There is a Herschel Science Centre short videotutorial that explains the products and how to access them. You can find the short videotutorial here:

https://www.youtube.com/watch?v=GlRiqfzvFvk

The short videotutorial will introduce you briefly to the various types of observation context before showing a demonstration of how to examine the Observation Context in HIPE using a HIFI Mapping observation.

The Herschel Science Centre has produced a short videotutorial guide for users of SPIRE spectrometry data to show how to use the observation context and where to find it.

https://www.youtube.com/watch?v=D66Yxz8waVI&t=413s

This short videotutorial gives an introduction to the Observational Context and how to look at it in the Herschel Science Archive. It describes the two types of products at level-2, for Sparse Mode and Mapping Mode. It looks at the Sparse Mode High+Low resolution mode and the detectors and continues with the Mapping Mode, describing the cube structure and the Spectrum2d table. It then looks at the tar Archive file, how to inspect the Observational Context in HIPE and how to use HIPE with data cubes.


If you are an absolutely beginner with Herschel data and want to learn how to do photometry on PACS observations there are two places to start before working with data.

First, you should look at the Quick Start Guide for Photometry (CSG), otherwise known as the PACS Data Reduction Launchpad: Photometry. This will teach you some of the basics of PACS data analysis. It will also guide you towards more detailed documentation that will explore photometry in more depth.

Second, you should consult the Herschel Science Centre short videotutorial on Starting with PACS Photometry. You can find it here:

https://www.youtube.com/watch?v=yLHLA5Jjtiw

The short videotutorial will talk you through the PACS photometry data products that are available in the Herschel Science Archive (HSA) and which products are best for the analysis of what type of object. You will be shown the structure of a PACS photometry observation when downloaded from the HSA and how the filename structure of a product tells you what type of product it is. You will be shown also what Highly Processed Data Products (HPDPs) - also known as "Expert Reduced Data Products" - are available and how they may be used.

There are two excellent starting points for learning how to use PACS spectroscopy data. We recommend strongly that you start by reading the Quick Start Guide to Spectroscopy (also known as the PACS Data Reduction Launchpad: Spectroscopy), which summarises the PACS spectroscopy observing modes, data products, calibration uncertainties and science-readiness of the different PACS spectroscopy products.

There is also a Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy, which will walk you through the basics of PACS Spectroscopy. You can find the short videotutorial here:

https://www.youtube.com/watch?v=rcY487Z9Kyg

This short videotutorial will explain the three basic observing modes for PACS spectroscopy and their uses, how to determine from the Herschel Science Archive (HSA), or from the FITs header in which observing mode an observation was made, detailing the main keywords that users should be familiar with. For each product, you will find out which data products you need and what products are available at different levels for the different observing modes. You will learn what the PACS cubes are and their contents for different modes, PACS data tables and so-called slices for each mode, which are the different presentations of the cubes and tables. You will learn what a PACS spectroscopy observation looks like in the HSA and what post-processing (if any) is required for sources of different sizes. Finally, you will see which Highly Processed Data Products (Expert Reduced Data Products) have been prepared by the experts at the Herschel Science Centre for particularly difficult cases.


If your observation is a spectral map, the products are 3-D spectral cubes and not 1-D spectra. This makes saving data more difficult, as no tools were implemented in HIPE to save a 3-D structure as a text file.

So, unlike for single point spectra, you cannot you cannot use the "Save to text file" option on a spectral cube. So, first you must extract a 1-D spectrum from a pre-selected cube pixel (aka spaxel).

The Herschel Science Centre has created a short videotutorial that explains how to do this. You can find it on the Herschel Academy YouTube channel at:

Saving a 3-D SPIRE FTS spectral cube as a 1D spectrum


The HSC has created a short videotutorial (6 minutes) on how to save an FTS spectrum to a csv file. It's available on the Herschel Academy YouTube channel on the following link:
 
Saving SPIRE FTS spectra as 1D CSV or ASCII files 

 
The HIPE script from the video is copied below. To run it, you need to be logged in to the HSA.

### example HIPE script
obsid =
1342268315
obs = getObservation(obsid,useHsa=True,instrument='SPIRE')
#
point_spectrum = obs.level2.getProduct("HR_spectrum_point")
ext_spectrum = obs.level2.getProduct("HR_spectrum_ext")
#
asciiTableWriter(table=point_spectrum["0000"]["SSWD4"], file='/Users/ivaltcha/test_spec_sswd4.csv')
asciiTableWriter(table=point_spectrum["0000"]["SLWC3"], file='/Users/ivaltcha/test_spec_slwc3.csv')
#### end script

For a simple ASCII file you can also view your spectrum in the Editor window in the HSA. Right click on the spectrum that you wish to save and select "Send to" --> "Text file" in the pop-up menu. These instructions are also shown in the short videotutorial referenced above.

The Herschel Science Archive (HSA) allows multiple categories of searches to be carried out using different criteria. In particular:
  • Search by target name.
  • Search by coordinates (in a radius around a particular position).
  • Search by catalogue (for example, in the Herschel PACS, or Herschel SPIRE Point Source Catalogues, or for data published as a Highly Processed Data Product, or... etc.)
  • Search for unpublished observations.
The Herschel Science Centre has produced a short videotutorial that guides you through the different methods of search in the HSA. It guides you through the use of the HSA Search Bar and how to use the three, individual tabs for specific searches: the search tab, the catalogues tab and the publications tab, as well as some of the more advanced features available when carrying out searches to tailor your results.

You can find the short videotutorial here:

https://www.youtube.com/watch?v=sHWQqp2zGR4&feature=youtu.be


Assuming that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

And the Herschel Science Centre short videotutorial Point Source Spectroscopy with PACS:

https://www.youtube.com/watch?v=nYWTccEQz_M

Or, at least, are familiar with the contents of both short videotutorials and their associated documentation, you will find that the Herschel Science Centre short videotutorial on Point source spectroscopy with PACS: point sources in the central 3x3 spaxels, which you can find here:

https://www.youtube.com/watch?v=v1W5pT9DYUk

will show you how to proceed if your point source is in the central 3x3 spaxels, but not in the central spaxel itself. This case is valid if you have a point source, with a pointed observation, in which the target is centred within one of the eight spaxels surrounding the central spaxel, or for a mapping observation, in which, for one of the raster positions, the point source is centred within one of the eight spaxels surrounding the central spaxel. You will learn to identify cases for which the process may not work well (contamination, poorly centred point sources, ...) You will learn how to identify the spaxel coordinates of a source and which HIPE scripts and tasks to apply in the different possible cases. It will also show you a "last resort" case that you can use if your source is not even centred in the central 3x3 spaxels. You will also be guided towards the other processing and post-processing that may be helpful, such as the Pointing Offset Correction pipeline, or the Extended-to-point Correction.


Assuming that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

And the Herschel Science Centre short videotutorial Point Source Spectroscopy with PACS:

https://www.youtube.com/watch?v=nYWTccEQz_M

Or, at least, are familiar with the contents of both short videotutorials and their associated documentation, you will find that the Herschel Science Centre short videotutorial on Point source spectroscopy with PACS: point sources in the central spaxel, which you can find here:

https://www.youtube.com/watch?v=_cVncMct3NE

will show you how to proceed if your point source is in the central spaxel itself. The short videotutorial will show you how to use the Product Decision Tree to decide which products and tools to use. This case is valid if you have a point source, with a pointed observation, in which the target is centred within the central spaxel, or for a mapping observation, in which, for one of the raster positions, the point source is centred within one of the eight spaxels surrounding the central spaxel. You will learn to identify cases for which the process may not work well (contamination, poorly centred point sources, ...) You will learn how to identify the spaxel coordinates of a source and which HIPE scripts and tasks to apply in the different possible cases. 

The short videotutorial will show you how to use the extractCentralSpectrum task, which is designed for these cases, how it operates and the corrections that it applies. It will show you how to decide which extraction to apply to the data (c1 - extracts from just the central spaxel, c9 - extracts from the central 3x3 (i.e. 9) spaxels , or c129 - which extracts both and scales the central spaxel to the central 3x3 spaxel box), given the various possible combinations of source flux and centering, although as a general guide, whichever extraction gives the highest flux is the best one to use.

You will see examples of extractions of data showing you what the different extractions show and how to select the most adequate one for you, as well as simple Quality Control checks to apply on the results. It will also show you what other documentation and scripts may be useful to you, as well as giving you indications on how you can do these analysis outside HIPE too.

 

Name searches in the HSA only work by cross-referencing with either the SIMBAD database of (mainly) stellar objects or the NED database of extragalactic objects that both, logically, exclude Solar System Objects. These must be identified using their NAIF ID, selecting the “NAIF ID” tab, as there is no name-searcher available to the HSA for Solar System Objects.

For comets and asteroids, the NAIF ID is a unique, 7-digit identifier. The NAIF ID for comets has the format 100****, while for numbered asteroids, it is 2******, with the digits following the leading “2” being the asteroid number. A few objects observed by Herschel were un-numbered asteroids with lower-quality orbits, which have the format 3******. You may also input a text file with a list of NAIF IDs to check.

The HSA uses wild-cards automatically. If you enter 100 and press Search, you will obtain a listing of all comet observations.

For planets and their satellites, the NAIF ID is 3-digits starting with the planet number in order of distance from the Sun (Mars = 4, Jupiter = 5, etc). The planet itself is 499, 599, 699, … while the satellites are the planet number followed by the satellite number so, Titan, or Saturn VI, is 606, while Callisto, Jupiter IV, is 504.

To return the NAIF ID for any named Solar System Object, enter the following line in a browser:

https://ssd.jpl.nasa.gov/x/smb_spk.cgi?OPTION=Look+up&OBJECT=ceres

Just replace the name "ceres" with the solar system object that you wish to identify (name - if named [e.g. Haumea], number [e.g. 2000], or provisional designation [e.g. 1950 DA]).

The Herschel Science Centre has produced a useful short videotutorial, How to use the Herschel Science Archive User Interface, to show new users how to access and use the Herschel Science Archive and about the data products that are available within it. You can find this short videotutorial here:

http://youtu.be/IgGNpUgubkw

Answer supplied by Eva Verdugo:

The Herschel Science Archive includes in the “Basic Search” panel a field called "Access Status" that allows searching by "Public", or by "Restricted" observations. Some observations were taken in non-standard modes, or with special set-ups only used for calibration and engineering purposes and their access is therefore restricted by default to Herschel experts only. If for any reason you still think you need to gain access to any of these observations please raise a ticket at this HSC Helpdesk indicating the reason and we will assist you providing technical advice on the additional caveats you need to consider and the limitations affecting the observation for any potential use.

 

 



We have prepared a short videotutorial to explain to users how to use the Herschel Science Archive User Interface. You can find it here:

http://youtu.be/IgGNpUgubkw

Assuming that you have watched the Herschel Science Centre short videotutorial called Starting with PACS Spectroscopy:

https://www.youtube.com/watch?v=rcY487Z9Kyg 

And the Herschel Science Centre short videotutorial Point Source Spectroscopy with PACS:

https://www.youtube.com/watch?v=nYWTccEQz_M

Or, at least, are familiar with the contents of both short videotutorials and their associated documentation, you will find that the Herschel Science Centre short videotutorial on PACS Cubes in Detail here:

https://www.youtube.com/watch?v=CxZF9-9D8UM

The PACS spectrometer produces five types of data cube and one table as products, although not all types of data have all types of cube. You will learn about each type of cube (re-binned cubes, mosaic cubes - including interpolated, projected, drizzled equidistant mosaic cubes), how and why it is produced and what data it contains, as well as the re-binned cube tables. You will learn also what useful scripts are available to transform cubes to a product of your own personal specifications (e.g. one that has spaxels of a different size).


Ancillary Data Products (ADPs) are data products that correspond to an instrument or too the observatory rather than to a single science observation. These include a wide variety of engineering and diagnostic data. The Herschel Science Centre has produced a short videotutorial with an overview of the many HIFI ADPs that are available. You can find the short videotutorial here:

https://www.youtube.com/watch?v=_PTe-SkVVrc

The HIFI ADPs include the PSF for the instrument, the prime calibrator models (Mars), plus a large quantity of engineering, trend analysis and quality data. Examples of the ADPs and the data that they contain are shown in the short videotutorial, such as the HIFI 2D beam model, the azimuthally averaged profile and the Encircled Energy Fraction and the gas cell spectra from pre-launch.


Highly processed data products are sets of data that have been reduced by experts working for the Herschel Science Centre because they have issues that makes them extremely difficult to reduce optimally by automatic pipeline, or even by a non-expert in interactive mode. The Herschel Science Centre has produced a short videotutorial that gives a guided tour of the different HIFI Highly Processed Data Products (HPDPs) that are available. You can find the short videotutorial here:

https://www.youtube.com/watch?v=AV9HwZYQt_I

The short videotutorial will introduce you to the types of HIFI HPDPs that are available, before showing you the scope and content of each HPDP dataset, with a description and examples of  the products that it contains: spectral scans, spectral maps, off spectra (reference position spectra), the HIFI lines catalogue, and flux uncertainty budget tables.

The Herschel Science Archive (HSA) provides a large amount of information in response to a search for observations of a particular target. In particular:
  • How many observations there are in the HSA for this target, their status and quality.
  • What publications have been produced from each observation.
  • What User Provided Data Products are available for each observation.
  • What Highly Processed Data Products are available for each observation and how these compare with the standard pipeline products.
  • How to filter searches to obtain more, or less information.
The Herschel Science Centre has produced a short videotutorial that guides you through the results of searchs in the HSA. It guides you through the use of the HSA tabs and how to use the  individual tabs to obtain specific information about individual observations.

You can find the short videotutorial here:

https://www.youtube.com/watch?v=sHWQqp2zGR4&feature=youtu.be