Tutorial: developing LSST Science Pipelines packages in the Notebook Aspect¶

The Notebook Aspect combines all the facilities you need to develop for, or alongside, the LSST Science Pipelines:

A base lsst_distrib installation.
Butler repositories in /datasets.
Compilers and other developer tools.
Jupyter notebooks and console.
A fully-featured bash shell.

This tutorial shows you a workflow for developing LSST Science Pipelines packages in the Notebook Aspect.

Set up Git¶

Before you get started with any development, be sure to configure Git and set up your GitHub credentials (particularly if you will be pushing code).

Step 1: open a terminal and set up lsst_distrib¶

First open a terminal. From the file browser click the + button to open the launcher (or type command-shift-L). Then click on the Terminal icon.

Follow the onscreen instructions to activate the LSST environment:

source /opt/lsst/software/stack/loadLSST.bash
setup lsst_distrib

Step 2: clone an existing package¶

In this tutorial, you’ll add a new Task to LSST’s pipe_tasks package. First, you need to clone it from GitHub. In the same terminal, run:

cd ~/notebooks
git clone https://github.com/lsst/pipe_tasks
cd pipe_tasks

Step 3: set up the package¶

Next, you need to set up this cloned-version of pipe_tasks, replacing the version built into the Notebook Aspect’s image. In the same terminal, run:

setup -k -r .

You can see that the pipe_tasks package that’s set up is your local copy:

eups list pipe_tasks

The other packages from lsst_distrib are still set up:

eups list -s

Step 4: build the package¶

All LSST Science Pipelines packages, even pure-Python packages like pipe_tasks, need to be built before they can be imported and used. In the same terminal, run:

scons

Step 5: set up the package for notebooks¶

In Step 3 you set up the cloned pipe_tasks package for that specific terminal session. This change isn’t carried over to notebooks. Instead, you need to add this setup command to the ~/notebooks/.user_setups file.

In a terminal text editor like Vim or Emacs, create or open ~/notebooks/.user_setups and edit the file to be:

setup -k -r ~/notebooks/pipe_base

You can check that this works by opening a new notebook with the LSST kernel and running:

import lsst.pipe.tasks
print(lsst.pipe.tasks.__file__)

As you can see, the module’s path is your clone in ~/notebooks/, rather than the preinstalled package in /opt/lsst/software/stack.

Step 6: write some code¶

There’s a lot that can be done in this step, but as a demonstration we’ll create a simple Task called MyTask.

First, create a Git branch from the terminal:

git checkout -b my-task

Second, create a new file for Task at python/lsst/pipe/tasks/myTask.py (inside ~/notebooks/pipe_base) and paste these contents into it:

__all__ = ('MyTask',)

from lsst.pipe.base import Task
from lsst.pex.config import Config


class MyTask(Task):

    _DefaultName = 'MyTask'
    ConfigClass = Config

    def run(self):
        print("Running MyTask")

Step 7: run the new code in a notebook¶

Go back to the notebook and reload the kernel. Then run the task:

from lsst.pipe.tasks.myTask import MyTask
myTask = MyTask()
myTask.run()

Tip

Instead of restarting the notebook’s kernel, you can sometimes reload a module that you’ve previously imported. See the Python documentation for importlib.reload, including caveats for when this function will not work.

Step 8: cleaning up¶

At this point, you will typically use Git to commit this work and push your new branch to GitHub.

After your work is done, you will want to revert the ~/notebooks/.user_setups file so that notebooks use the LSST Science Pipelines packages built into the Notebook Aspect image, instead of your local clone. Delete any lines with setup commands you no longer need.

Summary¶

Keep these steps in mind while developing LSST Science Pipelines software in the Notebook Aspect:

In terminals:
1. Load the LSST environment (loadLSST.bash).
2. setup lsst_distrib.
3. Set up the specific package you’re developing with setup -k -r {{path}}.
4. Build the package by running scons.
For notebooks, add a setup -k -r {{path}} command for your package to ~/notebooks/.user_setups.

Navigation