The Jupyter tool is a useful place to develop code and analyses in a notebook style. Hub users can easily share their notebooks with other users by publishing notebooks as tools. A published Jupyter notebook enables other users to interact with the notebook, stepping through its cells and even changing them. But, when users run your published notebook, any changes they make to it will not persist.
This set of instructions takes you through publishing a Hub tool based on your existing Jupyter notebook. Here, we'll assume that the short name for your tool is toolname. To develop the notebook tool, all you need is access to Jupyter. You'll navigate between your Jupyter home directory, the Jupyter terminal, and your tool's status page from the Tool Pipeline.
These instructions do not assume great familiarity with the Hubzero tool deployment process. However, you can learn more about deploying tools by referring to the other KB articles on tool deployment, including: Tool Development Workflow
Jupyter version
Note that if you are developing Jupyter notebooks or Jupyter based tools you should use the most recent version of Jupyter, jupyter6deb10. The same holds if you are making substantial changes to an already-published notebook-based tool: Ensure you are using the Jupyter on Deb 10 tool.
To deploy a Jupyter notebook:
1. CREATE THE TOOL
To create the tool for your Jupyter notebook, navigate to Tools and click "Create a New Tool" on the upper left. Fill in the Create Tool page that the system displays:
- Give your tool a brief name (no spaces or hyphens), a full title, and the at-a-glance description.
- Select "Deploy as Jupyter notebook", and add your username in the development team.
- The Access section enables you to restrict tool access to a specific Hub Group, if you wish.
- For other fields, you may accept the defaults, and submit.
- Finally, flip the tool status to Registered, and click Apply Change.
1B. Register as a Debian10 tool
The final step of tool registration: submit a ticket on this gateway, indicating the short name of your tool, and asking that it be registered as a Debian10 tool. This will ensure that the new tool uses current packages and kernels.
2. ADD THE TOOL REPO
Next we may need to add the code repository that will contain the code for your Jupyter notebook tool. If the Add Repo button is now available to you in your tool status page:
- Click the Add Repo button.
- Then, if the page shows a success message, flip the status to Created and click Apply Change.
Otherwise, an administrator must add the repo for you, so you should wait for a status email indicating the repo has been created.
3. CHECK OUT THE TOOL REPO
Now the new notebook tool's code repo is Created and ready to use. To do so, we must first check out the repo.
Open the Jupyter tool, navigate to your home notebooks directory, and open a terminal by selecting New, and then Terminal. Using the terminal, check out the newly created tool repo locally using this command (toolname is the brief name you gave your tool on the Create Tool page):
4. ADD NOTEBOOK CODE
It's now time to add the code that will run for your notebook. Back in the Jupyter tool file listing, you should see the toolname directory under your home notebooks directory. Into that directory, copy a working notebook (or develop one in place).
You can configure your notebook to access additional Python packages by loading an alternate kernel in the Jupyter notebook UI. Kernels are available now with packages such as xarray, cdo, and others. File a ticket or get in touch to let us know what packages you need.
import data.pythonfile
5. EDIT INVOKE SCRIPT
Finally, to tell the hub how to launch the notebook, you need to edit the invoke script that was automatically created at tool creation time. This is also done from the Jupyter file listing. The invoke shell script is found in the toolname/middleware/ directory. To edit it, double-click on the invoke script and the editor will launch.
In the invoke script you will specify the filename of your Jupyter notebook, the version of Anaconda you are using (presently, use Python 3, and specify anaconda-6), and other parameters. Here we suppose that your notebook is called your-jupyter-notebook-name.ipynb.
A basic script should look like the following for a Jupyter notebook with Python 3 (Use your own notebook's name in the call):
#!/bin/sh /usr/bin/invoke_app "$@" -C "start_jupyter -T @tool your-jupyter-notebook-name.ipynb" -r none -u anaconda-6
If your notebook needs additional modules, list them at the end after the -u.
For details on invoke script command line options, refer to the Hubzero invoke documentation.
6. TESTING
Next, you'll test that your working notebook starts properly as a Hub tool. Refer to the Knowledge Base article, Testing Jupyter Notebooks to see how to verify that your notebook works in the tool context. When the notebook passes testing, you are ready to proceed.
7. COMMIT CHANGES
Once you have saved your invoke script and your notebook, check them in to the repository management software, subversion. To do so: From a Jupyter terminal, navigate to your tool's directory (get there as we did in step 3. above). First, add the notebook to svn (similarly, add any other needed files, using "svn add");
svn add your-jupyter-notebook-name.ipynb
svn commit
Subversion will now prompt you for a commit message. Type in a suitable initial commit message ("initial commit of toolname" will do) and press ctrl-X to exit.
To alert the administrator that your tool is ready for installation, you can now visit your tool's status page, either from the Tool Pipeline, or specifying a URL like this:
https://vhub.org/tools/toolname/status
Here, click the link that reads, "My code is committed, working, and ready to be installed." If you have special instructions, caveats, compile steps, or other dependencies for your installation, enter them in the available text box now. The tool administrators will be alerted about your tool status and perform the installation along with any required steps you describe.
8. INSTALL
It's time to install the tool source. This action will depend on your access privileges; you may need the help of an administrator. On the hub, visit your tool's status page, either from the Tool Pipeline, or specifying a URL like this:
Here you can click the Install button and then on success message, flip the status to Installed and apply the change.
If the Install button is not available to you, this task will be executed by an administrator. You will receive a status email when it is complete.
Final step for jupyter6deb10 tools: To publish the tool you'll need to let us know the tool name so we can update the docker image mapping so the tool loads in the deb10 env. To do this, enter a hub ticket letting us know tool name and asking for an update of the docker image mapping.
9. TEST AND PUBLISH
To test your tool, go to the Hub Tool Pipeline and select your tool's link, or specify the tool URL directly:
In the status page, click the button to test run the tool. If the tool does not display or otherwise fails your test, there is still work to do. Revisit your development steps, starting with the TEST section above.
If the notebook test is successful, and it displays and functions as expected, you are almost done! Return to the tool status page. There you can indicate to administrators that you Approve the tool for publication. If special instructions or compilation steps need to be performed for your tool, indicate that here.
Depending on your access privileges, you may be able to set the tool as Published. If the link is available to you, click Publish, and then on display of the green success message, flip the status to Published and apply the change.
You will receive a status change email when the tool has transitioned to Published. When you receive word that your tool is Published, you should verify again that it works as expected.
That should do it--your Jupyter notebook is now published and available on the Hub as a tool. If you have questions, concerns, or run into a snag, please email the Hub administrator (include all error messages as available) and we'll give you a hand.
10. MAKING CHANGES
To make changes to a published notebook, you must only revisit some of the steps outlined above.
If you are planning to edit an existing gateway tool, please convert it into a Debian10 tool first (See 1B. above). To do so, submit a ticket on this gateway, indicating the short name of your tool, and asking that it be registered as a Debian10 tool. This will ensure that the tool has access to current packages and kernels. Hubzero staff will let you know when the change has been made.
Next, to make edits to the tool:
- Change your notebook code as necessary, revisiting the TEST and COMMIT CHANGES steps above when complete.
- INSTALL your changed code as above
- TEST AND PUBLISH the notebook tool as above