DOE Workflow Workshop and Hackathon

Posted on

March 19-22, 2019, Livermore, CA

You are hereby invited to participate in the DOE workflow workshop and hackathon to be held March 19-22, 2019 in Livermore, CA.  This event is intended to bring together people in the larger DOE community who are developing and using workflow technology.  Days 1 and 2 will be the workshop followed by a 2-day workflow hackathon, where people will have the opportunity to do hands-on work with the various tools/capabilities used around the DOE.  Details for both events are provided below.

Link to 2020 DOE Workflow Workshop and Hackathon

Materials from the workshop will be posted on this site for archival purposes.  

Registration

Registration is required, as is badging for this event.  

Please find the workshop and hackathon calls for participation below.  If you plan to join us, we request that you register by sending an email to the org committee at:

doe-workflow-workshop@llnl.gov

indicate the following:

  • Which days you plan to attend the workshop (March 19 & 20).
  • Which days you plan to join us for one or more hackathon sessions (March 21 & 22).
  • Affiliation (e.g., ANL, ORNL, …)
  • Citizenship
  • Email
  • Phone

Registration Fee

  • There is no registration fee for the workshop and hackathon  
  • However, attendees are requested to bring $25/day (cash) of attendance for food and drinks, which will consist of:
    • Coffee in the morning
    • Lunch
    • Coffee and snacks in afternoon break
    • Any dietary restrictions should be reported directly to Letty

Badging

  • Badging is required for site access
  • Learn more about Badging Info
  • Badging should be done well in advance, particularly for non-US citizens

Registering for the workshop will subscribe you to an email list for news and announcements.  Alternatively, anyone may request to be added to the doe-workflow-workshop-news@llnl.gov list as follows:

Workflow Email List

We have created a mailing list that will persist after this event for purposes of sharing workflow-related info amoungst interested parties.  To subscribe:

  • Send an email to: DOE-WORKFLOW-WORKSHOP-NEWS-request@LISTSERV.LLNL.GOV
  • With the following command in the body of the email:
  • subscribe doe-workflow-workshop-news

Organizing Committee (doe-workflow-workshop@llnl.gov)

  • Jay Billings, ORNL
  • Julien Capul, CEA
  • Robert Clay, SNL
  • Dan Laney, LLNL
  • Dave Montoya, LANL

Local POCs

Administrative POC

Special NIF Tour (CANCELED DUE TO SHOT MONDAY NOON)

We regret to inform those who had signed up for the NIF tour that the tour was canceled due to a “high yield” shot which is now scheduled for noon Monday.  

Remote Connection

By popular demand from a small group, we are providing a means to Skype into the workshop event.  We will try to get the Skype online by 8:30AM both days, so you can bring up your system in advance to work out the kinks if you’re not a regular Skype user. 

………………………………………………………………………………………………………………………..

Join Skype Meeting

Trouble Joining? Try Skype Web App

Join by Phone

1-888-771-4500,,6206911#

1-925-294-4500,,6206911#

1-877-720-1159,,6206911#

1-505-844-5300,,6206911#

Find a local number

Conference ID: 6206911

Forgot your dial-in PIN? | First online meeting?

………………………………………………………………………………………………………………………..

DOE Workflow Workshop – Tues-Weds March 19-20, 2019, Livermore, CA

Date: Tues-Weds, March 19-20, 2019, 9:00-17:00 (PST)

Location: Bldg 904 Auditorium, Sandia National Labs, Livermore, CA

Call for Participation

The goal of this workshop is to bring together users of workflows and those developing workflow tools in the broader DOE community to update one another on the state of practice, present emerging technologies and tools, and to enable new collaborations in the workflow area.  All topics related to workflow are welcome, including but not limited to:

  • Workflow management systems
  • Data Management [simulation and experimental]
  • Application of workflow systems to HPC use-cases
  • Resource management and scheduling
  • Containers and containerized computing as they apply to workflow systems

If you are interested in presenting at the workshop send a title, abstract, and bio to doe-workflow-workshop@llnl.gov. Speakers will be notified of acceptance on or before February 25th.   Talks will be limited to 30 minutes, including Q&A.

In addition to the scheduled talks, we will provide time on day 2 for 5-minute lightening talks, timeslots for which will be first-come-first-serve on sign up sheet provided day 1.  

Workshop Agenda

Tuesday March 19, 2019

Opening Session (09:00-10:30)

  • 09:00 Welcome and Introduction
  • 09:15 Keynote: G.E. Orient – Workflow Platform Foundational to Authoring, Communicating and Executing Credible Computational Models
  • 10:00 S. Jha – Toward Common Components for Open Workflow Systems: The RADICAL-Cybertools Experience

Break (10:30-11:00)

Workflow Systems (11:00-12:30)

  • 11:00 E. Friedman-Hill – The Sandia Analysis Workbench Next Generation Workflow System
  • 11:30 J.M. Wozniak – Swift/T: An HPC-oriented Workflow Tool
  • 12:00 K. Chard – Parsl: Pervasive Parallel Programming in Python

Lunch (12:30-14:00) [GROUP PHOTO OUT FRONT OF BLDG 904]

Workflow Systems and In-transit Analysis (14:00-15:00)

  • 14:00 J. Koning – Merlin Workflow: Machine Learning Driven Ensembles
  • 14:30 J. Capul – Executing HPC workflows in-transit with PaDaWAn and Redis

Mini Break (15:00-15:10)

Workflow Tools (15:10-16:10)

  • 15:10 P.P. Pébaÿ – The Automatic Report Generator (ARG)
  • 15:40 T. Dahlgren – Sina: A Gateway to Modern Simulation Data Management

Panel: Day 1 Speakers (16:10-16:50)

Day 1 Wrap Up (16:50-17:00)

Day 1 Adjourned (17:00)

No-host Dinner at Uncle Yu’s in downtown Livermore (18:00-?)

Wednesday March 20, 2019

Opening Session (09:00-10:30)

  • 09:00 Welcome and Introduction
  • 09:15 Keynote: H. Krishnan – Streaming Realtime Workflows at the Light Sources
  • 10:00 C. Snavely – Complex Workflows at Scale: the NERSC Superfacility Project

Break (10:30-11:00)

Workflow Container Solutions (11:00-12:30)

  • 11:00 D. Montoya – Opening Remarks
  • 11:10 S. Canon – Containers at NERSC: Shifter for HPC and Spin for Services
  • 11:30 T. Stirm – ACE’s Genesis
  • 11:50 J. Kincl – Hosting Workflow Systems on Kubernetes at OLCF
  • 12:10 P. Grubel – Containerized Application Workflows using the Build and Execution Environment (BEE)

Lunch (12:30-14:00)

Workflow Management Systems (14:00-15:30)

  • 14:00 T. Patki – Flux: Using Next-Generation Resource Management and Scheduling Infrastructure for Exascale Workflows
  • 14:30 M. Salim – Balsam: Simplifying Large-scale Science Campaigns on a Supercomputer
  • 15:00 Lightening Talks – sign up sheet out front 

Break (15:30-16:00)

Panel: Day 2 Speakers (16:00-16:50)

Day 2 Wrap Up (16:50-17:00)

Workshop Adjourned (17:00) 

DOE Workflow Hackathon – Thurs-Fri March 21-22, 2019, Livermore, CA

Dates:Thurs-Friday, March 21-22, 2019 (PST), 9:00-17:00 (PST)

Location: Sandia National Labs, Livermore, CA (rooms to be announced later)

Call for Participation

Persons interested in organizing a session are invited to send a title and abstract describing their proposed session(s) to doe-workflow-workshop@llnl.gov.  Session organizers will be expected to build the necessary software stacks and prepare materials as needed in advance of the hackathon.  An open-network Linux server will be provided, or session organizers are welcome to host their own.  Hackathon participants will be provided network access, and are expected to bring their own laptops with Unix-type OS (i.e., Linux or Mac OS) and tested applications they would like to integrate into workflows.  Participants must send an email to doe-workflow-workshop@llnl.govto request accounts on the open server (accounts on the open server will be limited to US citizens only). Session organizers will be given early access to ensure smooth operation of their software stack.

Description and Goals

  • Bring together people actively developing and/or using workflow tools in the community.
  • Provide an opportunity for application developers and users to interact with authors and experts of workflow tools.
  • Provide an opportunity to get hands-on experience with the current workflow tools/capabilities guided by expert developers and users.

Draft Agenda

  • Thurs March 21, 2019, two 3-hour sessions (9:00-12:00, 13:30 – 16:30) (PST)
  • Friday March 22, 2019, two 3-hour sessions (9:00-12:00, 13:30 – 16:30) (PST)
  • Thurs & Friday, Daily wrap up & open discussion, 16:30-17:00
  • Friday 17:00 Event Adjuourned

Proposed Hackathon Topics

  • LLNL – data management tools + Cassandra
  • CEA – in-situ workflow infrastructure (PaDaWAn)
  • SNL – next-gen workflow (NGW) (getting started wtith NGW)
  • SNL – credibility workflow (NGW) (building and running complex workflows)
  • NGA – automatic report generation (ARG)
  • LANL/NERSC – Containers in workflow, including BEE / Charliecloud and Shifter
  • ANL – Swift/T
  • LANL – Expanding GitLab CI to support workflow

The final schedule for hackathon sessions will be determined based on community interest and response.  All topics listed above will be accommodated if there is sufficient interest for the session.