Pipelines: Real-World PyRun Examples 🧪
PyRun Pipelines are ready-to-run, end-to-end examples demonstrating practical applications of PyRun with frameworks like Lithops and Dask. Instead of building from scratch, launch a pipeline to see PyRun solve a real-world problem, then explore the underlying code and configuration. Pipelines are excellent for learning best practices and discovering PyRun's capabilities.
What are PyRun Pipelines?
Each pipeline represents a complete, pre-configured project:
- Defined Use Case: Addresses a specific task (e.g., performance benchmarking, scientific data analysis, hyperparameter tuning).
- Pre-configured Runtime: Includes all necessary Python packages and dependencies, automatically built by PyRun.
- Ready-to-Run Code: Contains the complete script(s) needed for the task.
- Data Integration: May include sample data or connect to public data sources.
- Automated Execution: PyRun handles launching the job on your AWS account and directing you to monitoring.
- Code Access (Post-Execution): After running, you can typically access the workspace containing the pipeline's code to understand how it works.
Available Pipelines
Here's a selection of available pipelines:
| Name | Description | Framework Used | Key Concepts Shown | Data Format | Est. Complexity |
|---|---|---|---|---|---|
| FLOPS benchmark | Measures floating-point computation performance using parallel functions. | Lithops | Basic map, performance measurement | In-memory | Low |
| METASPACE Annotation | Metabolite annotation pipeline for mass spectrometry imaging data (imzML). Involves multiple processing stages. | Lithops | Multi-step workflow, complex data handling, imzML | imzML, Databases | High |
| Mandelbrot Classic | Generates the Mandelbrot set fractal using parallel computation across a defined space. | Lithops | Parallel computation, image generation (map) | Numerical Grid | Medium |
| Hyperparameter Tuning | Performs grid search hyperparameter tuning for a text classification model on Amazon reviews data. | Lithops | ML preprocessing, Grid search, parallel model eval | Compressed Text | Medium |
Running a Pipeline
Navigate to Pipelines: Find "Pipelines" in the PyRun sidebar.
Browse: Review the available pipelines and their descriptions.
Select & Launch: Click the "Run" button for the desired pipeline.
Monitor: PyRun automatically initiates the execution on your AWS account and shows you the Real-Time Monitoring page for that job. Watch the logs and metrics.
Exploring Pipeline Code
- Access Workspace: Once a pipeline execution is complete (or sometimes even while running, depending on implementation), PyRun usually provides a link or way to open the associated Workspace.
- Examine Files: Inside the workspace, you'll find the Python scripts (
.py), runtime definition (.pyrun/environment.yml), and any other configuration files used by the pipeline. - Learn: Study how the code utilizes Lithops or Dask, how data is handled, and how the workflow is structured. This is a great way to learn practical techniques.
Benefits of Using Pipelines
- Learn by Doing: See concrete examples of how PyRun works for real tasks.
- Quick Evaluation: Assess PyRun's suitability for different types of problems without initial coding effort.
- Best Practices: Observe effective patterns for structuring cloud-based Python applications.
- Code Templates: Adapt code snippets or entire structures from pipelines for your own projects.
- Discover Features: See integrations like Data Cockpit or specific framework features in action.
More Pipelines Coming!
We are continuously developing and adding new pipelines covering more domains (like advanced ML, geospatial analysis, genomics) and showcasing more PyRun features and framework integrations. Check back often!
Start exploring PyRun Pipelines today and accelerate your journey into effortless cloud computing!