PyRun Documentation

Appearance

Core Concepts in PyRun ​

Understanding these fundamental concepts will help you navigate and utilize PyRun effectively.

Workspace ​

  • What it is: Your primary project environment within the PyRun web interface. It's an isolated space containing your code files, runtime definition, configuration, and execution history.
  • Think of it as: A cloud-based IDE and control center for a specific project or task.
  • Key Features: File browser, code editor, terminal access (for interactive use), Run button, monitoring tabs, Lithops configuration access.
  • Isolation: Each workspace has its own runtime definition and execution history, preventing conflicts between projects.

Runtime ​

  • What it is: The specific software environment where your Python code executes during a job run on your cloud infrastructure (AWS).
  • Defined by: Either the .pyrun/environment.yml file (for Conda/pip packages) or a .pyrun/Dockerfile within your workspace.
  • Includes: Python version, installed Python packages, and potentially system-level dependencies.
  • Managed by PyRun: You define the requirements; PyRun automatically builds the runtime (e.g., creates a Conda environment, builds a Docker image) and deploys it (e.g., as an AWS Lambda layer) for job execution.
  • Learn more: Runtime Management

Job / Execution ​

  • What it is: A single instance of running a Python script through the PyRun platform. Initiated when you click the "Run" button in a workspace or launch a Pipeline.
  • Lifecycle: Includes packaging code, building/deploying runtime (if changed), provisioning cloud resources (via Lithops/Dask), running the script, collecting logs/metrics, and tearing down resources.
  • Monitored: Each job has associated real-time monitoring data and logs accessible within PyRun.

Task (in parallel frameworks) ​

  • What it is: A smaller unit of work within a larger parallel job, typically associated with frameworks like Lithops and Dask.
  • Example (Lithops): When you use lithops.map(my_func, data_list), each invocation of my_func for an item in data_list is often considered a task, executed potentially on a separate cloud function instance.
  • Example (Dask): Dask breaks down large computations into a graph of tasks, which are then executed by Dask workers.
  • Relevance: PyRun's monitoring (especially the Gantt chart) often visualizes the execution and timing of these individual tasks within a job.

Template ​

  • What it is: A pre-configured starting point for creating a new Workspace.
  • Purpose: To quickly set up an environment tailored for a specific framework (like Lithops or Dask).
  • Includes: A basic .pyrun/environment.yml with necessary framework dependencies and a simple example.py script.
  • Usage: You select a template when creating a new workspace to jumpstart your development.
  • Learn more: Templates

Pipeline ​

  • What it is: A pre-built, ready-to-run, end-to-end example application showcasing a real-world use case within PyRun.
  • Purpose: To demonstrate PyRun's capabilities, teach best practices, and allow users to quickly see results without writing code initially.
  • Includes: Complete code, pre-defined runtime, potential sample data, and automated execution setup.
  • Usage: You launch a pipeline directly from the "Pipelines" section. Often, you can explore the underlying workspace code after execution.
  • Learn more: Pipelines