PyRun Documentation

Appearance

Roadmap: The Future of PyRun 🛣️ ​

PyRun is rapidly evolving. We have a clear vision for the future, focused on expanding capabilities, enhancing performance, broadening accessibility, and building a powerful, user-friendly platform.

This roadmap outlines our planned directions and key areas of focus. Please note that this is a living document; priorities may shift based on technological advancements and, most importantly, your feedback!

Key Focus Areas & Future Directions ​

1. Platform Foundation & Performance ​

Our top priority is ensuring PyRun is robust, reliable, and efficient.

  • Stability Enhancements: Continuously improving the core platform's stability and resilience to ensure smooth and dependable operation.
  • Performance Optimization: Identifying and eliminating bottlenecks in job execution, runtime builds, and the overall platform responsiveness. We aim to make PyRun even faster and more resource-efficient.
  • Scalability Improvements: Ensuring PyRun can handle a growing number of users, larger workloads, and more complex scenarios gracefully.
  • Security Enhancements: Implementing best practices and adding security features to protect user code, data, and cloud resources.

2. Expanded Cloud Provider Support ​

We understand that flexibility is key. While PyRun currently integrates tightly with AWS, we plan to bring its power to other major cloud platforms.

  • Google Cloud Platform (GCP) Integration: Adding support for running PyRun workloads and managing resources on GCP.
  • Microsoft Azure Integration: Enabling PyRun capabilities within the Azure ecosystem.
  • Cross-Cloud Capabilities (Longer-term): Exploring ways to facilitate workflows that might span multiple cloud providers.

3. Broader Framework & Tool Ecosystem ​

PyRun is your central hub for distributed computing and data processing tools.

  • ✅ Currently Supported: Dask, Lithops, Ray, and Cubed — with first-class support for Dask and Lithops, plus ready-to-run pipelines for Ray and Cubed.
  • Coming Soon:
    • Deeper Framework Integration: Moving beyond pipelines to deliver more native, first-class support and advanced configuration for Ray and Cubed.
    • Expanded Tooling: Exploring integration with additional data processing, workflow orchestration, and scientific computing libraries based on user feedback.

4. End-to-End AI/ML Workflow Integration ​

We want PyRun to be the go-to platform for the entire AI/Machine Learning lifecycle.

PyRun is your central hub for distributed computing and data processing tools.

  • ✅ Now Supported: Foundational AI Pipelines. Initial support for AI workloads is now available, with pipelines for Audio Recognition, Image Classification, Dask-ML, and local LLM execution.
  • Coming Soon:
    • Enhanced Data Ingestion & Preparation. Streamlining the process of connecting to various data sources and performing large-scale data transformations.
    • Integrated Model Training & Tuning. Providing dedicated features for training models at scale, including hyperparameter optimization and experiment tracking.
    • Model Deployment & Serving. Adding capabilities to easily deploy trained models as scalable endpoints directly from PyRun.
    • MLOps Features. Incorporating features for a model registry and workflow orchestration tailored for ML lifecycles.
    • Expanded LLM Support: Enhancing capabilities for working with Large Language Models, including more integrations and optimized execution environments.

5. User Experience & Productivity Features ​

Making PyRun even more intuitive and powerful for everyday use.

  • Expanded Pipeline Library: Adding more diverse and complex real-world examples to the Pipelines section, covering a wider range of domains and tools.
  • Collaboration Features: Enabling teams to share workspaces, code, runtimes, and results more effectively.
  • Enhanced Monitoring & Visualization: Adding more sophisticated metrics, customizable dashboards, and visualization tools for deeper insights into job performance and resource usage.
  • Improved Data Cockpit: Expanding the capabilities of the Data Cockpit for easier data discovery, management, and integration with compute jobs.

Stay Tuned! ​

This roadmap provides a glimpse into the exciting future of PyRun. We are dedicated to making cloud computing more accessible, efficient, and powerful for everyone. Keep an eye on our Release Notes for the latest updates and feature announcements.

We value your input! Let us know what features and improvements are most important to you. GitHub Bug Report. Together, we can shape the future of PyRun!