PyRun Documentation

Appearance

Roadmap: The Future of PyRun 🛣️ ​

PyRun v1.0.0 marks the beginning of our journey to simplify cloud computing for Python users. We have a clear vision for the future, focused on expanding capabilities, enhancing performance, broadening accessibility, and building a powerful, user-friendly platform.

This roadmap outlines our planned directions and key areas of focus. Please note that this is a living document; priorities may shift based on technological advancements and, most importantly, your feedback!

Key Focus Areas & Future Directions ​

1. Platform Foundation & Performance ​

Our top priority is ensuring PyRun is robust, reliable, and efficient.

  • Stability Enhancements: Continuously improving the core platform's stability and resilience to ensure smooth and dependable operation.
  • Performance Optimization: Identifying and eliminating bottlenecks in job execution, runtime builds, and the overall platform responsiveness. We aim to make PyRun even faster and more resource-efficient.
  • Scalability Improvements: Ensuring PyRun can handle a growing number of users, larger workloads, and more complex scenarios gracefully.
  • Security Enhancements: Implementing best practices and adding security features to protect user code, data, and cloud resources.

2. Expanded Cloud Provider Support ​

We understand that flexibility is key. While PyRun currently integrates tightly with AWS, we plan to bring its power to other major cloud platforms.

  • Google Cloud Platform (GCP) Integration: Adding support for running PyRun workloads and managing resources on GCP.
  • Microsoft Azure Integration: Enabling PyRun capabilities within the Azure ecosystem.
  • Cross-Cloud Capabilities (Longer-term): Exploring ways to facilitate workflows that might span multiple cloud providers.

3. Broader Framework & Tool Ecosystem ​

PyRun aims to be your central hub for various distributed computing and data processing tools.

  • Ray Integration: Adding native support for Ray, a popular framework for building distributed applications, particularly in AI/ML.
  • Cube Integration: Exploring integration with Cubed, a newer framework focused on efficient processing of large N-dimensional arrays using serverless functions.
  • Additional Tooling: Investigating support for other relevant data processing, workflow orchestration, and scientific computing libraries based on user demand.

4. End-to-End AI/ML Workflow Integration ​

We want PyRun to be the go-to platform for the entire AI/Machine Learning lifecycle.

  • Enhanced Data Ingestion & Preparation: Streamlining the process of connecting to various data sources and performing large-scale data transformations within PyRun, potentially leveraging frameworks like Dask, Ray, or Lithops more deeply.
  • Integrated Model Training & Tuning: Providing dedicated features and optimizations for training machine learning models at scale, including support for popular ML libraries (TensorFlow, PyTorch, scikit-learn) and hyperparameter optimization techniques.
  • Model Deployment & Serving: Adding capabilities to easily deploy trained models as scalable endpoints directly from PyRun.
  • MLOps Features: Incorporating features for experiment tracking, model registry, and workflow orchestration tailored for ML lifecycles.

5. User Experience & Productivity Features ​

Making PyRun even more intuitive and powerful for everyday use.

  • Expanded Pipeline Library: Adding more diverse and complex real-world examples to the Pipelines section, covering a wider range of domains and tools.
  • Collaboration Features: Enabling teams to share workspaces, code, runtimes, and results more effectively.
  • Enhanced Monitoring & Visualization: Adding more sophisticated metrics, customizable dashboards, and visualization tools for deeper insights into job performance and resource usage.
  • Improved Data Cockpit: Expanding the capabilities of the Data Cockpit for easier data discovery, management, and integration with compute jobs.

Stay Tuned! ​

This roadmap provides a glimpse into the exciting future of PyRun. We are dedicated to making cloud computing more accessible, efficient, and powerful for everyone. Keep an eye on our Release Notes for the latest updates and feature announcements.

We value your input! Let us know what features and improvements are most important to you. GitHub Bug Report. Together, we can shape the future of PyRun!