PyRun Documentation

Appearance

Roadmap: The Future of PyRun 🛣️ ​

PyRun is rapidly evolving. We have a clear vision for the future, focused on expanding capabilities, enhancing performance, broadening accessibility, and building a powerful, user-friendly platform.

This roadmap outlines our planned directions and key areas of focus. Please note that this is a living document; priorities may shift based on technological advancements and, most importantly, your feedback!

Key Focus Areas & Future Directions ​

1. Platform Foundation & Performance ​

Our top priority is ensuring PyRun is robust, reliable, and efficient.

  • Stability Enhancements: Continuously improving the core platform's stability and resilience to ensure smooth and dependable operation.
  • Performance Optimization: Identifying and eliminating bottlenecks in job execution, runtime builds, and the overall platform responsiveness. We aim to make PyRun even faster and more resource-efficient.
  • Scalability Improvements: Ensuring PyRun can handle a growing number of users, larger workloads, and more complex scenarios gracefully.
  • Security Enhancements: Implementing best practices and adding security features to protect user code, data, and cloud resources.

2. Expanded Cloud Provider Support ​

We understand that flexibility is key. While PyRun currently integrates tightly with AWS, we plan to bring its power to other major cloud platforms.

  • Google Cloud Platform (GCP) Integration: Adding support for running PyRun workloads and managing resources on GCP.
  • Microsoft Azure Integration: Enabling PyRun capabilities within the Azure ecosystem.
  • Cross-Cloud Capabilities (Longer-term): Exploring ways to facilitate workflows that might span multiple cloud providers.

3. Broader Framework & Tool Ecosystem ​

PyRun is your central hub for distributed computing and data processing tools.

  • ✅ Currently Supported: Dask, Lithops, Ray, and Cubed — with first-class support for Dask and Lithops, plus ready-to-run pipelines for Ray and Cubed.
  • Coming Soon:
    • Deeper Framework Integration: Moving beyond pipelines to deliver more native, first-class support and advanced configuration for Ray and Cubed.
    • Expanded Tooling: Exploring integration with additional data processing, workflow orchestration, and scientific computing libraries based on user feedback.

4. End-to-End AI/ML Workflow Integration ​

We want PyRun to be the go-to platform for the entire AI/Machine Learning lifecycle, from discovery to deployment.

  • ✅ Recently Shipped:

    • Integrated AI Hub: A new dedicated "AI" section in the UI.
    • HuggingFace Model Explorer: Discover and integrate models from the HuggingFace Hub directly into your workspace in just a few clicks.
    • AI Code Assistant: An intelligent assistant in the editor to accelerate AI/ML development.
    • Foundational AI Pipelines: Ready-to-run examples for Audio Recognition, Image Classification, Dask-ML, and local LLM execution.

  • Coming Soon:

    • Deeper HuggingFace Integration: Pre-built pipelines for common tasks like fine-tuning and inference with models selected from the explorer.
    • Integrated Model Training & Tuning: Dedicated features for training models at scale, including experiment tracking and hyperparameter optimization.
    • Model Deployment & Serving: Capabilities to easily deploy trained models as scalable endpoints directly from PyRun.
    • MLOps Features: Incorporating features for a model registry and workflow orchestration tailored for ML lifecycles.
    • Advanced AI Code Assistant: Expanding the assistant with debugging help and performance optimization suggestions.

5. User Experience & Productivity Features ​

Making PyRun even more intuitive and powerful for everyday use.

  • ✅ Recently Shipped:

    • Enhanced Data Cockpit: A new dedicated "Data Cockpit" section featuring a public dataset explorer (e.g., for COVID-19 sequences) alongside its powerful data partitioning tools. This makes discovering and preparing data for pipelines seamless.

  • Coming Soon:

    • Expanded Pipeline Library: Adding more diverse and complex real-world examples that leverage the new Data Cockpit and AI Hub features.
    • Collaboration Features: Enabling teams to share workspaces, code, runtimes, and results more effectively.
    • Expanded Dataset Explorer: Integrating more public data registries (e.g., Kaggle, Google Public Datasets) into the Data Cockpit.
    • Enhanced Monitoring & Visualization: Adding more sophisticated metrics, customizable dashboards, and visualization tools for deeper insights into job performance and resource usage.

Stay Tuned! ​

This roadmap provides a glimpse into the exciting future of PyRun. We are dedicated to making cloud computing more accessible, efficient, and powerful for everyone. Keep an eye on our Release Notes for the latest updates and feature announcements.

We value your input! Let us know what features and improvements are most important to you. GitHub Bug Report. Together, we can shape the future of PyRun!