Fast Genomic Downloads and Reduced Cloud Costs for GenomeUp

GenomeUp offers clinical laboratories and researchers the JuliaOmix cloud platform for advanced analysis of omics data. However, managing the download of massive genomic outputs was an operational bottleneck. We re-engineered the data transfer infrastructure by introducing Download 2.0, a system that unlinks downloads from the browser, eliminates server-side compression costs, and gives users full control over their clinical data.

The Challenge

Files at the core of the JuliaOmix platform — BAM, FASTQ, and outputs of complex genomic processing — routinely weigh tens or hundreds of gigabytes. The original workflow was not scalable for these volumes, creating critical issues both technically and in terms of business.

Process Limitations and Compromised User Experience

In the previous system, the process was entirely managed by the browser via a pre-signed URL and could take a long time. In the event of a connection interruption or accidental tab closure, the entire download had to be restarted from scratch. This dependency on the browser forced users to tie up a computer resource for hours, without the ability to download partial selections or single files, a limitation that frustrated researchers and undermined laboratory productivity.

Furthermore, files stored on cold storage (such as Glacier or Deep Archive) were simply not included in the workflow, disappearing without any notification or explanation to the user, an unacceptable limitation in a context where data completeness has strong clinical value.

Infrastructure Costs and Server Inefficiency

Before initiating the transfer, the server had to read and compress the entire selection into a ZIP archive. This approach resulted in long waiting times for the user and recurring, massive consumption of infrastructural resources (compute, temporary storage, and bandwidth) for every single request, even for downloads that subsequently failed on the client side.

Our Approach

We thoroughly analyzed the existing infrastructure, concluding that a paradigm shift in architecture was necessary: moving from a model where the web application moves bytes to a model where the application orchestrates the process, delegating heavy lifting to a dedicated external actor.

Architectural Redesign and Progressive Migration

To ensure business continuity, we structured the new solution around a new dedicated API. Maintaining the previous v1 version, occupied by the legacy subsystem, allowed for progressive migration of the platform section by section. This dual-track approach guaranteed a release plan with natural rollback capabilities, eliminating the risk of interrupting the existing service.

The Solution: Download 2.0

The new Download 2.0 module completely reorganizes the workflow around three main actors, ensuring stability, visibility, and scalability.

Orchestration and the Local Agent

The web application no longer transfers any bytes. When the user confirms an operation, the platform creates the download record and entrusts the task, along with session credentials, to jodownloader-studio, an agent installed directly on the user's machine. It is this local daemon that directly interacts with the backend and performs the data transfer, immediately freeing the browser and allowing the user to continue working without blocking the system.

Advanced Management of Multiple Downloads and Cold Storage

The system finally supports initiating downloads for single files, multiple selections, or entire analysis contents with a single operation.
Crucial is the new management of archived data. Download creation operates in two phases: if the API detects files on Glacier or Deep Archive (not immediately downloadable), the process stops and presents the user with a detailed report. The user is transparently informed about which files will be transferred and which will remain excluded, thus allowing them to consciously decide whether to proceed or cancel, transforming a potential silent error into an informed clinical decision.

Security by Design for Hospital Environments

Dealing with health and genomic data, the architecture has been hardened. Security is structured on multiple levels:

• Host Validation: The application exclusively communicates the backend host to the agent. It is the agent that composes the full URL, validating the input against a predefined pattern, thereby protecting the system from attempts to hijack credentials to arbitrary or hostile endpoints.
• Authentication and Permissions: The platform uses Cognito authentication (with OAuth2 integration for SSO already provisioned at the contract level). The entire download functionality responds to dedicated application permissions, satisfying the strict access governance requirements of hospital clients.

Results and Business Value

The project, currently integrated and in the final refinement phase regarding credential delivery, with future evolutions already outlined, brings concrete operational and economic advantages for GenomeUp and its clients:

• Elimination of infrastructural overhead: By removing the need to generate server-side ZIP archives, the company has eradicated the computation, temporary storage, and bandwidth consumption previously associated with preparing downloads.
• Uninterrupted operation: Researchers can initiate massive downloads that proceed in the background via the local agent, freeing themselves from browser instabilities and reducing downtime.
• Reliability and clinical transparency: Fewer failed transfers translate into a lighter load for the support team. Immediate visibility into files in cold storage has increased confidence in the platform.
• Future-proof architecture: The v2 system is highly decoupled and secure, creating a solid foundation to pass future hospital security audits and support the scalability of GenomeUp's data volumes.

Why it's replicable

Any B2B or SaaS platform that distributes massive files — whether genomic datasets, engineering projects, machine learning outputs, or video archives — encounters the physiological limits of browsers and the hidden costs of infrastructure.

If your application suffers from failed transfers, bottlenecks related to server-side package generation, or unjustified costs for data movement, an architecture orchestrated via a local agent represents the definitive solution. Contact us to discover how we can transform your data delivery system into a secure, reliable, and scalable business asset.