Architecting a Hybrid Cloud B2B Infrastructure: Swarm vs. Multi-Server Strategy

[emirthab]

I am managing a complex hybrid infrastructure for my B2B software business. My setup includes:

• 2 high-availability servers on Hetzner and Microsoft Azure.
• A dedicated GPU machine at home for AI tasks.
• A physical machine in my office running multiple VMs, which has lower reliability due to occasional power outages.
• Customer-owned servers for clients who insist on hosting their data on their own infrastructure.
• A centralized Data-Vault server (Remote Server) handling shared databases, a private docker registry, and S3 storage.
• The entire network is interconnected via Wireguard.

My projects are B2B solutions that do not require massive scaling or millions of users, but they involve heavy background tasks like CRON jobs and frequent backups.

Current Setup:

• Client-specific data: Managed via the Remote-Server feature for total isolation (Proxy is still managed by the main Dokploy instance).
• Internal & Flexible Projects: Managed within a single Cluster where I use labeling (env=dev / env=prod) to separate environments on the same nodes.
• A centralized Data-Vault server (Remote Server) handling shared databases, a private registry, and S3 storage.
• To eliminate the burden of manual Traefik configurations and static file providers, I have developed a custom Service Discovery API. I’ve integrated this into my main Traefik instance as an HTTP Provider.

The Dilemma & Questions:

1. Database Placement & State Management:
   Since Dokploy’s Swarm implementation restricts databases to the Master node for backup/management purposes, how should I handle DBs for production? For my B2B apps, is it safer to move away from Swarm and use the Multi-server (Remote Server) feature for production DBs to gain more control over placement and reliability?

2. Dev/Prod Isolation Strategy:
   For projects hosted on my own cloud, is it better to stick with one cluster using labeling (env=dev / env=prod), or should I provision two separate Remote-Servers for Dev and Prod? I am worried that a failure in one of my less reliable office nodes (as a Swarm worker) might destabilize the Traefik configuration or the production traffic within the cluster.

3. Centralized Traefik in a Hybrid Network:
   I want to maintain a central Traefik proxy. When routing traffic to office VMs or customer servers (handling SSL/FQDN), what is the most stable approach? Should I rely on Dokploy's built-in remote routing, or deploy independent Traefik instances on each remote node and forward traffic from the main proxy?

4. Handling "Flaky" Nodes (Office/Home):
   My office machine is prone to restarts. If I keep it as a Swarm Worker, does the constant rescheduling of containers during downtime negatively impact the Manager node’s health? Would it be wiser to add these unreliable machines as standalone Remote-Servers instead of Swarm nodes to prevent cluster-wide side effects?

5. Performance & Latency:
   Given that everything is connected via Wireguard, are there known "best practices" for managing B2B apps where some components (like the database or S3) might be on a Remote-Server while the app is in a Cluster?

In conclusion, this is less about Dokploy and more about the DevOps architecture. I am quite confused about which path offers the best balance between ease of management and system stability. Any architectural advice would be greatly appreciated.