Question:

I extend my gratitude for the wealth of knowledge this forum offers. As a newcomer, I’ve found the discussions here immensely educational and I’m keen to engage more actively. Our family-run enterprise in Europe, with a dedicated team of 20, is in the process of reassessing our IT infrastructure. We currently operate a dual-node VMware setup with SAN storage for our VMs. My role, though part-time, involves overseeing our IT systems, and my background in computer science and DevOps has led me to contemplate switching to a Proxmox-based environment. This shift is partly motivated by the need to eliminate our reliance on the SAN, a potential single point of failure, and recent industry changes involving Broadcom have only added to the appeal of this transition.

Our Goal

: We aim to establish a dual-node Proxmox system with ZFS replication. While we recognize the data loss risk within the replication window, it’s a manageable risk for our business. A single node could meet our performance needs, but the additional redundancy from a second node is crucial, especially considering our remote location. Our focus is on a straightforward, economical setup that guarantees high availability without excessive costs. I’ve requested a quote for this setup from a Managed Service Provider (MSP), but I’m eager to hear this community’s perspectives and recommendations for enhancements.

Network Design Considerations

: Our proposed layout includes two Proxmox nodes and a Proxmox Backup Server (PBS), linked via a 25 Gbit/s network to facilitate ZFS replication, backups, and internal traffic. Each node would connect to a switch using 10 Gbit/s links, with VLANs for segmenting the network. Additionally, one of the Proxmox hosts would function as a firewall, managing VLANs and internet traffic, using OpnSense or PfSense.

I’ve outlined my network proposal here: [Network Diagram Link]

Specifications for Proxmox Nodes

:

• Model: ASUS RS500A-E11-RS12U
• CPU: AMD EPYC 7543P – 32 Cores, 64 Threads, 2.8GHz Base, 3.7GHz Max Turbo
• RAM: 256 GB DDR4 3200 ECC Registered, 32GB Sticks | Brand: Micron
• System Storage: Two 512GB NVMe SSDs for Proxmox installation
• Data Storage: Four 6.4TB NVMe Enterprise Drives, configured in ZFS as a stripe of mirrors, offering 12.8 TB of usable space
• Networking: 25Gbit Quad-Port and Dual-Port NICs
• Service: Next Business Day parts service, with a Proxmox support subscription

The Proxmox Backup Server will also serve as a quorum device for clustering.

Backup Host Specifications

:

• Model: Asus RS520A-E11-RS12U
• CPU: AMD Epyc 7232P – 8 Cores, 16 Threads
• RAM: 64 GB DDR4 3200 ECC Registered
• System Storage: Two 512GB NVMe SSDs for PBS
• Data Storage: Four 12TB HDDs for backup storage
• Networking: 25Gbit Quad-Port NIC

Seeking Expert Advice On

:

• Do you see any major issues with the proposed solution, or does it seem viable?
• Is a direct connection between the three hosts reliable, or would it be better to use two SFP28-capable switches with MLAG for added redundancy and network efficiency, despite the higher costs?
• What are your thoughts on the reliability of ASUS servers, particularly in scenarios without onsite service?
• Should we opt for the newer AMD Genoa CPUs over the current AMD Milan ones, considering the additional expense?

Your expert insights and guidance will be instrumental in helping us build a robust and future-proof IT infrastructure. Thank you for your valuable input!

Warm regards,

[Your

Name]”

Answer:

Thank you for reaching out and sharing the details of your IT infrastructure transition plan. It’s clear that you’ve put considerable thought into the potential move from VMware to a Proxmox and ZFS-based solution for your family business. Let’s delve into the aspects you’ve highlighted.

Your proposed dual-node Proxmox setup with ZFS replication seems well-considered and viable. The redundancy provided by the second node is a wise choice, particularly given your remote location. The risk associated with the replication window is acknowledged, and it’s good to hear that it’s within acceptable limits for your operations.

Network Design and Connectivity

The network design you’ve proposed, with two Proxmox nodes and a PBS connected via a 25 Gbit/s network, is robust. The use of VLANs for segmentation and a Proxmox host as a firewall using OpnSense or PfSense is a sound strategy. Direct connections between hosts can be reliable and cost-effective. However, if future expansion or additional redundancy becomes a priority, considering SFP28-capable switches with MLAG might be beneficial, despite the higher initial investment.

Hardware Specifications

The hardware specifications for both the Proxmox nodes and the backup host are impressive. ASUS servers are generally known for their reliability, and the components you’ve chosen, such as the AMD EPYC CPUs and Micron RAM, are top-notch. The system and data storage configurations are apt for your needs, and the networking capabilities are more than sufficient.

Service and Support Considerations

While ASUS servers are reliable, the lack of onsite service could be a concern. It’s crucial to have a contingency plan, especially if hardware issues arise that require immediate attention. The next business day parts service and Proxmox support subscription are positive steps, but ensure you have a local IT professional or service that can address any urgent hardware failures.

CPU Selection

Regarding the CPU choice, the AMD Milan CPUs are powerful and should serve your needs well. Upgrading to the newer AMD Genoa CPUs would future-proof your setup to some extent, but it’s essential to weigh the cost against the performance benefits. If budget constraints are a significant factor, sticking with the Milan CPUs might be the more prudent choice for now.

In conclusion, your plan for transitioning to a Proxmox and ZFS solution is well-founded. It’s tailored to provide high uptime and redundancy while being mindful of costs. As you move forward, keep an eye on potential service gaps and consider the long-term benefits of investing in newer technology if it aligns with your business goals.

Best regards,

[An IT Infrastructure Expert]