Yeah I think it's both more effective and cheaper to have dual/quad socket systems with 1 "normal" CPU and the rest filled with FPGAs without CPU cores, just to max out on the raw crunching ability. The PCIe block on the FPGA chips could be flexible enough to (re-?)wire directly into the programmable logic, maybe even reconfigurable to other protocols (e.g. 100GE). Also in "normal" NUMA fashion each FPGA would have the memory channels associated with that socket (presumably through the interconnect as if it were a CPU, so the CPU can access it too.)

I'm just looking at this from a logical chain of "who needs FPGAs in their computers?" => "cases with loooots of specific data crunching" => "want a controlling/driving CPU for the complicated parts, but then just concentrate as much FPGA in as possible." => Multi-socket with 1 CPU & rest FPGAs.

(There currently is no commodity Quad-socket SP3 mainboard, not sure if this is a design limitation or just no one made one yet? I'd still say the approach works great with only 2 sockets.)

I wouldn't expect to see anything like this on SP3 anyway, since it would take some time to do the work and by then the current generation would likely be whatever they replace SP3 with in order to support DDR5.