The radio geometry change should reduce self-interference by perhaps 3dB. But was that a problem to start with? If so, why was a geometry change selected as the solution? (Or part of it?)
An optimal solution would include managing transmissions, to ensure that no radio is transmitting on (or near) a channel that is being listened to by another Relay radio. Ideally, the "whole system" configuration (including the user's WiFi) should try to ensure the radios never share (compete for or interfere with) any other channels. (Having multiple Relay radios listening on a channel is never a problem, because only one would understand the signal.)
Then again, managing transmitter collisions may be "too much work", and the geometry change was made as a "better than nothing" step. Transmitter management could be added later.
It could also be the case that some of the transmitters are "messy", spilling side lobes into other channels (or, like WiFi, can straddle multiple channels at once), in which case the geometry change is all you can do, short of physically separating the antennas, or reducing the number of available channels, or reducing transmit power, or any of a number of similar RF-domain fixes. Or it could also be that some or all of the receivers lack good "notch" tuning, and are thus more susceptible to interference.
The key is always in the testing: Near-field tests are OK for basic functionality, but far-field tests are needed to ensure coverage extent and quality. Ideally, you'd have a Frankenstein box (containing radios for each protocol) that you'd walk around with to detect communication dropouts and unexpected range limitations. You may be able to use a Relay for this, if you can replace the Host protocol layers with Device layers (or if the protocol has Host-to-Host capability).
I wonder: Do the Relay radios exhibit equivalent range when tested with "common" devices for each protocol? It shouldn't be much of an issue for mesh protocols like 6LowPAN, but may prove to be a real limitation with non-mesh protocols.
