With three transmitters and two antennas for this job, we needed a switch that could connect anything to anything, so that if a transmitter failed, a spare could be switched in to replace it. Because the site would be unmanned, a remote controlled switch would be best. Most large broadcasters take the same approach.
I thought I'd done the right thing when I decided that we shouldn't rely on the usual supplier for our antenna matrix switch, because I couldn't see how it could possibly be good enough, and our own transmitter experts agreed with me.
The problem is that everything needs to be able to handle the transmitter power, and the bigger the cables, the more power they can handle. This supplier insisted that seven-eighths inch cables and connectors were fine, but we'd worked out that everything else it would connect to had to be a much bigger size to avoid blowing up at our power level.
So I took a contact's advice, and went to another switch supplier that I'd never heard of before, but who were a long-term government supplier in their own country. And as it turned out, the switches were indeed rather good - and already used the same size connector as we had already found we needed.
Where it all went wrong was that my project manager insisted that because they were the experts in their own product, we should get them to design the remote control system, and he wouldn't permit me to do it myself.
Perhaps you can already see a problem here? If it was their own motor driven product, so that it was intended for remote control, it was suspicious that they didn't already have a remote control system of their own.
Yes, of course they would do it - but I would have to design the status panel alyout for them. You can see this is already getting more ominous.
When it arrived on site in a remote location and we powered it up, the first thing that I noticed wrong was the indicator lamps. They were so dim that they were hard to see, and when I flipped switches so that half of them switched off as they should, then the brightness of the remaining lamps doubled. Very wrong indeed.
Worse was when I checked on the redundant power supply, which my engineering manager had insisted we needed. When I pulled the power plug on either one, the other one died. So it was actually worse than having one power supply.
I could have just packed it up and sent it back, which is what my project manager would have insisted on had he known, but we were already running very late, and I had no confidence in getting a supplier who was on the other side of the world to fix it, when it increasingly looked like he had no idea what he was doing. So I decided to pull it to bits myself.
What I found only confirmed my worst fears. The power supplies had 5 volt regulator chips, which had been connected directly to 2.2 volt LED lamps, without the necessary "dropping" resistors. And although redundant power supplies are always connected via diodes so that the one with lower voltage will drop out, those diodes were missing. As a result of these very basic mistakes, both of the regulator chips had burned out.
So having to pack it all up, hold up the rest of the commissioning until it came back fixed, and explain very basic electronics to a supplier who I now knew had no idea of what he was doing and was on the other side of the world was an unwelcome thought. It was quickest and cheapest to just get my hands dirty and fix it myself. I removed the failed chips (which weren't needed anyway), bought a bundle of resistors and a few diodes from the local electronics shop, and set to work with a soldering iron. It worked first time, and probably took less time than packing it up, shipping it back and writing up an unpleasant letter would have done.
Eventually, I wrote a letter to the supplier, explaining in simple terms what he'd got wrong. I guess it was too embarrassing - I never got an answer.
I nevcer really did find out what on earth had gone wrong, but I think that all the old guys must have retired, and there was a very new engineer trying to find his way to keep the company alive (because his was the only name that appeared on all of their documentation). I hope he learnt and succeeded, it was a nice bit of kit apart from that remote control.
Postscript: Since I first wrote this, I've been told that it wasn't as nice a bit of kit as I thought. Those motors were attached to what they were supposed to turn with two tiny screws. Over time, the screws stopped gripping properly, needing brute force solutions.