Tag Archives: hardware

Bad day at work? – How I destroyed a £2300 piece of equipment

It’s been longer than I care to admit since my last post, but sit comfortably because this is a tale worth telling. It is about all the planning and thought in the world can be let down with one careless oversight.

We use a fairly inexpensive HiPot tester (Clare H101) at work to check that some passive circuits are isolated from each other. This involves putting 850V on one circuit and checking that the hipotleakage to an adjacent circuit is less that 5mA. We currently use a custom switch box to manually dial between tests and perform the hipot test, remembering which combination of tests fail to determine which circuits require rework. This is a fairly quick (40 second) test but relies on the operator to connect the unit under test (UUT) correctly, dial through the tests correctly, record the results correctly, and stamp the correct section of the associated paperwork. With all the workplace distractions it is easy to forget (or overlook) one of those steps. Granted, the operator is working with equipment that has the potential (pun intended) to kill someone, so you’d be forgiven for thinking that additional care must be taken. But with all things, complacency settles in pretty quickly.

So a switch box was designed and software was written to control the hipot tester (using an partially documented protocol) and switch between circuits. This would check for the presence of the UUT (although various constraints prevented continuity checks of the individual circuits), and only proceed with the test in the UUT was connected. The HAL 101 included a guard circuit that h101is designed as a dead-man’s switch, but ours was fitted with a wire link instead.

The guard circuit calls for a no-volt switch to be used, whereby the test would only start if the contacts were joined. The connector was physically located with Mains parts (IEC inlet, fuses, 230/120V selector) and the connector was rated to 230V with L and N labels on the screw posts, but there was no mention of what voltage the guard circuit operated on.

It was decided that it would be safer if the hipot tester couldn’t initiate a test without software control, and that breaking the guard circuit would achieve this goal. The circuit was designed with track separation for 230V and a 230V relay was spec’ed for use with the guard circuit. All testing was carried out by bypassing the guard circuit until confirmation was received from the manufacturer of the working voltage of the guard circuit. This was received this morning, and all wiring/connectors/etc. needed to be rated to at least 5V 20mA. Perfect! The relay and wiring were completely overspec’ed but it meant that I could use a panel mount 3-pole 3.5m TRS connector instead of a large 230V rated connector. The wiring was finalised, and everything was soldered, connected, and screwed together for the final test.

The first test went through alright, but then the communications to the hipot tester went down. Maybe there’d been a software issue after all. Hardware and software were restarted but the comms were still down. Time to crack out RealTerm as an ASCII protocol had been used. Still nothing. Maybe the comms settings had been changed or corrupted, but everything checked out. What followed was an EE’s (almost) worst nightmare – the smell of magic smoke. Oh dear. Something going wrong is completely manageable; you can examine everything, evaluate possible failure modes, determine what was the cause and propose a fix. But what magic relaysmoke does is alert everyone in the room that you have messed up. Everything was quickly powered off and unplugged, to the sound of cheering from around the office. The only thing that had been changed was connecting the guard circuit so that seemed like a good place to start. Even if there was a solder defect or etching problem on the board then the worst thing was that the relay contacts were shorted together, which wouldn’t cause magic smoke. The connectors were taken apart and all the wiring was checked. Everything seemed alright. Time to take the hipot tester apart. The hipot tester was now already broken, so the ‘VOID if removed’ sticker wasn’t going to stop me.

The Clare H101 is available for around £2300, but accidents happen and I was outside of my probation period so I wasn’t fearful for my job. Opening the hipot tester revealed 2 screws rolling around the case. Maybe it was my lucky day, maybe it was just a coincidence that I plugging something in for the first time at the same time nte0505mcas it went bang. Unlikely… but possible. It didn’t take long to discover a slightly charred and cracked isolated DC-DC converter that powered the external interfaces (remote buttons, lights, beacon, serial interface, and guard circuit). I didn’t really want to send a unit back for a £300 fixing charge when a £5 component had failed (rest assured that my colleagues also picked up on my re-framing of “I’ve blown up a £2300 bit of kit” to “this £5 component has failed”).  But what caused it to fail?

I looked over everything again. The connectors had no stray bits of wire, the soldering was perfect, the relay contacts were switching like they should, the COMMON terminal was connected to 0V… WHAT?! Why is that connected to 0V. I opened the schematics and PCB artwork, the relay was only connected to a 5.08mm pitch connector. There was no way that this relay could be attached to 0V. I’d even checked this before and there were no shorts then. What else had I changed? Something must be different. And then it occurred to me, I had added an Earth bonding wire between the front and rear panels. My panel mount 3-pole 3.5m TRS earthconnector also happened to be metal, and so had shorted the sleeve (what I had designated common on the relay) to ground. Obviously when the relay switched across to close the guard circuit I had inadvertently shorted the isolated 5V of the hipot tester to ground (with the isolated 0V connected to the PC through the comms cable). The isolated power supply did not like this, and promptly died. I held my hands up to this. I had even added a cable gland to not use the TRS connector but decided against it at the last minute.

This is where it pays to understand the system as a whole. Yes, I was the only engineer to work on this and so I should’ve known better. What this meant was the avoidance of the fruitless exercise of software engineers blaming electronic engineers blaming mechanical engineers etc – I had to work with myself to ignore blame and work out what and why something had gone wrong. It was my fault that the Clare 5V was shorted to ground but I would learn from that mistake and make sure that it wouldn’t happen again. What actually happened was that I blamed Clare for not designing a more protected interface.

I don’t have access to any circuit diagram, but it is clear that the guard circuit did not include sufficient protection. Any inputs from the outside world should limit the voltage and current (as much as possible) before interfacing with anything sensitive like a micro-controller or logic gate. I tend to use the following circuit.input-protection

This limits the voltage and current to the gate of a MOSFET where I can then have voltage level conversion to my micro-controller VDD. This is by means not the only method, and other people may have other ideas, but it is a good place to start. However, having this alone will not protect against what actually happened, and that is that the voltage out drew too much current that the regulator burned out. Again, there are many ways of preventing this. As a starting point I would use a regulator that had over-current protection or thermal cutout. The hipot tester used a Murata NE0505MC for around £4.80 in 1000’s. A cursory check has turned up a BurrBrown part, DCP010505BP, for only £1 more. This features thermal cutout that would prevent the component failure. However, this is only part of it. What happened if the guard circuit was connected to something outputting 24V (like a light gate), or accidentally shorted to ground? Again, then the output should be current limited (using a resistor or PTC fuse) along with diodes to clamp the voltage. This obviously wouldn’t protect against connecting the circuit to mains voltage but it is a start.

If you have read this far then please take two things from this. Firstly, if you are interfacing with the outside world then please use protection. Protect what is going out and what is going in. You don’t need to go overboard, but if there is a chance that something will get shorted to ground or a power rail then limit that current. If you are powering with a DC socket, then include over-voltage and reverse polarity protection. A diode, resistor, or MOSFET are a lot easier and cheaper to replace than every IC on the board. Secondly, if you are the outside world, do not assume that the other designer has read this. Before plugging something in, check, check, and check again. If you are connecting to something that says it requires no-volt connection then don’t short it to a rail, just provide a relay. Obviously I could’ve taken the 5V into my circuit and then supplied my own 5V output, but in this case a relay was supposed to be safer as I may not have had the same 0V reference. Even though you are sure, check continuity between the relay contacts and any current source or sink – that means your voltage rails, case, ground, any IO etc. Read the manual and email the manufacturer for clarification. If something smells hot then be prepared to switch it off quickly. Limit current if you can. The manufacturer said that the wiring had to be capable of withstanding 5V 20mA so I could’ve included a resistor to limit that current. Would it have saved the isolated DC-DC converter? It’s tough to say, but it might have dragged the voltage down enough to affect communications and point to a potential issue.

I hope this has been informative and/or entertaining. To finish the story, my boss had a good laugh at my expense, we chalked it down to a learning experience and a replacement DC-DC converter is on order for me fit. It’s great being a double-E.

Everyone plays guitar, but more people watch TV

Guitar pedals are harder than they look – at least this one has been. I’ve learnt some valuable lessons that I’ve never had to worry too much about before. The most valuable lesson is that noise is not fun. Living in the digital domain is brilliant because is it far less affected by noise. Analogue is the complete opposite.

If you remember from last time, I had to bias my guitar input at Vcc/2, so that I could feed it into my first op-amp. This meant that any noise present on the voltage rail was put straight on my input, and amplified – not a good start. Thankfully, all is not lost as I just need to convert the input stage to an inverting op-amp with the reference set on the non-inverting input. It should just be a case of moving a few components around as I has enough forsite to scatter passives around the op-amp.

In the meantime I want to talk about my TV. It is a low-end 32inch LCD Hitachi with IO that fits the price tag – no USB, only 2 HDMI, Component, Composite, and VGA. It’s pretty basic, but it fulfils MOST requirements of it, and was a bargain when we bought it. However, over the last two years my needs have changed, but I cant justify buying a new TV. I’m sure modern TV’s are extremely complicated, but like all good hackers/engineers, I needed to know just how complicated. More than a few years ago I would have just taken the back off and gone oscilloscope crazy, but I’ve learnt my lesson by poking around inside said oscilloscope so instead started with the service manual.

In fact I started by searching for spares for my TV (Hitachi 32LD30UA). That gave me a part number of my main control board – the Vestel 17MB35. I can only assume it is a popular control board, purely based on the amount of information I can find about it. Amazingly I found a service manual on the ArgosSpares website. I don’t know why I say that’s amazing, as Argos is where I bought the TV, but I didn’t expect to find a service manual there. Anyway, hunting through the manual I found out that my control board should have footprints for 4 (not just 2) HDMI connectors, 2 USB connectors, and even a “Ye Olde” DSub9 connector. Thinks were looking up, as I needed a USB port just to power my Raspberry Pi. I have been tempted to open the back of the TV and piggy-back off of a voltage regular, but not all I need to do it fit a USB socket instead – a nice and clean modification.

At this point, I still haven’t actually opened up the TV, but I have ordered and received a spare control board. I figured when I eventually open the TV I can copy the contents of my TV’s EEPROM into this board, and see what happens when I switch them over.

My aim is to have the Raspberry Pi powered from a USB port on my TV, have a third HDMI channel, and enable HDMI-CEC. I am pretty confident in the USB power, but who knows about the rest – wish me luck.

Everyone plays guitar…

It has dawned on me that whilst I don’t pay a lot a month for hosting, I do actually pay something and so not keeping this blog up-to-date could be construed as a waste of money.

With that in mind I have an update – hooray.

I built my first guitar more than 10 years ago, whilst still at secondary school. I had only been playing for a few years, and I wanted something that would stand out both visually and aurally.

I started with a strat body, and a random neck I bought off eBay. I cut the scratch plate out of some clear acrylic, and mounted combination of a P-90, humbucker and single coil pickups. These were all wired together through a vast array of switches to enable me to switch any combination of pickups in both parallel or series, and either in or out of phase. Looking back it was a hideous rats nest of wiring, but that was nothing compared to the paint job.

I sat in the garage with my sister, who was 5 at the time, and we painted it with all the paints I could find. It was gloss, matte, and metallic, with a crackled finish in areas.

Since then I’ve bought 2 more guitars, and modified the wiring each time, but now I am more selective over the components. I try to select parts that match the current hardware, or add functionality without adding to the part count. All of this is leading to the real point of this post. I am designing a guitar pedal.

I have been meaning to design a pedal for a number of years, and I had no real reason to not do it. Every time I would sit down and work out what it needed, but I would always be put off by one aspect or another. There are hundreds of questions that stop me from progressing. What do I want the pedal to do? How do I control the modification to the sound? Is a 12bit ADC and DAC going to be enough? Do I have an analogue front end to mix the effect in, or deal with it in digital? Do I want a “true bypass” pedal, or is all the signal going into the processor at all times? Do I stick with what I know and use Microchip’s PIC32, go for a multi-core Parallax Propeller, or try my hand at one of Freescale’s DSP chips? Instead of letting the questions buzz around, I took the plunge and started the design of my pedal.

The signal is first passed through a capacitor, effectively removing the DC component. The signal is the biased at Vcc/2, and pass through an op-amp – TI’s LMV321. This will apply some gain to the signal, before feeding into the ADC – Microchip’s MCP3202; a 12bit ADC. It’s not the best ADC on the market, but it’s pretty cheap and will do the job for now. The digital data is then read in by Microchip’s PIC32MX764F128H. With a core frequency of 80MHz, this should be more than enough to perform some basic effects. Following manipulation, the data is then sent to the DAC – Microchip’s MCP4822; a 12bit DAC. Again, it’s not the best DAC around, but I am not aiming for that yet. Finally, I used a unity gain buffer to match any impedances, remove any DC component, and allow the next device in the chain to handle the signal. The parts selected were spares left over from previous projects. The only new purchases were the 6.35mm jacks and metal 1590A enclosure. It should be obvious that this is not going to be the best pedal in the world. I’m sure that any analogue aficionado will berate me for my choice of op-amp, and any audiophile will say that the minimum number of bits to consider would be 24. But they would be missing the point.

This pedal is my start line. It will allow me to see the weak points in the design. I should have mapped the control rotary encoders to an “Interrupt-On-Change” pin as the are a bit slow to respond, or sometimes appear to run in reverse. I should have use a codec IC with build-in 24bit ADC/DAC. I should have used analogue switches to bypass the circuit when disabled. By this time next week, I will have some answers.

After antagonising over this pedal for years, I have finally started. I spent around 2 days drawing a schematic and laying out a PCB. There’s some code still to be done, but a lot of it was written in the 2 weeks waiting for PCB’s to arrive. I’ve got PCBs from iTeadStudio for $27.59 (under £20), and all my components for around £25. For under £50, I have designed a programmable multi-fx guitar pedal. And I have no doubt that I’ll be doing the same next week.

Obscene Monitor for FREE!!!

A few months ago my place of employment and enjoyment suffered a catestrophic electrical failure. We recently installed some power factor correction equipment on our three phase mains setup – presumably to save money when powering running some of our more inductive loads.
For those that dont know, power factor is the ratio of real power and apparent power. We use a large number of transformers and motors, thus presenting a large inductive load to the power supply. Inductive loads cause the current waveform to lag behind the voltage waveform. Apparent power is measured in VA and calculated by multiplying the RMS of voltage by the RMS of current. Real power is measured in Watts and calculated by xxxxxx. Typically, power companies charge residential users based on real power, whereas industrial users are charged for apparent power. This means that cost savings may be acheived by ultising power factor correction in industry.

Cutting a long story short, apparently something went wrong whereby the neutral had become disconnected, and one of the three phase lines was used as a return path – or something like that. Needless to say, work sure was fun that day compiling a list of all broken equipment. Around ninety percent of PC’s required new power supplies, all the printers requred replacing, and approx fifty percent of monitors had a nice puddle of eletrolytic sitting at the base. Annoyingly my monitor was the only one in my department to remain completely opperational so everyone else has been gifted with new 24inch screens and I have to make do with a 21inch. The only other monitor to struggle on was a Samsung SyncMaster 245B, that suffered a flickering backlight a week later. This also made it to the scrap pile, where it stayed – until today.

I cracked the case open expecting to find puddles of electrolyte or scorch marks, but found none. Inside were three PCBs – control board, power board and backlight inverter. A casual check over the boards didnt reveal any blown componentry, although some of the resistors showed signs of extensive heat. The display was in perfect condition, other than the backlight, so I instantly dismissed the control board. I didn’t fancy going near the inverter quite yet, as neither my multimeter of myself could withstand a jolt of 3kV. I checked all the obvious voltage points and all the values read as expected.

I reconnected the inverter board, and all 6 CCFL tubes so see if the flickering settled down or got worse over time – in a bit to identify a temperature dependant behaviour. Instead I noticed that something was buzzing along with the flicker of the backlight. As carefully as I dared, I moved my ear closer to the 3kV and 240V voltage sources in a bid to locate the buzzing. As it turned out, the buzzing was coming from the power board, so the inverter was eliminated for now. The power board supplies the inverter with 24V, which measured dead on 24V without the invertor connected. Once connected, this 24V showed a 4V ripple which was well in excess of anything I was expecting.

In my experience, a ripple is caused by unsuitable de-coupling. Granted, my experience is purely low power stuff, but let’s see where this goes. Taking my unsuitable de-coupling theory, and the fact that a load of input protection had been “tested”, gave me a somewhat high expectation that the component in question was the 82uF 250V capacitor. I de-soldered the capacitor and it only measured 6uF. Clearly, if this wasnt responsible for the flickering, it would still require replacing as it was somewhat out-of-spec. I managed to source a 82uF 240V replacement capacitor and soldered it in place.

Tentitively, I powered the monitor back on, and… it worked. I mean, of course it worked – how could it not work. Unfortunately, I sourced an 85deg rated capacitor as a replacement for the 105deg version, but it should hold for now. Hopefully, this replacement will be a statistical outlier, and give me a few years of use – but if the backlight starts flickering I actually ordered 2 replacements.

Stereotypical beginners electronics project pt3

The LCM7215’s arrived as I was writing the last post, so I need to get the PCB’s cut. As I have access to a LPKF ProtoMat C60 PCB cutting machine I have designed a number of boards that fit together to give the case.

The idea is that the first board sits on the underside of the battery back with the electronics on the bottom side. Then two lots of the third PCB sit underneath, with the second PCB being the bottom of the case. These will be glued together, and M3 nuts will be glued in place to receive some M3 bolts mounting the battery case securely. There are two 5mm holes to accept a suitably sized screw.

To finish it off I’ve embedded a video.

[youtube ]UeT_E9ZLQX4[/youtube]