Bad day at work? – How I destroyed a £2300 piece of equipment

It’s been longer than I care to admit since my last post, but sit comfortably because this is a tale worth telling. It is about all the planning and thought in the world can be let down with one careless oversight.

We use a fairly inexpensive HiPot tester (Clare H101) at work to check that some passive circuits are isolated from each other. This involves putting 850V on one circuit and checking that the hipotleakage to an adjacent circuit is less that 5mA. We currently use a custom switch box to manually dial between tests and perform the hipot test, remembering which combination of tests fail to determine which circuits require rework. This is a fairly quick (40 second) test but relies on the operator to connect the unit under test (UUT) correctly, dial through the tests correctly, record the results correctly, and stamp the correct section of the associated paperwork. With all the workplace distractions it is easy to forget (or overlook) one of those steps. Granted, the operator is working with equipment that has the potential (pun intended) to kill someone, so you’d be forgiven for thinking that additional care must be taken. But with all things, complacency settles in pretty quickly.

So a switch box was designed and software was written to control the hipot tester (using an partially documented protocol) and switch between circuits. This would check for the presence of the UUT (although various constraints prevented continuity checks of the individual circuits), and only proceed with the test in the UUT was connected. The HAL 101 included a guard circuit that h101is designed as a dead-man’s switch, but ours was fitted with a wire link instead.

The guard circuit calls for a no-volt switch to be used, whereby the test would only start if the contacts were joined. The connector was physically located with Mains parts (IEC inlet, fuses, 230/120V selector) and the connector was rated to 230V with L and N labels on the screw posts, but there was no mention of what voltage the guard circuit operated on.

It was decided that it would be safer if the hipot tester couldn’t initiate a test without software control, and that breaking the guard circuit would achieve this goal. The circuit was designed with track separation for 230V and a 230V relay was spec’ed for use with the guard circuit. All testing was carried out by bypassing the guard circuit until confirmation was received from the manufacturer of the working voltage of the guard circuit. This was received this morning, and all wiring/connectors/etc. needed to be rated to at least 5V 20mA. Perfect! The relay and wiring were completely overspec’ed but it meant that I could use a panel mount 3-pole 3.5m TRS connector instead of a large 230V rated connector. The wiring was finalised, and everything was soldered, connected, and screwed together for the final test.

The first test went through alright, but then the communications to the hipot tester went down. Maybe there’d been a software issue after all. Hardware and software were restarted but the comms were still down. Time to crack out RealTerm as an ASCII protocol had been used. Still nothing. Maybe the comms settings had been changed or corrupted, but everything checked out. What followed was an EE’s (almost) worst nightmare – the smell of magic smoke. Oh dear. Something going wrong is completely manageable; you can examine everything, evaluate possible failure modes, determine what was the cause and propose a fix. But what magic relaysmoke does is alert everyone in the room that you have messed up. Everything was quickly powered off and unplugged, to the sound of cheering from around the office. The only thing that had been changed was connecting the guard circuit so that seemed like a good place to start. Even if there was a solder defect or etching problem on the board then the worst thing was that the relay contacts were shorted together, which wouldn’t cause magic smoke. The connectors were taken apart and all the wiring was checked. Everything seemed alright. Time to take the hipot tester apart. The hipot tester was now already broken, so the ‘VOID if removed’ sticker wasn’t going to stop me.

The Clare H101 is available for around £2300, but accidents happen and I was outside of my probation period so I wasn’t fearful for my job. Opening the hipot tester revealed 2 screws rolling around the case. Maybe it was my lucky day, maybe it was just a coincidence that I plugging something in for the first time at the same time nte0505mcas it went bang. Unlikely… but possible. It didn’t take long to discover a slightly charred and cracked isolated DC-DC converter that powered the external interfaces (remote buttons, lights, beacon, serial interface, and guard circuit). I didn’t really want to send a unit back for a £300 fixing charge when a £5 component had failed (rest assured that my colleagues also picked up on my re-framing of “I’ve blown up a £2300 bit of kit” to “this £5 component has failed”).  But what caused it to fail?

I looked over everything again. The connectors had no stray bits of wire, the soldering was perfect, the relay contacts were switching like they should, the COMMON terminal was connected to 0V… WHAT?! Why is that connected to 0V. I opened the schematics and PCB artwork, the relay was only connected to a 5.08mm pitch connector. There was no way that this relay could be attached to 0V. I’d even checked this before and there were no shorts then. What else had I changed? Something must be different. And then it occurred to me, I had added an Earth bonding wire between the front and rear panels. My panel mount 3-pole 3.5m TRS earthconnector also happened to be metal, and so had shorted the sleeve (what I had designated common on the relay) to ground. Obviously when the relay switched across to close the guard circuit I had inadvertently shorted the isolated 5V of the hipot tester to ground (with the isolated 0V connected to the PC through the comms cable). The isolated power supply did not like this, and promptly died. I held my hands up to this. I had even added a cable gland to not use the TRS connector but decided against it at the last minute.

This is where it pays to understand the system as a whole. Yes, I was the only engineer to work on this and so I should’ve known better. What this meant was the avoidance of the fruitless exercise of software engineers blaming electronic engineers blaming mechanical engineers etc – I had to work with myself to ignore blame and work out what and why something had gone wrong. It was my fault that the Clare 5V was shorted to ground but I would learn from that mistake and make sure that it wouldn’t happen again. What actually happened was that I blamed Clare for not designing a more protected interface.

I don’t have access to any circuit diagram, but it is clear that the guard circuit did not include sufficient protection. Any inputs from the outside world should limit the voltage and current (as much as possible) before interfacing with anything sensitive like a micro-controller or logic gate. I tend to use the following circuit.input-protection

This limits the voltage and current to the gate of a MOSFET where I can then have voltage level conversion to my micro-controller VDD. This is by means not the only method, and other people may have other ideas, but it is a good place to start. However, having this alone will not protect against what actually happened, and that is that the voltage out drew too much current that the regulator burned out. Again, there are many ways of preventing this. As a starting point I would use a regulator that had over-current protection or thermal cutout. The hipot tester used a Murata NE0505MC for around £4.80 in 1000’s. A cursory check has turned up a BurrBrown part, DCP010505BP, for only £1 more. This features thermal cutout that would prevent the component failure. However, this is only part of it. What happened if the guard circuit was connected to something outputting 24V (like a light gate), or accidentally shorted to ground? Again, then the output should be current limited (using a resistor or PTC fuse) along with diodes to clamp the voltage. This obviously wouldn’t protect against connecting the circuit to mains voltage but it is a start.

If you have read this far then please take two things from this. Firstly, if you are interfacing with the outside world then please use protection. Protect what is going out and what is going in. You don’t need to go overboard, but if there is a chance that something will get shorted to ground or a power rail then limit that current. If you are powering with a DC socket, then include over-voltage and reverse polarity protection. A diode, resistor, or MOSFET are a lot easier and cheaper to replace than every IC on the board. Secondly, if you are the outside world, do not assume that the other designer has read this. Before plugging something in, check, check, and check again. If you are connecting to something that says it requires no-volt connection then don’t short it to a rail, just provide a relay. Obviously I could’ve taken the 5V into my circuit and then supplied my own 5V output, but in this case a relay was supposed to be safer as I may not have had the same 0V reference. Even though you are sure, check continuity between the relay contacts and any current source or sink – that means your voltage rails, case, ground, any IO etc. Read the manual and email the manufacturer for clarification. If something smells hot then be prepared to switch it off quickly. Limit current if you can. The manufacturer said that the wiring had to be capable of withstanding 5V 20mA so I could’ve included a resistor to limit that current. Would it have saved the isolated DC-DC converter? It’s tough to say, but it might have dragged the voltage down enough to affect communications and point to a potential issue.

I hope this has been informative and/or entertaining. To finish the story, my boss had a good laugh at my expense, we chalked it down to a learning experience and a replacement DC-DC converter is on order for me fit. It’s great being a double-E.

Everyone plays guitar (Part 2)

It has been over a year, and what a year it has been. Fortunately I don’t have to go into any of that here.

The TV project has all but died. I still have the spare PCB kicking about, but I’ve not got the motivation to fix my BusPirate probes and try to add USB to my TV. The Rasperry Pi2 is out now, and I don’t think my TV could supply enough current regardless. So one rainy day I might look into this again, but for now – everyone plays guitar, right?

As this post is going to deal with the guitar pedal project I think we should go over some history.

The MP-01 design was released 6th July 2013 (wow, that was a long time ago). It used a single PIC32 with external SPI DAC and ADC. Due to quite a lack of foresight, the rotary encoders were not on change notification pins, and any rail noise was put straight onto the guitar signal. That was a major issue that resigned this version to the junk pile, and the less said about this the better.

MP-02 was to be the savour. The design for that was released almost 1 year ago to the day (tomorrow in fact). I learnt my lessons from the first design. The op-amp stage was re-architected to utilise the op-amps supply noise rejection capabilities. I’d been hit by scope creep quite severely by now. I had moved to a 2-PIC solution; a PIC24 to handle the non audio functionality (which went from just LEDs to include an external port for an expression pedal and USB to allow direct loading of patches from a PC), and a dsPIC33 to handle the real-time audio activities (using it’s in-built ADC and DAC along with the rotary encoders, footpedal switch and EEPROM). After building the pedal I had real difficulties getting the thing to work as I expected it to. For some reason I just couldn’t sample quickly enough and there was still far too much noise. I finally fixed the sampling issue by changing how I was clocking everything, and I was having a pedal that was actually passing the guitar signal through, albeit with a lot of white noise on top. This noise was causing me some heartache as I just couldn’t pin down what was causing it; poor layout, inherent ADC and DAC noise issues, op-amp selection, dodgy power supply. I was fighting against so much that I decided that enough was enough. Plus I blew up the dsPIC twice as apparently the metal enclosure is made of metal.

MP-02 v1.1 was to be the savour! I refused to play any more games. The dsPIC range do have inbuilt ADCs and DACs but they just can’t compete against a proper chip. So I enlisted the help of something made for audio; the Wolfson WM8731. This blows the 10 and 12 bit capabilities of the dsPIC out of the water with a whopping 24bits at 192kHz. It supports pass through of the audio without requiring it to even enter the dsPIC. We have volume control, anti-thump mute, and filters coming out of our ears. This is definitely going to go some way to helping the sound. The LMV321 op-amps are out. I looked over the specs and the limited slew rate is fighting against us. I am looking at a couple of different op-amps to use instead, and I am favouring the LMV793. A slew rate of 24V/us (instead of 1V/us), and a noise density of 6.2nV/rt(Hz) (instead of 46 nV/rt(Hz)) means that this pedal will be singing. I’ve added some external SPI SRAM into the mix too. At 512Kb we have enough storage for over 5 seconds of audio (16bit mono 48kHz). This doesn’t sound a lot but it is more than enough for a decent echo. Plus unlike EEPROM, SRAM can be written to again and again and again… I’ve still got the EEPROM though, boosted to 512Kb too. This allows me to bootload both the PIC24 and dsPI33 from SPI or UART without having to support the entire USB stack in the bootloader. Scope creep got me again, and now we support USB OTG. This means that whilst you can plug the pedal into a PC to download new patches, you can download from your phone instead or use that as an expression pedal, or simply plug in a USB memory stick to load patches directly. This just opens up the possibilities. Finally I’ve added filtering and protection. I probably could’ve added extra but it will hopefully that will be enough.

These changes complicated the design no end, but I still managed to get everything on two layers (this may be the problem, we shall have to see), within my 50x100mm outline, and mostly single sided population. I want this pedal to be silent, as well as whisper, sing, and scream. I’ve done all I can to help with this task from additional components to filter the noise to segregation to help keep any noise where it can do no harm. I’ve even gone to a dedicated audio chip. This marks the beginning of the end of my attempt to design a guitar pedal. All will be revealed in a week or two when the new design gets posted through the door.

I have been fairly lax in updating this website, but hopefully the next post will bring good news.

More people watch TV (Part 3)

This post I am going to be demonstrating how to read an I2C device using a BusPirate. The BusPirate is a brilliant little tool that provides a command line interface to various communication buses. I am using the BusPirate V4, but the theory will be the same for the V3 and V3.5. The first device I am going to try and read is the ST24LC21. This ST part is a 1Kbit EEPROM that stored the TV’s VGA information that allows a PC to detect the capabilities of the display.

Attach your probes as follows: Vcc -> +5, GND -> GND, SCL -> CLK, SDA -> MOSI, WP -> AUX. Connect to the BusPirate using your favourite terminal program, and type the following “m 4 1 3″ to enter I2C mode. Next enable the power “P”, connect the pull-ups “e”, and enable the pull-ups “W”. Next you can search for any devices on the bus “(1)”. This should show a device on 0xA0 and 0xA1. This is the same device, but the write and read address. To read the data from the chip we set up the address “[0xA0 0]”, and send 128 read commands “[0xA1 r:1EDID28]”. Actually deciphering the numbers means delving into the EDID specification (or at least the Wikipedia article), but with that it is possible to determine manufacturing details of the TV, and its capabilities.
For example, byte7-8 (counting on from byte0) contain a compressed ASCII string. 0x58B3 is 0b0101100010110011, which in this scheme turns into 0bx 10110 00101 10011, or 22 5 19, or VES. Microsoft are responsible for allocating these values, and checking here reveals that the owner of VES is indeed Vestel Elektronik Sanayi ve Ticaret, the manufacturers of the control panel. Looking further on, byte17 (counting from byte0) is the year of manufacture minus 1990. In this instance the value is 0x0A, or 10 in decimal. This puts the manufacture year at 2000. I was expecting a much later year, but maybe the EEPROM image hasn’t been updated over all these years.

That’s enough for today, especially as I managed to break one of my BusPirate probe leads. Next time I will be looking through the main NVM chip to see if there is anything recognisable that I can modify. Wish me luck.

More people watch TV (Part 2)

I have three problems that I need to tackle; USB, 3x HDMI ports, HDMI-CEC. So again, lets head to the service manual to avoid embarrassment.



There are footprints for 2 USB connectors. Even though I don’t want to plug a memory stick in, it did pique my interest as to the differences. The top-most connector (CN125) is labelled DIGITAL_USB, as opposed to the plain USB of the bottom connector (CN112). Neither of the connectors have a direct connection to the 5V_VCC rail, but instead go through an IC labelled STMP2161. On my spare PCB this part is missing (as I expected), and I couldn’t find a datasheet or anywhere to get one from. Thankfully, the part has both pins and nets labelled, so it isn’t too difficult to at least identify the function of the part, allowing a suitable replacement to be found. The device features ENABLE and FAULT pins, of which the FAULT pin is connected to a net labelled USB_OCD. This device is probably a current limited load switch (OCD being short for Over Current Detected).


USB SchematicTests have shown the Raspberry Pi has a maximum current consumption of between 500-600mA. This is just a guess, but I doubt that the STMP2161could handle 600mA, so if anything I am lucky that I have to source my own current limited load switch. Failing that I could bypass the switch circuitry and use a resettable fuse instead.

Back to the differences between the two connectors. the DIGITAL_USB goes to a CT216T-R – the Cheertek Integrated DVB-T Receiver. Given this device has digital video and audio decoding capabilities, it is likely that the DIGITAL_USB connection would be used for playing video from a memory stick.

The plain USB connector goes to the MST6WB7GQ-3 – the main MSTAR microcontroller. This is the brains that bring everything together. I couldn’t find any other information about the MSTAR processor, or its USB capabilities. The service manual makes reference to an “Analog USB” port which I didn’t even think was possible.

Enabling the power for the DIGITAL_USB connector is easy, but I wonder how much work would be involved in getting the actual USB data side to work. I am going on the presumption that all MSTAR microprocessors go out the same, and it is just a configuration page in external EEPROM that enables the features. This idea explains a number of things found in the Service Menu (found by pressing 4725 whilst in the TV’s menu). Firstly, there are options for “Digital USB Hotplug”, “CEC Enable”, HDMI3, HDMI4 but they are all greyed out. Secondly, there is the ability to modify NVM (non-volatile memory) data. I think that modifying the correct byte in NVM will enable the greyed out features.

Before I modify the NVM, it makes sense to make a backup. But first I need to find the NVM. Hunting for NVM on the schematic leads to a net labelled SCL_NVM. This is an I2C clock line for the NVM device, in this case a Microchip’s 24C32. Admittedly, I’m getting a bit ahead of ourselves, but it is obvious how the building blocks all come together and what started as a horrifically complicated design is getting simpler.

Back to the USB, and there is another unidentified device – a AZ099-04S. I am guessing this is probably something to help with ESD – maybe an array of bi-directional transient voltage suppressor (TVS) diode clamps. The last interesting part of the schematic is “Impedance 90R olacak”. USB is a differentially driven bus with a characteristic impedance of 90 ohms, and olacak is Turkish for “it will happen”. As to why there are Turkish instructions of the schematic, Vestel is a Turkish firm.

Jobs before the next post are dump the contents of any EEPROM on the board. This includes the main NVM EEPROM (U103) that hopefully enables features, and the VGA EEPROM (U112) responsible for identifying the TV as a PC monitor and presumably including EDID (Extended Display Indification Data). Additionally, the HDMI switch contains an EDID table that should be visible over I2C. There are two I2C flash chips; one on the MPEG decoder (U114), and one on the main microcontroller (U132), that can also be read.

Everyone plays guitar, but more people watch TV

Guitar pedals are harder than they look – at least this one has been. I’ve learnt some valuable lessons that I’ve never had to worry too much about before. The most valuable lesson is that noise is not fun. Living in the digital domain is brilliant because is it far less affected by noise. Analogue is the complete opposite.

If you remember from last time, I had to bias my guitar input at Vcc/2, so that I could feed it into my first op-amp. This meant that any noise present on the voltage rail was put straight on my input, and amplified – not a good start. Thankfully, all is not lost as I just need to convert the input stage to an inverting op-amp with the reference set on the non-inverting input. It should just be a case of moving a few components around as I has enough forsite to scatter passives around the op-amp.

In the meantime I want to talk about my TV. It is a low-end 32inch LCD Hitachi with IO that fits the price tag – no USB, only 2 HDMI, Component, Composite, and VGA. It’s pretty basic, but it fulfils MOST requirements of it, and was a bargain when we bought it. However, over the last two years my needs have changed, but I cant justify buying a new TV. I’m sure modern TV’s are extremely complicated, but like all good hackers/engineers, I needed to know just how complicated. More than a few years ago I would have just taken the back off and gone oscilloscope crazy, but I’ve learnt my lesson by poking around inside said oscilloscope so instead started with the service manual.

In fact I started by searching for spares for my TV (Hitachi 32LD30UA). That gave me a part number of my main control board – the Vestel 17MB35. I can only assume it is a popular control board, purely based on the amount of information I can find about it. Amazingly I found a service manual on the ArgosSpares website. I don’t know why I say that’s amazing, as Argos is where I bought the TV, but I didn’t expect to find a service manual there. Anyway, hunting through the manual I found out that my control board should have footprints for 4 (not just 2) HDMI connectors, 2 USB connectors, and even a “Ye Olde” DSub9 connector. Thinks were looking up, as I needed a USB port just to power my Raspberry Pi. I have been tempted to open the back of the TV and piggy-back off of a voltage regular, but not all I need to do it fit a USB socket instead – a nice and clean modification.

At this point, I still haven’t actually opened up the TV, but I have ordered and received a spare control board. I figured when I eventually open the TV I can copy the contents of my TV’s EEPROM into this board, and see what happens when I switch them over.

My aim is to have the Raspberry Pi powered from a USB port on my TV, have a third HDMI channel, and enable HDMI-CEC. I am pretty confident in the USB power, but who knows about the rest – wish me luck.