Tag Archives: IP Camera

Hello Caller, You Are Through To The Helpdesk Part4

Posted on by
5

If you remember my last post, I had finished sorting out the hardware, and had just taken the plunge with fishing around for working embedded software. I’ll be honest, it didn’t take many attempts to find a “working” solution, but each attempt felt like an age due to the switching between programs to use XMODEM. For some reason RealTerm doesn’t support XMODEM, and TeraTerm simply wouldn’t send the data. I found ClearTerminal, which although managed absolutely fine with XMODEM there were issues with interrupting the boot process of the camera; a necessary step when dealing with experimental Linux images. 

I hit various brick walls, including the camera having a “Kernel panic”, and numerous I2C errors. The Kernel Panic was due to a completely incompatible firmware, and the I2C error seemed to be triggered by some “missing” hardware. Missing in that the camera would support it, but it simply wasn’t included in this model.

On the forth attempt I tried software designed for the HooToo camera – and not only had the camera booting up, but I also had a web interface to use. This wasn’t the nicest interface in the world, but at least I could see the video stream and control the camera. The hardware was fixed, some software was running; job done, or so I thought.

This camera originally came with some software that when installed prompted the user to enter the camera’s serial number. This serial number is used to tie together the camera and a DDNS service. DDNS stands for Dynamic Domain Name Service, and once set up it allows you to view the IP camera from anywhere in the world. The HooToo software has an unknown DDNS hard coded, or allows you to specify a different DDNS service. Unfortunately, this was restricted to DynDNS.org and a couple dodgy Chinese services that my browser warned me against visiting.After some digging around I realised that the DDNS information is held in the romfs.bin file, and that it might be possible to use parts of my original extracted romfs.bin file to overwrite HooToo’s default DDNS service information.

At which point it struck me, I already had a working webui.bin and romfs.bin as it was just my linux.zip file that the camera had decided it didn’t need anymore. So I reflashed my original working files and the working linux.zip from HooToo and … it didn’t work. Well, that is not strictly true. The camera was working, and I could use an additional piece of software to control the camera, and get its video feed, but the website was a pink background with “Page not found” plastered on it. I assume there are some checks in the implementation of that uCLinux that checks the versions of romfs.bin and webui.bin.

For me, the challenge had been won with the HooToo firmware. I had traced the problem back to some shorted pins, and theorised that the problem was with the position of the ARM processor relative to the mounting hole of the camera. The camera was now in a usable state, and the intellectual challenge was over.All that was left was to brute force the firmware issues. I emailed StorageOptions on the off chance that they would take me at my word, and not think of me as a backstreet manufacturer too lazy to play with embedded software. They have yet to reply to me, so I have resorted to creating a table of every possible combination of linux.zip, romfs.bin and webui.bin to find other working combinations. If nothing, this will be some potentially useful information to feed back into the community.

And now, I stalled again. I made the mistake of sharing my progress with the customer (i.e. my father-in-law), and he was so impressed that I’d brought his camera back from the dead that he wanted it back. The website and DDNS functionality weren’t working, but he is fine with that at the moment as you can still view the stream and control the camera from an additional piece of software. I would be surprised if a mobile app wasn’t in existence or in progress, and the DDNS issue can be resolved by remembering your home’s IP address. So it is with a heavy heart that I bid farewell to the camera. It has been fun.

 

Hello Caller, You Are Through To The Helpdesk Part3

Posted on by
8

Following the advice of the people at www.openipcam.com I downloaded the entire contents of flash using Kermit95. Kermit95 is a program developed by Columbia University, but was retired in July of this year. The full source code is available, but as of writing this, there have been no compiled binaries. There exists a command “d” that dumps the data in a specified memory location and Kermit95 was used along with a script found at www.openipcam.com to record every result.

As I’ve mentioned before, the flash chip is split into 5 areas; bootloader, uCLinux, romfs, settings, and webui.99.9% of bootloaders for the Foscam IP camera (and its derivatives, and clones) use the same bootload, and by comparing bootloaders it is possible to see some address and data line problems. My flash chip is a 16bit device, meaning it deals with data in 16bits – and has 16 data lines. The bootloader sits in the bottom 64k, and so uses 12 address lines. If the two bootload sections are identical then you can safely assume that the 16 data lines, and 12 address lines are intact and working correctly.

Moving on to the uCLinux partition, and this is where things got interesting – and we found the fault. We know that the uCLinux is in a ZIP archive, because the Camera tried to unzip “image 7″ (our uCLinux file). Dumping from 0x7f020000 to 0x7F0DF700 gives the *.ZIP file that contains our uCLinux data, as evidenced by the presence of a ZIP header – “50 4B 03 04″ or which includes “PK” in ASCII. PK stands for Phil Katz best known for his work on, you guessed it, ZIP compression algorithms. Part of the header includes the file size, and so we could convert the dump.txt file into a HEX, and onwards to a binary ZIP file, just to check if the unzip functionality of the bootloader was borked. Although I went through the motions, I came to the same conclusion as the bootloader; the file was knackered. As it turns out, memory location 0x7f03000 contained some interesting information.

Upon power up the camera loads the bootloader, and then looks for the BOOT_INFO file at location 0x7f010000. BOOT_INFO contains the camera’s MAC address, IP address, DHCP information, Serial number etc. If the camera can’t find this information, it would query the other hardware, load the values into memory location 0x7f010000 and carry on with its boot sequence. Now image for a second what happens if some damage had happened to the pins responsible for those memory locations. One particular failure mode would be that the ARM would look in the flash at memory location 0x7f010000, but due to an addressing failure, the flash would return the data at location 0x7f030000 (pin A16 being tied with pin A15). Of course 0x7f030000 didn’t contain the BOOT_INFO and so the ARM polled the attached devices for information, and set up a new BOOT_INFO at 0x7f010000. Which would be great, apart from the fact that the flash actually wrote the new data back into 0x7f030000. For those that missed it, the uCLinux partition starts at 0x7f020000 and so NOW has a hole 8k wide. By all accounts, the camera did this to itself

There was no way I would be getting that 8k back, and the only way to move forward was to write to flash a working program. I was 99% certain that I had fixed the hardware fault, and that I would not only write to flash, but the correct location in flash too. But that 1% kept niggling away at me. If for some reason I managed to overwrite the bootloader in flash then I had convinced myself that it would be the end. As it turns out that never happened, and I could always read back the bootloader code and verify it against a previously dumped version – a dodgy address line wouldn’t even matter, as I wrote to the wrong location, so reading from that wrong location is easy. What wouldn’t be easy though is doing anything in Windows7. TeraTerm was supposed to have XMODEM capabilities, but for the life of me I couldn’t get it to work. Fortunately I managed to download an alternative program (ClearTerminal) so all was not lost. Side note: If the developers of RealTerm ever happen to read this, please implement XMODEM.

It took me some time to rationalise my following action; the camera was not usable in its current state – by deleting the current uCLinux there is the possibility that I could find a working version and all would be well with the world. So I started trawling OpenIPCam for a compatible dump. I now knew that my camera was a branded Storage Options, and that it was a clone of a EasyN IP camera which reduced the number of possible candidates. All that is left to do, is find a compatable and appropriate version of software – how hard can that be?

 

 

Hello Caller, You Are Through To The Helpdesk Part2

Posted on by
4

Hands up who thought I would have fixed it by now. You are not alone, I figured this would be an easy fix too. If you remember last post (or care to click on previous post), you would remember that I was concerned by those few pins that looked damaged. Well damaged they were, and a quick touch-up with a soldering iron soon fixed that problem. Did I say fixed; I clearly meant changed.

I have jumped the first hurdle, only to be greeted by a brick wall. Yes, I can see more of the flash. No, the flash is not complete – or at least it is corrupt. I will take this opportunity to thank the people over at http://www.openipcam.com as they have been/are being fantastically helpful. It is pretty obvious that the Flash chip is responsible for all my woes – and also for forcing me to understand ARM processors a bit more. These work differently to how a PIC or Atmel (including the arduino) works. The 4M flash chip is split into a number of sections; the bootloader, uCLinux, the file system, and additional settings or website interface. The ARM takes these pieces of data and copies them onto the 16M SDRAM chip, and runs the software from there. This realisation led me into thinking of this tiny embedded system as a mini-computer instead of a microcontroller; something that I am sure is either entirely obvious to people, or lost on them. Either way, my thinking had changed.

I still had a problem that needed breaking down and I came up with a number of different theories. Either the ARM, flash, or SDRAM couldn’t communicate properly, or the flash had gotten corrupted. I knew that some of the flash was fine, and some of the communication was working as I already had the bootloader loading up, and now I could see what was actually in the flash. However, I couldn’t trust anything I was seeing from the flash, as I knew part of it wasn’t working. If only there was a way to load the SDRAM directly. Luckily we can do just that.

Using the command “mx 0×8000” it is possible to upload a version of uCLinux directly into SDRAM, and “mx 0×700000” allows us to upload a new file system. On XP this is easy. All I need to do is open HyperTerminal, click Transfer -> Send File. All I needed to remember was I was using the XMODEM protocol. HyperTerminal was actually written by a company called Hilgraeve, and was licensed to Microsoft to use in their communications package. Apparently Microsoft did not want to continue paying a fee for this incredibly useful piece of software, and so HyperTerminal is no longer packaged with either Windows Vista, or Windows 7. To get around this, I used RealTerm as my serial communication program, and fall back on TeraTerm when I need to use the XMODEM protocol. These tools are purely my preference, and not the only ones capable. Uploading the two files takes some time, but once they are done all that was left was to type “g 0×8000”. This tells the ARM to start running whatever is found at data position 0×8000, which just happens to be where I stored the uCLinux program. I was able to ping Google, and so I was happy that the SDRAM was working properly, and I started on the long journey on sorting the flash out.

People always talk about data as “ones and noughts”, or “zeros and ones”. This is fine as that is how a computer stored that data – but it is not how the data is represented. Letters are just a collection of straight and curved lines, and yet we that is not how we think of them. I am not going to go into bytes, nybbles, hexadecimal as you already know it, or you don’t want to know it. I will say that the flash chip has numerous address and data lines; the idea being you punch in the address that you want and the chip will pump out the data held in that address. The “ones and noughts” become important now as they make or break those address and data values. Imagine we wanted data location 13. This becomes 00001101, so we would set our address lines high or low depending on those values. Now unfortunately we have a fault on the board such that bit5 is always tied high. Although we want 00001101, the flash chip sees 00011101 (or location 29) and gives us the data held in that location. This works the other way too, as writing to location 13 would not only overwrite location 29, but also leave 13 alone. There is no easy way to diagnose these problems, other that a multimeter and a steady hand. Alternatively, I could solder flying leads to each of the pins of the flash chip, and the appropriate pins on the ARM, and check the transitions with my Open Bench Logic Sniffer.

Again, if you checked my last post, you would see that I gave up on using the Logic Sniffer. Or should I say I gave up on it for the night. The next day I took it to work, followed the same steps and “hey presto” it worked first time. I am now running the latest PIC and FPGA code, and it works great…at work. For some reason, it still doesn’t play nice with my home PC. There must be some quirk with the combination of Windows 7 and a 64bit OS that means it just won’t play nice. It appears as a comms port, but software doesn’t appear to be able to talk to it. I am chalking this up to be an issue with the Microchip drivers, as I can’t open the port in any other program either. I went as far as downloading Microchips latest Application Library (containing the latest USB Framework), in an attempt to get a working virtual com port, but even that didn’t work.

Moving back to the camera. Although the boss is aware that I am bringing this problem into work, I set it apart just for breaks and lunch time, and the occasional early start and late finish. As part of the process I have had to take the PCB out of the case. This makes probing a lot easier, but also has the added benefit of disguising its true purpose if the MD walks past my desk.

 

Hello Caller, You Are Through To The Helpdesk Part1

Posted on by
2

I was getting ready to have an enjoyable weekend catching up on my free Stanford University courses when I received a phone call from my father-in-law. This usually means that some piece of technology has gone wrong and I take the role of helpdesk.  Before you all start drawing conclusions from this; he is fairly technical and so this call means something has gone seriously wrong, and I actually enjoy these challenges. Today’s challenge was a no-name pan/tilt IP Camera.

 

The first step to troubleshooting a problem is to isolate and repeat it. Normally upon power-up the camera would perform a single sweep on both axes, and sit in a centre position. This was not happening – so either the thing wasn’t powered, or the firmware wasn’t running. Plugging a network cable in lit up the status and activity LED’s so at least we knew that it was getting power. I didn’t really want to open the camera yet (purely as it wasn’t mine), so opened up AngryIPScanner and Wireshark. I couldn’t find the device on any subnet despite the activity light flickering. There was no other option, I had to crack this bad boy open.

 

There are a number of chips on the PCB (# ESTESPCB613M136) , including a Nuvoton W90N745CDGARM (main ARM processor), Winbond W9812G6JH (2M SRAM), DM9161 (phy chip), and a WM8731 (audio codec). The other side included a Stansion S29GLO32 (another Flash chip) and a WiFi daughter-board. All in all, not an overly complex board. Due to a poor design choice, the threaded mounting hole is directly above the ARM chip. There was a light witness mark on the ARM, but I was more concerned by the marking on a number of pins – almost as if the solder was squashed. Unfortunately, I don’t have a decent iron at home, so a fix will have to wait until I get back to work. I turned to Google to get more information on the camera. I didn’t have access to the box, so I had no idea who the manufacturer was, but I found out that Foscam used to OEM extremely similar looking cameras in the past, and the likelihood is that this is either one of the OEM variants or a clone thereof.

A typical feature of the Foscam camera range is the presence of a UART header connection, and this was no exception. Although it wasn’t labelled, a quick check of the N745 chip datasheet identified the pins as thus. This enables a terminal connection to the ARM’s bootloader upon power-up. I was using my only remaining Max232 in a different project so I took the opportunity to revive my Open Bench Logic Sniffer. It needed updating to bring it to the current revision, but once that was done I could start on the hard stuff. And here we hit a snag. I updated the Logic Sniffer, and now the USB won’t enumerate. Clearly I am not having a very good day.

 

So before I could start fixing the IP Camera I had to now fix my Logic Sniffer. This was as easy as soldering some header pins to the ICSP and use my PICkit3 to reprogram the PIC. And by easy, I meant I gave up and hit the forums. I had to resort to hunting down my old PICDEM 2 PLUS board (circa 2002) and use the MAX232 onboard. I had to use the same trick to get the keys off my 360 DVD-drive, but that is another story. With fresh wires soldered onto the camera board, and plugged onto the PICDEM2 board, I loaded up RealTerm. Powering up the camera board revealed that the ARM is working (at least part of it), as I get a nice serial interface.

 

Routing around in the terminal revealed that only the bootloader is present, and there is no sight of the linux.bin or romfs.img files that I was expecting. Given this I can draw one of two conclusions; either the data on the flash chip has become corrupt, or the pins with witness marks are actually broken and the ARM can’t talk to the flash chip.

Stay tuned for Part2, where we find out if the camera can be recovered.