Browser-Based Data Logging

Finally! After several months the Black Tower has a general-purpose data logging program for the Atmel interface. There were some distractions: I had to take the cylinder head off my truck engine and put the thing back together. And there was the dead tree that fell across the power lines coming into my house. But after several week-ends coding away in Perl, the results can now be shared.

The idea was to have a completely HTML-based GUI to the Atmel ATMega32 midcrocontroller interface. Data logging could be configured from anywhere in the world via the Internet. Each input channel of the ATMega32 could be configured so that the data was recorded in engineering units or Vdc. The readings from each channel can be displayed on demand and data files downloaded directly into Excel using FTP.

In addition, I completely customized a webpage to generate graphs of the collected data using a Java freeware linegraph applet. Up to five traces can be defined and the X and Y limits of the graph set by the user. A sample screen is shown below.

I have been running the data logger continuously for a week or so and so far no problems. There are a few bugs still to fix in some of the Perl code, but it pretty much works the way I had intended.

A detailed description of the data logger is given in the Projects section of the University Linux website at www.documatrix-usa.com. If you have any suggestions or want a copy of the perl programs, e-mail me.

Browser-based Data Logger for Serial Interface 

Adding ATMega32L Digital I/O to the Webpage

Once the ATMega32L analog inputs were accessible from a browser, it was time to add the digital I/O. The easiest thing to add was the digital input states. It was just a matter of having the ATMega32 analog input server also request the digital input states and then appending this to the output string of analog data sent to the client. The digital input string consists of four characters, one for each input, and are 1 (for an open input) and 0 (for a closed input). The ATMega32 has internal pull-up resistors on each input, so you simply connect an SPST switch from the input to ground. When the switch is closed, the input goes low; open the switch and the input is pulled high. On the client side, the four character string is parsed out channel by channel and printed to the bottom of the webpage containing the analog input readings. The designations are Open or Closed depending on the character received in the string from the server.

All this seemed to work well enough. I wired in a dual channel DIP switch to ATMega32L digital inputs 0 and 1 on the protype breadboard, and could view their state on the webpage over the LAN. The next step will be to use some contact closure type sensors. This might be a limit switch to detect position of some moving part. Or maybe a door sensor to detect if the door is closed or open. (These sensors are available anywhere home security devices are sold.) Magnetic switches are another possibility. These close in the presence of a permanent magnet. So you could know the position of a moving part without any physical contact with the switch.

Setting the digital output bits proved to be a bit more complicated. The ideal webpage interface for this is a checkbox that reflects the state of the output. But the state of the output bits must be stored somewhere so that the webpage can be initialized on first viewing. When the state on an output is changed, the ATMega32L must be updated, the new output states stored and the page re-printed thru the browser reflecting the current values.

The original idea was to have a single server handle all of the ATMega32L serial communications for reading inputs and setting outputs . The server for reading the analog and digital inputs is very simple. A client calls it and receives a string of values in return. Modifying the server to first accept a string output states added another layer of complications, so I took the easy way out.

The Perl code called by the browser that displayed and saved the digital output states was modified to include the serial port calls to the ATMega32L for setting the state of each digital output bit. I added two new navigation links on the left margin of the Black Tower homepage: ATMega32 Inputs and ATMega32 Outputs. The inputs page will fetch the analog readings and digital input states and displays them on a webpage. The outputs page fetches the current state of the digital outputs as checkboxes on a webpage. On both pages buttons are used to actuate the new settings or fetch new readings.

For an initial experiment, I wired an LED to digital output 1 and turned the photocell light sensor to be near the LED. The analog readings can pick up the change in photocell resistance when the LED is lit. So now there is positive feedback for the state of the digital output. So far there have been no glitches in the serial communications to and from the ATMega32L.

I now have two channels of digital inputs and two channels of digital outputs working smoothly. The next step is to add some relays and contact closure sensors so that real work can be done.

ATMega32L Controller Webpage 

Perl Code for the ATMega32

Once the ATMega32L circuit was operational it was time to write some Perl code so that it could be accessed in University Linux. The first priority was to have a way to view the analog input channels from a browser. The best way to do this is to write a server in Perl so that any application running anywhere on the Black Tower can call up data coming from the ATMega32L.

I adapted the Perl server from the temperature sensor. I simply changed the port number (to avoid conflicts) then added some code to request the counts from each analog input channel via a call to the serial port. Perl treats any device just like a file, so it was a matter of opening /dev/ttyS0. Perl is a bit fussy about reading and writing to the same file, so the eventual open statement looked like this:

open (SPOut, "+>/dev/ttyS0");

I wrote a "0" to the serial port, and then read in the reply from the ATMega32L in counts, multiplied this by 5 and divided by 1024 to get the Vdc reading for analog input channel 0. Since there are 8 channels, I put this into a loop. Turned out that the serial port had to be opened and closed to get each separate reading. This was not as elegant as I had hoped, but it seemed to work OK.

The next problem was how to pass the values back to the client. I decided a single long string would be easiest, and individual readings could be parsed out from that. For this to work, however, each channel reading had to have the same number of digits. A reading of 4 volts, for example, would have to be 4.000 and not 4.0 or something else. I decided that three digits to the right of the decimal would be sufficient since the 10-bit A/D converter was reading only to 5 mV.

The solution to this was to take each reading and pad it by concatenating three 0's to it, then extracting a substring consisting of the first five characters. This seemed to work in every case and produced consistent 5 character readings for all channels.

All 8 readings, now five characters each, were concatenated together, separated by a space. This was passed back to the client by the Perl server. The client code parsed out each reading from the large string, and created a webpage displaying all the readings in Vdc with a channel label. A button on the webpage calls the server and generates a new page of readings. When the page is first viewed the readings are blank, but pressing the "Get Readings" button fills the page with current values.

The update goes fairly fast considering that the webpage has to enable a Perl client that calls a server that has to open and close a 2400 baud serial port 8 times. It has proven reliable in a few days of continuous testing; serial ports can be prone to locking up. Everything is situated on the Gamma machine in the Black Tower which is the newest and fastest PC. The ATMega32L controller page now is accessible from the Black Tower homepage, as shown in the screen shot.

The first application for the new interface was to connect a photocell in series with a resistor and the +5 V supply to form a light level sensing circuit. When it is dark, the photocell circuit puts out about 4.9 V, and when in bright light, about 1.5 V. This signal was connected to channel 1 of the ATMega32L analog inputs. I put the work light directly over the photocell and used the X10 control page to turn the light on or off and to vary the brightness. The photocell signal gives a positive indication of a change in light level, so now I can confirm the state of the X10 light switch remotely using the ATMega32L web page.

The next step is to add digital input and output controls. But to do this, I will have to figure out how have the Perl client pass a value (desired switch state) to the server calling the ATMega32L. Taking over the world is never easy...

Microcontroller Interfacing

Everyone knows the world will be taken over by microprocessors. To control the world, however, you need a microcontroller. A microcontroller is a sort of muscular cousin to the microprocessor: slow, not too smart, but able to deal effectively with the outside world. So to bring the Black Tower to its full potential, there needs to be an easy way for Linux to interface to the real world through microcontrollers.

There are thousands of microcontrollers and choosing one can be confusing. Probably the most popular microcontrollers are the PIC series, but after looking around, I settled on the Atmel ATMEGA32L. The ATMEGA32L has a number of advantages and features that made it the best choice for the Black Tower project. First of all, the ATMEGA32 is available as a through-hole component. That means you can use it with breadboards and other easy-to-get prototyping tools. Most of the newer microcontrollers are coming out as surface-mount only. And this is a big problem for those of us who aren't going to spend hundreds of dollars developing a circuit board and contracting for outside assembly.

Another feature of the ATMEGA32L is it has a lot of I/O capability: 8 channels of 10-bit analog input, 8 bits of digital I/O and 2 counter-timers. Most microcontrollers have only 8-bit A/D converters and a limited number of other I/O. With the ATMEGA32L you can specify each of the digital bits as input or output in software. And the output bits can sink 20 mA so they can be used with solid-state and low-current reed relays to switch real 110 VAC power. And the ATMEGA32L is self-contained: no clock crystal is needed and it has a built-in serial UART so it can be configured to communicate with a PC running University Linux.

But the most compelling reason to use the Atmel microcontrollers is the software support. BASCOM offers a Basic compiler for the Atmel microprocessors that supports all of the I/O and communication functions. So you can write code for the ATMEGA32L in a high language, compile and burn it into non-volatile memory and you are ready to go. I found an Atmel training board from Futurelek that provided all the cables and supporting circuitry needed to develop and test my ATMEGA32L software.

There were the usual false starts and mistakes to be made, as with any new project, but I quickly got the Bascom code to upload and work on my ATMEGA32L in just few days. After that I built up a stand-alone circuit for mounting to the Black Tower. You need a level-shifting IC for the serial port because the ATMEGA32L port is TTL and the PC serial ports are based on 12 V signals. But the whole thing fit on a protoyping board I got from Radio Shack, including a +5 V regulator circuit for power. A description of how to build this will appear on the Documatrix website as a new project.

The ATMEGA32L accepts a series of commands from a PC serial port and then responds to those commands with information. If you want the analog voltage reading from one of the A/D input channels, the PC simply sends the channel number. The ATMEGA32L then responds with an integer value between 0 and 1023 that represents the magnitude of the voltage reading from that channel. To set one of the four digital output bits, the PC sends the letter A and the ATMGA32L sets bit 0 low; sending the lowercase a sets bit 0 high. (This seems backwards but to light an LED or drive a relay, you need to set the bit low. "A" seemed a better choice for "ON".) The input bits are read from commands W, X, Y or Z and you get back a value of 0 or 1 that reflects the state of that input bit. There is some handshaking and error messages, etc also.

So now all that is needed is some Perl code to communicate to the ATMEGA32L over the PC serial port. And then the fun begins, because you can easily enable the Perl code from a browser page and get readings and control things using the ATMEGA32L from anywhere over the Internet.

Adding Temperature Readings

The next phase for the Black Tower was to get the temperature sensor hooked up and running. The sensor interfaces to the PC parallel port and provides a 12 bit reading for ambient temperature. This is a project on the UL website, so I reconstructed the Perl code that reads the sensor, converts the bytes to degrees and writes a webpage. I FTP'd the relevent files into Sigma, connected the sensor and tested it successfully.

The next step was to organize the webpages. What with the X10 control page, and the three temperature pages, I needed some way to access everything conveniently without typing in the explicit URL and web page each time. I made a simple webpage that included frames so the navigation is by links in frame that makes up the left margin of the web page. Now I can go to the X10 pages, and do control, or view the temperature pages and get the current reading or look at a file of readings.

I soon had everything working, more or less. The graphing script for the temperature file seems to be malfunctioning. I expect I will have to do some debugging on the Perl code that parses the readings out of the ASCII file. But this should be straightforward.

The Black Tower Control Page now looks like that shown below. As functions are added, I can just add a new link to the frame on the left. New webpages will appear in the main center frame. This scheme also eliminates the need to do a lot of browser backspacing when trying to get several current temperature readings, for example. The Perl scripting is fairly complex and actually re-builds the entire temperature webpage. Just clicking on a link in the left-hand navigation frame is much faster.

Once I get the temperature graph working agin, I will start integrating a new serial analog/digital interface and attempt to hook up more sensors...

Control Web Page 

Telnet Magic

You really have to love Telnet. So simple, so unassuming yet so very useful. Now that I have Sigma running as the server with X10 functions operational, I decided to try an experiment.

Today I powered up both the Sigma and Beta machines. Sigma at IP 192.168.0.125 has all the Internet ports and services forwarded to it from the router. I changed the IP for Beta to 192.168.0.122, so it is just another machine on the LAN and normally not accessible from the Internet.

When I got to work, first thing I did was to Telnet into Sigma at my static Internet IP address. The log-in prompt came up, as usual, and I logged in as root. After starting the webserver, I decided to try Telnetting from Sigma to Beta. This proved to be as simple as it sounds. While logged into Sigma, I typed in "telnet 192.168.0.122" and the Beta log-in screen appeared. I logged in as root, and now I was inside Beta looking at files, directories, etc, just as if I was sitting at the Black Tower console.

Another plus for Telnet is that when I use HyperTerminal from my Win XP computer at work, I can run Pico on the target University Linux machine. So I can edit and write files remotely. This makes remote programming possible.

Backing out was similarly straightforward. While in Beta, I typed "logout" and I was out of Beta, but still inside Sigma. I logged out of Sigma and the Internet connection was terminated, just as always.

For my next experiment, I telnetted back into Sigma and invoked FTP to the Beta machine. The FTP log-in came up and I logged into Beta as ftp (since no password required using that username). I changed directories under FTP and used the GET command to transfer a file from Beta into Sigma. And it worked, just as if I was sitting next to the Black Tower.

The robust character of simple telnetting suggests that I can probably use this to control various operations as I add them to the different machines on the Black Tower. One thing is for sure, I will have to remember how to add a machine identifier to the screen prompts. I will also probably do some housekeeping with the hosts file so I can use machine names instead of IP address. This will make telnet navigation from Sigma much easier.

I will also probably try to write some scripts that invoke FTP to move files between the various Black Tower machines. I don't have NFS running on UL, so maybe I can come up with something similar, if a bit kludgy, with some clever Perl programming. Anyway, lots to try and do!

Adding More X10 Functions

With the ON/OFF and light level control for my X10 lamp module now working via webpage, it was time to add a second X10 control module. In addition to the A1 lamp controller/dimming module, I have an X10 controller for inductive loads like fans or small motors and appliances. I dug out the motor control module, set it to X10 address A2, and plugged it into a handy power strip. Next, I plugged my table radio into the A2 module and verified that everything was functioning correctly by switching A2 with the handheld X10 control pad. The new X10 module was working reliably, even at some distance from the receiver module.

The next step was to test it with the Bottlerocket drivers running on Sigma. I typed in a command line for the new A2 module, and, as expected, Bottlerocket was able to switch A2 on and off reliably. Next came the hard part.

In order to make A2 accessible from the Internet, I had to modify the existing X10 webpage form and Perl scripting needed to call Bottlerocket. I decided to control the A2 module from a checkbox on the webpage. I had already used some radio buttons for the state of the A1 module. And using a checkbox is actually a bit simpler. Even so, I had to revise the reading and writing of data to the status file, detect the state of the checkbox and create the correct Bottlerocket line command syntax. After about two hours I had it all working, but it was not easy.

I started by using two Bottlerocket system calls, one for A1 and one for A2. This caused confusion in the X10 system. A1 was coming on when it was supposed to be off and A2 was supposed to be on. After some trial and error i discovered that the Bottlerocket command line can be structured to send commands to more than one module. So I created one system call to include the state of both modules and the dimming parameter for A1.

This fixed the problem. The new X10 control screen is shown in the posting below. it was a bit of an exercise to get the form working, and to pass the correct state of each module to the X10 status file, and to subsequent webpages. (When you Submit the page, it executes a Perl program that creates the Bottlerocket command line from the form controls, saves the status of each control to the file, calls the Bottlerocket driver and then re-builds the webpage/form with the state of all controls reflecting the previous choices.) So there was a fair amount of de-bugging involved.

But now I have complete control via the Internet over all the X10 modules that I own. Time to move on to a more challenging interface...

New X10 Controller Screen