Saturday, May 8, 2010

Tcl operations, and many different zeroes

One reason for the title of this blog that I've explored fairly little up to now is that I'm one of the gnomes who make the Tcl programming language go. For this reason, many of my computer applications, when they start up, depart from a script - a Tcl script.

Last night, a colleague pointed me to a thread on linuxquestions.org that begins with a misunderstanding of how data are interpreted in Tcl, and diverts into a screed that Tcl is "objectively bad."

Fundamentally, the original misunderstanding appears to be one of notation. The original poster complains how Tcl is 'clumsy in dealing with binary data.' He has a string comprising a single character, for example 'B', and he expects to be able to write, [expr {$char & 0x80}]. Needless to say, that expression fails:

can't use non-numeric string as operand of "&"

Why is this? Well, fundamentally, in Tcl, Everything Is A String - it's one of the language's guiding principles. That principle means that there is no difference between the string "12" and the number 12: they comprise the same characters, hence they are the same entity. A side effect of this rule is that there is no difference between the character 0 and the number 0: again, they are the same string of characters. This unity is a tremendous convenience when you are using Tcl as a shell (or for writing scripts that serve the same purpose as shell scripts): you don't have to add a lot of excess quotes just to distinguish the types of things.

It gets in the way only when you are trying to do low-level stuff like parsing, such as is done in C:

for (i = 0; (c = str[i]) != '\0'; ++i) {
    /* do something with the character c */
}


Tcl allows similar processing that (in my opinion) is just as convenient:

foreach c [split $str] {
    # do something with the character c
}

But inside the loop, the character c is a one-character string in Tcl; in C, it's an integer whose value is the representation of the character in ASCII (or some other encoding).

If you do want, for some reason, to iterate over the bytes of a string as integers, it's easily done:

binary scan $str c* chars
foreach c $chars {
    # do something with the integer c
}

If there were demand for it - and as far as I'm aware, nobody's ever asked - Tcl's expression engine could be modified to accept single-quoted characters as integer constants. It appears that to Tcl's users, the existing facilities to manipulate bytes are Good Enough - partly because of Tcl's culture of extensibility. If you need high-performance, low-level programming, you do it in C and provide a Tcl API.

And so the original poster's question could be easily dismissed as "not understanding the language" - were it not for the somewhat vitriolic posts that followed. The one that rankled in particular said,

Well, TCL is an objectively bad language because it is (used to be ?) a true source code interpreter. So, syntax errors are not found until containing them code is executed.

For example, in VLSI synthesis may take several days, and it's quite a pity if the whole process ultimately fails because of a silly syntax error at the script end.

That's why static TCL checkers have been invented ...

"Objectively bad?" Oh, my.

Given that static Tcl checkers like Nagelfar do exist, most of that poster's screed can be reduced to a complaint that Tcl doesn't force you to use them. Surely, if you're going to invest days in a VLSI synthesis, you'll want to ensure that you've bulletproofed your program in every way possible: perhaps those who are dealing with lighter-weight tasks where you can fix a script and rerun in literally seconds have different sets of tradeoffs? Does "objectively bad" reduce to "unfit for this purpose without some extra help?"

Admittedly, I am aware of studies claiming that programmers presented with strongly-typed languages debug their programs faster. But most of them are old, and compare the strong typing of a language like Pascal against the untyped pointers of PL\1 : anyone else remember the disaster that those were? I'm not aware of any studies that both are conducted with experienced programmers (beyond the level, say, of first- or second-year undergraduates) and are evaluating modern languages. The folklore says, "the more checking up front, the better," but the actual experience with up-front program checking always seems to reach a point of diminishing, nay, negative returns. After that point, adding additional strictness simply makes the programmer jump through hoops arguing to the compiler that the program is correct before the system will deign to run it. When that point is reached seems to vary, so there's certainly still room for subjectivity in evaluating it; in my opinion "objectively bad" is far too strong a statement.

And the "strongly typed" languages fall prey to the other problem: runtime exceptions (SIGSEGV in C, NullPointerException in Java, etc.). Tcl essentially never gets those: if a Tcl script manages to cause one without help from extension code in another language, it's "all hands on deck" among the maintainers to fix the bug. Even if the script is obviously erroneous, Tcl's maintainers pride themselves on the ability to catch and report all runtime errors.

But Tcl is unquestionably not fit for all purposes. No language is. If you find that you have an application where it does poorly, I probably have similar applications somewhere, and I'd be happy to recommend other languages to you. For a good many "high-level" tasks, it seems to fit the way I think. Argue if you will that I'm not "right-thinking," but don't assert that your right thinking is objectively better without data to back it up!

(Hmm, is this attitude Tcl's marketing problem? The Tcl community seems to be remarkably free of zealots that assert that it is objectively better than its competitors. If that is a problem, it's a problem that I enjoy having; I tend to struggle to deal with zealots of any stripe.)


Read more...

Tuesday, October 27, 2009

The Information Technology Life Cycle

From the bad old days of Hulking Giant Mainframes to the era of Grid Computing and Cloud Computing, I've observed a distinct life cycle of Information Techology organisations. This little piece is intended to share my anecdotal observations. No properly controlled study has been conducted to validate these findings; they are merely intended as a framework that might guide future research.

It is critically important to note that no criticism is intended of the hard workers who labour in the IT coal mines. They are no more in control of the forces that drive the IT life cycle than their clients (or their managers). Rather, the life cycle appears to be an inexorable consequence of organisational ecology.

Phase 1. Someone goes out and buys a shiny new computer. The proud owner puts some software on it, and uses it to solve some problems. Life is good, and then...

Phase 2. The owner's colleague realizes that the computer provides an effective solution to a problem that the colleague shares. "Hey, can I borrow your computing resources for my problem too?" asks the colleague. "Sure," replies the owner, because most owners of shiny new computers are eager to help their colleagues (and show off that their shiny new computers weren't a total waste of money and time). Now two people are getting their problems solved. Life is even better. Perhaps even more people join in the fun, until...

Phase 3. The computer becomes too big to handle as a side project. Either enough people are using it that they interfere with the owner's own work, or they are starting to interfere with each other, or there has been a failure and people realize that hey, maybe backups would have been a good thing after all. However it happens, there are two outcomes: either the original owner winds up having his job redefined to comprise mostly the care and feeding of the computer, or else the original owner (or his boss) hires someone to attend to it. An Information Technology organisation is born. The newfound support from management allows even more colleagues to get their problems solved. Life is sublime. Except for the poor IT guy, who finds out that...

Phase 4. More and more users come to the IT organization with more and more problems to be solved. Not all of these users are good friends of the computer owner; now they are people that the boss sent over. The IT guy works really hard to solve their problems, but discovers that it's an uphill battle. Some users want one thing, and some want another. Everyone wants an unreasonable quality of service. Work still gets done, but life is deteriorating, until eventually...

Phase 5. The IT guy eventually, to preserve sanity, begins to realize that the real mission of Information Technology is to protect the computer from the users. Anything new runs the risk of breaking existing functionality, so nothing can be done for the first time, ever. New development becomes forbidden. Management starts buying products to lock down system configurations, lest some user change something. The IT department may be outsourced. Work still gets accomplished, because by now the system is capable of handling most of the business's processes. Nevertheless, businesses continue to grow and change, and the system becomes a progressively worse match to the real needs. Eventually, someone decides that in order to make progress, they have to...

Phase 1.Buy a shiny new computer (which by now has become so cheap that they might even do it on their own nickel). Put some applications on it to do the actual work. Use it happily. The cycle has begun anew.

The turning of the cycle leaves systems and IT organisations in its wake. I've been to sites where there is a Hulking Giant Mainframe running business applications like Payroll; a successor to a Department Minicomputer running things like Sourcing; a Scientific Computer Cluster serving a bunch of Engineering Workstations; a whole lot of Personal Computers running Office Automation applications; Web Servers; Database Servers; Application Servers; maintained by half a dozen different Information Technology organizations - all of them ossified, and none of them speaking to any of the others (or to the users) in anything but a sneering tone.

In the spirit of Departing From The Script, I wonder:

Is there a way to step off this endless cycle? Or is it as inevitable as the Software Upgrade Treadmill, and just another manifestation of the Wheel of Life?

The solution, offered by some organizations, of requiring all new computer purchases to go through the IT Department, attempts to break the cycle. This so-called solution merely breeds stagnation, discontent, and eventual mutiny. The mutiny, if successful, merely has the effect of returning the cycle to Phase 1 for the mutineers.


Read more...

Monday, September 14, 2009

Allen organ project - card reader enhancement is done!

The hardware for the card reader enhancement is built! Excuse me while I crow.

Terminal in place on the console

It sits there on the organ console, looking as if it was built to go there. Well, it was.

Mounting keyboard and display was a matter of screwing in a whole bunch of #2-64 screws and nuts:

Keypad and display mounted

Then wiring the circuitry was a pretty routine perfboard job. The front of the board (toward the front panel) has sockets for the keypad and display, and mounts the trimpot that controls display contrast. It also has most of the actual wiring. There's a little scorch mark on one wire that would not stay out of the way of the soldering iron. Nobody's going to see it inside the box, and it's not near anything that's going to short it, so I'm just going to leave it alone. Nobody's perfecʇ.
Front side of the board

The back of the board (the side that faces the inside of the box) mounts the RBBB, the DIN jack for the PS/2 keyboard cable, and a 4-pin socket where the serial interface goes for flashing the firmware.
Back side of the board

Of course, I wound up making up some circuit modifications while I was wiring it up.

First off, I rearranged the pin assignments to give a neater layout on the perfboard with the RBBB turned sideways.

Next, I added a 470 Ω resistor and green LED to provide a 2.5 Hz "blinky light" status indicator that shows the firmware's main scan loop is running. (I've half a mind to have it send the firmware revision number in Morse.)

Finally, I decided that driving the LED backlight for the display - which is rated for 240 mA at a 4V forward voltage - off of the voltage regulator on board the RBBB was a losing proposition. Instead, I put together a little current regulator circuit. The 2N3053 transistor (good for a couple of Watts of collector dissipation) is switched with base current supplied from an RBBB pin via the 2.2K resistor. The 2.7Ω 2W wirewound resistor (it's mismarked 2.4Ω, which is why it turned up in surplus!) senses the emitter current. When the drop across it exceeds the Vbe of the little 2N2222A transistor, the little transistor turns on and robs the base current from the big one. Result is that the LED is driven at a 240 mA constant current, and then is pulse-width modulated for brightness. It's both brighter than the 6.8Ω resistor made it, and more stable with supply voltage and temperature fluctuations. Only thing is that the 2N3053 pass transistor was getting a little warm (on the borderline of exceeding the safe operating area of the "fingertip thermometer"), so I added a little homemade heatsink cut out of aluminum foil from a pie plate in the recycle bin. This technique works pretty well for medium-size devices as long as you use lots of grease to get good thermal contact with the foil.
Hpmebrew heatsink

The board goes into the project box, held in place by the keypad and display sockets in two corners and #4-40 screws, nuts and washers on fiber standoffs in the other two corners.
Putting it in the box

So now I try to flash the firmware (with the revised pin assignments). And ... nothing! Nada! Bupkis! The TX light flashes a couple of times, and avrdude refuses to sync with the chip. Several hours of puzzlement ensue.

Oh, [Nixonian expletive deleted once again]. I already made this mistake once, with the other RBBB inside the organ! And I've gone and done exactly the same thing again. I have RX and TX interchanged. (RX on the Ladyada serial interface goes to TX on the RBBB, and vice versa. Only this time, when I correct it - still nothing. Yes indeed, I've blown the chip. It's pretty forgiving, but this was too much for it.

Oh well, I still have one more Atmega168, sitting in the Arduino. (And I've been wanting to upgrade that one to a 328 anyway). So I swap chips...

And the display shows the menus nicely, and programs the organ.

Stop programming menu

I won't say that the firmware is done, but it's getting there. The big pieces that I want to add are that there's nothing behind menu #5 (where I want to put Hammond drawbar registrations), and I want to add an interface for scaling stops (making them louder or softer). Aside from that, though, it's more usable than the card reader - I don't have to worry about cards falling apart and lamps burning out.

I'm a happy camper for the moment.

A big thanks to Steve for getting me started with his surplus Arduino stuff, and to Carl K2YR for the fabrication assistance (and circuit sanity checking and general willingness to serve as the teddybear that I explain things to).

And, of course, thanks to Mary Ann and Cathy for putting up with all this geekiness.

Read more...

Friday, September 11, 2009

Allen organ project - display terminal front panel

Once again, I've been spending time doing the project and not writing about it. I've come up with a front panel layout for the display terminal, and cut some holes in the project box.

Front panel cutouts


I started by printing a template on paper and putting it on the panel with spray glue. That gave me all the cut locations neatly laid out. As I did the template, I even remembered to displace the display unit a little bit upward so that the pins of the display and keypad lined up to holes on 0.1 inch perfboard.

Next came drilling some 5/16" holes in the middle of the two rectangular cutouts that hold the keypad and display. Then it was crunch-crunch-crunch with an Adel nibbling tool making rectangular cutouts that were (intentionally) a shade too small. (I can cut away more material, but it'd be real hard to put any back!) Then, with a file and patience, I enlarged the holes until they gave a snug fit for the components. (The keypad has a radius on each corner, and I just made that with a Dremel tool).

I used the components themselves as templates to mark the locations for #42 holes for the 2-64 mounting screws that hold the keypad and display, and drilled holes to mount the perfboard and to provide screwdriver access to the display contrast control. The screw holes came out a little ragged, what with being drilled from the wrong side, but the screw heads will cover that up anyway.

Perfboard backs the keyboard and display

The fit of the perfboard inside is "perfect enough." If you click through to the original photo, you can see the socket pins lined up with the holes. That'll give me a base to wire on. The two nylon studs are for mounting the RBBB. [UPDATE: Decided not to use them, and just mount the RBBB flush to the perfboard.]

I also machined a slot into the side of the box below the keyboard to hold the DIN connector for the PS/2-style keyboard cable, and a mating groove on the connector. That one worked out nicely. The connector feels like a rigid fit even without backing it up with hot glue, which I plan to do anyway.

RBBB with PS/2 connector

And I got another RBBB built. I soldered a 6-pin female connector to go to the DIN connector breakout. This box draws its power from the DIN connector and has the data lines going to two of the digital inputs. You can see where I ground away the breakout board so that the slot in the project box can engage the groove on the connector body. (Grinding a FR-4 fiberglass board takes patience and a dust mask.)

Plugging the DIN connector into the LED controller, the pilot light on the power supply lights and the Vcc pins check out at 5V, so I guess I got that part right.


Read more...

Saturday, September 5, 2009

Allen organ project - proof of concept achieved. Facepalm.

Now that the LED control is integrated with the card reader, it's time to see if it actually can program a stop!

Open up the organ, and in it goes. Take the breadboard display terminal, select "Blockflöte 8'" and send it over.

WOOHOO! First try!

Try again - Octa-Bell. "Bing bong bing." Kinura. "Blaaaaat." Festival Trumpet. "ta-ta-ta-DAAAA!"

Time to close up the organ and work on suitable packaging for the terminal.

Oh dear. Oh dear. Oh [Nixonian expletive deleted].

The two rotary switches above the card reader leave absolutely no clearance behind the lamp strip.
There's no room for the rearward extension of the board.

Oh dear.

Fortunately, there's plenty of room underneath the card reader. Back to the bench.

I cut the card reader perfboard in half, and rewire everything so that I have two halves joined with a strip of ribbon cable. I'm a little worried about switching noise getting into the audio, so I try to control it some by having each LED have an independent ground return on an adjacent conductor. (Desolder all the LEDs and resistors. Solder in both ends of all 20 conductors of the ribbon cable. Replace the LED that I overheated in the rework process.)

After all this rework, Frankenreader goes back in the organ and looks like this:

Frankenreader - rear view
From the rear. The logic board extends a little way under the reader.

Frankenreader - top view
From the top. The ribbon cable clears all the hardware above.

Read more...

Allen organ project - integrating with the organ

With the LED control apparently working on breadboard, it's time to get it aboard the organ. Mounting the LEDs and resistors, the RBBB, the programming port, and the PS/2 connector is a fairly simple matter of perf-board construction. (You'll notice that I replaced the voltage regulator on board with a heat-sinked 7805. The current draw of the LEDs was making the itty-bitty 5V regulator that comes with the RBBB get, uhm, kind of toasty. The 7805 is nice and cool.)

The Frankenreader - take 1

The LEDs need to be on quarter-inch spacing, while the perfboard hole spacing is 0.1 inch. But quarter-inch spacing is easily done by mounting every other LED on a diagonal pair of rather than a h0rizontal pair. The row 8 LED (second from top in the picture) is mounted vertically, which displaces is 0.05 inch toward the rear of the organ.

With a little tweaking, the LEDs fit the holes in the card reader beautifully:
The LED control aboard the card reader
The LED pin numbering was a little inconvenient for the perfboard layout, so I did a quick firmware flash to renumber them.

And the PS/2 communications crapped out. I'm getting spurious interrupts.

After a couple of evenings of very frustrating debugging, I finally get it. When I put in the keyboard cable, rather than the short wires on the breadboard, I added enough capacitance that the LOW->HIGH transition got really sloppy (remember, these are open collector outputs!). Adding a line of code to drive the pin HIGH at low impedance for a brief period (just long enough to change the pin mode to INPUT) makes the transition clean enough that the spurious interrupts go away. The data transmission looks solid as a rock.

And as I was testing to make sure that the card slot was unobstructed, I realized.

The photocell for LED 8 is displaced to the rear. Not the front. So it's not in quadrature with the others. Instead, it gets fired independently when all the others are off. A quick bout of editing and another few firmware flashes. (Slow the clock down again, to watch the lights blink, and then speed it back up. Plus debugging.)

The firmware is now at http://kbk.is-a-geek.net:2303/ci/95f796cc4d.

And it's time to try this thing out for real!

Read more...

Allen organ project - lighting the LEDs

Last time, I breadboarded the LED controller and got its serial communications working. Now it's time to fire those LED's.

When the controller is idle, all the LED's are on, as if there is no card in the slot. (Among other things, this convention allows me to continue programming the organ with cards, if I wish.)

Breadboard LED control - powered on

Wow, those LED's are bright.

Now, let's look in detail at what happens when a card is inserted and removed.

[Update: I was wrong about the quadrature signal. The next post explains.]
  • First, the paper blocks all the lamps (LEDs are all off). This condition persists until column 6 lines up with the photoresistors.
  • Next, for each group of three columns, the row 9 hole, and the data in rows 0-6, come into view. LED 9 and the data byte are lit.
  • The row 8 hole comes into view early, because its photocell is displaced half a column. LED 8 lights.
  • The data column leaves the photocells (LEDs 0-6 and 9 go off).
  • The timing column leaves the photocell (LED 8 goes off).
  • About 1 1/2 column of blank space (all LEDs off) passes over, and then the cycle repeats for the next fifteen data bytes.
Once all this is done, the LEDs stay off for a short pause (the card blocks the light from all of them, andt then the entire cycle repeats, reversed in time, as the card is withdrawn. Finally, all the lamps are lit as the card exits the reader.

http://kbk.is-a-geek.net:2303/ci/41508259dd has the code baseline that does this. With the clock rate slowed down to where my human eyeball can track what's going on, the data look right.

Now to move the thing onto the card reader and try it out on the organ!

Read more...