Dec 98 Factory Floor
Volume Number: 14
Issue Number: 12
Column Tag: From The Factory Floor
A PowerPlant Update, Part 3
by Frank Vernon and Dave Mark, ©1998 by Metrowerks, Inc., all rights reserved.
This month's column is the third installment in our series on PowerPlant. In the past
two months, we heard from two key members of the PowerPlant team, Greg Dow and
John Daub. This month, we'll hear from another key member of the team, Frank
Vernon.
Frank Vernon has been working for Metrowerks since CW9. Before joining
Metrowerks, Frank worked at Apple doing R&D for first eWorld and later Internet
Services in their Boulder, Colorado group. When Frank joined Metrowerks, he took
over the networking classes from Eric Scouten and completely rewrote them for
CW11. The first of the Internet Classes were introduced around CW9 as well. Stuffit
classes and Internet Config classes were added right after CW11.
Frank also worked on the 1.1 update to Marionet for Allegiant Software (SuperCard,
etc.). Marionet is a faceless background application that utilized Apple events. It
supported SMTP, POP3, NNTP, FTP, and a custom chat client/server.
In addition to his engineering background, Frank also has a degree in music.
Dave: Tell me a little bit about how you got started with PowerPlant.
Frank: Before MW I was doing R&D for Apple in a small autonomous group based in
Boulder, Colorado. Our group initially began working on eWorld and later evolved into
the Internet Services division as eWorld was being dismantled. This was an interesting
R&D group to be involved with in that we did simultaneous technology R&D and
business modeling. We were focused on creating technology directly applicable to the
business models we thought would be successful rather than being a "pure" R&D
technology group. I personally found this type of business and results driven R&D to be
quite rewarding.
After leaving Apple in early 1996 and while at MacWorld San Francisco I heard a
rumor that Eric Scouten, the original author of the PowerPlant Networking Classes,
was looking for someone to take them over so he could concentrate on Constructor
full-time. I eventually got in touch with Eric and began working on the classes. An
interesting story here is that I still, to this day, have never met Eric face to face.
Despite having worked relatively closely with him for almost 2 years, as I first
started taking over the classes and then began rewriting them we have never met in
person. We played "booth tag" that year while at MacWorld and have continued to miss
each other at subsequent MacWorlds and Developer Conferences. The majority of our
work was done via Email and occasionally we would have a phone call to hash out the
nastier details.
Dave: Tell us about the evolution of the networking classes.
Frank: Long before working at Metrowerks, Eric Scouten had created a very nice set
of MacTCP based networking libraries called TurboTCP. When OT was first being
released, he had the idea to abstract MacTCP and OT into a single library so developers
would not have to deal with developing seperately for both in their applications. Since
he was working at Metrowerks at the time, he created them as PowerPlant classes.
Being heavily involved in Constructor, however, he didn't have time to really get them
into shape. In fact, when I took them over for the CW9 release they were still in the
PowerPlant "In Progress" folder.
When I took over the Networking Classes the basic groundwork had been laid, but they
did have several major limitations. The first thing I added was "Listen" support. This
allowed them to accept incoming connections so they could be used for network servers.
Prior to this they operated in client mode only. I was also fixing lots of bugs and
preparing them for release outside of the "In Progress" folder.
Abstracting OT & MacTCP into a common interface was an interesting prospect to say
the least. Besides the obvious complexity of trying to make the two disparate stacks
behave in the same way, we also encountered numerous little problems. Not the least of
which was simply testing them.
For each feature or bug we encountered they had to be tested in all possible
combinations of MacTCP and OT, 68K and PPC, Asynchronous mode and Threaded mode,
and with and without Interrupt notifications. This alone made for many long and
frustrating nights of debugging. As anyone who's done network programming can tell
you, even on a good day, debugging network code can be like searching for a needle in a
needle factory.
In addition, Eric's initial designs were quite ambitious. Not only were we trying to
support two radically different TCP/IP stacks through a single interface, but we were
also attempting to address all of the common network programming styles such as
asynchronous and threaded operations while also maintaining an interrupt driven
notification model. These interrupt driven notifications were particularly difficult to
manage due to their memory allocation limitations and debugging restrictions. While
not impossible to comprehend, most developers simply didn't want to deal with all the
complexity that these disparate paradigms introduced into the classes. After going
through several releases we had a lot of feedback that they were simply getting too
complex for the needs of most developers.
After CW10 Eric and I worked out a major overhaul to the classes which is what you
see today in the Networking Classes. From the feedback we received we decided to
streamline the classes and make them as stable and as easy to use as possible while
still maintaining the initial design requirement of abstracting MacTCP and OT into a
single interface.
One of the major advantages that we had in redesigning the classes was that the Thread
Manager was also reaching maturity. Prior to this time, most network programmers
used heavily asynchronous, event driven, programming models. Without having
multiple threads available you simply couldn't block and wait for a network event to
complete. This could tie up your application or even the entire machine for each call
which of course meant that your interface and the whole machine would be
unresponsive while you waited for completion.
By settling on a threaded model for the new classes we were able to make several
major improvements. The present classes, for instance, hide 99% of the issues related
to interrupt driven events and associated memory and debugging issues from the
developer. Also, in my mind, the threaded model of network programming is much
easier for casual network programmers to understand and work with.
The threaded model allows for radically simplified state machines with most higher
level protocols. Code that might have been thousands of lines before could often be
reduced to hundreds.
Subtle side effects of this model also allow for more sophisticated objects like the
current LInternetAddress object. This object now transparently translates between
canonical DNS and IP dotted decimal style addresses. Without the straight-forward
flow control of the threaded model this would not possible.
For example, the simple act of converting an IP address into a canonical host name in
the past would have required a multi-stage state machine to negotiate the DNS
interaction to resolve the address. In the present Networking classes you can simply do
the following:
LInternetAddress theAddress("208.226.102.11");
Str255 theConnonicalName;
theAddress.GetDNSDescriptor(theConnonicalName);
Dave: What about support for other networking protocols such as
AppleTalk within the Networking classes?
Frank: For strictly historical reasons, the initial concept behind the Networking
classes was only to abstract MacTCP and TCP/IP aspects of OT into a common library.
We do receive sporadic requests to support other things like AppleTalk and some of the
more sophisticated OT features from time to time. With Apple in the midst of major
updates around Carbon and beyond we are in the process of considering our options for
the future. We are actively seeking input from the developer community in this
matter. As always, we encourage people to contact Metrowerks tech support with any
suggestions and requests they have along these lines.
Dave: Speaking of the future, what about Apple's planned changes to a
Sockets-based networking model? How will this effect the Networking
classes?
Frank: As I mentioned, we are still in the process of determining how to move
forward towards Carbon and beyond. I can tell you however that Metrowerks
recognizes the importance and growing ubiquity of Internet protocols and networking
in general to modern applications. We are going to do our best to support whatever
paradigms Apple chooses moving forward.
Dave: Can you describe the difference between the Networking classes
and the Internet classes?
Frank: Based of course on the Networking Classes, the Internet Classes address
specific Internet protocols. Presently these classes support: HTTP, SMTP, POP3, FTP,
NTP, FINGER, and IRC. In addition, they support associated standards such as Internet
Config, MIME, MD5, BinHex, UUEncode/UUDecode, MacBinary II, URL's, and others.
The design philosophy behind the Internet protocols was to represent the protocols as
faithfully as possible while also supplying simple interfaces. For instance, many
protocols have interfaces such as LFTPConnection::PutFile which take all the necessary
parameters such as the name of the remote host, username, password, the name of the
file in question, etc. and simply send or retrieve the files. The developer doesn't need
to know much if anything about the protocol itself. If however they need to do more
sophisticated procedures they also have complete control over the protocol at each step
in the process.
As much as possible I've also tried to leverage other features of PowerPlant to keep the
usage of the Internet Classes familiar to anyone who has used PowerPlant. For
example, I use the standard Broadcast/Listen mechanism for progress and status
reporting during the protocol execution. This allows for easy and familiar interface
creation just like any other control.
The Internet classes also have a few nice features like the transparent use of
temporary file buffering to allow the sending and receipt of arbitrarily large files (up
to available disk space.) This buffering mechanism (represented in LDynamicBuffer)
will automatically and transparently make decisions about either keeping data in
memory or swapping out to temporary files when necessary. This is all hidden behind
standard PP Stream classes but the developer never need know if the data is in a file on
disk or resident in application memory. This greatly simplifies things such as large
FTP file transfers.
In addition to the simplified interfaces available with the protocols themselves, I have
also recently added a new utility class called UInternetProtocol. These utility methods
even further simplify the process by simply taking a standard URL and doing all the
work behind the scenes. These methods will automatically do things such as creating a
new thread, parsing the URL, and resolving the URL. These are especially useful for
adding simple URL based Apple event support to any application. (Little known tip: The
Internet Example application also demonstrates use of the GURL and FURL Apple events
and includes the AETE resources. Try setting the Internet Example Application as your
default FTP client in Internet Config and click on an FTP URL in your browser.)
UInternetProtocol FTP Example:
LURL theURL("ftp://user:pass@host/filepath");
UInternetProtocol::FTP_GetFile(nil, theURL, nil);
These two lines are all that are necessary to get the FTP file at the host specified using
the name and password imbedded in the URL. If no username and password are included
then anonymous login will be attempted.
Wherever possible defaults from Internet Config will be used for all the
UInternetProtocol methods. This includes download file directories, default servers,
usernames, passwords, email addresses, etc.
Dave: What can you tell me about the StuffIt classes?
Frank: The StuffIt classes grew out of a number of requests we had to somehow
integrate file compression and decompression into the Internet classes. The people at
Aladdin were quite helpful in allowing us to include the API to the StuffIt engine with
the CodeWarrior releases.
What I attempted to do with these classes was to simplify the StuffIt Engine API by
making assumptions about some of the more common usage. For the most part these
also hide a lot of the necessary API requirements about properly opening and closing
the engine, handling error conditions, and properly navigating archives. I've also
attempted to represent the Engine's API into OOP methodologies so C++ developers
could easily integrate its usage into their applications.
UStuffItSupport Example:
void
StuffItExample::StuffItGetFile()
//Do a StandardGetFile for the file to Stuff
StandardFileReply theReply;
::StandardGetFile(nil, -1, nil, &theReply);
if (!theReply.sfGood)
return;
//Stuff the file assuming the same file name, etc.
UStuffItSupport::StuffFile(theReply.sfFile);
DisplayStuffItError(err);
}
}
By comparison, the direct interface to the StuffIt Engine API to stuff a file is:
extern pascal OSErr StuffFSList (long magicCookie,
FSSpecArrayHandle fileList,
FSSpecPtr archiveSpec,
FSSpecPtr resultFSSpec,
Boolean compressOriginal,
Boolean deleteOriginal,
Boolean encryptOriginal,
Boolean resolveAliases,
Boolean noRecompression,
short conflictAction);
As you can tell, the StuffIt Classes go a long way towards simplifying the common usage
of the StuffIt Engine.
Another class, LstuffItArchive, allows you to work directly with StuffIt Archive files.
With this class you can walk an archive, insert or delete files from an archive, create
new archives, etc..
By the way, I should mention that a license from Aladdin is still required to ship the
StuffIt engine with an application.
Dave: I know you've done some work on email servers. What kinds of
changes have you seen recently in email server technology?
Frank: Actually, there have not been a lot of significant changes in email server
technology for quite a while. The basic protocol, SMTP, has remained virtually
unchanged for quite some time. While extensions are added to SMTP from time to time,
it has generally proved to be a very effective, if not sophisticated, protocol. I believe
in fact that its lack of sophistication is probably directly responsible for its longevity.
(A lesson from which we could probably all learn something.)
The primary change is that you are now seeing more and more customized servers
acting as gateways into other non-SMTP based systems such as Lotus Notes.
In addition, many of the older servers were never designed to deal with the larger
volumes of mail you now encounter on the rapidly growing Internet. Many of the larger
ISPs for instance have a lot of problems simply with volume. These older mail servers
often require either complex reconfigurations or complete rewrites to handle high
volumes.
We're also beginning to see more task specific email servers. Servers that focus on
mailing list management and other automated mail handling tasks.
There are lots of startup companies out there dealing with email issues. Things like
automated mail responders for customer support, customized email reminder
services, and purchase transaction systems just to mention a few.
Email is arguably the most popular protocol on the Internet. It's the original "Push
technology. It's strange in way that only now are people really concentrating on email
as "the" way to interact directly with customers.