New paradigm of introspective OO development

==  Introduction to DAF ==

  I am interested in the next generation of approaches to software
development.  Over the last decades, there has been an obvious shift
in how our most complex software is created.  We started off with
punch cards (or so I am told -- I was born in 1981 so apologies if my
history is off or my ideas are naive), punching in a stream of ones
and zeros.  The code contained only 0 and 1 and everything else was
abstracted in the mind of the programmer, even the machine-level
instructions.  Then we moved to assembly language, where code more or
less explicitly contains the machine-level instructions, while the
meaning of variables, and the control structures were still largely
abstract.  From there, with the advent of C, control structures and
variable names are now in the code.  With C++, objects are in the
code, while system-level architecture, design patterns, and object
interoperability are still mostly abstracted.

  The general trend is a move towards more intelligent code -- from
1's and 0's, to objects with explicit inheritance, interfaces, and
data members.  Each time more abstractions are made explicit in the
code, the code becomes easier to implement, test, and reuse.

  I am interested in exploring the next step (perhaps even determining
the nature of or actualizing the final step), of moving more
abstractions to code, making the code more intelligent, and easier to
develop, as the developer needs to carry less abstraction in his/her
own head.

  What I am proposing is a strictly minimalistic paradigm that
facilitates more intelligent code.  More intelligent code facilitates
better and easier-to-implement RAD design tools, as well as more
intelligently adaptive and self-aware software components.  The
working name used here is DAF, which stands for Dynamic Application
Framework.

  == Basic Concept ==

  The idea I have is for a purer form of object-oriented development,
or really, aiming for objectively pure objects that perfectly
represent any conceptual abstraction a software programmer or
architect could make, on the object / component level.  (On a system
level, abstraction arises from the interactions of the objects, and so
is beyond the scope of object definition, or object oriented
programming, but arises out of it anyway.)

  Conceptually speaking, what is an object?  It is a concept, an idea.
 It only has meaning once things are attached to it.  The decorator
pattern (or at least my concept of it), in its general conceptual
form, can be used to attach capability to objects.  Objects have two
primary capabilities: they can accept input from their environment,
and they affect their environment.  Automata such as computer code
only meaningfully affects its external runtime environment in response
to input.  (Whether various "agents" such as humans or software agents
act spontaneously is a philosophical matter requiring a theory of
reality, like the CTMU, or a semantic issue, for people who like to
think they are writing spontaneously intelligent agents.)  Taking this
into consideration, we get the general principles of interface (which
is basically equivalent to environmental input or event handling) and
implementation (internal aspect of doing something that affects the
outer world) and their natural separation.

  Thus each object intrinsically has decorators that provide
interfaces, along with implementation of those interfaces.  I think
that the progression towards easier programming will inevitably lead
and is leading towards smarter programming tools, eventually towards
the point where the programming objects themselves will support
reflexive self-modification, which could potentially be dropped at
either compilation time or runtime for the sake of proprietary opacity
or memory/CPU efficiency, where desired, or left in for the sake of
dynamically intelligent, self-aware and/or user runtime-extendable
applications.

  (There is also a shift towards using scripting languages where
certain application behavior is treated more as data than as
statically hard-wired information.  The divide between data vs code is
shrinking and will continue to shrink as our software demands more
cross-integration and adaptive capability such as self-repair,
self-optimization, and transparency of object distribution.)

  Self-modifying objects therefore require two basic decorators: one
for managing interfaces, and one for managing decorators (which
encapsulate implementors).  In DAF, there are three fundamental types
of objects, data, executable, and generic.
 - Data primitives contain a data decorator, and provide get/set
interfaces for compatible types in the host language (char*, long,
etc.), with the implementation possibly converting numeric strings to
binary integers, for example.
 - Executable primitives are guaranteed to implement an "execute"
method, which executes code in a supported language (perhaps a call to
shared library, or C code, or a lua script).
 - Generic objects are not primitives in that they contain no Data or
Executable that interface with the host machine architecture.  They
are completely abstract objects, containing nothing and having no
capability by default.
 
  == Extending Capability with Decorators == 

  Generic, or non-primitive objects must be extended to be useful. 
The DAF development environment contains a executable primitive to
create a Generic with self-modifying capabilities.  Its inextricable
interfaces provide access to both the interface manager, and the
decorator manager.  From here, all imaginable capability can be added.

  By adding a string decorator to an object, for example, the string
decorator factory will invoke operations in the Interface Manager to
add the supported Get/Set accessor interfaces to the object.  The
decorator factory will also use the object's Decorator Manager to add
a string decorator, which is responsible for handling the
implementation of the string accessor methods appropriately, and
ensuring the value is stored in memory.  (Prototypes for efficient
creation of frequently used objects is mentioned later.)

  == Refining Capability via Chain of Command == 

  Another decorator could be added, to handle thread synchronization
for the object.  A 'synchronization' decorator would be attached,
containing some kind of mutex or semaphore as its private
implementation data, and the existing interface methods for accessing
the string would be replaced by a chain of command: when GetString, or
whatever is invoked, the Interface Manager will pass execution to
locking/waiting implementation in the sync decorator, followed by the
string get implementation in the string decorator, followed by the
unlocking implementation.

  Another decorator could be added to control permissions of the
object, inserting a permission check at the beginning fo the chain of
command.  The implementation would verify a security token object's
security domain (referenced inside the decorator, described in a
central location) before the chain was allowed to continue, if at all.

  All sorts of kinds of decorators could be added to objects, such as
ones that implement transparent access of objects over the net,
time-variable caching from net or disk, database access, child
containers, parent accessors, observer pattern, documentation info,
etc.  As such, this could be seen as a sort of middleware that could
compete (once it evolves to this point, of course) with the likes of
CORBA, DCOP, ZeroC ICE, etc.  (ICE's feature list is probably the
closest match I've seen to what I envision here.)  The distinction
between DAF and these being that all capability for distributed
objects, as well as everything else, is optional, not constrained to
or focused on any one application.

  == Typing == 

  Objects are not described by a single type, but rather conform to
one or more schemata which indicate their capabilities.  A schema
comprises a set of one or more interfaces (or interface groups, not
described here), or one or more decorators, or both, with the
openendedness of DAF allowing more complex schema systems to be
implemented, such as the one used by XML (.xsd).  For convenience of
organization, a schema may include other sub-schemata.

  Schemata can be generated at runtime, against which introspective
objects can be validated.  Alternately, non-introspective objects or
non-modifiable objects can be hardcoded to indicate they implement a
certain schema.

  == Prototyping / Flyweighting ==

  Factories may be fed a particular schema to use to create an object.
 Detailed schemata may effectively be prototypes, which factories
could automatically generate for efficient creation, or which user
could manually specify.
  I won't go through all the details, but with DAF there is
opportunity to set up the creation flyweights as well as copy-on-write
objects, transparently and conveniently for the developer.

  == Full Reflexivity and Self-Documentation == 

  Documentation decorators could be added to every object prototype as
desired, at runtime (in a RAD environment) with the documentation
being physically stored in a separate location as the code.  DAF
allows for multiple group (and parent accessor) decorators,
facilitating multi-dimensional heirarchies.
  For example, a schema object could simultaneously be all of:
 - the member of a multi-level index of all schemas currently
available,
 - the member of a index of schemas provided by a certain pluggable
library,
 - the dereference point of another object's reference (perhaps
hardcoded, or cached from a schema check) indicating that object
conforms to the schema,
 - the parent object of a documentation object describing that schema,
the member of an index containing the set of objects authoritative by
an object server,
 - a member of a revision control system tracking this version of the
object
 - or any other number of examples.

  When subsystems are dropped or become irrelevant, such as a
development subsystem, all related decorators are dropped (revision
control, code documentation, etc.), and may be added again at runtime
(depending on how much introspective capabilities the objects have).

  The intent is to establish one way (via group/reference decorators)
and traceable one way (via group/parent decorators) connectivity
between objects in a way that explicitly captures the abstract
relationships between objects, moving the intelligence into the code.

  The resulting codebase becomes a self-documenting, self-accessible
network with broad scope, allowing a developer to traverse it similar
to how TheBrain traverses broad information rich networks.

  == Example Applications ==

Theoretically there should be no restriction on what couldn't be done,
the idea being that at some point, runtime dynamicity and reflection
can be traded in for runtime functionality and performance comparable
to a C program.  That said, here are a few random examples where the
dynamic capabilities might be helpful:

 - A web based knowledge base that allows arbitrary relationships
providing multilevel heirarchies.  Nodes of information could have
discussions linked to them, people's votes on appeal or validity,
external links, etc.
 - A multi-server scale or peer-to-peer computer game requiring
transparent relocation of objects, benefitting from adaptive load
balancing and self-optimization.
 - Any application where introspection is needed, applications that
learn, system doctor programs, better versions of clippy.
 - Any application benefitting from a minimal or maximal
object-oriented approach, and design patterns.

  == Flexibility ==

The idea is that DAF is a simple abstraction layer, and does not
prevent the developer from making use of existing libraries and their
APIs, or from designing 80% of their application in Java or C++ or
perl and abstracting the 20% of the high-level stuff and scripting
abilities with DAF.

In theory, DAF is just a way to abstract concepts, and should not
impose any restrictions on design.  Any run-time dynamic
self-modification and introspective abilities should be optional
similar to how C++ RTTI is optional.

  == Facilitation of Adaptivity ==

  A quick example of how introspection could facilitate Adaptive Agent
Intelligence: a group decorator (container of keyed child objects),
could have an AI agent attach itself to the "find object" interface
chain of command to profile a single or multiple algorithms for
looking up children, and either a) switch to another algorithm /
storage mechanism that is known or believed to be better, or b)record
performance of algorithms to disk so that common executions of the
application can reveal the best algorithm to use (hash table (of
varying sizes and algorithms), b-tree, map, etc.).
  In general, this kind of introspection allows profiling that can be
more automated and facilitate intelligent run-time adaptivity in code
execution.

  == Implementation ==

  DAF is a simple and obvious way of defining application in terms of
objects, interfaces and decorators.  The matter of actually executing
code is arbitrary and could be handled by any compiled or interpreted
language with sufficient bindings to the DAF specification.  If DAF
actually gets implemented, and it's up to me, I'm thinking that a BSD
or GPL licensed DAF runtime environment would be implemented using
C++, with support for executing shared libraries on Win32 / Linux /
other OS's, and bindings to a scripting language like Lua.  At some
point, it may make sense to create a dedicated language for working
with DAF for the sake of syntax convenience and aesthetics (so far
I've been creatively overloading C++ operators.)

  == Summary ==

DAF is a proposal to facilitate and make explicit the implementation
of several important design patterns, including decorators, facade,
chain-of-command, command objects (instead of parameters to executable
objects).  Generic support for observers, marshals, factories and
probably several other patterns will be provided as a part of DAF's
core decorator libraries, as will commonly useful applications such as
threading, network/database transparency, load balancing, etc.

DAF aims to ease the development and debugging bottlenecks by leverage
today's increased CPU and memory availability to make applications
fully interactive, both to use and to develop introspectively without
necessarily enforcing any of these features.  Introspection
facilitates adaptively intelligent program dynamicity to the point
where applications can start to become intelligently self-analytical.

  == Conclusion ==

  I am a recent SW engineering grad without several years of serious
programming work experience, so I present this with humility and an
open mind.  I am hoping that people can direct me to a) why this is a
bad idea, b) what is out there that already does this, or c) that this
is an avenue worth exploring, or d) same as c and that they're
interested in helping make it or e) something else.  In addition to
being someone who would like better RAD tools for software
development, I am also an independent researcher in the realm of
psychology and cognitive science, and would like to see intuitive
methods of programming that more closely represents good theory of
mind (requiring less overall time and effort from the software
developer), as well as a theory of mind (AI) that is easier to
implement in and interact with software.

I will try to monitor the forums where I post this, but if you want to
be sure to get through to me, email me at [e-mail]
(removing the 123 for my actual address).

Regards,
Jared Thirsk
http://daf.melsib.net