From: jaredthirsk@... (Jared Thirsk) Date: 2004-02-25T16:59:50+09:00 Subject: New paradigm of introspective OO development == Introduction to DAF == I am interested in the next generation of approaches to software development. Over the last decades, there has been an obvious shift in how our most complex software is created. We started off with punch cards (or so I am told -- I was born in 1981 so apologies if my history is off or my ideas are naive), punching in a stream of ones and zeros. The code contained only 0 and 1 and everything else was abstracted in the mind of the programmer, even the machine-level instructions. Then we moved to assembly language, where code more or less explicitly contains the machine-level instructions, while the meaning of variables, and the control structures were still largely abstract. From there, with the advent of C, control structures and variable names are now in the code. With C++, objects are in the code, while system-level architecture, design patterns, and object interoperability are still mostly abstracted. The general trend is a move towards more intelligent code -- from 1's and 0's, to objects with explicit inheritance, interfaces, and data members. Each time more abstractions are made explicit in the code, the code becomes easier to implement, test, and reuse. I am interested in exploring the next step (perhaps even determining the nature of or actualizing the final step), of moving more abstractions to code, making the code more intelligent, and easier to develop, as the developer needs to carry less abstraction in his/her own head. What I am proposing is a strictly minimalistic paradigm that facilitates more intelligent code. More intelligent code facilitates better and easier-to-implement RAD design tools, as well as more intelligently adaptive and self-aware software components. The working name used here is DAF, which stands for Dynamic Application Framework. == Basic Concept == The idea I have is for a purer form of object-oriented development, or really, aiming for objectively pure objects that perfectly represent any conceptual abstraction a software programmer or architect could make, on the object / component level. (On a system level, abstraction arises from the interactions of the objects, and so is beyond the scope of object definition, or object oriented programming, but arises out of it anyway.) Conceptually speaking, what is an object? It is a concept, an idea. It only has meaning once things are attached to it. The decorator pattern (or at least my concept of it), in its general conceptual form, can be used to attach capability to objects. Objects have two primary capabilities: they can accept input from their environment, and they affect their environment. Automata such as computer code only meaningfully affects its external runtime environment in response to input. (Whether various "agents" such as humans or software agents act spontaneously is a philosophical matter requiring a theory of reality, like the CTMU, or a semantic issue, for people who like to think they are writing spontaneously intelligent agents.) Taking this into consideration, we get the general principles of interface (which is basically equivalent to environmental input or event handling) and implementation (internal aspect of doing something that affects the outer world) and their natural separation. Thus each object intrinsically has decorators that provide interfaces, along with implementation of those interfaces. I think that the progression towards easier programming will inevitably lead and is leading towards smarter programming tools, eventually towards the point where the programming objects themselves will support reflexive self-modification, which could potentially be dropped at either compilation time or runtime for the sake of proprietary opacity or memory/CPU efficiency, where desired, or left in for the sake of dynamically intelligent, self-aware and/or user runtime-extendable applications. (There is also a shift towards using scripting languages where certain application behavior is treated more as data than as statically hard-wired information. The divide between data vs code is shrinking and will continue to shrink as our software demands more cross-integration and adaptive capability such as self-repair, self-optimization, and transparency of object distribution.) Self-modifying objects therefore require two basic decorators: one for managing interfaces, and one for managing decorators (which encapsulate implementors). In DAF, there are three fundamental types of objects, data, executable, and generic. - Data primitives contain a data decorator, and provide get/set interfaces for compatible types in the host language (char*, long, etc.), with the implementation possibly converting numeric strings to binary integers, for example. - Executable primitives are guaranteed to implement an "execute" method, which executes code in a supported language (perhaps a call to shared library, or C code, or a lua script). - Generic objects are not primitives in that they contain no Data or Executable that interface with the host machine architecture. They are completely abstract objects, containing nothing and having no capability by default. == Extending Capability with Decorators == Generic, or non-primitive objects must be extended to be useful. The DAF development environment contains a executable primitive to create a Generic with self-modifying capabilities. Its inextricable interfaces provide access to both the interface manager, and the decorator manager. From here, all imaginable capability can be added. By adding a string decorator to an object, for example, the string decorator factory will invoke operations in the Interface Manager to add the supported Get/Set accessor interfaces to the object. The decorator factory will also use the object's Decorator Manager to add a string decorator, which is responsible for handling the implementation of the string accessor methods appropriately, and ensuring the value is stored in memory. (Prototypes for efficient creation of frequently used objects is mentioned later.) == Refining Capability via Chain of Command == Another decorator could be added, to handle thread synchronization for the object. A 'synchronization' decorator would be attached, containing some kind of mutex or semaphore as its private implementation data, and the existing interface methods for accessing the string would be replaced by a chain of command: when GetString, or whatever is invoked, the Interface Manager will pass execution to locking/waiting implementation in the sync decorator, followed by the string get implementation in the string decorator, followed by the unlocking implementation. Another decorator could be added to control permissions of the object, inserting a permission check at the beginning fo the chain of command. The implementation would verify a security token object's security domain (referenced inside the decorator, described in a central location) before the chain was allowed to continue, if at all. All sorts of kinds of decorators could be added to objects, such as ones that implement transparent access of objects over the net, time-variable caching from net or disk, database access, child containers, parent accessors, observer pattern, documentation info, etc. As such, this could be seen as a sort of middleware that could compete (once it evolves to this point, of course) with the likes of CORBA, DCOP, ZeroC ICE, etc. (ICE's feature list is probably the closest match I've seen to what I envision here.) The distinction between DAF and these being that all capability for distributed objects, as well as everything else, is optional, not constrained to or focused on any one application. == Typing == Objects are not described by a single type, but rather conform to one or more schemata which indicate their capabilities. A schema comprises a set of one or more interfaces (or interface groups, not described here), or one or more decorators, or both, with the openendedness of DAF allowing more complex schema systems to be implemented, such as the one used by XML (.xsd). For convenience of organization, a schema may include other sub-schemata. Schemata can be generated at runtime, against which introspective objects can be validated. Alternately, non-introspective objects or non-modifiable objects can be hardcoded to indicate they implement a certain schema. == Prototyping / Flyweighting == Factories may be fed a particular schema to use to create an object. Detailed schemata may effectively be prototypes, which factories could automatically generate for efficient creation, or which user could manually specify. I won't go through all the details, but with DAF there is opportunity to set up the creation flyweights as well as copy-on-write objects, transparently and conveniently for the developer. == Full Reflexivity and Self-Documentation == Documentation decorators could be added to every object prototype as desired, at runtime (in a RAD environment) with the documentation being physically stored in a separate location as the code. DAF allows for multiple group (and parent accessor) decorators, facilitating multi-dimensional heirarchies. For example, a schema object could simultaneously be all of: - the member of a multi-level index of all schemas currently available, - the member of a index of schemas provided by a certain pluggable library, - the dereference point of another object's reference (perhaps hardcoded, or cached from a schema check) indicating that object conforms to the schema, - the parent object of a documentation object describing that schema, the member of an index containing the set of objects authoritative by an object server, - a member of a revision control system tracking this version of the object - or any other number of examples. When subsystems are dropped or become irrelevant, such as a development subsystem, all related decorators are dropped (revision control, code documentation, etc.), and may be added again at runtime (depending on how much introspective capabilities the objects have). The intent is to establish one way (via group/reference decorators) and traceable one way (via group/parent decorators) connectivity between objects in a way that explicitly captures the abstract relationships between objects, moving the intelligence into the code. The resulting codebase becomes a self-documenting, self-accessible network with broad scope, allowing a developer to traverse it similar to how TheBrain traverses broad information rich networks. == Example Applications == Theoretically there should be no restriction on what couldn't be done, the idea being that at some point, runtime dynamicity and reflection can be traded in for runtime functionality and performance comparable to a C program. That said, here are a few random examples where the dynamic capabilities might be helpful: - A web based knowledge base that allows arbitrary relationships providing multilevel heirarchies. Nodes of information could have discussions linked to them, people's votes on appeal or validity, external links, etc. - A multi-server scale or peer-to-peer computer game requiring transparent relocation of objects, benefitting from adaptive load balancing and self-optimization. - Any application where introspection is needed, applications that learn, system doctor programs, better versions of clippy. - Any application benefitting from a minimal or maximal object-oriented approach, and design patterns. == Flexibility == The idea is that DAF is a simple abstraction layer, and does not prevent the developer from making use of existing libraries and their APIs, or from designing 80% of their application in Java or C++ or perl and abstracting the 20% of the high-level stuff and scripting abilities with DAF. In theory, DAF is just a way to abstract concepts, and should not impose any restrictions on design. Any run-time dynamic self-modification and introspective abilities should be optional similar to how C++ RTTI is optional. == Facilitation of Adaptivity == A quick example of how introspection could facilitate Adaptive Agent Intelligence: a group decorator (container of keyed child objects), could have an AI agent attach itself to the "find object" interface chain of command to profile a single or multiple algorithms for looking up children, and either a) switch to another algorithm / storage mechanism that is known or believed to be better, or b)record performance of algorithms to disk so that common executions of the application can reveal the best algorithm to use (hash table (of varying sizes and algorithms), b-tree, map, etc.). In general, this kind of introspection allows profiling that can be more automated and facilitate intelligent run-time adaptivity in code execution. == Implementation == DAF is a simple and obvious way of defining application in terms of objects, interfaces and decorators. The matter of actually executing code is arbitrary and could be handled by any compiled or interpreted language with sufficient bindings to the DAF specification. If DAF actually gets implemented, and it's up to me, I'm thinking that a BSD or GPL licensed DAF runtime environment would be implemented using C++, with support for executing shared libraries on Win32 / Linux / other OS's, and bindings to a scripting language like Lua. At some point, it may make sense to create a dedicated language for working with DAF for the sake of syntax convenience and aesthetics (so far I've been creatively overloading C++ operators.) == Summary == DAF is a proposal to facilitate and make explicit the implementation of several important design patterns, including decorators, facade, chain-of-command, command objects (instead of parameters to executable objects). Generic support for observers, marshals, factories and probably several other patterns will be provided as a part of DAF's core decorator libraries, as will commonly useful applications such as threading, network/database transparency, load balancing, etc. DAF aims to ease the development and debugging bottlenecks by leverage today's increased CPU and memory availability to make applications fully interactive, both to use and to develop introspectively without necessarily enforcing any of these features. Introspection facilitates adaptively intelligent program dynamicity to the point where applications can start to become intelligently self-analytical. == Conclusion == I am a recent SW engineering grad without several years of serious programming work experience, so I present this with humility and an open mind. I am hoping that people can direct me to a) why this is a bad idea, b) what is out there that already does this, or c) that this is an avenue worth exploring, or d) same as c and that they're interested in helping make it or e) something else. In addition to being someone who would like better RAD tools for software development, I am also an independent researcher in the realm of psychology and cognitive science, and would like to see intuitive methods of programming that more closely represents good theory of mind (requiring less overall time and effort from the software developer), as well as a theory of mind (AI) that is easier to implement in and interact with software. I will try to monitor the forums where I post this, but if you want to be sure to get through to me, email me at jaredthirsk123@yahoo.com (removing the 123 for my actual address). Regards, Jared Thirsk http://daf.melsib.net