Re: Extension Language for a Text Editor

On Fri, 10 Oct 2003 02:36:25 +0900
Nikolai Weibull <[e-mail]> wrote:

> * Ryan Pavlik <[e-mail]> [Oct, 09 2003 09:10]:
> [stuff about Emacs and Vim]
> > Well, whichever, just be aware that emacs is designed as an extensible
> > editor, and vim is not, even though such things have crept in with
> > varying degrees of usefulness.

> Yes, that, as I see it, is Emacs biggest win.  The big problem I see
> with Vim is that it hasn't undergone a major overhaul.  It is basically
> Emacs written in C now.

Not really.  It's vi with lots of hacks to do more stuff added.  So
much for being a tiny, bloat-free editor. ;-)

> Vim is, in my opinion, probably the best editor that exists right
> now.  It is, however, going to reach a point where adding new
> features will demand some sort of rewrite.  With Emacs, things like
> these are easy to alter.  Anyway, Vim is extensible, it is, however,
> not _changable_.  By that I mean that, if you want to change the way
> folding works, you must rewrite the core of Vim.

Actually that'd be the other way around.  vim is changeable... you can
alter the C core... but it's not extensible: you can't really add
extensions.  _This_ is what causes feature creep and bloat.  Not
having 20 million extensions that do everything imaginable.  Being
able to add extensions means you're able to _remove_ them.  Rolling
stuff into the core means you've got a bloated editor.

Somewhat ironic.

Now, I don't want to start an editor war here.  This could very easily
turn into one, and that would totally defeat the point.  Therefore,
let's look at emacs and vim in terms of the interface concepts they
embody, such as interactivity, modality, extensibility, etc, rather
than implementation faults.  Implementation is irrelevant since you're
implementing something new.

So basically, if you like a modal editor or not, go for editor of form
of your choice.  I've seriously considered switching to vim simply
because of the ruby support.  (A few things have kept me from doing
this, but they were merely technical issues.)

> And, as Bram pointed out in some interview, that means altering
> basically every file in the Vim distribution.  To Vim's salvation
> comes the ability to easily define new syntax definitions and
> indentation definitions, which one has to agree, are a lot easier to
> create and alter than with Emacs (Emacs being perhaps more powerful
> though).

Not really, the emacs syntax bits are pretty trivial to do.  They look
worse than they are; I was hacking around on the ruby-mode, and it was
much easier to modify than it appears.

Now, this isn't really emacs vs vim; it's just a matter of philosphy.
You could trivially add code in emacs to do vim-ish syntax
descriptions, and I'm sure you could add code in vim to do more
scripted syntax.

The point to note is that it'd be a lot less work to convert
simplified syntax into complex syntax than it is to add scripted
syntax later.

> [stuff about Ruby libraries and such]
> > Right.  They're there, people can write extensions that interface to
> > the web, or whatever.

> Yes, but I don't want an Operating System.  I guess, to a certain degree
> the library you get helps you, but it can also detract you from the
> central topic, namely editing text.

I know it's a typical joke to say "emacs isn't an editor, it's an OS",
but remember---it's a joke.  Don't build your editor based on jokes.

In truth the emacs core is very tight: it provides a buffer-oriented
language and interface elements.  Because it's general, people have
written lots of stuff, some of which is quite silly (tetris, web
browser, etc.), but is also a testament to flexibility.  You can be
sure that if you can make it play tetris, you can also make it edit
text in any conceivable way.

Of course this isn't limited to emacs-style editing.  Plenty of
editors are scriptable; you could implement scripting capabilities and
provide any interface you want.  Same deal.  Flexibilty, however, is
something worth considering.

> > See http://ogmo.mephle.org/tabular-alignment.org for the Lisp
> > version.  The ruby one I deleted, as it was pretty simple to
> > reproduce, I'm sure someoone can whip up an example.

> Yeah, OK.  I see your point.  It is, however, very easy to alter to fit
> your own needs.  Change some global variables and you can make it work
> for almost anything.  I can't tell, but I'm guessing your code in Ruby
> wouldn't allow for this?

Actually the ruby version and the lisp version are almost identical,
API-wise.  I just wrote a function in ruby that took a string and a
regexp, and returned the aligned string.  The fact the emacs one goes
through and replaces buffer text is the only real difference.

> I'm not trying to contract you, only point out that Emacs extensions
> are, as oposed to Vim extensions, very flexible and well thought
> out.  This does, of course, not mean that it can't be done in Ruby.
> I just get the feeling that Lisp excels at this.

Well.  I definitely agree about the emacs extensions vs vim ones.
This is mostly philosophical IMO, as stated above.  What makes
extensions more flexible is not the language, exactly, but the
philosophy you have for extensions.  If it's "you can run a script on
the text" (like the ruby vim module), you don't get a lot to work
with.  If it's "you can hook into every conceivable part of the
editor", or (like the sawfish window manager) "all higher-level
functionality is implemented in the scripting language", you have
extreme flexibility.

That said, one of the nice things about ruby is that it has taken a
_lot_ of direction from Lisp.  There are _many_ little things that
just make it more convenient to use (closures, for instance).
Convenience when writing extensions is greatly appreciated by
users. ;-)

> [stuff about functional vs. OO being more well suited for editing
> text]

> > I don't really find that.  I don't think functional programming is any
> > easier for editor-related tasks.  I'm not even sure how you would come
> > up with such an assumption. ;)

> My real point was that having OO around doesn't really help either.  It
> doesn't add anything.  Sure, you can make classes like Buffer and Window,
> were is the real gain?

The first gain is elegance.  Uniformity.  You know where to look when
you want a method to alter Window; you look in the Window class.  You
don't look for all the functions that may or may not have "-window" in
them.  (This is a major issue I've had with both sawfish and emacs;
the nomenclature is not uniform.  This may be considered purely
implementational, but people tend to be lazy, or they've approached
something from an unusual angle, and things break down.)

The second is extensibility.  If I want to alter how the buffer is
handled, I have to look for hooks (if they're provided) and make sure
to set all the right global variables (ugh) and change the keys, etc.
With the object-oriented approach, I just subclass Buffer and override
what I need.  (Or perhaps I merely make a new interface class.)  It's
much more straightforward, and again, more elegant.

> I have tried to envision some OO structure for implementing Emacs
> like Major/Minor Modes and such, but I haven't been able to come to
> any satisfactory results.  I mean, how is a Major Mode an object?
> Really? 

This is because you're trying to do a direct transliteration of emacs
to OOP.  This doesn't work.  Step back and see first what you're
trying to accomplish, and then design an object structure that handles
it, while sticking to the _philosophy_ of emacs ("script everything").

Just off the top of my head, you don't want modes, first off.  They're
too constraining.  The major problem is that you can't have multiple
"major" modes without a lot of hackery (if at all).  What you ask
yourself is: what do modes provide?  The answer: buffer behavior.

What might be better: behavior objects.  You have a class hierarchy to
modify various aspects, and you put these together to build your
favorite buffer behavior.  This way you could have multiple syntaxes
per buffer, for instance.

Configuration would simply be changing the object properties.  For
instance, if you get a hilighter object as 'syntax', you might:

            :
    syntax.comment_color  = BLUE;
    syntax.variable_color = RED;
            :

The 'syntax' object could be a subclass of a more general Formatter,
and you may even have a subclass per language, where you have the
ability to implement specifics if necessary.  A basic SyntaxHilighter
could provide simple regexp hilights that everyone would use anyway.

> I guess it has a syntax definition, a separate keybinding mapping,
> an indentation callback, maybe something else?  I just don't feel it
> adds anything though.  I am, of course open to suggestions ;-).

The main thing I'd suggest is to look at what things _mean_, and what
they _could_ be... don't necessarily mimic the way things are
implemented right now.

> > Right, tiny C core like emacs, everything higher-level in the language
> > of choice.  Ruby is highly suited for this task.

> Yeah, this is true.  Ruby would be well suited for this I do
> believe.  But note that Emacs C core isn't very small ;-)

The emacs C core includes elisp, which isn't exactly fair; count all
the lines of vim and perl, and you won't find it too small either. ;-)

> > People could even write high-performance ruby extensions in C...

> Yeah, this would be easy to do as well.  There is, of course, the
> inherent risk of not being portable enough.  Vim supports this in a way,
> and I have never seen it used to date.

True, but ruby extensions tend to be fairly portable.  Don't worry
about enforcing this.  If people want it, they'll port it.  It'd
likely be a rare occurance it happens at all.  (Maybe some really fast
text munger; I dunno.)

> [stuff about regular expressions]
> > Well, to be blunt, whatever you come up with won't be as popular or
> > useful as the existing regular expressions, just because they'll be a
> > nonstandard replacement of something already very common.  PCRE
> > regexps are extremely flexible and well-known.

> As useful?  Please, my dear sir, there has to be something better than
> the way we describe regular expressions now.

Better?  Perhaps.  I would first want to know what is wrong with
existing regular expressions.  (Now, I dislike the overuse of regexps,
and tend to avoid them in my code, but when you're writing editor code
or parsers it's a different story.)  Consider:

   1.  Do you feel they could be improved in form alone, or
       functionality?

   2.  What form should they take, it not the current form?

   3.  What function should they accomplish, that they do not
       currently accomplish?

   4.  Is the action you're trying to take actually something you
       should be using regexps for, or is there a better way?

> At least for searching text.  The syntax we have today for regular
> expressions is basically the same, only extended, as that that Ken
> Thompson uses in his 1968 paper on it.  Or that of _real_ regular
> expressions long before it.  And remember, real regular expressions
> only have * (Kleene star) and no +.

Just because something is old doesn't mean it's bad.  Text is text.
That hasn't changed since 1968 or 1908 or 1809 or long before. ;)
Only the content.

The method of handling it has gotten more complex, because we've gone
from purely academic uses to actual everyday useful uses.

> There has to be a simpler syntax that can be useful for
> interactive text search-and-replaces.

There might be.  There are simpler syntaxes, but they usually trade
off on functionality.  There are GUI regexp builders.

I'm definitely a fan of throwing things out the window when they need
to be.  But the first thing to determine is what you're trying to
accomplish, and how throwing things away lets you accomplish it.

> Look at Vim, Emacs, and Perl (and thus, basically, Ruby)'s syntax.
> They are all extensions of this, adding new short cryptic ways of
> saying things that you often don't need, and if you did you wouldn't
> want to do it that way anyway.  The real example of how it has
> gotten out of hand is the overuse of backslash (\).  It is
> everywhere.  <snip more obscure things>

There are really two things at play here.  The first is commonality.
Everyone uses backslashes for escaping, from C to bash.  You need to
escape things, so backslash is as good as any, and everyone is
familiar with it.  This is good.

Obscure feature creep is less good.  I am not a fan of Perl's "a
kitchen sink for every occasion" philosophy (don't get me started
here), but if you start getting tons of special cases, maybe your
instincts are right: maybe we need something new.  But maybe it's not
regular expressions, either.

> > That isn't to say people won't use them, especially if they're
> > simpler, but it probably won't be the main selling point of your
> > editor to _other people_.

> Nah OK.  You've got a point.  But, as with most free software, this
> one's for me ;-).  If anyone wants to tag along later on, fine.  But I
> won't care if no one is interested, Emacs and Vim are fine editors.
> Even notepad has its uses.  It can, for example, tell you if a file is
> smaller or greater than 65535 bytes very easily ;-).

Actually this is good, I think.  If you're writing an editor to be
popular, it probably won't.  If you're writing it to be useful to you,
it'll probably be popular with others who have similar needs.

> I have, perhaps, failed to describe the real winning here.  (Alas, I
> realize I forgot to mention it.)  As you perhaps know, Vim, and most
> other UNIX software, operate on a line-by-line basis.  This restriction
> would not impede the command language I'm contemplating.  If you take a
> look at the Sam editor[2], this is its main selling point, and this is
> another one I want to include.

Not sure what you're getting at here, but it seems interesting...

<snip>
> > I'm not sure how that's a problem.  The same applies to // regexps.
> > They're just basically strings, except stored in a different type of
> > object with a few flags.

> Nono, they don't do string escapes.  \n in a regex (//) means match a
> newline, not substitute this for 0x0a.  So, you don't have to quote it
> with an extra backslash to get that meaning.
<snip>

Hmm, I think I see what you mean.  You could just use '' though, which
does it for you:

    irb> s = '\n'
    => "\\n"

<snip>
> and to match a backslash itself:
> 	"\\\\"
> which is horrendous.  In a Ruby regex, /\\/ suffices.

Yes, this is a problem I've had with elisp regexps.  Too many \\'s.
I've seen lines like \\\\\\\\\\\\\\\\\\\\ before.  Really painful to
read.

ttyl,

-- 
Ryan Pavlik <[e-mail]>

"That's the first pratical line of thinking to
 come out of your word hole since we first met." - 8BT