# Re: [stack] Digest Number 24

## Mark van Gulik — 2000-05-28 22:56:02

    Massimo Dentico <m.dentico@...> wrote:

    [...]

     >> Ok, my worry was about the dimension of the data-set. OLAP application

    >> of which I have heard to speak involves usually TBytes (T = Tera = 10 ^ 12),

    >> if not PBytes (P = Peta = 10 ^ 15), of data.

    >>

    >> Apologize me, I have misunderstood what you have written, I have

    >> understood the data products by 20*10^9 (giga) transactions and not 20 GBytes

    >> of data; estimating 1 KB of data per transaction, the data-set is 20 TB ...

    >> managing this amount of data in reasonable time are not peanuts.

    >

    > agreed.  commercial olap products (arbor essbase for example) claim

    > to scale to multi-terabyte data, but experience (viz. mine)

    > shows the practical limits to be much lower (for example, there is a

    > massive explosion of the data during the precalculation phase.)  the

    > only petabyte database i know of is at cern, and i have no idea what

    > they're using to analyze it.  in any case, 100 gig databases are still

    > rare in the world of commerce, and that's probably for the best, since

    > all the commercial rdbms products choke at well under this limit, but

    > pose little difficulty for 32 bit k, and none at all for the 64 bit

    > version.  ;-)

     Actually, SLAC (Stanford Linear Accelerator Center) is using Objectivity, an

    ODBMS.  Last year they surpassed 25 Terabytes, and by now they must have

    much, much more data.  I believe they expect to cross into the Petabyte

    range in the next couple of years.

    Also, the Sloan Digital Sky Survey uses an Objectivity database which will

    contain 40 Terabytes of data when completed.

    I myself have worked on a smaller scale project with an Objectivity database

    containing information for about 50 million customers (about 800 Gig, if I

    recall).  We did most of the development in *Smalltalk*.  No regrets -- it

    was a *lot* of fun, and a complete success.

    Languages like Forth would be completely inappropriate for such constructs.

    The amount of semantic information in the models easily exceeds Forth's

    complexity limits (in my estimation).  *Parts* of the database could be

    dealt with by Forth, but its entirety is too intrinsically complex (e.g.,

    all the indexing structures, keyed by a new kind of spherical coordinate

    system and the spectra, must allow well-choreographed storage clustering on

    disk and tape without overcomplicating the high-level code requiring the

    access).

    Clearly these projects need the cooperation of many software developers.

    What kind of tools are available for Forth in this regard?  As far as I'm

    aware, Forth doesn't even have namespaces.  It also lacks (if memory serves)

    any ability to specify interfaces separately from implementations.  High

    complexity parallel development is simply not possible (at any price) in

    such an environment.

    Your mileage may vary.

## Samuel A. Falvo II — 2000-05-28 23:09:08

    On Sun, 28 May 2000, Mark van Gulik wrote:

     > Clearly these projects need the cooperation of many software developers.

    > What kind of tools are available for Forth in this regard?  As far as I'm

    > aware, Forth doesn't even have namespaces.  It also lacks (if memory serves)

     This is not a defense for Forth, but merely an "update."  Almost all Forth

    implementations today support "vocabularies", which are equivalent to

    namespaces in other languages.

    The other points you bring up, however, are valid; I *love* Forth over any

    other programming language I've used, including Lisp (another one of my

    favorites, BTW).  But because Forth doesn't properly handle the clean

    separation of interface and implementation, I can't use it for the

    majority of my projects, as it doesn't support separately compilable

    source files, and can't be easily integrated with other programs written

    in other languages.

    One could use CORBA for this facility, and indeed, an ORB in Forth is well

    within the realm of possibility.  However, the ORB will necessarily be

    Forth-implementation specific, for precisely the same reasons outlined in

    your original message.  And what's more, it'll be bloated by the same

    mechanisms that Steve was talking about wrt SwiftForth, though not

    necessarily by the same degree, as CORBA gives much more freedom in the

    ABI used to implement an ORB than an operating system does.

    It's simply a matter of economics -- in many cases, Forth cannot meet the

    project's many budgets: financially, chronologically, and available

    resource consumption.

## Massimo Dentico — 2000-05-30 01:48:56

    Mark van Gulik on Concatenative mailing-list wrote:

     > [.. discussing about OLAP ..]

    > I myself have worked on a smaller scale project with an Objectivity database

    > containing information for about 50 million customers (about 800 Gig, if I

    > recall).  We did most of the development in *Smalltalk*.  No regrets -- it

    > was a *lot* of fun, and a complete success.

    >

    > Languages like Forth would be completely inappropriate for such constructs.

    > The amount of semantic information in the models easily exceeds Forth's

    > complexity limits (in my estimation).  *Parts* of the database could be

    > dealt with by Forth, but its entirety is too intrinsically complex (e.g.,

    > all the indexing structures, keyed by a new kind of spherical coordinate

    > system and the spectra, must allow well-choreographed storage clustering on

    > disk and tape without overcomplicating the high-level code requiring the

    > access).

     [Well, I'm not here to sell Forth but to highlight its qualities because

    I'm interested about the design of a better concatenative language of the kind

    that Bill Tanksley has illustrated previously. I'm convinced that it needs

    improvement, in particular about its safety, thus my interest in a hypothetical

    type system that fits well in the general framework of Forth: IMHO the approach

    "the language is the type system", that you also seem to embrace with Avail,

    looks very promising.]

    You miss completely my point. Forth is a meta-tool, this is its point of

    strength. You know better than me (you say that you have written one) that

    Forth is extensible, at the syntactic/semantic level thanks to its

    open interpreter/compiler, in whatever direction: toward low level (direct

    access to hardware, embedding of assemblers) and toward high level (domain

    or even application specific languages). Smalltalk, to a certain extent,

    shares these characteristics of uniformity and extensibility.

    Numerous Object Oriented extensions, with very different characteristics,

    and a Garbage Collector are available in Forth, written in it. I have seen

    also subsets of other programming languages embedded in it, like Lisp and

    Prolog or a tagging language like HTML.

    All that in a coherent, uniform framework, *without* leaving the language

    (or better, the environment) and on-the-fly (load, use and then discard, the

    extensions you need). Note that this is different in comparison to statically

    compiled modules, it's a dynamic precess: every time you load a screen (a little

    chunk of code) you interpret/compile it perhaps in a different context; this

    opens the door to adaptive interpretation/compilation/execution which is specific

    to actual context, so any sort of specialization/optimization which exploit

    this dynamic information is potentially possible (depending of what make sense).

    Note even that only most frequent executed parts of the program need to be

    heavily optimized and these parts could vary during the execution itself or

    between different executions.

    Traditional batch compilers don't know much about the dynamic execution context

    of the code, they compile alla and blindly apply every possible optimization

    technique statically. This means long compilation times and calls for separate

    module compilation. But separate compilation introduces complexity and constrains

    the semantics of the language: every feature that is not statically known,

    at compile time *and* at the module boundary, is discarded or restricted (for

    example, separate compilation in Standard ML, which have a powerful type system,

    is quite difficult and limits the language). Obviously, moving this to link-time

    or load-time (of compiled code, a.k.a. DLL) complicates the process and don't

    solve anything: the problem here is too much pre-compilation. Forth avoids this

    issue entirely and Smalltalk too (I know that, for some implementation, is possible

    to compile to C/C++ but usally this feature is used at the end of a development

    cycle).

     > Clearly these projects need the cooperation of many software developers.

    > What kind of tools are available for Forth in this regard?  As far as I'm

    > aware, Forth doesn't even have namespaces.  It also lacks (if memory serves)

    > any ability to specify interfaces separately from implementations.  High

    > complexity parallel development is simply not possible (at any price) in

    > such an environment.

    >

    > Your mileage may vary.

     Experienced Forth programmers assert that their productivity is greater of

    an order of magitude when they use Forth rather than other programming

    languages. I think that, thanks to the characteristics discussed above, they

    tend and succeed to avoid complexity, thus the necessity of big teams of

    programmers and complex management.

    But forthers are no more alone in their commitment toward simplicty.

    "Do the simplest thing that could possibly work" [1] is also the motto of

    Extreme Programming [2], a lightweight, humanistic discipline of software

    development, grown in a part of the Smalltalk community:

    --------------------------------------------------------------------------------

    [...]

    We find that we have to remind ourselves of this rule continually. Developers

    like to develop, and most of us have years of experience in creating a "general

    solution" to whatever were asked for. Real progress against the real problem

    is maximized if we just work on what the problem really is.

    [...]

    --------------------------------------------------------------------------------

    This is surprisingly similar to what Charles Moore said [3]:

    --------------------------------------------------------------------------------

    [...]

    Don't Complexify

    Simplify the problem you've got or rather don't complexify it. I've done it

    myself, it's fun to do. You have a boring problem and hiding behind it is a much

    more interesting problem. So you code the more interesting problem and the one

    you've got is a subset of it and it falls out trivial. But of course you wrote

    ten times as much code as you needed to solve the problem that you actually had.

    [...]

    --------------------------------------------------------------------------------

    This seems to contradict what I have written in another e-mail, my concern with

    generality, but it's not so: abstraction, taking out of context, is a process that

    necessarily needs more concrete "objects" by which to abstract. Infact in Forth

    and Smalltalk communities (and probably other, like Self community) this process

    of abtraction is called factoring or refactoring and moves from more specific/

    concrete pattern of code/data to more abstract one, in an utilitarian way because

    only what is really general, in the actual context, is generalized. This contributes

    to the compactness and then tractability of code (no redundances). I think that

    generalization and specialization are not necessarily in contrast.

    References

    [1] eXtreme Programming web site

        - http://xprogramming.com/

    [2] "Do the simplest thing that could possibly work"

        - http://www.xprogramming.com/Practices/PracSimplest.html

    [3] "1x Forth"

        - http://www.ultratechnology.com/1xforth.htm

    --

    Massimo Dentico