[forum] Some perspective from the cheap seats...

[Allen Akin]

On Mon, Mar 24, 2003 at 11:29:18PM -0500, Havoc Pennington wrote:
| I think the answer to that is "no" - in general, we have near-zero
| understanding of where hardware is going. ;-) Some people might.

It's too big a topic to handle in a short note, and I might not be the
best-qualified person to attempt it.  But for desktop systems:

    1.	Graphics hardware design runs in cycles -- roughly,
    general-purpose programmable hardware gets supplanted by
    special-purpose hardware that offers dramatically better
    performance, which in turn is supplanted by new general-purpose
    hardware that offers more flexibility.  Currently we're in a phase
    of moving toward general-purpose programmable hardware.

    2.  Nevertheless, there's a fairly persistent conceptual model --
    the graphics pipeline.  Data structures representing the scene are
    "traversed" to generate "vertices" which are processed and
    "assembled" into low-level primitives (mostly triangles).  These
    primitives are "rasterized", which involves converting them (by
    sampling) into "fragments" (of pixels), combining them with zero or
    more "textures", and finally combining the result with a
    framebuffer.

    3.  Traversal is still largely in the hands of the applications,
    though some performance-critical simple traversals (e.g. indexing
    arrays of vertices to form triangles) are handled by the hardware.

    4.  Vertex processing is handled by short programs (tens to hundreds
    of instructions) written by the application developer using an
    instruction set oriented towards four-element floating-point vector
    ops.  Small numbers (tens) of scratch registers are available for
    intermediate results.  Severe constraints are placed on the program
    semantics; for example, one vertex in -> one vertex out, with no
    state information from one vertex affecting any other.  (This is
    done to permit unordered parallel processing of vertices.)  The
    information associated with a vertex, other than its position, is
    essentially arbitrary.  So quite sophisticated functionality is
    possible despite the constraints.

    5.  Fragment processing is also handled by short programs (tens to
    hundreds of instructions) written by the application developer using
    a vector-oriented instruction set.  The hardware shipping today is
    the transition from fixed-point to floating-point operands.  Again,
    significant constraints are placed on the program semantics; for
    example, operations on one fragment don't affect adjacent fragments
    directly (though this may change somewhat in the next generation).
    A few operations are still implemented directly in the silicon, but
    most operations are under programmable control.

    6.  Texturing is key to most nontrivial fragment programs (and in
    the next generation, vertex programs).  Texturing is best thought of
    as general 1D, 2D, 3D, or 4D table lookup with certain hardwired
    interpolation options, using index(es) supplied by the application
    or generated by the vertex or fragment programs.  Compositing,
    masking, convolution, scaling, warping, colorspace conversion, and
    traditional 3D image-on-geometry operations are all aspects of
    texturing.

As far as the API goes, for the Open Source world, OpenGL is the most
widely available option.  The learning curve can be steep, particularly
because things are in transition and there's a lot of legacy support to
be worked around.

The OpenGL Programming Guide (3rd edition) by Woo et al. is probably the
best place to start.

Currently programmability in OpenGL is provided by extensions to the
core API.  The most important are:

	oss.sgi.com/projects/ogl-sample/registry/ARB/vertex_program.txt

	oss.sgi.com/projects/ogl-sample/registry/ARB/fragment_program.txt

(For the full list of extension specs see
oss.sgi.com/projects/ogl-sample/registry/ )

We're moving away from assembly-level languages toward high-level
languages for both vertex and fragment processing.  NVIDIA offers Cg
(www.cgshaders.org) which is compatible with the high-level language
used in DirectX.  The Stanford graphics group has a shading language
also used with both OpenGL and DirectX.  The OpenGL Architecture Review
Board is working on the next-generation OpenGL core spec, which has a
"shading language" that incorporates both vertex and fragment
programming; see www.opengl.org/developers/documentation/gl2_workgroup/
.

| It might be worth trying to think about how UI graphics (say, drawing
| a button, or drawing a word processor document) are different from 
| graphics in general.
| 
| Here are some things I can think of quickly:
| 
|  - WYSIWYG. We need the same API to draw to screen, printer, and a
|    raster image to be dumped to a file, generating results that are
|    basically the same. A pixel off here and there is OK in most cases,
|    but using a different font or doubling line widths or something
|    would not be. And we need client-side code available that can draw
|    the primitives, even if there's also a server-side option.

When you say you want to draw to devices that differ as much as screens
and some printers, that tells me we also need to think about larger
issues in the rendering model.  (For example, where the responsibility
for sorting lies when the rendering operations are order-dependent, as
in blending.)  Solving this in a completely general way requires a
higher-level knowledge of the scene to be rendered, something that
low-level APIs like OpenGL offer assistance for, but don't address
entirely.

As for client-side drawing code, I'm not clued-in enough to understand
all the issues.  For example, do you intend to rule out
hardware-assisted direct rendering when running locally?  Is this
something that the application can be expected to decide, or is it up to
the rendering library?  What are the key factors in making the decision?

|  - Multiple apps. We need to use the API to draw basically every
|    window on the server. So if there's a special visual or other
|    limitations involved this may not work. ...

GLX Visuals don't have to differ from core X11 Visuals, unless the app
wants to perform some operation that the core X11 Visual doesn't support
(e.g. Z buffering or blends using destination alpha).  Even those
constraints can often be worked around with texturing, so at this point
I can only say "I don't have enough information to tell."

|                                        ... Also, this means that 
|    a simple double-buffer for a single window isn't enough; to 
|    make things smooth, double-buffering spanning multiple windows 
|    (or some other solution) seems to be needed.

Yes, this is true in the 3D world as well.  (And I think we talked about
it on the Render list many months ago, but I can't remember exactly
when.)  GLX extensions for this purpose have existed since 1997.  I
believe Ian Romanick is working on the DRI-based implementation for
XFree86.

|  - Performance not such an issue. We care about performance, but not 
|    *as much* as others might. Not that it hurts to be faster.

:-)

| I don't really understand at all why people feel these things are at
| odds with using the hardware well (more performance isn't going to
| make people complain). But, that is probably because I don't
| understand the hardware. I would like to understand the issues, if
| someone feels like writing them up.

I think most of us in the OpenGL community are worried about lower-level
things -- reinvention of concepts related to texturing and fragment
programming but with slight mismatches that hinder acceleration, data
formats (especially floating-point) for drawing surfaces and textures,
antialiasing techniques and precision requirements, polygon sampling
rules, etc.

Maybe at a higher level there's concern about creating a fixed-function
architecture when the hardware is going programmable, relying too
heavily on imaging operations when geometry processing has become
relatively cheap, and making too-optimistic assumptions about the amount
of available graphics memory (or the cost of moving images into and out
of it).

Most of these things have been discussed in the past, and Keith tried in
good faith to bridge the gaps.  I guess I felt that (with the exception
of a few people like Raster) there wasn't enough common experience
between the hardware world and the UI world to actually close the gaps,
so we just had to let Render go its way and try again once we had more
experience.

There really is a stupendous amount of effort going into graphics
hardware right now.  It would be great to take full advantage of that if
we can find the resources to do so.  Microsoft and Apple are already
headed down that path.

| Ah, one other issue I've repeatedly encountered when trying to do nice
| animations. Say you want to do the "window swooshes into its icon"
| genie effect from OS X, for example. I haven't found a good way to do
| this animation.

Depends a lot on the assumptions you make about backing-store for
windows (or the ways in which apps handle redraw).  Maybe a good topic
for a separate thread.

I'm reminded of a conversation I had in the Olden Days with Dave Brown
shortly after he left SGI and joined DEC.  Dave described how people
were stunned by the text-scrolling speed on the then-current SGI machine
(probably a GTX, I don't remember for sure).  Dave laughed and explained
that the terminal emulator didn't scroll; it just cleared the window and
re-rendered the text in its new position.  When you've got enough
horsepower to render 3D scenes at interactive rates, you might be able
to find new ways to handle 2D effects, including brute force. :-)

| I think Raster has probably made more progress on getting X to do nice
| animations than anyone else, so he may have found better approaches.

Yep.

Allen