With all the talk of the 6800 series and Pixel Shader 3.0, when will we actually start seeing games that use this? We are just now starting to see PS2.0 and 3Dc games. Far Cry is the only PS3 game I know of. Also, I've heard people say that PS3.0 does nothing with image quality, it only increases fps. However, I have seen screenshots of Far Cry supposedly with PS3 that show the complete opposite. Will PS3 be just an fps increaser, or will amazing new things be possible with the graphics because of it?
Pixel Shader 3.0
- Thread starter beedubaya
- Start date
- Status
NanoWarrior
In Runtime
- Messages
- 268
I've looked at screenshots of farcry that used PS.3 and it looked amazing. The shines on the rocks and other matallic objects just look amamzing from what I saw. If what I saw is true than I can see ps.3 uping the graghics in games to another level.
PS3.0 is only the 3rd step in a long staircase .
Pixel Shrader has everything to do with the future .
Dont be surprised when PS 5.0 comes around
.
Without Pixel Shrader
future Game makers are limited to what they can bring to the table for us to eat .
Heres a good article read if you really want to no whats behind the driving force of gaming and graphic wizardry .
If you program or play computer games or even recently attempted
to purchase a video card, then you will have no doubt heard the terms
"Vertex Shader" and "Pixel Shader". What do they mean,
other than that marketdroids will never cease to invent
crappy technology jargon? Hear all the noise about "Hardware T&L"
a couple of years ago? It's all related and this article will attempt
to describe it all neatly for you.
Warning: mathematics ahead; this devolves into a basic
3D graphics tutorial. It's nothing more complicated than
what you'll see in your final year of high school (linear algebra).
If you're totally non-technical, run screaming now.
First of all, you need to have a basic understanding of how a
consumer render pipeline (stop panicking already) works and how
this is modelled by Direct3D or OpenGL. I shall concentrate
on OpenGL here since it's a tad more portable and it's what
I'm familiar with. If you're not a programmer, skip down to
the bottom where there's links to demos and pretty pictures.
Pipeline Overview
The information & logic that makes up what you see
on the screen in a 3D game (or other visualisation) is composed of:
Vertex data for models
(optional) Index data into vertex arrays
Object positioning logic: vertex transformations
Textures
Yet more textures (eg lightmaps, bumpmaps)
Texture combining logic (add, multiply, dependent, etc)
Fragment operations (stencil, depth test, alpha test)
Blending logic (for transparency)
For each object (eg teapot, knife, player) drawn, the following occurs:
The application tells the card where to find a chunk of memory containing
all the vertex data
The geometry pipeline transforms each vertex from model space to clip
space and may perform some lighting calculations
(optional) the geometry pipeline generates texture coordinates from
mathematical formulae
Primitives (triangles, points, quadrilaterals) are rasterized, ie
converted from a set of corner vertices into a huge pile of fragments
(a piece of information representing what will become a pixel)
The colour of each fragment is determined by passing it through the
fragment pipeline (which performs texture lookups, amongst other things)
Some tests are performed on the fragment to decide if it should be kept
or discarded
The pixel colour is calculated based on the fragment's colour, the
current pixel colour and some logical operations involving the fragment
or pixel's alpha channel
You see a pixel
This article will give a bit of an overview of what happens
at each stage and show how a programmable pipeline ("shaders")
is more powerful than the equivalent fixed-function pipelines.
Model Mesh Data
It all starts with a 3D model - a collection of and knowledge of
how they form a mesh of triangles or quadrilaterals.
A vertex typically contains the following properties,
with the Position field being mandatory:
Position
Normal (vector perpendicular to face)
Colour (1 or 2)
Texture coordinates (1 to 4 for each of 1 to 8 texture units)
Other properties for advanced functionality (eg matrix skinning)
The position and normal vectors are in model space, meaning that
they are relative to some origin that is significant to the model,
eg for a homonid it might be between their feet, a teapot might
have its origin at the centre of the base, etc. The colours are
used to specify simple material properties. Texture coordinates
specify which part of a texture each vertex corresponds;
collectively they describe how the texture is stretched over the model.
The typical method of getting this data to the renderer
is to put it all in a big array and tell it how large
(in bytes) each vertex is and how many of them there are
. The vertices are numbered from 0 upwards. The vertex soup is
organised into triangle by passing a list of indices to the
renderer that tells it what combinations of vertices make up triangles.
For example, the index list {0, 10, 1, 11, 2, 12} in triangle strip mode
will cause 4 triangles to be rendered: {0, 10, 1}, {10, 1, 11}, {1, 11, 2},
{11, 2, 12}. Organising the mesh into long strips and fans
means you need to pass (on average) about 1 index per triangle
rather than 3 because after the first two indices, every new
index causes a triangle to be rendered.
All vectors are in homogenous coordinates, a neat form that
deals very neatly with projective geometry. A nice oversimplification
is that there is an extra (4th) coordinate called w; the effective
position of a vector is {x/w, y/w, z/w} rather than just {x, y, z}.
Directional vectors (eg normals) have w=0 so are (sort-of) infinite
length while still maintaining a magnitude. Position vectors
(eg vertex positions) typically have w=1 so are unchanged.
All vectors therefore have 4 elements, matrices are 4x4.
The application needn't always supply all 4 fields of a vector;
the latter ones have defaults.
Geometry Pipeline
The job of the geometry pipeline is to convert vertex data
from model space to world space, view space and then clip space.
Each of these transformations is a matrix*vector multiplication,
so all of the matrices are pre-multiplied together before vertex
processing begins; this works because matrix multiplication is associative.
Pworld = Mm Pmodel
Pview = Mv Pworld
Pclip = Mp Pview
Pclip = (Mp Mv Mm) Pmodel
World space has the origin at some globally unique position -
the corner of a map, the centre of a solar system, whatever.
The camera viewing the scene must also typically move,
so another transformation is necessary - from world space
to view space where the camera is at the origin and looking along the Z axis.
The modelview matrix is the product of the model transform
and view transform and is directly manipulatable by an application.
So, to draw a mesh object moving through space, the only varying
state is (probably) the modelview matrix. In the case of software
"T&L", the transformation and lighting is performed by software on
the host CPU. In other words, the CPU performs a matrix*vector
multiplication for every vertex in every model drawn, as well as
calculating lighting at each vertex (probably in view space).
Hardware T&L provides silicon with exactly this functionality,
relieving the CPU of this task so that it may concentrate on other things like AI.
The conversion from view space to clip space is performed by the projection matrix,
this is what defines the shape of your view: is it a cuboid (orthographic view /
parallel projection, as used in CAD programs) or a frustum
(perspective view that divides object screen positions by their depth)
? Remember that it's all in homogenous coordinates;
for perspective division the projection matrix contains a
-1 in the bottom row, second from the right. The result is
that when a vertex position is multiplied by this matrix,
w is set to -z, thereby causing the x and y screen positions
to be divided by depth.
There are yet more (optional) matrices in operation:
Colour matrix: transforms vertex colours
Texture matrix (one per texture unit):
transforms texture coordinates
It may not be necessary to store texture coordinates
in the model if they are planar (vary linearly across some plane in view space or clip space).
The geometry pipeline can generate the coordinates at render time,
this process is known as TexGen.
Having been transformed to clip space, primitives
(triangles, etc) are clipped against the view volume.
If they're outside, they're discarded; if partially
inside then new primitives are generated that lie entirely
within the clip volume and coplanar with the original primitives.
Still, the transformation and lighting is fixed -
there is only one way to do it and the hardware does just that.
Positions are transformed by the modelview matrix, normals are transformed by its inverse transpose, colours and textures are transformed by the colour and texture matrices (if they're not identity), lighting is calculated with a standardised formula based on light positions, vertex positions and a simple model involving dot products.
Programmable Geometry Pipeline
To gain flexibility of transformation, modern
(since about GeForce3 for consumers) hardware
has provided a mechanism whereby the hardware
behaves more like a CPU than a fixed-function pipeline of matrix operations.
The programmer has the opportunity to write simple assembly programs that
define exactly what transformations are to be applied to each vertex,
completely bypassing the default fixed-function vertex pipeline.
The deformations are of course not limited to position, but effects
can be implemented that modify colours, texture coordinates, normals.
The programmable pipeline encompasses modelview transformation of
the positions and normals, texture and colour transformation (or generation)
but not projection, perspective division or clipping.
Initially it's not obvious why you would bother to do such a thing,
but what it gives you is flexibility in a very high performance part
of the system. Common applications of this flexibility are:
Custom lighting models
Matrix skinning: transforming a vertex using multiple modelview
matrices and blending between them, this gives realistic elbows in models
Procedural geometry: waves, grass, moving blobs, etc
Deformable objects: dents, elastic stretching/bending, etc
Hair, cloth
Arbitrary scrolling, twisting, bending, pinching and swirling
of objects & their textures
Grass is a good example of the use of vertex programs.
The CPU holds a model of a single blade of grass (plus textures, etc)
and sends a few thousand copies of it to the renderer.
The vertex program can use the position of each vertex as the
argument to sin() functions and use the result to translate
the vertex to one side. By varying the phase of the sin() with each frame drawn,
the grass appears to wave in a grasslike fashion; each blade sways side to side
and the net effect has waves travelling through the grass. All with zero effort
from the CPU except to update a couple of numbers each frame.
Because the renderer bends each blade of grass,
it can accurately generate the normal for each vertex
and have the grass accurately lit so that it changes
colour as it bends and interacts differently with the light.
In OpenGL, vertex program functionality is currently
provided by the ARB_vertex_program extension; Direct3D
calls these things Vertex Shaders.
So any time you see lots of objects with similar dynamics,
objects bending, objects with multiple parts and skinned elbows,
etc, chances are you're looking at vertex shaders in action.
Rasterization
For each primitive in eye space (ie the screen coordinates of
the vertices are known), the screen coordinates for each pixel
are determined and values for position, colour, texture coordinates,
etc are determined by interpolation between the vertex values of the primitive.
This means that colour, texture, etc, all vary continuously over a primitive,
(mostly) as expected. Having determined where in view-,
texture- and colour-space each pixel will come from,
a fragment is evaluated from that information by the fragment pipeline.
Fragment Pipeline
The purpose of the fragment pipeline is to determine what
colour (r, g, b, a) a fragment will be, based upon lighting
calculations, texturing, etc. In the fixed-function pipeline,
a set of textures are downloaded (one per texture unit) and
a texture environment (texenv) specified for each texture unit.
A texenv specifies how to combine (eg add, multiply) a texture
sample with some other colour (iterated colour, result from other texture unit).
Special-case texenv operations are available in extensions to perform
things like bumpmapping (deflecting the normal vector per-pixel to modify
lighting and reflection calculations); this must all be manually and
explicitly supported by the card driver and the application.
Consider a knife model - it has a wooden handle and a chromed blade
with a bloodstain on it. We want to draw the knife in a single pass
(ie render it only once and not rely on slow framebuffer blending
operations to give us extra texture layers) from a single model
(not using separate meshes for the handle and blade).
We want it to interact with lighting from the scene (colours and shadowing)
as well as have the blade reflect its surroundings.
A simple implementation might use four textures:
Diffuse colour: woodgrain texture, black on the blade
Lightmap: propagation of light through the scene, including shadows and colours
Glossmap: reflectivity of the knife, dark grey on the handle,
white on the chrome and pale pink on the bloodstained chrome
Environment cubemap: 6 faces of a cube, contains view of world as seen from the knife
The lighting equation we'll use is:
Cfrag = lightmap * diffuse + glossmap * environment
Therefore the wooden handle interacts as expected with scene
lighting (neon signs, candles, some static shadows) and you
can see reflections of the room in the blade. Where there's blood on it,
the reflection is tinted.
Setting all of that up takes wads of code and a fair bit of effort.
It's also quite inflexible because the texture environments typically
only allow you to do arithmetic on outputs (colours) rather than inputs
(texcoords) and therefore special effects (eg bumpmapping)
must be performed with special and specific extensions.
Like a vertex program, a fragment program is a short program
that replaces the old fixed-function pipeline, in this case for fragments.
A fragment program can sample textures and do arithmetic, giving it at
least the same capabilities as the old pipelines. However,
the arithmetic is not restricted to operations on colours,
it can be performed on the texture coordinates before the texture is sampled.
This leads to dependent texture reads, whereby the result of one texture
read is used as (or to modify) the texture coordinates to read from another;
most forms of bumpmapping are special cases of dependent texture reads and
are neatly implemented as such.
But of course, having gained the flexibility, it is much more powerful
than just reimplementations of the fixed-function pipeline. No longer
is texturing restricted to just "drawing pictures on surfaces",
it is now a generalised function sampling operation.
Any function of 1, 2 or 3 variables can be stored at
arbitrary resolution in a texture
(lots of memory in the 3D case but 1D and 2D are quite tractable)
and evaluated on a per-pixel basis. Uses for such technology include:
Bumpmapped per-pixel lighting (dot-product per pixel instead of per-vertex)
Bumpmapped reflections and refraction
Advanced lighting models (pp 67-97): fresnel terms,
subsurface scattering (the "glow" of healthy skin), etc
Scattering: why the sky is blue and sunsets are red (8MB pdf - pretty pictures)
Cartoon rendering (1.8MB pdf)
Arbitrary signal processing, including filtering & post-processing
(2.4MB pdf - pretty pictures)
Procedural Textures (eg marble)
OpenGL exposes this functionality via the ARB_fragment_program extension,
Direct3D refers to it as Pixel Shaders.
Its not always obvious that you're looking at the result of a fragment program;
many programmers use them to implement shaders very similar to the default
pipeline or a basic extension. However, they are often used to implement
very subtle little lighting effects that enhance the realism of a scene
without you consciously noticing their effect - things like scattering,
peach-fuzz on faces (see the nVidia Dawn demo), etc.
The effect doesn't have to be smack-you-in-the-face dramatic
like heavy bumpmapping of a reflective surface. Of course,
if you don't notice the effect and how it got there, its probably done its job.
Per-Fragment Operations
Having decided the fragment colour, it is passed (or not!)
through the stencil, depth and alpha tests, each of which may
decide to keep or discard the fragment. If it is kept, it is
combined with the framebuffer using operations specified by
the program (eg replace, blend using source alpha, etc).
Cg
This discussion would not be complete without a mention of Cg ("C for graphics"),
a high-level language from nVidia that compiles down to the assembly instructions
you would otherwise be required to write to use the above extensions or their
D3D equivalents. Cg does indeed look like C, and its existence makes the writing
of both vertex and fragment programs trivial - you just write down the equations
you want and it "just works". Well, nearly - there is a certain amount of effort
required but nothing onerous.
Cg supports a variety of target architectures; obviously NV20 and NV30 are first
among them. However, it is capable of generating assembly opcodes for the two ARB
[1] extensions described above as well as the DirectX equivalents. I am currently
using it to write both vertex and fragment programs on an ATI card using the ARB
targets ("arbvp1", "arbfp1").
Even if you're not interested in programming these video cards,
looking at the manufacturers' demos can be enlightening or just plain cool:
ATI Demos, nVidia OpenGL Demos, Cg examples and yet more Cg.
If you're an artist/modeler rather than a programmer,
it would appear that there now exists a free version of Maya for personal,
non-commercial use (I haven't tried it); there is a Cg plugin for it that
should allow you to fiddle with shaders without having to write your own 3D engine.
Pixel Shrader has everything to do with the future .
Dont be surprised when PS 5.0 comes around
.Without Pixel Shrader
future Game makers are limited to what they can bring to the table for us to eat .
Heres a good article read if you really want to no whats behind the driving force of gaming and graphic wizardry .
If you program or play computer games or even recently attempted
to purchase a video card, then you will have no doubt heard the terms
"Vertex Shader" and "Pixel Shader". What do they mean,
other than that marketdroids will never cease to invent
crappy technology jargon? Hear all the noise about "Hardware T&L"
a couple of years ago? It's all related and this article will attempt
to describe it all neatly for you.
Warning: mathematics ahead; this devolves into a basic
3D graphics tutorial. It's nothing more complicated than
what you'll see in your final year of high school (linear algebra).
If you're totally non-technical, run screaming now.
First of all, you need to have a basic understanding of how a
consumer render pipeline (stop panicking already) works and how
this is modelled by Direct3D or OpenGL. I shall concentrate
on OpenGL here since it's a tad more portable and it's what
I'm familiar with. If you're not a programmer, skip down to
the bottom where there's links to demos and pretty pictures.
Pipeline Overview
The information & logic that makes up what you see
on the screen in a 3D game (or other visualisation) is composed of:
Vertex data for models
(optional) Index data into vertex arrays
Object positioning logic: vertex transformations
Textures
Yet more textures (eg lightmaps, bumpmaps)
Texture combining logic (add, multiply, dependent, etc)
Fragment operations (stencil, depth test, alpha test)
Blending logic (for transparency)
For each object (eg teapot, knife, player) drawn, the following occurs:
The application tells the card where to find a chunk of memory containing
all the vertex data
The geometry pipeline transforms each vertex from model space to clip
space and may perform some lighting calculations
(optional) the geometry pipeline generates texture coordinates from
mathematical formulae
Primitives (triangles, points, quadrilaterals) are rasterized, ie
converted from a set of corner vertices into a huge pile of fragments
(a piece of information representing what will become a pixel)
The colour of each fragment is determined by passing it through the
fragment pipeline (which performs texture lookups, amongst other things)
Some tests are performed on the fragment to decide if it should be kept
or discarded
The pixel colour is calculated based on the fragment's colour, the
current pixel colour and some logical operations involving the fragment
or pixel's alpha channel
You see a pixel
This article will give a bit of an overview of what happens
at each stage and show how a programmable pipeline ("shaders")
is more powerful than the equivalent fixed-function pipelines.
Model Mesh Data
It all starts with a 3D model - a collection of and knowledge of
how they form a mesh of triangles or quadrilaterals.
A vertex typically contains the following properties,
with the Position field being mandatory:
Position
Normal (vector perpendicular to face)
Colour (1 or 2)
Texture coordinates (1 to 4 for each of 1 to 8 texture units)
Other properties for advanced functionality (eg matrix skinning)
The position and normal vectors are in model space, meaning that
they are relative to some origin that is significant to the model,
eg for a homonid it might be between their feet, a teapot might
have its origin at the centre of the base, etc. The colours are
used to specify simple material properties. Texture coordinates
specify which part of a texture each vertex corresponds;
collectively they describe how the texture is stretched over the model.
The typical method of getting this data to the renderer
is to put it all in a big array and tell it how large
(in bytes) each vertex is and how many of them there are
. The vertices are numbered from 0 upwards. The vertex soup is
organised into triangle by passing a list of indices to the
renderer that tells it what combinations of vertices make up triangles.
For example, the index list {0, 10, 1, 11, 2, 12} in triangle strip mode
will cause 4 triangles to be rendered: {0, 10, 1}, {10, 1, 11}, {1, 11, 2},
{11, 2, 12}. Organising the mesh into long strips and fans
means you need to pass (on average) about 1 index per triangle
rather than 3 because after the first two indices, every new
index causes a triangle to be rendered.
All vectors are in homogenous coordinates, a neat form that
deals very neatly with projective geometry. A nice oversimplification
is that there is an extra (4th) coordinate called w; the effective
position of a vector is {x/w, y/w, z/w} rather than just {x, y, z}.
Directional vectors (eg normals) have w=0 so are (sort-of) infinite
length while still maintaining a magnitude. Position vectors
(eg vertex positions) typically have w=1 so are unchanged.
All vectors therefore have 4 elements, matrices are 4x4.
The application needn't always supply all 4 fields of a vector;
the latter ones have defaults.
Geometry Pipeline
The job of the geometry pipeline is to convert vertex data
from model space to world space, view space and then clip space.
Each of these transformations is a matrix*vector multiplication,
so all of the matrices are pre-multiplied together before vertex
processing begins; this works because matrix multiplication is associative.
Pworld = Mm Pmodel
Pview = Mv Pworld
Pclip = Mp Pview
Pclip = (Mp Mv Mm) Pmodel
World space has the origin at some globally unique position -
the corner of a map, the centre of a solar system, whatever.
The camera viewing the scene must also typically move,
so another transformation is necessary - from world space
to view space where the camera is at the origin and looking along the Z axis.
The modelview matrix is the product of the model transform
and view transform and is directly manipulatable by an application.
So, to draw a mesh object moving through space, the only varying
state is (probably) the modelview matrix. In the case of software
"T&L", the transformation and lighting is performed by software on
the host CPU. In other words, the CPU performs a matrix*vector
multiplication for every vertex in every model drawn, as well as
calculating lighting at each vertex (probably in view space).
Hardware T&L provides silicon with exactly this functionality,
relieving the CPU of this task so that it may concentrate on other things like AI.
The conversion from view space to clip space is performed by the projection matrix,
this is what defines the shape of your view: is it a cuboid (orthographic view /
parallel projection, as used in CAD programs) or a frustum
(perspective view that divides object screen positions by their depth)
? Remember that it's all in homogenous coordinates;
for perspective division the projection matrix contains a
-1 in the bottom row, second from the right. The result is
that when a vertex position is multiplied by this matrix,
w is set to -z, thereby causing the x and y screen positions
to be divided by depth.
There are yet more (optional) matrices in operation:
Colour matrix: transforms vertex colours
Texture matrix (one per texture unit):
transforms texture coordinates
It may not be necessary to store texture coordinates
in the model if they are planar (vary linearly across some plane in view space or clip space).
The geometry pipeline can generate the coordinates at render time,
this process is known as TexGen.
Having been transformed to clip space, primitives
(triangles, etc) are clipped against the view volume.
If they're outside, they're discarded; if partially
inside then new primitives are generated that lie entirely
within the clip volume and coplanar with the original primitives.
Still, the transformation and lighting is fixed -
there is only one way to do it and the hardware does just that.
Positions are transformed by the modelview matrix, normals are transformed by its inverse transpose, colours and textures are transformed by the colour and texture matrices (if they're not identity), lighting is calculated with a standardised formula based on light positions, vertex positions and a simple model involving dot products.
Programmable Geometry Pipeline
To gain flexibility of transformation, modern
(since about GeForce3 for consumers) hardware
has provided a mechanism whereby the hardware
behaves more like a CPU than a fixed-function pipeline of matrix operations.
The programmer has the opportunity to write simple assembly programs that
define exactly what transformations are to be applied to each vertex,
completely bypassing the default fixed-function vertex pipeline.
The deformations are of course not limited to position, but effects
can be implemented that modify colours, texture coordinates, normals.
The programmable pipeline encompasses modelview transformation of
the positions and normals, texture and colour transformation (or generation)
but not projection, perspective division or clipping.
Initially it's not obvious why you would bother to do such a thing,
but what it gives you is flexibility in a very high performance part
of the system. Common applications of this flexibility are:
Custom lighting models
Matrix skinning: transforming a vertex using multiple modelview
matrices and blending between them, this gives realistic elbows in models
Procedural geometry: waves, grass, moving blobs, etc
Deformable objects: dents, elastic stretching/bending, etc
Hair, cloth
Arbitrary scrolling, twisting, bending, pinching and swirling
of objects & their textures
Grass is a good example of the use of vertex programs.
The CPU holds a model of a single blade of grass (plus textures, etc)
and sends a few thousand copies of it to the renderer.
The vertex program can use the position of each vertex as the
argument to sin() functions and use the result to translate
the vertex to one side. By varying the phase of the sin() with each frame drawn,
the grass appears to wave in a grasslike fashion; each blade sways side to side
and the net effect has waves travelling through the grass. All with zero effort
from the CPU except to update a couple of numbers each frame.
Because the renderer bends each blade of grass,
it can accurately generate the normal for each vertex
and have the grass accurately lit so that it changes
colour as it bends and interacts differently with the light.
In OpenGL, vertex program functionality is currently
provided by the ARB_vertex_program extension; Direct3D
calls these things Vertex Shaders.
So any time you see lots of objects with similar dynamics,
objects bending, objects with multiple parts and skinned elbows,
etc, chances are you're looking at vertex shaders in action.
Rasterization
For each primitive in eye space (ie the screen coordinates of
the vertices are known), the screen coordinates for each pixel
are determined and values for position, colour, texture coordinates,
etc are determined by interpolation between the vertex values of the primitive.
This means that colour, texture, etc, all vary continuously over a primitive,
(mostly) as expected. Having determined where in view-,
texture- and colour-space each pixel will come from,
a fragment is evaluated from that information by the fragment pipeline.
Fragment Pipeline
The purpose of the fragment pipeline is to determine what
colour (r, g, b, a) a fragment will be, based upon lighting
calculations, texturing, etc. In the fixed-function pipeline,
a set of textures are downloaded (one per texture unit) and
a texture environment (texenv) specified for each texture unit.
A texenv specifies how to combine (eg add, multiply) a texture
sample with some other colour (iterated colour, result from other texture unit).
Special-case texenv operations are available in extensions to perform
things like bumpmapping (deflecting the normal vector per-pixel to modify
lighting and reflection calculations); this must all be manually and
explicitly supported by the card driver and the application.
Consider a knife model - it has a wooden handle and a chromed blade
with a bloodstain on it. We want to draw the knife in a single pass
(ie render it only once and not rely on slow framebuffer blending
operations to give us extra texture layers) from a single model
(not using separate meshes for the handle and blade).
We want it to interact with lighting from the scene (colours and shadowing)
as well as have the blade reflect its surroundings.
A simple implementation might use four textures:
Diffuse colour: woodgrain texture, black on the blade
Lightmap: propagation of light through the scene, including shadows and colours
Glossmap: reflectivity of the knife, dark grey on the handle,
white on the chrome and pale pink on the bloodstained chrome
Environment cubemap: 6 faces of a cube, contains view of world as seen from the knife
The lighting equation we'll use is:
Cfrag = lightmap * diffuse + glossmap * environment
Therefore the wooden handle interacts as expected with scene
lighting (neon signs, candles, some static shadows) and you
can see reflections of the room in the blade. Where there's blood on it,
the reflection is tinted.
Setting all of that up takes wads of code and a fair bit of effort.
It's also quite inflexible because the texture environments typically
only allow you to do arithmetic on outputs (colours) rather than inputs
(texcoords) and therefore special effects (eg bumpmapping)
must be performed with special and specific extensions.
Like a vertex program, a fragment program is a short program
that replaces the old fixed-function pipeline, in this case for fragments.
A fragment program can sample textures and do arithmetic, giving it at
least the same capabilities as the old pipelines. However,
the arithmetic is not restricted to operations on colours,
it can be performed on the texture coordinates before the texture is sampled.
This leads to dependent texture reads, whereby the result of one texture
read is used as (or to modify) the texture coordinates to read from another;
most forms of bumpmapping are special cases of dependent texture reads and
are neatly implemented as such.
But of course, having gained the flexibility, it is much more powerful
than just reimplementations of the fixed-function pipeline. No longer
is texturing restricted to just "drawing pictures on surfaces",
it is now a generalised function sampling operation.
Any function of 1, 2 or 3 variables can be stored at
arbitrary resolution in a texture
(lots of memory in the 3D case but 1D and 2D are quite tractable)
and evaluated on a per-pixel basis. Uses for such technology include:
Bumpmapped per-pixel lighting (dot-product per pixel instead of per-vertex)
Bumpmapped reflections and refraction
Advanced lighting models (pp 67-97): fresnel terms,
subsurface scattering (the "glow" of healthy skin), etc
Scattering: why the sky is blue and sunsets are red (8MB pdf - pretty pictures)
Cartoon rendering (1.8MB pdf)
Arbitrary signal processing, including filtering & post-processing
(2.4MB pdf - pretty pictures)
Procedural Textures (eg marble)
OpenGL exposes this functionality via the ARB_fragment_program extension,
Direct3D refers to it as Pixel Shaders.
Its not always obvious that you're looking at the result of a fragment program;
many programmers use them to implement shaders very similar to the default
pipeline or a basic extension. However, they are often used to implement
very subtle little lighting effects that enhance the realism of a scene
without you consciously noticing their effect - things like scattering,
peach-fuzz on faces (see the nVidia Dawn demo), etc.
The effect doesn't have to be smack-you-in-the-face dramatic
like heavy bumpmapping of a reflective surface. Of course,
if you don't notice the effect and how it got there, its probably done its job.
Per-Fragment Operations
Having decided the fragment colour, it is passed (or not!)
through the stencil, depth and alpha tests, each of which may
decide to keep or discard the fragment. If it is kept, it is
combined with the framebuffer using operations specified by
the program (eg replace, blend using source alpha, etc).
Cg
This discussion would not be complete without a mention of Cg ("C for graphics"),
a high-level language from nVidia that compiles down to the assembly instructions
you would otherwise be required to write to use the above extensions or their
D3D equivalents. Cg does indeed look like C, and its existence makes the writing
of both vertex and fragment programs trivial - you just write down the equations
you want and it "just works". Well, nearly - there is a certain amount of effort
required but nothing onerous.
Cg supports a variety of target architectures; obviously NV20 and NV30 are first
among them. However, it is capable of generating assembly opcodes for the two ARB
[1] extensions described above as well as the DirectX equivalents. I am currently
using it to write both vertex and fragment programs on an ATI card using the ARB
targets ("arbvp1", "arbfp1").
Even if you're not interested in programming these video cards,
looking at the manufacturers' demos can be enlightening or just plain cool:
ATI Demos, nVidia OpenGL Demos, Cg examples and yet more Cg.
If you're an artist/modeler rather than a programmer,
it would appear that there now exists a free version of Maya for personal,
non-commercial use (I haven't tried it); there is a Cg plugin for it that
should allow you to fiddle with shaders without having to write your own 3D engine.
Cappy said:PS3.0 is only the 3rd step in a long staircase .
Pixel Shrader has everything to do with the future .
Dont be surprised when PS 5.0 comes around
Without Pixel Shrader
future Game makers are limited to what they can bring to the table for us to eat .
Heres a good article read if you really want to no whats behind the driving force of gaming and graphic wizardry .
If you program or play computer games or even recently attempted
to purchase a video card, then you will have no doubt heard the terms
"Vertex Shader" and "Pixel Shader". What do they mean,
other than that marketdroids will never cease to invent
crappy technology jargon? Hear all the noise about "Hardware T&L"
a couple of years ago? It's all related and this article will attempt
to describe it all neatly for you.
Warning: mathematics ahead; this devolves into a basic
3D graphics tutorial. It's nothing more complicated than
what you'll see in your final year of high school (linear algebra).
If you're totally non-technical, run screaming now.
First of all, you need to have a basic understanding of how a
consumer render pipeline (stop panicking already) works and how
this is modelled by Direct3D or OpenGL. I shall concentrate
on OpenGL here since it's a tad more portable and it's what
I'm familiar with. If you're not a programmer, skip down to
the bottom where there's links to demos and pretty pictures.
Pipeline Overview
The information & logic that makes up what you see
on the screen in a 3D game (or other visualisation) is composed of:
Vertex data for models
(optional) Index data into vertex arrays
Object positioning logic: vertex transformations
Textures
Yet more textures (eg lightmaps, bumpmaps)
Texture combining logic (add, multiply, dependent, etc)
Fragment operations (stencil, depth test, alpha test)
Blending logic (for transparency)
For each object (eg teapot, knife, player) drawn, the following occurs:
The application tells the card where to find a chunk of memory containing
all the vertex data
The geometry pipeline transforms each vertex from model space to clip
space and may perform some lighting calculations
(optional) the geometry pipeline generates texture coordinates from
mathematical formulae
Primitives (triangles, points, quadrilaterals) are rasterized, ie
converted from a set of corner vertices into a huge pile of fragments
(a piece of information representing what will become a pixel)
The colour of each fragment is determined by passing it through the
fragment pipeline (which performs texture lookups, amongst other things)
Some tests are performed on the fragment to decide if it should be kept
or discarded
The pixel colour is calculated based on the fragment's colour, the
current pixel colour and some logical operations involving the fragment
or pixel's alpha channel
You see a pixel
This article will give a bit of an overview of what happens
at each stage and show how a programmable pipeline ("shaders")
is more powerful than the equivalent fixed-function pipelines.
Model Mesh Data
It all starts with a 3D model - a collection of and knowledge of
how they form a mesh of triangles or quadrilaterals.
A vertex typically contains the following properties,
with the Position field being mandatory:
Position
Normal (vector perpendicular to face)
Colour (1 or 2)
Texture coordinates (1 to 4 for each of 1 to 8 texture units)
Other properties for advanced functionality (eg matrix skinning)
The position and normal vectors are in model space, meaning that
they are relative to some origin that is significant to the model,
eg for a homonid it might be between their feet, a teapot might
have its origin at the centre of the base, etc. The colours are
used to specify simple material properties. Texture coordinates
specify which part of a texture each vertex corresponds;
collectively they describe how the texture is stretched over the model.
The typical method of getting this data to the renderer
is to put it all in a big array and tell it how large
(in bytes) each vertex is and how many of them there are
. The vertices are numbered from 0 upwards. The vertex soup is
organised into triangle by passing a list of indices to the
renderer that tells it what combinations of vertices make up triangles.
For example, the index list {0, 10, 1, 11, 2, 12} in triangle strip mode
will cause 4 triangles to be rendered: {0, 10, 1}, {10, 1, 11}, {1, 11, 2},
{11, 2, 12}. Organising the mesh into long strips and fans
means you need to pass (on average) about 1 index per triangle
rather than 3 because after the first two indices, every new
index causes a triangle to be rendered.
All vectors are in homogenous coordinates, a neat form that
deals very neatly with projective geometry. A nice oversimplification
is that there is an extra (4th) coordinate called w; the effective
position of a vector is {x/w, y/w, z/w} rather than just {x, y, z}.
Directional vectors (eg normals) have w=0 so are (sort-of) infinite
length while still maintaining a magnitude. Position vectors
(eg vertex positions) typically have w=1 so are unchanged.
All vectors therefore have 4 elements, matrices are 4x4.
The application needn't always supply all 4 fields of a vector;
the latter ones have defaults.
Geometry Pipeline
The job of the geometry pipeline is to convert vertex data
from model space to world space, view space and then clip space.
Each of these transformations is a matrix*vector multiplication,
so all of the matrices are pre-multiplied together before vertex
processing begins; this works because matrix multiplication is associative.
Pworld = Mm Pmodel
Pview = Mv Pworld
Pclip = Mp Pview
Pclip = (Mp Mv Mm) Pmodel
World space has the origin at some globally unique position -
the corner of a map, the centre of a solar system, whatever.
The camera viewing the scene must also typically move,
so another transformation is necessary - from world space
to view space where the camera is at the origin and looking along the Z axis.
The modelview matrix is the product of the model transform
and view transform and is directly manipulatable by an application.
So, to draw a mesh object moving through space, the only varying
state is (probably) the modelview matrix. In the case of software
"T&L", the transformation and lighting is performed by software on
the host CPU. In other words, the CPU performs a matrix*vector
multiplication for every vertex in every model drawn, as well as
calculating lighting at each vertex (probably in view space).
Hardware T&L provides silicon with exactly this functionality,
relieving the CPU of this task so that it may concentrate on other things like AI.
The conversion from view space to clip space is performed by the projection matrix,
this is what defines the shape of your view: is it a cuboid (orthographic view /
parallel projection, as used in CAD programs) or a frustum
(perspective view that divides object screen positions by their depth)
? Remember that it's all in homogenous coordinates;
for perspective division the projection matrix contains a
-1 in the bottom row, second from the right. The result is
that when a vertex position is multiplied by this matrix,
w is set to -z, thereby causing the x and y screen positions
to be divided by depth.
There are yet more (optional) matrices in operation:
Colour matrix: transforms vertex colours
Texture matrix (one per texture unit):
transforms texture coordinates
It may not be necessary to store texture coordinates
in the model if they are planar (vary linearly across some plane in view space or clip space).
The geometry pipeline can generate the coordinates at render time,
this process is known as TexGen.
Having been transformed to clip space, primitives
(triangles, etc) are clipped against the view volume.
If they're outside, they're discarded; if partially
inside then new primitives are generated that lie entirely
within the clip volume and coplanar with the original primitives.
Still, the transformation and lighting is fixed -
there is only one way to do it and the hardware does just that.
Positions are transformed by the modelview matrix, normals are transformed by its inverse transpose, colours and textures are transformed by the colour and texture matrices (if they're not identity), lighting is calculated with a standardised formula based on light positions, vertex positions and a simple model involving dot products.
Programmable Geometry Pipeline
To gain flexibility of transformation, modern
(since about GeForce3 for consumers) hardware
has provided a mechanism whereby the hardware
behaves more like a CPU than a fixed-function pipeline of matrix operations.
The programmer has the opportunity to write simple assembly programs that
define exactly what transformations are to be applied to each vertex,
completely bypassing the default fixed-function vertex pipeline.
The deformations are of course not limited to position, but effects
can be implemented that modify colours, texture coordinates, normals.
The programmable pipeline encompasses modelview transformation of
the positions and normals, texture and colour transformation (or generation)
but not projection, perspective division or clipping.
Initially it's not obvious why you would bother to do such a thing,
but what it gives you is flexibility in a very high performance part
of the system. Common applications of this flexibility are:
Custom lighting models
Matrix skinning: transforming a vertex using multiple modelview
matrices and blending between them, this gives realistic elbows in models
Procedural geometry: waves, grass, moving blobs, etc
Deformable objects: dents, elastic stretching/bending, etc
Hair, cloth
Arbitrary scrolling, twisting, bending, pinching and swirling
of objects & their textures
Grass is a good example of the use of vertex programs.
The CPU holds a model of a single blade of grass (plus textures, etc)
and sends a few thousand copies of it to the renderer.
The vertex program can use the position of each vertex as the
argument to sin() functions and use the result to translate
the vertex to one side. By varying the phase of the sin() with each frame drawn,
the grass appears to wave in a grasslike fashion; each blade sways side to side
and the net effect has waves travelling through the grass. All with zero effort
from the CPU except to update a couple of numbers each frame.
Because the renderer bends each blade of grass,
it can accurately generate the normal for each vertex
and have the grass accurately lit so that it changes
colour as it bends and interacts differently with the light.
In OpenGL, vertex program functionality is currently
provided by the ARB_vertex_program extension; Direct3D
calls these things Vertex Shaders.
So any time you see lots of objects with similar dynamics,
objects bending, objects with multiple parts and skinned elbows,
etc, chances are you're looking at vertex shaders in action.
Rasterization
For each primitive in eye space (ie the screen coordinates of
the vertices are known), the screen coordinates for each pixel
are determined and values for position, colour, texture coordinates,
etc are determined by interpolation between the vertex values of the primitive.
This means that colour, texture, etc, all vary continuously over a primitive,
(mostly) as expected. Having determined where in view-,
texture- and colour-space each pixel will come from,
a fragment is evaluated from that information by the fragment pipeline.
Fragment Pipeline
The purpose of the fragment pipeline is to determine what
colour (r, g, b, a) a fragment will be, based upon lighting
calculations, texturing, etc. In the fixed-function pipeline,
a set of textures are downloaded (one per texture unit) and
a texture environment (texenv) specified for each texture unit.
A texenv specifies how to combine (eg add, multiply) a texture
sample with some other colour (iterated colour, result from other texture unit).
Special-case texenv operations are available in extensions to perform
things like bumpmapping (deflecting the normal vector per-pixel to modify
lighting and reflection calculations); this must all be manually and
explicitly supported by the card driver and the application.
Consider a knife model - it has a wooden handle and a chromed blade
with a bloodstain on it. We want to draw the knife in a single pass
(ie render it only once and not rely on slow framebuffer blending
operations to give us extra texture layers) from a single model
(not using separate meshes for the handle and blade).
We want it to interact with lighting from the scene (colours and shadowing)
as well as have the blade reflect its surroundings.
A simple implementation might use four textures:
Diffuse colour: woodgrain texture, black on the blade
Lightmap: propagation of light through the scene, including shadows and colours
Glossmap: reflectivity of the knife, dark grey on the handle,
white on the chrome and pale pink on the bloodstained chrome
Environment cubemap: 6 faces of a cube, contains view of world as seen from the knife
The lighting equation we'll use is:
Cfrag = lightmap * diffuse + glossmap * environment
Therefore the wooden handle interacts as expected with scene
lighting (neon signs, candles, some static shadows) and you
can see reflections of the room in the blade. Where there's blood on it,
the reflection is tinted.
Setting all of that up takes wads of code and a fair bit of effort.
It's also quite inflexible because the texture environments typically
only allow you to do arithmetic on outputs (colours) rather than inputs
(texcoords) and therefore special effects (eg bumpmapping)
must be performed with special and specific extensions.
Like a vertex program, a fragment program is a short program
that replaces the old fixed-function pipeline, in this case for fragments.
A fragment program can sample textures and do arithmetic, giving it at
least the same capabilities as the old pipelines. However,
the arithmetic is not restricted to operations on colours,
it can be performed on the texture coordinates before the texture is sampled.
This leads to dependent texture reads, whereby the result of one texture
read is used as (or to modify) the texture coordinates to read from another;
most forms of bumpmapping are special cases of dependent texture reads and
are neatly implemented as such.
But of course, having gained the flexibility, it is much more powerful
than just reimplementations of the fixed-function pipeline. No longer
is texturing restricted to just "drawing pictures on surfaces",
it is now a generalised function sampling operation.
Any function of 1, 2 or 3 variables can be stored at
arbitrary resolution in a texture
(lots of memory in the 3D case but 1D and 2D are quite tractable)
and evaluated on a per-pixel basis. Uses for such technology include:
Bumpmapped per-pixel lighting (dot-product per pixel instead of per-vertex)
Bumpmapped reflections and refraction
Advanced lighting models (pp 67-97): fresnel terms,
subsurface scattering (the "glow" of healthy skin), etc
Scattering: why the sky is blue and sunsets are red (8MB pdf - pretty pictures)
Cartoon rendering (1.8MB pdf)
Arbitrary signal processing, including filtering & post-processing
(2.4MB pdf - pretty pictures)
Procedural Textures (eg marble)
OpenGL exposes this functionality via the ARB_fragment_program extension,
Direct3D refers to it as Pixel Shaders.
Its not always obvious that you're looking at the result of a fragment program;
many programmers use them to implement shaders very similar to the default
pipeline or a basic extension. However, they are often used to implement
very subtle little lighting effects that enhance the realism of a scene
without you consciously noticing their effect - things like scattering,
peach-fuzz on faces (see the nVidia Dawn demo), etc.
The effect doesn't have to be smack-you-in-the-face dramatic
like heavy bumpmapping of a reflective surface. Of course,
if you don't notice the effect and how it got there, its probably done its job.
Per-Fragment Operations
Having decided the fragment colour, it is passed (or not!)
through the stencil, depth and alpha tests, each of which may
decide to keep or discard the fragment. If it is kept, it is
combined with the framebuffer using operations specified by
the program (eg replace, blend using source alpha, etc).
Cg
This discussion would not be complete without a mention of Cg ("C for graphics"),
a high-level language from nVidia that compiles down to the assembly instructions
you would otherwise be required to write to use the above extensions or their
D3D equivalents. Cg does indeed look like C, and its existence makes the writing
of both vertex and fragment programs trivial - you just write down the equations
you want and it "just works". Well, nearly - there is a certain amount of effort
required but nothing onerous.
Cg supports a variety of target architectures; obviously NV20 and NV30 are first
among them. However, it is capable of generating assembly opcodes for the two ARB
[1] extensions described above as well as the DirectX equivalents. I am currently
using it to write both vertex and fragment programs on an ATI card using the ARB
targets ("arbvp1", "arbfp1").
Even if you're not interested in programming these video cards,
looking at the manufacturers' demos can be enlightening or just plain cool:
ATI Demos, nVidia OpenGL Demos, Cg examples and yet more Cg.
If you're an artist/modeler rather than a programmer,
it would appear that there now exists a free version of Maya for personal,
non-commercial use (I haven't tried it); there is a Cg plugin for it that
should allow you to fiddle with shaders without having to write your own 3D engine.
Wow this is a 5 star post! It's like an encyclopedia.
- Status
