1. Install the necessary build tools

You may find that you need some additional development libraries depending on what is already installed on your system.
sudo apt-get install build-essential libglut3-dev libxi-dev libxmu-dev

2. Download the CUDA driver, Toolkit, and SDK from developer.nvidia.com
3. Install the CUDA driver

The nVidia driver cannot be installed while X is running, so you will need to stop gdm and install the driver in a virtual console by pressing ctrl+alt+F1. If you have already installed the Ubuntu restricted driver for your nVidia graphics card like I did, you will probably run into some trouble installing the CUDA driver. I solved my problems by following the nVidia driver manual installation guide here. The normal process if there are no conflicting drivers already installed are as follows.
chmod +x NVIDIA-Linux-x86-xxx-pkg1.run
sudo /etc/init.d/gdm stop
sudo ./NVIDIA-Linux-x86-xxx-pkg1.run
sudo /etc/init.d/gdm start

4. Install the CUDA toolkit

chmod +x NVIDIA_CUDA_Toolkit_2.1_ubuntu8.04_x86.run
sudo ./NVIDIA_CUDA_Toolkit_2.1_ubuntu8.06_x86.run

5. Modify Environment Variables

The simplest way to do this is to edit your ~/.bashrc file. Changes take effect when you start a new terminal session.
PATH=$PATH:/usr/local/cuda/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib
export PATH
export LD_LIBRARY_PATH

6. Install the CUDA SDK

chmod +x ./NVIDIA_CUDA_SDK_xxx_linux.run
./NVIDIA_CUDA_SDK_xxx_linux.run

7. Compile and Run the example programs

cd ~/NVIDIA_CUDA_SDK
make

Now execute any of the example programs in ~/NVIDIA_CUDA_SDK/bin/linux/release/

Check out http://lifeofaprogrammergeek.blogspot.com/2008/05/cuda-development-in-ubuntu.html for similar details and how to copy the template project and start coding your own CUDA program.