Speeding up Drupal
Nobody likes to wait. Especially people surfing the web. They want to see what they are looking for fast and will judge your website within a fraction of a second. Based on their first impression and user experience they make choices. A slow loading website is bad experience and will have a negative impact on conversion.
See the 'waterfall' load comparison test below between an setup (A) without any cache or optimizations, and (B) a setup which is almost fully optimized.
Now, Drupal does a good job of optimization 'out-of-the box'. Under settings -> performance you will find a number of options which optimize performance. You can enable server-side caching and aggregate all js and css files. That's pretty nice but if you'll use a tool like YSlow or PageSpeed and analyze your site, you will probably still see a number of red flags showing up. These are tips for further optimizing your page load.
In addition to faster load times, this one can potentially save your server a lot of bandwidth too!
This trick requires you to setup a number of subdomaines which point to the main site. Browsers are limited to a number of concurrent connections to a hostname. For modern browsers this amount is six, but older ones (or even IE7) have as few as two (see full comparison). You can workaround this limitation by providing images, css and js files from different domains and thus speeding things up significantly.
Guess what. There is a Drupal module for this: parallel.
You might check out if you have any seo issues (images are taken from another domain, so this might influence your search visibility if you receive many visitors from image search.
Content Delivery Network
An even better way is to use a CDN and put it all in the cloud, but this unnecessary for many sites. There is a Drupal CDN module which - in anaddition to supporting real CDN's - also offers the same functionality as parallel. The CDN module is maintained by Wim Leers which also wrote a great but a bit older post on Drupal page loading performance. Also check out his slideshow and video presentation from DrupalCon Paris on this topic if you want to know more.
Leverage browser caching
You might not see this one pop up in PageSpeed at all if your server is already properly configured. Browsers tend to cache files they downloaded so future requests are pulled of your harddisk instead of the website. This requires your files (images) to provide an expiry date in the HTTP headers so your browser will know when to look for a new version. Drupal already this for you, and sets all expiry dates to two weeks in the future. However this is done in the .htaccess file and requires the mod_expires Apache module. If you get a red flag on this one like in the image above, you probably don't have it running.
This one can have a big impact on image rich websites where people are browsing image galleries with a lot of thumbnails: the browser will download all images which don't have an expiry date set on each page request!
These are some of the things which will probably have the biggest impact. But, depending on your needs, you might try a number of other techniques. For example, use sprites for your theme images, or optimize your PNG's. For the Drupal 7 icon set, ImageOptim(mac only) was used to smush core images.
In case your bottleneck is at the server-side, you might want to look at accelerators to speed up php like APC or XCache. Also check out this great comparison chart for Drupal performance modules.