I wouldn't go that far. If you don't know where you are going or are some hick from the country, by all means, stay downtown. If you make sure to avoid certain areas, there are plenty of nice places (restaurants, clubs, hotels, etc.) away from the harbor that are also much easier on the wallet.

Now the Orioles...they suck.