In principle you only need the exe and the dll's. If you have config information you want to change, you should add the config file (yourapp.exe.config). In that case you do what we call an XCopy deployment, simply copying the relevant files.
The best way to ensure you get everything is to add an installer to your project. You can do that in VS by adding a Setup project to the solution, add Primary output to get your executables into the msi. Your program can then be installed properly, and also deinstalled from the Add/Remove Program panel.
The last, and possibly "smartest" way, is to do as suggested in the previous post, use a ClickOnce approach. You app can then be installed and upgraded automatically from a website.
- terje