If your site is bad enough that people would rather click the back button then maybe you should rethink your design or content. Have you ever been to www.2advancedstudios.com or www.brokenpath.com? They loop music. Short little ambient clips. I wouldnt say the key would be to not have music, just have a variety of it so it doesnt get old. Their sites are insanely awesome compared to mine because I dont really have any content except for the hidden secret stuff that only a few friends know how to access, or clients. But thats besides the point, I just think youre wrong about playing music on a web site.
All good points in regards to music on web sites. If you want to appeal to the widest audience possible, then I would agree that music is probably something you would want to avoid. One has to think if they want the visitor to the site to focus on the content, or the music, and hunting for the not always conspicuous "music off" button. I believe background music music can really enrich a user's experience, but it must be used cleverly and not cart blanche as it seems to be on most flash web sites. I like your comment about the music being in a default off position, that seems to be a good solution. The visitor then makes the decision on whether to listen to music. I know when I surf, I usually have my own music playing, and when the site's sound kicks in, I have a few seconds of unpleasantness trying to either turn down my music, or the sites.
-peace!