It's a mistake because, if you only allow your baby to sleep when it's perfectly quiet and they're in their crib, they'll never be able to sleep in any other situation.
First let's address noise: From day one vary the amount of noise that is around your baby while they're sleeping, during the day. One day have it quiet, the next day vacuum in the room while they are sleeping. Play a radio during nap time one day put on talk radio and the next play music. Make sure that the sound is up to a normal speaking voice level. Leave the door open to their room so that the everyday noises get in. If you have kept it silent and your baby is older, gradually increase the amount of noise around them while they're sleeping. It may take a few weeks but they'll get used to it.
Location: When you put your baby down for a nap vary the places that they sleep. Don't only put them in their crib!!!!! Place them on a blanket on the floor, in the stroller, in a baby swing, on a couch (make sure that they can't roll off!!!) Switch the location for each nap and also don't change the activities that you need to take care of. Bang the dishes, vacuum the floors, talk, let the other kids run around and play. Carry them sleeping in their car seat to run errands.
If you do this, your baby will be able to sleep when they need, to no matter where you are or what's going on around them. How many crying kids have you seen at a store and the mother says, "I've got to get them home! It's their nap time and they're so tired."
I didn't want to have to go home just because it was nap time or have a simple phone call wake them up. My kids would just drop off wherever they were, when they got tired. They still can! Noises and location made no difference to them. Get your baby in the habit of sleeping wherever and whenever they need to and you won't regret it.