Only when you ovulate is there an unfertilised egg waiting for a sperm cell. Once an egg comes into contact with sperm, it becomes fertilised and attaches itself to the womb. This starts the conception process and you become pregnant.
If you have sex before you ovulate, sperm can hang around for days in your fallopian tube waiting for an egg to pop out. If you have sex after you ovulate, the egg will be ready and waiting for the sperm to arrive.
But remember, having sex only at other times of the month is not a reliable way to avoid getting pregnant!