Sarah, this is normal for the burner to stay on, until the water reaches the thermosat temperture. It is normal to shut off untill the water tempeture drops below the thermostat setting, then it kicks back on till it reaches its thermostat setting again. and pilot light should stay on at all times ...