Create the config/provision directory after checking that config_path is writable and it doesn't exist.