Using custom containers with AI Platform Training Reviews

Baptiste S. · Reviewed over 3 years ago

Benedikt S. · Reviewed over 3 years ago

bugs

MunkhOrgil E. · Reviewed over 3 years ago

Miguel Z. · Reviewed over 3 years ago

Everton L. · Reviewed over 3 years ago

command completes with error

Serhii S. · Reviewed over 3 years ago

PRATIK G. · Reviewed over 3 years ago

Mugdha S. · Reviewed over 3 years ago

Navin k. · Reviewed over 3 years ago

David G. · Reviewed over 3 years ago

Jeremy C. · Reviewed over 3 years ago

Ilya O. · Reviewed over 3 years ago

Error during notebook run: INFO 2022-07-25 17:50:26 +0000 service Validating job requirements... INFO 2022-07-25 17:50:26 +0000 service Job creation request has been successfully validated. INFO 2022-07-25 17:50:27 +0000 service Job JOB_20220725_175025 is queued. INFO 2022-07-25 17:50:36 +0000 service 4 Waiting for job to be provisioned. INFO 2022-07-25 17:50:36 +0000 service 3 Waiting for job to be provisioned. INFO 2022-07-25 17:50:36 +0000 service 1 Waiting for job to be provisioned. INFO 2022-07-25 17:50:36 +0000 service 2 Waiting for job to be provisioned. INFO 2022-07-25 17:50:38 +0000 service 2 Waiting for training program to start. INFO 2022-07-25 17:50:39 +0000 service 3 Waiting for training program to start. INFO 2022-07-25 17:50:39 +0000 service 1 Waiting for training program to start. INFO 2022-07-25 17:50:39 +0000 service 4 Waiting for training program to start. NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 2 Opening GCS connection... NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 2 Mounting file system "gcsfuse"... NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 2 File system has been successfully mounted. NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 1 Opening GCS connection... NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 1 Mounting file system "gcsfuse"... NOTICE 2022-07-25 17:51:10 +0000 master-replica-0.gcsfuse 1 File system has been successfully mounted. NOTICE 2022-07-25 17:51:11 +0000 master-replica-0.gcsfuse 3 Opening GCS connection... NOTICE 2022-07-25 17:51:11 +0000 master-replica-0.gcsfuse 3 Mounting file system "gcsfuse"... NOTICE 2022-07-25 17:51:11 +0000 master-replica-0.gcsfuse 3 File system has been successfully mounted. NOTICE 2022-07-25 17:51:14 +0000 master-replica-0.gcsfuse 4 Opening GCS connection... NOTICE 2022-07-25 17:51:14 +0000 master-replica-0.gcsfuse 4 Mounting file system "gcsfuse"... NOTICE 2022-07-25 17:51:14 +0000 master-replica-0.gcsfuse 4 File system has been successfully mounted. ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 Traceback (most recent call last): ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "train.py", line 88, in <module> ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 fire.Fire(train_evaluate) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 141, in Fire ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 component_trace = _Fire(component, args, parsed_flag_args, context, name) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 target=component.__name__) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 component = fn(*varargs, **kwargs) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "train.py", line 35, in train_evaluate ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 df_train = pd.read_csv(training_dataset_path) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 return _read(filepath_or_buffer, kwds) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 413, in _read ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 filepath_or_buffer, encoding, compression) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 223, in get_filepath_or_buffer ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 mode=mode) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/pandas/io/gcs.py", line 15, in get_filepath_or_buffer ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 filepath_or_buffer = fs.open(filepath_or_buffer, mode) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1043, in open ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 **kwargs, ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1208, in _open ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 **kwargs, ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1325, in __init__ ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 **kwargs, ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1385, in __init__ ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 self.size = self.details["size"] ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1398, in details ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 self._details = self.fs.info(self.path) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 86, in wrapper ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 return sync(self.loop, func, *args, **kwargs) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 raise return_result ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 result[0] = await coro ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 724, in _info ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 out = await self._ls(path, **kwargs) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 760, in _ls ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 out = await self._list_objects(path, prefix=prefix) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 504, in _list_objects ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 return [await self._get_object(path)] ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 458, in _get_object ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 393, in _call ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 method, path, *args, **kwargs ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/decorator.py", line 221, in fun ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 return await caller(func, *(extras + args), **kw) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 return await func(*args, **kwargs) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 384, in _request ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 validate_response(status, contents, path, args) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 84, in validate_response ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 raise FileNotFoundError(path) ERROR 2022-07-25 17:52:19 +0000 master-replica-0 2 FileNotFoundError: b/qwiklabs-gcp-01-ec0ff30e09da-kubeflowpipelines-default/o/%2Fdata%2Ftraining%2Fdataset.csv ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 Traceback (most recent call last): ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "train.py", line 88, in <module> ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 fire.Fire(train_evaluate) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 141, in Fire ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 component_trace = _Fire(component, args, parsed_flag_args, context, name) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 target=component.__name__) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 component = fn(*varargs, **kwargs) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "train.py", line 35, in train_evaluate ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 df_train = pd.read_csv(training_dataset_path) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 return _read(filepath_or_buffer, kwds) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 413, in _read ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 filepath_or_buffer, encoding, compression) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 223, in get_filepath_or_buffer ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 mode=mode) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/pandas/io/gcs.py", line 15, in get_filepath_or_buffer ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 filepath_or_buffer = fs.open(filepath_or_buffer, mode) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1043, in open ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 **kwargs, ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1208, in _open ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 **kwargs, ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1325, in __init__ ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 **kwargs, ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1385, in __init__ ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 self.size = self.details["size"] ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1398, in details ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 self._details = self.fs.info(self.path) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 86, in wrapper ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 return sync(self.loop, func, *args, **kwargs) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 raise return_result ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 result[0] = await coro ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 724, in _info ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 out = await self._ls(path, **kwargs) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 760, in _ls ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 out = await self._list_objects(path, prefix=prefix) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 504, in _list_objects ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 return [await self._get_object(path)] ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 458, in _get_object ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 393, in _call ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 method, path, *args, **kwargs ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/decorator.py", line 221, in fun ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 return await caller(func, *(extras + args), **kw) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 return await func(*args, **kwargs) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 384, in _request ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 validate_response(status, contents, path, args) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 84, in validate_response ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 raise FileNotFoundError(path) ERROR 2022-07-25 17:52:21 +0000 master-replica-0 1 FileNotFoundError: b/qwiklabs-gcp-01-ec0ff30e09da-kubeflowpipelines-default/o/%2Fdata%2Ftraining%2Fdataset.csv ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 Traceback (most recent call last): ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "train.py", line 88, in <module> ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 fire.Fire(train_evaluate) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 141, in Fire ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 component_trace = _Fire(component, args, parsed_flag_args, context, name) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 target=component.__name__) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 component = fn(*varargs, **kwargs) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "train.py", line 35, in train_evaluate ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 df_train = pd.read_csv(training_dataset_path) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 return _read(filepath_or_buffer, kwds) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 413, in _read ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 filepath_or_buffer, encoding, compression) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 223, in get_filepath_or_buffer ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 mode=mode) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/pandas/io/gcs.py", line 15, in get_filepath_or_buffer ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 filepath_or_buffer = fs.open(filepath_or_buffer, mode) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1043, in open ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 **kwargs, ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1208, in _open ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 **kwargs, ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1325, in __init__ ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 **kwargs, ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1385, in __init__ ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 self.size = self.details["size"] ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1398, in details ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 self._details = self.fs.info(self.path) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 86, in wrapper ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 return sync(self.loop, func, *args, **kwargs) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 raise return_result ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 result[0] = await coro ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 724, in _info ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 out = await self._ls(path, **kwargs) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 760, in _ls ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 out = await self._list_objects(path, prefix=prefix) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 504, in _list_objects ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 return [await self._get_object(path)] ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 458, in _get_object ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 393, in _call ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 method, path, *args, **kwargs ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/decorator.py", line 221, in fun ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 return await caller(func, *(extras + args), **kw) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 return await func(*args, **kwargs) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 384, in _request ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 validate_response(status, contents, path, args) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 84, in validate_response ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 raise FileNotFoundError(path) ERROR 2022-07-25 17:52:22 +0000 master-replica-0 3 FileNotFoundError: b/qwiklabs-gcp-01-ec0ff30e09da-kubeflowpipelines-default/o/%2Fdata%2Ftraining%2Fdataset.csv ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 Traceback (most recent call last): ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "train.py", line 88, in <module> ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 fire.Fire(train_evaluate) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 141, in Fire ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 component_trace = _Fire(component, args, parsed_flag_args, context, name) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 target=component.__name__) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 component = fn(*varargs, **kwargs) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "train.py", line 35, in train_evaluate ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 df_train = pd.read_csv(training_dataset_path) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 return _read(filepath_or_buffer, kwds) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 413, in _read ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 filepath_or_buffer, encoding, compression) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 223, in get_filepath_or_buffer ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 mode=mode) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/pandas/io/gcs.py", line 15, in get_filepath_or_buffer ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 filepath_or_buffer = fs.open(filepath_or_buffer, mode) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1043, in open ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 **kwargs, ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1208, in _open ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 **kwargs, ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 1325, in __init__ ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 **kwargs, ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1385, in __init__ ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 self.size = self.details["size"] ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/spec.py", line 1398, in details ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 self._details = self.fs.info(self.path) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 86, in wrapper ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 return sync(self.loop, func, *args, **kwargs) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 raise return_result ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 result[0] = await coro ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 724, in _info ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 out = await self._ls(path, **kwargs) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 760, in _ls ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 out = await self._list_objects(path, prefix=prefix) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 504, in _list_objects ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 return [await self._get_object(path)] ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 458, in _get_object ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 393, in _call ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 method, path, *args, **kwargs ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/decorator.py", line 221, in fun ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 return await caller(func, *(extras + args), **kw) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 return await func(*args, **kwargs) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 384, in _request ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 validate_response(status, contents, path, args) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 File "/opt/conda/lib/python3.7/site-packages/gcsfs/retry.py", line 84, in validate_response ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 raise FileNotFoundError(path) ERROR 2022-07-25 17:52:25 +0000 master-replica-0 4 FileNotFoundError: b/qwiklabs-gcp-01-ec0ff30e09da-kubeflowpipelines-default/o/%2Fdata%2Ftraining%2Fdataset.csv ERROR 2022-07-25 17:52:29 +0000 service 2 The replica master 0 exited with a non-zero status of 1. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=823357052520&resource=ml_job%2Fjob_id%2FJOB_20220725_175025&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22JOB_20220725_175025%22 ERROR 2022-07-25 17:52:31 +0000 service 3 The replica master 0 exited with a non-zero status of 1. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=823357052520&resource=ml_job%2Fjob_id%2FJOB_20220725_175025&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22JOB_20220725_175025%22 ERROR 2022-07-25 17:52:31 +0000 service 1 The replica master 0 exited with a non-zero status of 1. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=823357052520&resource=ml_job%2Fjob_id%2FJOB_20220725_175025&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22JOB_20220725_175025%22 ERROR 2022-07-25 17:53:01 +0000 service 4 The replica master 0 exited with a non-zero status of 1. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=823357052520&resource=ml_job%2Fjob_id%2FJOB_20220725_175025&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22JOB_20220725_175025%22

Everton L. · Reviewed over 3 years ago

Harsh G. · Reviewed over 3 years ago

PRATIK G. · Reviewed over 3 years ago

Luis G. · Reviewed over 3 years ago

Jeffrey G. · Reviewed over 3 years ago

Steve S. · Reviewed over 3 years ago

Vincenzo P. · Reviewed over 3 years ago

Swapna S. · Reviewed over 3 years ago

Leonard L. · Reviewed over 3 years ago

Corneel V. · Reviewed over 3 years ago

Niklas M. · Reviewed over 3 years ago

Very interesting, come additional time would be appreciated. Thank you.

Miguel A. · Reviewed over 3 years ago

Nice

Ajay k. · Reviewed over 3 years ago