Categories
Misc

No dashboards are active for the current data set

I am trying detecting objects using tensorflow and Google colab. The steps is given in the link below: https://medium.com/swlh/tensorflow-2-object-detection-api-with-google-colab-b2af171e81cc

When I came to the step starting tensorboard, I’m facing :

 No dashboards are active for the current data set. 

After two steps, training the model, now I’m facing a lot warnings and eventually an error:

Traceback (most recent call last): File "model_main_tf2.py", line 113, in <module> tf.compat.v1.app.run() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 303, in run _run_main(main, args) File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "model_main_tf2.py", line 110, in main record_summaries=FLAGS.record_summaries) File "/usr/local/lib/python3.7/dist-packages/object_detection-0.1-py3.7.egg/object_detection/model_lib_v2.py", line 639, in train_loop loss = _dist_train_step(train_input_iter) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 888, in _call return self._stateless_fn(*args, **kwds) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__ filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 560, in call ctx=ctx) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[100,51150] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Loss/Compare_9/IOU/Intersection/Minimum_1 (defined at /local/lib/python3.7/dist-packages/object_detection-0.1-py3.7.egg/object_detection/core/box_list_ops.py:257) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[Func/Loss/localization_loss_1/write_summary/summary_cond/then/_0/input/_71/_348]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. (1) Resource exhausted: OOM when allocating tensor with shape[100,51150] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Loss/Compare_9/IOU/Intersection/Minimum_1 (defined at /local/lib/python3.7/dist-packages/object_detection-0.1-py3.7.egg/object_detection/core/box_list_ops.py:257) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. 0 successful operations. 0 derived errors ignored. [Op:__inference__dist_train_step_51563] Errors may have originated from an input operation. Input Source operations connected to node Loss/Compare_9/IOU/Intersection/Minimum_1: Loss/Compare_9/IOU/Intersection/split (defined at /local/lib/python3.7/dist-packages/object_detection-0.1-py3.7.egg/object_detection/core/box_list_ops.py:250) Input Source operations connected to node Loss/Compare_9/IOU/Intersection/Minimum_1: Loss/Compare_9/IOU/Intersection/split (defined at /local/lib/python3.7/dist-packages/object_detection-0.1-py3.7.egg/object_detection/core/box_list_ops.py:250) Function call stack: _dist_train_step -> _dist_train_step 

How can I handle these issues? And tensorboard plays important role? Or it activating tensorboard plays an important part or it just is something like optional?

Thx in advance for your answers.

submitted by /u/kursat44
[visit reddit] [comments]

Leave a Reply

Your email address will not be published. Required fields are marked *