Month: December 2020

Categories
Misc

Model training stalls forever after just a few batches.

I posted
this as an issue on Github, maybe someone here will have a
magic solution:

  • TensorFlow version: 2.4.0-rc4 (also tried with stable
    2.4.0)
  • TensorFlow Git version: v2.4.0-rc3-20-g97c3fef64ba
  • Python version: 3.8.5
  • CUDA/cuDNN version: CUDA 11.0, cuDNN 8.0.4
  • GPU model and memory: Nvidia RTX 3090, 24GB RAM

Model training regularly freezes for large models.

Sometimes the first batch or so works, but then just a few
batches later and training seems stuck in a loop. From my activity
monitor, I see GPU CUDA use hovering around 100%. This goes on for
minutes or more, with no more batches being trained.

I don’t see an OOM error, nor does it seem like I’m hitting
memory limits in activity monitor or nvidia-smi.

I would expect the first batch to take a bit longer, then any
subsequent batches to take less than <1s. Never have a random
batch take minutes or stall forever.

Run through all the cells in the notebook shared below to
initialize the model, then run the final cell just a few times.
Eventually it will hang and never finish.


https://github.com/not-Ian/tensorflow-bug-example/blob/main/tensorflow%20error%20example.ipynb

Smaller models train quickly as expected, however I think even
then they eventually stall out after training many, many batches. I
had another similar, small VAE like in my example that trained for
5k-10k batches overnight before stalling.

Someone suggested I set a hard memory limit on the GPU like
this:

gpus = tf.config.experimental.list_physical_devices('GPU') tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024 * 23)])

And finally, I’ve tried using the hacky ptxas.exe file from CUDA
11.1 in my CUDA 11.0 installation. This seems to remove a warning?
But still no change.

Open to any other ideas, thanks.

submitted by /u/Deinos_Mousike

[visit reddit]
[comments]

Categories
Misc

newbie here^^ ;trying to build tensorflow to old gpu

i’ve geforce 840m. it is cuda 5.0.My project has dependencies as
tensorflow ,opencv ,cuda 7.5+ and cudnn 5.0+.(https://github.com/dvschultz/neural-style-tf)

i keep getting this error

“W tensorflow/stream_executor/platform/default/dso_loader.cc:59]
Could not load dynamic library ‘cudart64_101.dll’; dlerror:
cudart64_101.dll not found”

tensorflow doesnt see my gpu.

1-is it because i’ve higher cuda version than my gpu?

2-is it because my tensorflow version 2.3.1 ?

thx.

submitted by /u/elyakubu

[visit reddit]
[comments]

Categories
Misc

Inception to the Rule: AI Startups Thrive Amid Tough 2020

2020 served up a global pandemic that roiled the economy. Yet the startup ecosystem has managed to thrive and even flourish amid the tumult. That may be no coincidence. Crisis breeds opportunity. And nowhere has that been more prevalent than with startups using AI, machine learning and data science to address a worldwide medical emergency Read article >

The post Inception to the Rule: AI Startups Thrive Amid Tough 2020 appeared first on The Official NVIDIA Blog.

Categories
Misc

Shifting Paradigms, Not Gears: How the Auto Industry Will Solve the Robotaxi Problem

A giant toaster with windows. That’s the image for many when they hear the term “robotaxi.” But there’s much more to these futuristic, driverless vehicles than meets the eye. They could be, in fact, the next generation of transportation. Automakers, suppliers and startups have been dedicated to developing fully autonomous vehicles for the past decade, Read article >

The post Shifting Paradigms, Not Gears: How the Auto Industry Will Solve the Robotaxi Problem appeared first on The Official NVIDIA Blog.

Categories
Misc

Role of the New Machine: Amid Shutdown, NVIDIA’s Selene Supercomputer Busier Than Ever

And you think you’ve mastered social distancing. Selene is at the center of some of NVIDIA’s most ambitious technology efforts. Selene sends thousands of messages a day to colleagues on Slack. Selene’s wired into GitLab, a key industry tool for tracking the deployment of code, providing instant updates to colleagues on how their projects are Read article >

The post Role of the New Machine: Amid Shutdown, NVIDIA’s Selene Supercomputer Busier Than Ever appeared first on The Official NVIDIA Blog.

Categories
Misc

AI at Your Fingertips: NVIDIA Launches Storefront in AWS Marketplace

AI is transforming businesses across every industry, but like any journey, the first steps can be the most important. To help enterprises get a running start, we’re collaborating with Amazon Web Services to bring 21 NVIDIA NGC software resources directly to the AWS Marketplace. The AWS Marketplace is where customers find, buy and immediately start Read article >

The post AI at Your Fingertips: NVIDIA Launches Storefront in AWS Marketplace appeared first on The Official NVIDIA Blog.

Categories
Misc

How can I train a model on a HUGE dataset?

So I have a huge dataset that devours my 32GB memory and then
crashes every time before I can even begin training. Is it possible
to break the dataset into chunks and train my model that way?

I’m fairly new to tensorflow so I’m not sure how to go about it.
Can anyone help?

Thank you.

EDIT: the data is time series data (from a csv) that I’m loading
into a pandas dataframe. From there, the data is being broken up
into samples with a 10 step window. I have about 90M samples with
the shape (90M, 10, 1) that should then be fed into the LSTM. The
problem is that the samples crash the RAM and I have to start all
over again each time.

submitted by /u/dsm88

[visit reddit]
[comments]

Categories
Misc

Optimizing System Latency with NVIDIA Reflex SDK – Available Now

Measuring and optimizing system latency is one of the hardest challenges during game development and the NVIDIA Reflex SDK helps developers solve that issue.

Measuring and optimizing system latency is one of the hardest challenges during game development and the NVIDIA Reflex SDK helps developers solve that issue. NVIDIA Reflex is an easy to integrate SDK that provides API to both measure and reduce system latency – giving players a more responsive experience.  Epic, Bungie, Respawn, Activision Blizzard, and Riot have integrated the NVIDIA Reflex Low Latency mode  into their titles, giving gamers a responsive experience without  dips in resolution or framerate. 

The NVIDIA Reflex SDK offers developers:

  • Low Latency Mode – Aligns game engine work to complete just-in-time for rendering, eliminating the GPU render queue and reducing CPU back pressure in GPU-bound scenarios, thus reducing latency in GPU bound scenarios.
  • Latency Markers – Real time latency metrics broken down by game pipeline stage: Input, Simulation, Render Submission, Graphics Driver, Render Queue, and GPU Render.  Great for debugging and for real time in-game overlays. 
  • Flash Indicator – Using the marker system, the flash indicator marker draws a small white rectangle on the screen each click.  This is helpful when automating the use of a tool like the NVIDIA Reflex Latency Analyzer to measure latency. 

The NVIDIA Reflex SDK is a low latency suite of esports technologies designed to measure, analyze and reduce input latency. The SDK has been built to support custom engines as well as popular game engines such as UE4 and Unity.

Get Started with NVIDIA Reflex >

Categories
Misc

NVIDIA Announces Upcoming Events for Financial Community

SANTA CLARA, Calif., Dec. 17, 2020 (GLOBE NEWSWIRE) — NVIDIA will present at the following events for the financial community: J.P. Morgan Healthcare ConferenceMonday, Jan. 11, at 1:30 p.m. …

Categories
Misc

Updates to NVIDIA’s Unreal Engine 4 Branch, DLSS, and RTXGI Available Now

NVIDIA has released updates to DLSS, NVIDIA’s Unreal Engine 4 Branch, and RTXGl.

To help developers get the most out of Unreal Engine 4 as they head into the new year, NVIDIA RTX UE4.26 has just been released. We have also released the first DLSS plugin that can be used with both NVIDIA’s NvRTX branch and mainline UE4, along with an updated UE4 Plugin for RTX Global Illumination.

NVIDIA RTX UE4.26 

The new NVIDIA UE4.26 Branch offers all of the benefits of mainline UE4.26, while providing some additional features: 

  • Faster ray tracing
    • NVRTX includes a number of improvements to ray tracing performance. Some of these are tunable, some are automatic. 
  • New tools
    • New debugging tools like the BVH viewer and Ray Timing Visualization allows developers to get a handle on ray tracing cost in their scene and get it tuned for speed.
  • Hybrid Translucency
    • Another way to do ray traced translucency, with greater compatibility, speed and rendering options.  
  • World position offset simulation for ray traced instanced static meshes (beta)
    • Allows ambient motion of foliage like trees and grass.
    • Uses approximate technique of shared animations to reduce overhead for simulating a full forest.
    • Selectable per instance type.
  • Inexact Shadows (beta)
    • Deals with potential mesh mismatches of ray traced and raster geometry.
    • Dithers shadow testing to hide potential artifacts. 
    • Enables approximations that improve performance in the management of ray tracing data.

An updated build of NVIDIA RTX UE4.25 has also been released, which includes all of the new features listed. 

Both branches can be found here. 

NVIDIA DLSS Plugin for UE4

NVIDIA DLSS is a deep learning neural network that boosts frame rates and generates beautiful, sharp images for your games. It delivers the performance headroom needed to maximize ray tracing settings and increase output resolution. It is available for the first time for mainline UE4 (in beta), compatible with UE4.26 Enjoy great scaling across all RTX GPUs and resolutions, and the new ultra performance mode for 8K gaming. 

Request access to the beta for NVIDIA DLSS plugin for UE4 here.

NVIDIA RTXGI Plugin for UE4

Leveraging the power of ray tracing, NVIDIA RTX Global Illumination (RTXGI) provides scalable solutions to compute multi-bounce indirect lighting without bake times, light leaks, or expensive per-frame costs. RTXGI is supported on any DXR-enabled GPU and is an ideal starting point to bring the benefits of ray tracing to your existing tools, knowledge, and capabilities. We have updated our RTXGI UE4 plugin with bugs fixes, image quality improvements, and support for UE4.26.

Request access to the RTXGI plugin for UE4 here.