Month: June 2023

Siloed data has long been a challenge in architecture, engineering, and construction (AEC), hindering productivity and collaboration. However, new innovative…

Siloed data has long been a challenge in architecture, engineering, and construction (AEC), hindering productivity and collaboration. However, new innovative solutions are transforming the way that architects, engineers, and construction managers work together on BIM (building information management) workflows, offering new possibilities for real-time collaboration.

The new NVIDIA Omniverse Connector from Vectorworks exemplifies this potential, opening up exciting new workflow options. Vectorworks creates design software that serves the architecture, landscape, and entertainment industries. They specialize in hybrid 2D and 3D workflow solutions with an emphasis on visualization and non-proprietary collaboration.

Universal Scene Description (OpenUSD) helps Vectorworks provide their customers with even more flexibility in the design process and the ability to collaborate freely with anyone involved in a project. The connector helps optimize BIM workflows and provides real-time collaboration at every design phase. USD is an open and extensible framework and ecosystem for describing, composing, simulating, and collaborating within 3D worlds. It is the foundation of NVIDIA Omniverse.

What is building information modeling?

At its core, Vectorworks is building information modeling (BIM) software. BIM workflows nowadays are the defining force in the AEC industry, and they’re growing in popularity in landscape architecture, too.

BIM is a collaborative digital process that integrates design, construction, and operation information into a single model, enabling stakeholders to visualize, analyze, and coordinate all aspects of a building project. To get a short history of BIM and learn about its various use cases in AEC, see What is BIM | Building Information Modeling in the AEC Industry.

Although collaboration is essential to data-driven BIM workflows, there is currently a lack of software options for design collaboration and coordination.

Real-time BIM collaboration at every design phase

There are four standard phases involved in the design of a building.

First, architects must pull together all the necessary information to start their project for conceptual design, site planning, and analysis. With the Omniverse Connector, teams can pull this data from a wide variety of file formats and sources with Universal Scene Description (OpenUSD).

Next, teams explore schematic designs to refine initial design concepts and ideas. Intuitive drawing tools and a flexible modeling engine are easy to use with the connector, enabling you to explore different design options and evaluate their feasibility. You can easily transition from massing models to a BIM model, visualize concepts with integrated 3D rendering, and share the results in real-time with their entire team.

When moving into more detailed design phases, BIM tools can often become less creative in nature. The Vectorworks Omniverse Connector is a bit different. It enables you to freely sketch, model, and document your design ideas with precision-drafting capabilities so that you’re not limited by presets and strict parameters.

In the last phase of construction documentation, teams must coordinate with consultants and continue to verify and refine models to make sure that the models are ready for the real world. Omniverse makes this process seamless, enabling different users to collaborate in real time on models and have the models automatically updated in the construction documentation.

Almost any stakeholder in the design process can now collaborate in real time on design efforts with the new connector. This could be a building architect working with a landscape architect or a lighting designer collaborating with a set designer on a live event.

Combining models into Omniverse enables you to collaborate in a real-time, virtual environment in all project phases, enabling you to ensure that you’re developing cohesive design solutions.

Developing the connector in Omniverse

Implementing the Omniverse Connector was a straightforward and streamlined process. The Vectorworks developers followed the comprehensive NVIDIA documentation and referred to existing connectors for user interface and user experience design. They also asked questions in the Omniverse forums where answers provided directly from NVIDIA engineers and managers helped deliver the connector on schedule with the Vectorworks 2023 Service Pack 4 release.

Key to the development process was the Omniverse Connect HelloWorld sample, documentation that demonstrates how to build your own NVIDIA Omniverse Connector, along with a range of functionalities that enable you to interact with USD and the Omniverse Nucleus server.

The sample documentation made it easy to quickly start reaping the benefits of OpenUSD. It shows how to save projects to USD format so you can take advantage of creating and editing live layers. Then, you can create a USD stage, which serves as a container for organizing and manipulating the Vectorworks 3D scene data. This stage acts as a foundation for the subsequent operations.

Seamless workflow management is critical to the experience Vectorworks creates for users. The sample demonstrates how to create Omniverse Nucleus checkpoints, which serve as saved states of the stage. These checkpoints provide the ability to revert to a previous stage configuration, facilitating experimentation and version control.

The sample also demonstrates how to enhance communication and collaboration for users by implementing the capability to send and receive messages over a channel on the Nucleus server. This feature fosters interaction and coordination among users within a shared environment.

“Ultimately, the ease with which we were able to implement the Omniverse Connector speaks to how attainable the entry point is and enabled us to easily leverage Omniverse APIs that enhance workflows for our users,” said Dave Donley, senior director of rendering and research at Vectorworks.

“The collaborative, multi-disciplinary nature of Omniverse is a great fit, especially with OpenUSD at its foundation and we can’t wait to see the great designs our customers will produce with it.”

Maintaining a competitive advantage with next-gen tech

Staying up-to-date with technology trends is crucial for maintaining a competitive advantage and identifying new opportunities for innovation and growth in your field.

By providing customers with powerful solutions that have the potential to revolutionize workflows like the NVIDIA Omniverse Connector, Vectorworks is making good on the promise of their public development roadmap: to constantly evolve their technology in line with the needs of customers during this time of accelerated digital transformation.

“At Vectorworks, we continue to evolve our digital solutions to empower customers to create and share great designs. With the latest update, which delivers a direct connection to NVIDIA Omniverse, comes yet another reminder of our passion and commitment to serve customers as a design partner—embracing the power and possibilities afforded by next-generation technology,” said Steve Johnson, chief technology officer at Vectorworks.

Not only does the NVIDIA Omniverse Connecter provide Vectorworks users with access to a powerful real-time visualization tool, but it also paves the way for exciting developments in the future.

As the design industries continue to mature, use cases for OpenUSD will continue to present themselves. Vectorworks’ compatibility with USD puts them in an advantageous position to be able to implement future technology. By embracing Omniverse’s immense potential, Vectorworks and its users are poised to lead the charge toward a more collaborative and innovative future in AEC.

To start taking advantage of this powerful connection, see the NVIDIA Omniverse Connector page at Vectorworks.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. If you’re a developer, get started building your first extension or developing a Connector with Omniverse resources. Stay up-to-date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. For resources, check out our forums, Discord server, Twitch, and YouTube channels.

Whether animating fish fins or fashioning chic outfits for digital characters, creators can tap Marvelous Designer software to compose and tailor assets, clothes and other materials for their 3D workflows.

Amir Anbarestani, an accomplished 3D artist who goes by the moniker Kingsletter, had a “shell of a good time” creating his Space Turtle scene this week In the NVIDIA Studio.

NVIDIA DLSS Frame Generation is the new performance multiplier in DLSS 3 that uses AI to create entirely new frames. This breakthrough has made real-time path…

NVIDIA DLSS Frame Generation is the new performance multiplier in DLSS 3 that uses AI to create entirely new frames. This breakthrough has made real-time path tracing—the next frontier in video game graphics—possible.

NVIDIA has made it easier for you to take full advantage of this technology with the release of the Unreal Engine 5.2 Plugin and Streamline 2.1 SDK.

Unreal Engine developers can get started now. Coupled with the NVIDIA Reflex low-latency technology available through Unreal Engine 5, they have all the tools to boost game performance while providing a highly responsive experience for players.

If you’re looking to do an integration within your own custom engine, Streamline 2.1 greatly simplifies the manual API hooking for all necessary components needed for DLSS 3. Streamline is an open-source cross-IHV framework that simplifies the integration of features like DLSS 3.

Instead of manually integrating the DLSS Frame Generation libraries, you identify which resources (motion vectors, depth, and so on) are required for the desired plug-in and then trigger when to execute the plug-ins in the rendering pipeline. Here are the necessary steps to ensure that your integrations take full advantage of DLSS 3:

1. Integrate the Streamline 2.1 SDK: To add Streamline to your application, follow the Streamline Manual Hooking guide. Integrate without any features and focus on tasks such as manual hooking and resource state tracking.
2. Perform a security check: Verify the NVIDIA and Streamline dual signatures on sl.itnerposer.dll before loading the DLL. Follow the verification process within the Security section of the programming guide.
3. Check for system support: The DLSS 3 components (Super Resolution, Frame Generation, and NVIDIA Reflex) all have varied system requirements. Check for hardware and software system support and show appropriate error messages based on reported support.
4. Integrate DLSS Super Resolution through Streamline: Pass in the necessary input resources and set up the upscaling pipeline. Follow these integration steps before all other post-processing.
5. Evaluate integration: Validate and confirm image quality and performance benefits from DLSS Super Resolution.
6. Integrate NVIDIA Reflex through Streamline: Add Reflex and its sub-features to the rendering pipeline. Make sure to place Reflex markers in the appropriate location or where your application should sleep.
7. Confirm system latency reduction: There are three primary ways to check that input latency was reduced:
   • NVIDIA FrameView SDK
   • GeForce Experience in-game overlay
   • The Reflex latency analyzer
8. Integrate DLSS Frame Generation through Streamline: Follow these integration steps and pass in the appropriate constants, camera matrices, and input resources in your post-processing pipeline. Pass in all the input resources marked for DLSS Super Resolution (for example, hudless and UIColor Color with Alpha). Disable DLSS Frame Generation when appropriate, such as when in-menu or for scene transitions.
9. Validate DLSS Frame Generation inputs: Use the sl.imgui plugin to validate inputs (camera matrices, depth, MVEC, color, and so on). We recommend using ICAT to validate image quality and FrameView to validate latency. Lastly, buffer visualization using the development DLLs.
10. Swap to production DLLs: After image quality and performance benefits from DLSS Frame Generation are validated, replace the watermarked DLLs with non-watermarked, production-ready DLLs from NVIDIA.

For an integration checklist and the most asked questions for DLSS Super Resolution, Frame Generation, and NVIDIA Reflex, see Streamline Getting Started (registration required). To learn more about the new DLSS plugin in Unreal Engine 5, see the Unreal Engine page.

Game developers can find additional free resources to re-create fully path-traced and AI-driven virtual worlds on the NVIDIA Game Development page.

NVIDIA DLSS 3 is a neural graphics technology that multiplies performance using AI image reconstruction and frame generation. It’s a combination of three core…

NVIDIA DLSS 3 is a neural graphics technology that multiplies performance using AI image reconstruction and frame generation. It’s a combination of three core innovations:

• Super Resolution uses deep learning algorithms to upscale a lower-resolution input into a higher-resolution output, creating a sharp image with a boosted frame rate.
• Frame Generation uses AI rendering to generate entirely new frames with best-in-class quality and responsiveness.
• NVIDIA Reflex is a low-latency technology that minimizes input lag by synchronizing the CPU and the GPU for optimal responsiveness.

Powered by these three technologies, DLSS 3 enables upwards of 4x performance boosts, providing headroom for next-generation, path-traced rendering. 

DLSS Super Resolution has been available in Unreal Engine since 2021, making it easy to integrate NVIDIA AI scaling technology into Unreal Engine projects. NVIDIA has now released DLSS 3 for Unreal Engine 5.2, which includes Frame Generation and the latest NVIDIA Reflex version. For more information about Unreal Engine 5.1 and earlier, see step 2 in the installation guide later in this post.

To make integrating NVIDIA technology into your project as simple as possible, the new DLSS 3 Unreal Engine 5.2 package contains the Frame Generation, Super Resolution, and NVIDIA Reflex plugins all in a single download.

DLSS 3 technologies

The DLSS Frame Generation plugin uses Frame Generation to create entirely new frames by analyzing sequential frames and motion data from the Optical Flow Accelerator in GeForce RTX 40 Series GPUs.

Bundled inside the DLSS Frame Generation plugin is NVIDIA Reflex. Paired with DLSS 3, NVIDIA Reflex reduces onscreen latency by up to 2x compared to native rendering.

The DLSS Super Resolution plugin supports a variety of image quality modes—from Ultra Performance to Quality—determined by the native resolution relative to the DLSS output resolution. DLSS Super Resolution is customizable based on the needs of your game, with additional NVIDIA technologies included in the plugin:

• Deep Learning Anti-Aliasing Mode (DLAA) offers an AI-based anti-aliasing mode for users who have spare GPU headroom and want higher levels of image quality.
• NVIDIA Image Scaling is an open-source spatial upscaler and sharpening algorithm that is available for all platforms. 

The DLSS 3 Unreal Engine 5.2 plugin is delivered with the latest optimizations to NVIDIA AI algorithms, always learning and evolving with over-the-air updates.

How to install DLSS 3 for Unreal Engine

Follow these steps to download and install DLSS 3 for your Unreal Engine project.

1. Agree to the Terms of the License Agreement and download DLSS 3 for your version of Unreal Engine.
2. Unzip the DLSS folder. Only the 5.2 version of DLSS contains the Streamline/Frame Generation plugin.
3. Copy the plugin folders to install to the /Engine/Plugins/MarketPlace folder of your Unreal Engine directory. If you don’t currently have a /MarketPlace folder, create one.
4. Launch Unreal Editor, go to Plugins, and search for the plugins to activate. Search for “NVIDIA” to quickly list all of the included DLSS 3 plugins.
5. Activate and restart Unreal Editor.
6. Load the DLSS 3 Test project from the /Samples folder of the downloaded DLSS plugin file.

For prior versions of Unreal, you must build from the source and modify your source code with a small patch. For more information, see the included DLSS Frame Generation Quick Start Guide PDF in the download .zip file.

Tips for using DLSS 3 in Unreal Engine

After DLSS 3 is installed, follow these steps to verify that the Frame Generation, Super Resolution, and Reflex plugins are integrated into your project correctly.

• To confirm that DLSS Frame Generation is working, along with real-time statistics, navigate to project settings, and then to your preferences for the NVIDIA Streamline plugin. Toggle the Load Debug Overlay option.
  • The Load Debug Overlay option for Frame Generation works in the editor and can appear in development or debug builds, but won’t appear in production builds.
• To update Streamline automatically as well as DLSS AI algorithms with the latest improvements, use the same settings window to ensure that the Allow OTA Update option is enabled.
• In the Unreal Editor, Frame Generation only works from a new editor window (PIE) or in Standalone mode. It doesn’t work from the selected viewport or while editing.
• If any of the included DLSS 3 technologies aren’t working, check the output log or look for onscreen warning messages. A common issue may be that the NVIDIA drivers may have to be updated, for example.
• The DLSS 3 Unreal Engine plugin contains the latest NVIDIA Reflex technology, a newer version than the version currently built into Unreal Engine. While it’s possible to keep the earlier plugin enabled, and even use the earlier NVIDIA Reflex Blueprint scripts, we recommended that you disable the earlier NVIDIA Reflex plugin and use the new version bundled in DLSS 3 Streamline instead.
• We recommend that you set up all NVIDIA plugins through Blueprint scripts, as this enables you to conveniently activate plugins from menus and set preferences for users. However, if you need access to the console commands, they can be found under r.ngx. For more information about using console commands, see the DLSS Quick Start Guide PDF included in the DLSS 3 plugin download.
• When Frame Generation is on, we recommend that you disable VSYNC in your application. The DLSS 3 plugin can set VSYNC to behave incorrectly when active. VSYNC can be disabled with the r.vsync 0 console command.

Download DLSS 3 for Unreal Engine

DLSS 3 for Unreal Engine makes the latest NVIDIA advancements in neural rendering and performance multiplication easy to integrate into your UE project. Get started with the Frame Generation, Super Resolution, and Reflex plugins now.

DLSS 3 for Unreal Engine 5.2 is now available.

For more information, see NVIDIA technologies supported by Unreal Engine 5.

Learn how financial firms can build automated, real-time fraud and threat detection solutions with NVIDIA Morpheus.

Learn how financial firms can build automated, real-time fraud and threat detection solutions with NVIDIA Morpheus.

Generative AI will “supercharge” creators across industries and content types, NVIDIA founder and CEO Jensen Huang said today at the Cannes Lions Festival, on the French Riviera. “For the very first time, the creative process can be amplified in content generation, and the content generation could be in any modality — it could be be Read article >

The analysis of 3D medical images is crucial for advancing clinical responses, disease tracking, and overall patient survival. Deep learning models form the…

The analysis of 3D medical images is crucial for advancing clinical responses, disease tracking, and overall patient survival. Deep learning models form the backbone of modern 3D medical representation learning, enabling precise spatial context measurements that are essential for clinical decision-making. These 3D representations are highly sensitive to the physiological properties of medical imaging data, such as CT or MRI scans.

Medical image segmentation, a key visual task for medical applications, serves as a quantitative tool for measuring various aspects of medical images. To improve the analysis of these images, the development and application of foundation models are becoming increasingly important in the field of medical image analysis.

What are foundation models?

Foundation models, the latest generation of AI neural networks, are trained on extensive, diverse datasets and can be employed for a wide range of tasks or targets.

As large language models demonstrate their capability to tackle generic tasks, visual foundation models are emerging to address various problems, including classification, detection, and segmentation.

Foundation models can be used as powerful AI neural networks for segmenting different targets in medical images. It opens up a world of possibilities for medical imaging applications, enhancing the effectiveness of segmentation tasks and enabling more accurate measurements.

Challenges in medical image analysis

The application of medical foundation models in medical image analysis poses significant challenges. Unlike general computer vision models, medical image applications typically demand high-level domain knowledge.

Institutes have traditionally created fully annotated datasets for specific targets like spleens or tumors, relying solely on the association between input data features and target labels. Addressing multiple targets is more difficult, as manual annotations are laborious and time-consuming. Training larger or multi-task models is also increasingly challenging.

Despite recent advancements, there is still a long-standing issue in comprehending large medical imaging data due to its heterogeneity:

• Medical volumetric data is often extremely high-resolution, necessitating substantial computational resources.
• Current deep learning models have yet to effectively capture anatomical variability.
• The large-scale nature of medical imaging data makes learning robust and efficient 3D representations difficult, particularly when dealing with heterogeneous data.

However, the modern analysis of high-resolution, high-dimensional, and large-scale medical volumetric data presents an opportunity to accelerate discoveries and obtain innovative insights into human body functions, behavior, and disease.

Foundation models offer the capability to address the heterogeneous variations that complicate the rectification of inter– and intra-subject differences. AI has the potential to revolutionize medical imaging by enabling more accurate and efficient analysis of large-scale, complex data.

A platform for medical visual segmentation foundation models

MONAI Model Zoo serves as a platform for hosting medical visual foundation models. It contains a collection of pretrained models for medical imaging tasks developed using the Medical Open Network for AI (MONAI) framework.

The MONAI Model Zoo is a publicly available resource that provides access to a variety of pretrained models for different medical imaging tasks, such as segmentation, classification, registration, and synthesis. These pretrained models can be used as starting points or foundation models for training on new datasets or fine-tuning for specific applications.

The MONAI Model Zoo is designed to accelerate the development of new medical imaging applications and enable researchers and clinicians to leverage pre-existing models and build on top of them.

Whole-body CT segmentation

Segmenting the entirety of a whole-body CT scan from a single model is a daunting task. However, the MONAI team has risen to the challenge. They’ve developed models that segment all 104 anatomical structures from a single model:

• 27 organs
• 59 bones
• 10 muscles
• 8 vessels

Using the dataset released by the totalSegmentator team, MONAI conducted research and benchmarking to achieve fast inference times. For a high-resolution 1.5 mm model, the inference time using a single NVIDIA V100 GPU for all 104 structures is just 4.12 seconds, while the inference time using a CPU is 30.30 seconds. This is a significant improvement from the original paper’s reported inference time for a single CT scan, which took more than 1 minute.

To access the MONAI Whole Body CT Segmentation foundation model, see the MONAI Model Zoo.

For more information about the overview of all anatomical structures in whole-body CT scans, see the TotalSegmentator: robust segmentation of 104 anatomical structures in CT images whitepaper.

(Source: TotalSegmentator: robust segmentation of 104 anatomical structures in CT images)

Whole-brain MRI segmentation

Whole-brain segmentation is a critical technique in medical image analysis, providing a non-invasive means of measuring brain regions from clinical structural magnetic resonance imaging (MRI). However, with over 130 substructures in human brains, segmenting anything in the brain is a difficult challenge for MRI 3D segmentation. Unfortunately, detailed annotations of the brain are scarce, making this task even more challenging for the medical imaging community.

To address this issue, the MONAI team collaborated with Vanderbilt University to develop a deep learning model that can simultaneously segment all 133 brain structures. Using 3D Slicer, the MONAI model can infer the entire brain in just 2.0 seconds. The MONAI whole brain MRI segmentation model represents a promising development in medical imaging research, offering a valuable resource for improving the accuracy of brain measurements in clinical settings.

Visit the MONAI Model Zoo to access the MONAI Whole Brain MRI Segmentation Foundation Model.

How to access medical imaging foundation models

The use of foundation models in medical image analysis has great potential to improve diagnostic accuracy and enhance patient care. However, it’s important to recognize that medical application requires strong domain knowledge.

With the ability to process large amounts of data and identify subtle patterns and anomalies, foundation models have proven to be valuable tools in the medical image analysis field. The development and refinement of these models is ongoing, with researchers and practitioners working to improve their accuracy and expand their capabilities.

Although challenges such as patient privacy and potential biases must be addressed, the use of foundation models has already demonstrated significant benefits. It is expected to play a more prominent role in healthcare in the future.

As researchers, clinicians, and users continue to focus on foundation models, the MONAI Model Zoo, a platform hosting pretrained medical image models, is amplifying its impact. Fine-tuning pretrained models is crucial to the future of medical image analysis.

The MONAI Model Zoo provides access to a diverse collection of pretrained models for various medical imaging tasks, including segmentation, classification, registration, and synthesis. By using these pre-existing models as starting points, researchers and clinicians can accelerate the development of new medical imaging applications, saving time and resources.

Join us in driving innovation and collaboration in medical imaging research by exploring the MONAI Model Zoo today.