Patent
US 10,997,464Digital Image Layout Training using Wireframe Rendering within a Generative Adversarial Network (GAN) System
Jimei Yang, Jianming Zhang, Aaron Phillip Hertzmann, Jianan Li
US
Copyright © 2026 Matter42. All rights reserved.
Patent
Atlas literature
Patent
US 10,997,464Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1 is an illustration of a digital medium environment in an example implementation that is operable to employ digital image layout training techniques …
FIG. 2 to generate wireframe prediction data based on the refined training digital image layout using machine lea rn ing to train the generator module. [0013 1
FIG. 3 depicts a system in an example implementation showing operationof a generator module of
FIG. 4 depicts a system in an example implementation showing operation of a wireframe rendering discriminator module of
FIG. 5 depicts an example implementation showing examples of wireframe rendering of graphic elements. [
FIG. 6 is a flow diagram depicting a procedure in an example implementation in which a wireframe digital image layout is generated by rasterizing a refined …
FIG. 7 depicts a system in an example implementation showing operation of a relation-based discriminator module of
FIG. 8 is a flow diagram depicting a procedure in an example implementation of use of a relation-based discriminator module of the GAN system to train a …
FIG. 9 depicts a system in an example implementation in which a generator module trained as part of a GAN network receives an input digital image layout, and …
FIG. 10 is a flow diagram depicting a procedure in an example implementation in which an input digital image layout is used to generate a refined digital image …
FIG. 11 illustrates an example system including various components of an example device that can be implemented as any type of computing device as described …
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
In a digital medium digital image layout environment, a method implemented by at least one computing device, the method comprising: generating, by the at least one computing device, a digital image layout that includes a plurality of graphic elements by a generator module of a generative adversarial network (GAN) system; rasterizing, by the at least one computing device, the digital image layout into a wireframe digital image layout by a wireframe rendering discriminator module of the GAN system; comparing, by the at least one computing device, the wireframe digital image layout with at least one ground truth digital image layout using a loss function as a part of machine learning by the wireframe rendering discriminator module; and training, by the at least one computing device, the generator module by backpropagating a result of the comparing from the wireframe rendering discriminator module based on the loss function to the generator module.
The method as described in claim 1, further comprising: generating, by the at least one computing device, feature data from the digital image layout by a relation-based discriminator module of the generative adversarial network (GAN) system; modeling, by the at least one computing device, semantic and spatial relationships based on the generated feature data by the relation-based discriminator module; and classifying, by the at least one computing device, the digital image layout based on the modeling by the relation-based discriminator module, and wherein the training of the generator module is further based on backpropagating a result of the classifying from the relation-based discriminator module to the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, a training digital image layout by the generator module; and generating, by the at least one computing device, the digital image layout as a refined training digital image layout of the plurality of graphic elements from the training digital image layout by the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, an input digital image layout by the trained generator module; and generating, by the at least one computing device, a refined digital image layout from the input digital image layout by the trained generator module.
The method as described in claim 1, wherein the wireframe rendering discriminator module operates at a vector graphic level for the plurality of graphic elements in the digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a generative adversarial network (GAN) system including: a generator module implemented by at least one computing device to receive a training digital image layout, and from which, generate a refined training digital image layout using machine learning; and a wireframe rendering discriminator module implemented by the at least one computing device, the wireframe rending discriminator module including: a rasterization module to rasterize the refined training digital image layout into a wireframe digital image layout; a layout optimization module to optimize the rasterized digital image layout using a convolutional neural network (CNN) and a loss function and train the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 9, further comprising a relation- based discriminator module implemented by the at least one computing device, the relation-based discriminator module including: an encoder to generate feature data from the refined training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a classifier module to classify the refined digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 9, wherein the generator module includes: an encoder to generate feature data from the training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a decoder module to generate the refined training digital image layout from the training digital image layout based on the modeled semantic and spatial relationships.
The system as described in claim 9, wherein the generator module, once trained, is configured to receive an input digital image layout and generate a refined digital image layout from the input digital image layout.
The system as described in claim 9, wherein the wireframe rendering discriminator module operates at a vector graphic level for graphic elements in the refined training digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a system comprising: means for receiving an input digital image layout including a plurality of graphic elements having respective semantic parameters and geometric parameters; and means for generating a refined digital image layout from the input digital image layout using a generator module as part of machine learning, the generator module trained as part of a generative adversarial network (GAN) using a wireframe rendering discriminator module including: means for rasterizing a training digital image layout into a wireframe digital image layout; means for optimizing the rasterized digital image layout using a convolutional neural network (CNN) and a loss function; and means for training the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 16, wherein the generator module is further trained as part of the generative adversarial network (GAN) using a relation-based discriminator module implemented including: means for generating feature data from the training digital image layout; means for modeling semantic and spatial relationships based on the generated feature data; and means for classifying the training digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 16, wherein the generator module includes: means for generating feature data from the input digital image layout; means for modeling semantic and spatial relationships based on the generated feature data using a convolutional neural network; and means for generating the refined digital image layout based on the modeled semantic and spatial relationships.
Patent
Atlas literature
Patent
US 10,997,464Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1 is an illustration of a digital medium environment in an example implementation that is operable to employ digital image layout training techniques …
FIG. 2 to generate wireframe prediction data based on the refined training digital image layout using machine lea rn ing to train the generator module. [0013 1
FIG. 3 depicts a system in an example implementation showing operationof a generator module of
FIG. 4 depicts a system in an example implementation showing operation of a wireframe rendering discriminator module of
FIG. 5 depicts an example implementation showing examples of wireframe rendering of graphic elements. [
FIG. 6 is a flow diagram depicting a procedure in an example implementation in which a wireframe digital image layout is generated by rasterizing a refined …
FIG. 7 depicts a system in an example implementation showing operation of a relation-based discriminator module of
FIG. 8 is a flow diagram depicting a procedure in an example implementation of use of a relation-based discriminator module of the GAN system to train a …
FIG. 9 depicts a system in an example implementation in which a generator module trained as part of a GAN network receives an input digital image layout, and …
FIG. 10 is a flow diagram depicting a procedure in an example implementation in which an input digital image layout is used to generate a refined digital image …
FIG. 11 illustrates an example system including various components of an example device that can be implemented as any type of computing device as described …
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
In a digital medium digital image layout environment, a method implemented by at least one computing device, the method comprising: generating, by the at least one computing device, a digital image layout that includes a plurality of graphic elements by a generator module of a generative adversarial network (GAN) system; rasterizing, by the at least one computing device, the digital image layout into a wireframe digital image layout by a wireframe rendering discriminator module of the GAN system; comparing, by the at least one computing device, the wireframe digital image layout with at least one ground truth digital image layout using a loss function as a part of machine learning by the wireframe rendering discriminator module; and training, by the at least one computing device, the generator module by backpropagating a result of the comparing from the wireframe rendering discriminator module based on the loss function to the generator module.
The method as described in claim 1, further comprising: generating, by the at least one computing device, feature data from the digital image layout by a relation-based discriminator module of the generative adversarial network (GAN) system; modeling, by the at least one computing device, semantic and spatial relationships based on the generated feature data by the relation-based discriminator module; and classifying, by the at least one computing device, the digital image layout based on the modeling by the relation-based discriminator module, and wherein the training of the generator module is further based on backpropagating a result of the classifying from the relation-based discriminator module to the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, a training digital image layout by the generator module; and generating, by the at least one computing device, the digital image layout as a refined training digital image layout of the plurality of graphic elements from the training digital image layout by the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, an input digital image layout by the trained generator module; and generating, by the at least one computing device, a refined digital image layout from the input digital image layout by the trained generator module.
The method as described in claim 1, wherein the wireframe rendering discriminator module operates at a vector graphic level for the plurality of graphic elements in the digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a generative adversarial network (GAN) system including: a generator module implemented by at least one computing device to receive a training digital image layout, and from which, generate a refined training digital image layout using machine learning; and a wireframe rendering discriminator module implemented by the at least one computing device, the wireframe rending discriminator module including: a rasterization module to rasterize the refined training digital image layout into a wireframe digital image layout; a layout optimization module to optimize the rasterized digital image layout using a convolutional neural network (CNN) and a loss function and train the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 9, further comprising a relation- based discriminator module implemented by the at least one computing device, the relation-based discriminator module including: an encoder to generate feature data from the refined training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a classifier module to classify the refined digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 9, wherein the generator module includes: an encoder to generate feature data from the training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a decoder module to generate the refined training digital image layout from the training digital image layout based on the modeled semantic and spatial relationships.
The system as described in claim 9, wherein the generator module, once trained, is configured to receive an input digital image layout and generate a refined digital image layout from the input digital image layout.
The system as described in claim 9, wherein the wireframe rendering discriminator module operates at a vector graphic level for graphic elements in the refined training digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a system comprising: means for receiving an input digital image layout including a plurality of graphic elements having respective semantic parameters and geometric parameters; and means for generating a refined digital image layout from the input digital image layout using a generator module as part of machine learning, the generator module trained as part of a generative adversarial network (GAN) using a wireframe rendering discriminator module including: means for rasterizing a training digital image layout into a wireframe digital image layout; means for optimizing the rasterized digital image layout using a convolutional neural network (CNN) and a loss function; and means for training the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 16, wherein the generator module is further trained as part of the generative adversarial network (GAN) using a relation-based discriminator module implemented including: means for generating feature data from the training digital image layout; means for modeling semantic and spatial relationships based on the generated feature data; and means for classifying the training digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 16, wherein the generator module includes: means for generating feature data from the input digital image layout; means for modeling semantic and spatial relationships based on the generated feature data using a convolutional neural network; and means for generating the refined digital image layout based on the modeled semantic and spatial relationships.
Patent
Atlas literature
Patent
US 10,997,464Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1 is an illustration of a digital medium environment in an example implementation that is operable to employ digital image layout training techniques …
FIG. 2 to generate wireframe prediction data based on the refined training digital image layout using machine lea rn ing to train the generator module. [0013 1
FIG. 3 depicts a system in an example implementation showing operationof a generator module of
FIG. 4 depicts a system in an example implementation showing operation of a wireframe rendering discriminator module of
FIG. 5 depicts an example implementation showing examples of wireframe rendering of graphic elements. [
FIG. 6 is a flow diagram depicting a procedure in an example implementation in which a wireframe digital image layout is generated by rasterizing a refined …
FIG. 7 depicts a system in an example implementation showing operation of a relation-based discriminator module of
FIG. 8 is a flow diagram depicting a procedure in an example implementation of use of a relation-based discriminator module of the GAN system to train a …
FIG. 9 depicts a system in an example implementation in which a generator module trained as part of a GAN network receives an input digital image layout, and …
FIG. 10 is a flow diagram depicting a procedure in an example implementation in which an input digital image layout is used to generate a refined digital image …
FIG. 11 illustrates an example system including various components of an example device that can be implemented as any type of computing device as described …
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
In a digital medium digital image layout environment, a method implemented by at least one computing device, the method comprising: generating, by the at least one computing device, a digital image layout that includes a plurality of graphic elements by a generator module of a generative adversarial network (GAN) system; rasterizing, by the at least one computing device, the digital image layout into a wireframe digital image layout by a wireframe rendering discriminator module of the GAN system; comparing, by the at least one computing device, the wireframe digital image layout with at least one ground truth digital image layout using a loss function as a part of machine learning by the wireframe rendering discriminator module; and training, by the at least one computing device, the generator module by backpropagating a result of the comparing from the wireframe rendering discriminator module based on the loss function to the generator module.
The method as described in claim 1, further comprising: generating, by the at least one computing device, feature data from the digital image layout by a relation-based discriminator module of the generative adversarial network (GAN) system; modeling, by the at least one computing device, semantic and spatial relationships based on the generated feature data by the relation-based discriminator module; and classifying, by the at least one computing device, the digital image layout based on the modeling by the relation-based discriminator module, and wherein the training of the generator module is further based on backpropagating a result of the classifying from the relation-based discriminator module to the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, a training digital image layout by the generator module; and generating, by the at least one computing device, the digital image layout as a refined training digital image layout of the plurality of graphic elements from the training digital image layout by the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, an input digital image layout by the trained generator module; and generating, by the at least one computing device, a refined digital image layout from the input digital image layout by the trained generator module.
The method as described in claim 1, wherein the wireframe rendering discriminator module operates at a vector graphic level for the plurality of graphic elements in the digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a generative adversarial network (GAN) system including: a generator module implemented by at least one computing device to receive a training digital image layout, and from which, generate a refined training digital image layout using machine learning; and a wireframe rendering discriminator module implemented by the at least one computing device, the wireframe rending discriminator module including: a rasterization module to rasterize the refined training digital image layout into a wireframe digital image layout; a layout optimization module to optimize the rasterized digital image layout using a convolutional neural network (CNN) and a loss function and train the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 9, further comprising a relation- based discriminator module implemented by the at least one computing device, the relation-based discriminator module including: an encoder to generate feature data from the refined training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a classifier module to classify the refined digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 9, wherein the generator module includes: an encoder to generate feature data from the training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a decoder module to generate the refined training digital image layout from the training digital image layout based on the modeled semantic and spatial relationships.
The system as described in claim 9, wherein the generator module, once trained, is configured to receive an input digital image layout and generate a refined digital image layout from the input digital image layout.
The system as described in claim 9, wherein the wireframe rendering discriminator module operates at a vector graphic level for graphic elements in the refined training digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a system comprising: means for receiving an input digital image layout including a plurality of graphic elements having respective semantic parameters and geometric parameters; and means for generating a refined digital image layout from the input digital image layout using a generator module as part of machine learning, the generator module trained as part of a generative adversarial network (GAN) using a wireframe rendering discriminator module including: means for rasterizing a training digital image layout into a wireframe digital image layout; means for optimizing the rasterized digital image layout using a convolutional neural network (CNN) and a loss function; and means for training the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 16, wherein the generator module is further trained as part of the generative adversarial network (GAN) using a relation-based discriminator module implemented including: means for generating feature data from the training digital image layout; means for modeling semantic and spatial relationships based on the generated feature data; and means for classifying the training digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 16, wherein the generator module includes: means for generating feature data from the input digital image layout; means for modeling semantic and spatial relationships based on the generated feature data using a convolutional neural network; and means for generating the refined digital image layout based on the modeled semantic and spatial relationships.
Patent
Atlas literature
Patent
US 10,997,464Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1 is an illustration of a digital medium environment in an example implementation that is operable to employ digital image layout training techniques …
FIG. 2 to generate wireframe prediction data based on the refined training digital image layout using machine lea rn ing to train the generator module. [0013 1
FIG. 3 depicts a system in an example implementation showing operationof a generator module of
FIG. 4 depicts a system in an example implementation showing operation of a wireframe rendering discriminator module of
FIG. 5 depicts an example implementation showing examples of wireframe rendering of graphic elements. [
FIG. 6 is a flow diagram depicting a procedure in an example implementation in which a wireframe digital image layout is generated by rasterizing a refined …
FIG. 7 depicts a system in an example implementation showing operation of a relation-based discriminator module of
FIG. 8 is a flow diagram depicting a procedure in an example implementation of use of a relation-based discriminator module of the GAN system to train a …
FIG. 9 depicts a system in an example implementation in which a generator module trained as part of a GAN network receives an input digital image layout, and …
FIG. 10 is a flow diagram depicting a procedure in an example implementation in which an input digital image layout is used to generate a refined digital image …
FIG. 11 illustrates an example system including various components of an example device that can be implemented as any type of computing device as described …
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
In a digital medium digital image layout environment, a method implemented by at least one computing device, the method comprising: generating, by the at least one computing device, a digital image layout that includes a plurality of graphic elements by a generator module of a generative adversarial network (GAN) system; rasterizing, by the at least one computing device, the digital image layout into a wireframe digital image layout by a wireframe rendering discriminator module of the GAN system; comparing, by the at least one computing device, the wireframe digital image layout with at least one ground truth digital image layout using a loss function as a part of machine learning by the wireframe rendering discriminator module; and training, by the at least one computing device, the generator module by backpropagating a result of the comparing from the wireframe rendering discriminator module based on the loss function to the generator module.
The method as described in claim 1, further comprising: generating, by the at least one computing device, feature data from the digital image layout by a relation-based discriminator module of the generative adversarial network (GAN) system; modeling, by the at least one computing device, semantic and spatial relationships based on the generated feature data by the relation-based discriminator module; and classifying, by the at least one computing device, the digital image layout based on the modeling by the relation-based discriminator module, and wherein the training of the generator module is further based on backpropagating a result of the classifying from the relation-based discriminator module to the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, a training digital image layout by the generator module; and generating, by the at least one computing device, the digital image layout as a refined training digital image layout of the plurality of graphic elements from the training digital image layout by the generator module.
The method as described in claim 1, further comprising: receiving, by the at least one computing device, an input digital image layout by the trained generator module; and generating, by the at least one computing device, a refined digital image layout from the input digital image layout by the trained generator module.
The method as described in claim 1, wherein the wireframe rendering discriminator module operates at a vector graphic level for the plurality of graphic elements in the digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a generative adversarial network (GAN) system including: a generator module implemented by at least one computing device to receive a training digital image layout, and from which, generate a refined training digital image layout using machine learning; and a wireframe rendering discriminator module implemented by the at least one computing device, the wireframe rending discriminator module including: a rasterization module to rasterize the refined training digital image layout into a wireframe digital image layout; a layout optimization module to optimize the rasterized digital image layout using a convolutional neural network (CNN) and a loss function and train the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 9, further comprising a relation- based discriminator module implemented by the at least one computing device, the relation-based discriminator module including: an encoder to generate feature data from the refined training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a classifier module to classify the refined digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 9, wherein the generator module includes: an encoder to generate feature data from the training digital image layout; a relation module to model semantic and spatial relationships based on the generated feature data; and a decoder module to generate the refined training digital image layout from the training digital image layout based on the modeled semantic and spatial relationships.
The system as described in claim 9, wherein the generator module, once trained, is configured to receive an input digital image layout and generate a refined digital image layout from the input digital image layout.
The system as described in claim 9, wherein the wireframe rendering discriminator module operates at a vector graphic level for graphic elements in the refined training digital image layout and not at a pixel level.
In a digital medium digital image layout environment, a system comprising: means for receiving an input digital image layout including a plurality of graphic elements having respective semantic parameters and geometric parameters; and means for generating a refined digital image layout from the input digital image layout using a generator module as part of machine learning, the generator module trained as part of a generative adversarial network (GAN) using a wireframe rendering discriminator module including: means for rasterizing a training digital image layout into a wireframe digital image layout; means for optimizing the rasterized digital image layout using a convolutional neural network (CNN) and a loss function; and means for training the generator module by backpropagating a result of the optimization based on the loss function to the generator module.
The system as described in claim 16, wherein the generator module is further trained as part of the generative adversarial network (GAN) using a relation-based discriminator module implemented including: means for generating feature data from the training digital image layout; means for modeling semantic and spatial relationships based on the generated feature data; and means for classifying the training digital image layout based on the modeling and train the generator module based on backpropagating a result of the classifying.
The system as described in claim 16, wherein the generator module includes: means for generating feature data from the input digital image layout; means for modeling semantic and spatial relationships based on the generated feature data using a convolutional neural network; and means for generating the refined digital image layout based on the modeled semantic and spatial relationships.