Patent
US 11,379,722METHOD FOR TRAINING GENERATIVE ADVERSARIAL NETWORK (GAN), METHOD FOR GENERATING IMAGES BY USING GAN, AND COMPUTER READABLE STORAGE MEDIUM
Edward Chang
US
Copyright © 2026 Matter42. All rights reserved.
Patent
Atlas literature
Patent
US 11,379,722Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator, and a prediction network, comprising: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; 25 updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
The method according to claim 1, wherein the discriminating result is used to formulate a first loss function for training the discriminator and a second loss function for training the first generator and the second generator, and the second neural network weightings are updated subject to minimizing the first loss function.
The method according to claim 1, wherein the first category indication is defined as a first one-hot vector indicating the first type category or a first specific semantic embedding vector indicating the first type category.
The method according to claim 1, wherein the second category indication is defined as a second one-hot vector indicating the second type category or a second specific semantic embedding vector indicating the second type category.
The method according to claim 1, wherein the prediction network is an embedding regression network pre-trained with the available training samples belonging to the first type category.
The method according to claim 1, wherein the second type category has no training samples.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator and a prediction network 25 to execute steps of: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: 25 receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category, and only a plurality of training samples belonging to the second type category are previously used to train the first generator and the second generator.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator and a color estimator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region 25 color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
The method according to claim 10, wherein the first generator comprises a first convolutional neural network (CNN), a mask network, and a first combiner, and the method comprises: using the first CNN to generate a first foreground image based on the first input image and the category indication; using the mask network to generate a first probability map corresponding to the first input image, wherein each pixel in the first probability map is labelled with a probability of corresponding to the first specific region; using the first combiner to retrieve a first partial image in the first foreground image based on the first region in the first probability map, retrieve a second partial image in the first input image based on the second region in the first probability map, and combining the first partial image and the second partial image as the first output image.
The method according to claim 10, comprising: generating, by the discriminator, the discriminating result via discriminating the first output image with a real image; predicting, by the discriminator, a predicted category of the first output image; generating, by the discriminator, the classification result via comparing the predicted category with the first type category.
The method according to claim 10, wherein the discriminator is characterized by a plurality of second neural network weightings, the discriminating result and the classification result are used to formulate a first loss function for training the discriminator, and the method comprises: updating the discriminator via updating the second neural network weightings subject to minimizing the first loss function.
The method according to claim 10, wherein the discriminating result, the color comparing result, and the cycle-consistency result are used to formulate a second loss function for training the first generator and the second generator, and the method comprises: updating the first generator and the second generator via updating the first neural network weightings subject to minimizing the second loss function.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator, and a color estimator to execute steps of: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; 25 receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the training color samples are previously used to train the first generator and the second generator; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color 25 corresponds to the target color indication, and the second target color does not belonging to the first type category.
Patent
Atlas literature
Patent
US 11,379,722Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator, and a prediction network, comprising: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; 25 updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
The method according to claim 1, wherein the discriminating result is used to formulate a first loss function for training the discriminator and a second loss function for training the first generator and the second generator, and the second neural network weightings are updated subject to minimizing the first loss function.
The method according to claim 1, wherein the first category indication is defined as a first one-hot vector indicating the first type category or a first specific semantic embedding vector indicating the first type category.
The method according to claim 1, wherein the second category indication is defined as a second one-hot vector indicating the second type category or a second specific semantic embedding vector indicating the second type category.
The method according to claim 1, wherein the prediction network is an embedding regression network pre-trained with the available training samples belonging to the first type category.
The method according to claim 1, wherein the second type category has no training samples.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator and a prediction network 25 to execute steps of: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: 25 receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category, and only a plurality of training samples belonging to the second type category are previously used to train the first generator and the second generator.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator and a color estimator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region 25 color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
The method according to claim 10, wherein the first generator comprises a first convolutional neural network (CNN), a mask network, and a first combiner, and the method comprises: using the first CNN to generate a first foreground image based on the first input image and the category indication; using the mask network to generate a first probability map corresponding to the first input image, wherein each pixel in the first probability map is labelled with a probability of corresponding to the first specific region; using the first combiner to retrieve a first partial image in the first foreground image based on the first region in the first probability map, retrieve a second partial image in the first input image based on the second region in the first probability map, and combining the first partial image and the second partial image as the first output image.
The method according to claim 10, comprising: generating, by the discriminator, the discriminating result via discriminating the first output image with a real image; predicting, by the discriminator, a predicted category of the first output image; generating, by the discriminator, the classification result via comparing the predicted category with the first type category.
The method according to claim 10, wherein the discriminator is characterized by a plurality of second neural network weightings, the discriminating result and the classification result are used to formulate a first loss function for training the discriminator, and the method comprises: updating the discriminator via updating the second neural network weightings subject to minimizing the first loss function.
The method according to claim 10, wherein the discriminating result, the color comparing result, and the cycle-consistency result are used to formulate a second loss function for training the first generator and the second generator, and the method comprises: updating the first generator and the second generator via updating the first neural network weightings subject to minimizing the second loss function.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator, and a color estimator to execute steps of: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; 25 receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the training color samples are previously used to train the first generator and the second generator; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color 25 corresponds to the target color indication, and the second target color does not belonging to the first type category.
Patent
Atlas literature
Patent
US 11,379,722Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator, and a prediction network, comprising: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; 25 updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
The method according to claim 1, wherein the discriminating result is used to formulate a first loss function for training the discriminator and a second loss function for training the first generator and the second generator, and the second neural network weightings are updated subject to minimizing the first loss function.
The method according to claim 1, wherein the first category indication is defined as a first one-hot vector indicating the first type category or a first specific semantic embedding vector indicating the first type category.
The method according to claim 1, wherein the second category indication is defined as a second one-hot vector indicating the second type category or a second specific semantic embedding vector indicating the second type category.
The method according to claim 1, wherein the prediction network is an embedding regression network pre-trained with the available training samples belonging to the first type category.
The method according to claim 1, wherein the second type category has no training samples.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator and a prediction network 25 to execute steps of: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: 25 receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category, and only a plurality of training samples belonging to the second type category are previously used to train the first generator and the second generator.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator and a color estimator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region 25 color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
The method according to claim 10, wherein the first generator comprises a first convolutional neural network (CNN), a mask network, and a first combiner, and the method comprises: using the first CNN to generate a first foreground image based on the first input image and the category indication; using the mask network to generate a first probability map corresponding to the first input image, wherein each pixel in the first probability map is labelled with a probability of corresponding to the first specific region; using the first combiner to retrieve a first partial image in the first foreground image based on the first region in the first probability map, retrieve a second partial image in the first input image based on the second region in the first probability map, and combining the first partial image and the second partial image as the first output image.
The method according to claim 10, comprising: generating, by the discriminator, the discriminating result via discriminating the first output image with a real image; predicting, by the discriminator, a predicted category of the first output image; generating, by the discriminator, the classification result via comparing the predicted category with the first type category.
The method according to claim 10, wherein the discriminator is characterized by a plurality of second neural network weightings, the discriminating result and the classification result are used to formulate a first loss function for training the discriminator, and the method comprises: updating the discriminator via updating the second neural network weightings subject to minimizing the first loss function.
The method according to claim 10, wherein the discriminating result, the color comparing result, and the cycle-consistency result are used to formulate a second loss function for training the first generator and the second generator, and the method comprises: updating the first generator and the second generator via updating the first neural network weightings subject to minimizing the second loss function.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator, and a color estimator to execute steps of: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; 25 receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the training color samples are previously used to train the first generator and the second generator; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color 25 corresponds to the target color indication, and the second target color does not belonging to the first type category.
Patent
Atlas literature
Patent
US 11,379,722Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator, and a prediction network, comprising: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; 25 updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
The method according to claim 1, wherein the discriminating result is used to formulate a first loss function for training the discriminator and a second loss function for training the first generator and the second generator, and the second neural network weightings are updated subject to minimizing the first loss function.
The method according to claim 1, wherein the first category indication is defined as a first one-hot vector indicating the first type category or a first specific semantic embedding vector indicating the first type category.
The method according to claim 1, wherein the second category indication is defined as a second one-hot vector indicating the second type category or a second specific semantic embedding vector indicating the second type category.
The method according to claim 1, wherein the prediction network is an embedding regression network pre-trained with the available training samples belonging to the first type category.
The method according to claim 1, wherein the second type category has no training samples.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator and a prediction network 25 to execute steps of: receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category, and the first type category has available training samples; predicting, by the prediction network, a first semantic embedding vector corresponding to the first output image; generating a first comparing result by comparing the first semantic embedding vector with a second semantic embedding vector corresponding to the first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category; predicting, by the prediction network, a third semantic embedding vector corresponding to the second output image; generating a second comparing result by comparing the third semantic embedding vector with a fourth semantic embedding vector corresponding to the second type category; generating, by the discriminator, a discriminating result via discriminating between the first output image and at least one reference image belonging to the first type category, wherein the discriminator is characterized by a plurality of second neural network weightings; updating the second neural network weightings based on the discriminating result; updating the first neural network weightings based on the discriminating result, the first comparing result and the second comparing result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: 25 receiving, by the first generator, a first random input and a first category indication and accordingly generating a first output image, wherein the first generator and the second generator are both characterized by a plurality of first neural network weightings, the first category indication indicates that the first output image corresponds to a first type category; receiving, by the second generator, a second random input and a second category indication and accordingly generating a second output image, wherein the second category indication indicates that the second output image corresponds to a second type category, and only a plurality of training samples belonging to the second type category are previously used to train the first generator and the second generator.
A method for training a generative adversarial network (GAN), wherein the GAN comprises a first generator, a second generator, a discriminator and a color estimator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region 25 color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
The method according to claim 10, wherein the first generator comprises a first convolutional neural network (CNN), a mask network, and a first combiner, and the method comprises: using the first CNN to generate a first foreground image based on the first input image and the category indication; using the mask network to generate a first probability map corresponding to the first input image, wherein each pixel in the first probability map is labelled with a probability of corresponding to the first specific region; using the first combiner to retrieve a first partial image in the first foreground image based on the first region in the first probability map, retrieve a second partial image in the first input image based on the second region in the first probability map, and combining the first partial image and the second partial image as the first output image.
The method according to claim 10, comprising: generating, by the discriminator, the discriminating result via discriminating the first output image with a real image; predicting, by the discriminator, a predicted category of the first output image; generating, by the discriminator, the classification result via comparing the predicted category with the first type category.
The method according to claim 10, wherein the discriminator is characterized by a plurality of second neural network weightings, the discriminating result and the classification result are used to formulate a first loss function for training the discriminator, and the method comprises: updating the discriminator via updating the second neural network weightings subject to minimizing the first loss function.
The method according to claim 10, wherein the discriminating result, the color comparing result, and the cycle-consistency result are used to formulate a second loss function for training the first generator and the second generator, and the method comprises: updating the first generator and the second generator via updating the first neural network weightings subject to minimizing the second loss function.
A non-transitory computer readable storage medium, recording an executable computer program to be loaded by a training system for training a generative adversarial network (GAN) comprising a first generator, a second generator, a discriminator, and a color estimator to execute steps of: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the first generator and the second generator are partially characterized by a plurality of first neural network weightings; generating, by the discriminator, a discriminating result and a classification result based on the first output image; 25 receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color corresponds to the target color indication, and the second target color does not belonging to the first type category; estimating, by the color estimator, a region color corresponding to the second specific region in the second output image and generating a color comparing result by comparing the region color with the target color; generating, by the first generator, a cycle image according to the second output image and an original category indication and generating a cycle-consistency result by comparing the cycle image with the second input image; updating the discriminator based on the discriminating result and the classification result; updating the first generator and the second generator based on the discriminating result, the color comparing result, and the cycle-consistency result.
A method for generating images by using a generative adversarial network (GAN) comprising a first generator and a second generator, comprising: receiving, by the first generator, a first input image and a category indication and accordingly generating a first output image via replacing a first color of a first specific region in the first input image with a first target color, wherein the first target color belongs to a first type category having a plurality of training color samples, and the training color samples are previously used to train the first generator and the second generator; receiving, by the second generator, a second input image and a target color indication and accordingly generating a second output image via replacing a second color of a second specific region in the second input image with a second target color, wherein the second target color 25 corresponds to the target color indication, and the second target color does not belonging to the first type category.