Patent
Atlas literature
Patent
US 10,373,023Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
Alearning method for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising steps of (a) a lea rn ing device, if at least one first image, which is one of the real images, is acquired, (i) instructing a first transformer to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 1 _ 1-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images; (b) the learning device, if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fifth image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fifth image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images; and (c) the learning device calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi ft h image, the sixth image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.1.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The lea rn ing method of Claim 1, wherein, at the step of(c), the learning device, if(i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.3.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.4.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The lea rn ing method of Claim 1, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing method for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising a step of on condition that (1) a lea rn ing device (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii- 2) has instructed a second transformer to transform the second training image to at least one third train ing image, whose one or more characteristics are same as or similar to those of the real images for train ing (2) the learning device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fifth training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fifth training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; a testing device instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing method of Claim 10, wherein the transformed test image is used for f in e-tuning of parameters included in a virtual object detector.
The testing method of Claim 10, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Alea m ing device for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory that stores instructions; and at least one processor configured to execute the instructions to: perform processes of (1I), if at least one first image, which is one of the real images, is acquired, (i) instructing a first transfo rm er to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 11-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images, (I I) if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fi ft h image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fi ft h image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images, and (III) calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi fth image, the sixth image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.5.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The learning device of Claim 13, wherein, at the process of (III), the processor, if (i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.7.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.8.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The learning device of Claim 13, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing device for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory stores instructions; and at least one processor, on condition that a lea rn ing device, (1) (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training, to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training, (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training, and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii-2) has instructed a second transformer to transform the second training image to at least one third training image, whose one or more characteristics are same as or similar to those of the real images for training (2) the lea rn ing device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fi ft h training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fi ft h training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; configured to execute the instructions to: perform a process of instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and instructing the first transformer to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing device of Claim 22, wherein the transformed test image is used for fine-tuning of parameters included in a virtual object detector.
The testing device of Claim 22, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Patent
Atlas literature
Patent
US 10,373,023Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
Alearning method for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising steps of (a) a lea rn ing device, if at least one first image, which is one of the real images, is acquired, (i) instructing a first transformer to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 1 _ 1-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images; (b) the learning device, if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fifth image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fifth image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images; and (c) the learning device calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi ft h image, the sixth image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.1.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The lea rn ing method of Claim 1, wherein, at the step of(c), the learning device, if(i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.3.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.4.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The lea rn ing method of Claim 1, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing method for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising a step of on condition that (1) a lea rn ing device (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii- 2) has instructed a second transformer to transform the second training image to at least one third train ing image, whose one or more characteristics are same as or similar to those of the real images for train ing (2) the learning device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fifth training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fifth training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; a testing device instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing method of Claim 10, wherein the transformed test image is used for f in e-tuning of parameters included in a virtual object detector.
The testing method of Claim 10, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Alea m ing device for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory that stores instructions; and at least one processor configured to execute the instructions to: perform processes of (1I), if at least one first image, which is one of the real images, is acquired, (i) instructing a first transfo rm er to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 11-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images, (I I) if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fi ft h image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fi ft h image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images, and (III) calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi fth image, the sixth image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.5.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The learning device of Claim 13, wherein, at the process of (III), the processor, if (i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.7.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.8.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The learning device of Claim 13, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing device for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory stores instructions; and at least one processor, on condition that a lea rn ing device, (1) (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training, to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training, (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training, and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii-2) has instructed a second transformer to transform the second training image to at least one third training image, whose one or more characteristics are same as or similar to those of the real images for training (2) the lea rn ing device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fi ft h training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fi ft h training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; configured to execute the instructions to: perform a process of instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and instructing the first transformer to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing device of Claim 22, wherein the transformed test image is used for fine-tuning of parameters included in a virtual object detector.
The testing device of Claim 22, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Patent
Atlas literature
Patent
US 10,373,023Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
Alearning method for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising steps of (a) a lea rn ing device, if at least one first image, which is one of the real images, is acquired, (i) instructing a first transformer to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 1 _ 1-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images; (b) the learning device, if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fifth image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fifth image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images; and (c) the learning device calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi ft h image, the sixth image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.1.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The lea rn ing method of Claim 1, wherein, at the step of(c), the learning device, if(i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.3.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.4.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The lea rn ing method of Claim 1, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing method for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising a step of on condition that (1) a lea rn ing device (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii- 2) has instructed a second transformer to transform the second training image to at least one third train ing image, whose one or more characteristics are same as or similar to those of the real images for train ing (2) the learning device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fifth training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fifth training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; a testing device instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing method of Claim 10, wherein the transformed test image is used for f in e-tuning of parameters included in a virtual object detector.
The testing method of Claim 10, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Alea m ing device for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory that stores instructions; and at least one processor configured to execute the instructions to: perform processes of (1I), if at least one first image, which is one of the real images, is acquired, (i) instructing a first transfo rm er to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 11-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images, (I I) if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fi ft h image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fi ft h image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images, and (III) calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi fth image, the sixth image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.5.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The learning device of Claim 13, wherein, at the process of (III), the processor, if (i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.7.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.8.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The learning device of Claim 13, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing device for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory stores instructions; and at least one processor, on condition that a lea rn ing device, (1) (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training, to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training, (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training, and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii-2) has instructed a second transformer to transform the second training image to at least one third training image, whose one or more characteristics are same as or similar to those of the real images for training (2) the lea rn ing device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fi ft h training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fi ft h training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; configured to execute the instructions to: perform a process of instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and instructing the first transformer to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing device of Claim 22, wherein the transformed test image is used for fine-tuning of parameters included in a virtual object detector.
The testing device of Claim 22, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Patent
Atlas literature
Patent
US 10,373,023Patent drawings and their descriptions. Click a drawing to enlarge it.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
Alearning method for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising steps of (a) a lea rn ing device, if at least one first image, which is one of the real images, is acquired, (i) instructing a first transformer to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 1 _ 1-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images; (b) the learning device, if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fifth image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fifth image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images; and (c) the learning device calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi ft h image, the sixth image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.1.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The lea rn ing method of Claim 1, wherein, at the step of(c), the learning device, if(i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.3.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The lea rn ing method of Claim 1, wherein, at the step of(c), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.4.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The lea rn ing method of Claim 1, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing method for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising a step of on condition that (1) a lea rn ing device (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii- 2) has instructed a second transformer to transform the second training image to at least one third train ing image, whose one or more characteristics are same as or similar to those of the real images for train ing (2) the learning device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fifth training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fifth training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; a testing device instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing method of Claim 10, wherein the transformed test image is used for f in e-tuning of parameters included in a virtual object detector.
The testing method of Claim 10, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
Alea m ing device for learning transformation of one or more real images on a real world into one or more virtual images on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory that stores instructions; and at least one processor configured to execute the instructions to: perform processes of (1I), if at least one first image, which is one of the real images, is acquired, (i) instructing a first transfo rm er to transform the first image to at least one second image, whose one or more characteristics are same as or similar to those of the virtual images, (ii-1) instructing a first discriminator to determine whether the second image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images without a transformation from at least part of the real images, and the secondary virtual images are at least part of the virtual images with the transformation from at least part of the real images, to thereby generate a 11-st result, and (ii-2) instructing a second transformer to transform the second image to at least one third image, whose one or more characteristics are same as or similar to those of the real images, (I I) if at least one fourth image, which is one of the virtual images, is acquired, (i) instructing the second transformer to transform the fourth image to at least one fi ft h image, whose one or more characteristics are same as or similar to those of the real images, (ii-1) instructing a second discriminator to determine whether the fi ft h image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images without a transformation from at least part of the virtual images, and the secondary real images are at least part of the real images with the transformation from at least part of the virtual images, to thereby generate a 21-st result, and (ii-2) instructing the first transformer to transform the fi ft h image to at least one sixth image, whose one or more characteristics are same as or similar to those of the virtual images, and (III) calculating one or more losses by referring to at least part of the first image, the second image, the third image, the fourth image, the fi fth image, the sixth image, the 1 _ 1-st result, and the 21-st result, to thereby learn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.5.svg 0.8 5.15 Black and white a transformer loss included in said one or more losses is def in ed by a formula above, I is the first image, G (I) is the second image, D G (G (I))) is the 1_ 1-st result, F (G (I)) is the third image, X is the fourth image, F(X) is the fifth image, D F (F (X)) is the 21-st result, G (F (X)) is the sixth image, y and fl are constants for adjusting each of weights of each of I-F (G (I)) and I X-G(F (X)) I.
The learning device of Claim 13, wherein, at the process of (III), the processor, if (i) a virtual object detection result on at least part of the second image and the sixth image generated by a virtual object detector, which detects at least one virtual object included in its inputted image, and (ii) its corresponding GT are acquired, instructs a loss unit to generate at least part of said one or more losses by further referring to the virtual object detection result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.7.svg 0.28 3.57 Black and white an FD loss for the first discriminator included in the losses is def in ed by a formula above, VI is any arbitrary virtual image among the virtual images, D G (VI) is a 1_ 2-nd result, from the first discriminator, of determining the arbitrary virtual image, G (I) is the second image, and D G (G (I)) is the 1 _ 1-st result.
The learning device of Claim 13, wherein, at the process of (III), SVG 16258877.01-28-2019.JRGLB₀FTRXEAPX2.CLM.8.svg 0.28 3.59 Black and white an SD loss for the second discriminator included in the losses is defined by a formula above, R I is any arbitrary real image among the real images, D F (RI) is a 2 _ 2-nd result, from the second discriminator, of determining the arbitrary real image, F (X) is the fifth image, and D F (F (X)) is the 2 1-st result.
The learning device of Claim 13, wherein each of the first transformer and the second transformer includes at least part of one or more encoding layers and one or more decoding layers.
A testing device for testing transformation of one or more real images for testing on a real world into one or more virtual images for testing on a virtual world by using a cycle GAN (Generative Adversarial Network), comprising: at least one memory stores instructions; and at least one processor, on condition that a lea rn ing device, (1) (i) has instructed a first transformer to transform at least one first training image, which is one of real images for training, to at least one second training image, whose one or more characteristics are same as or similar to those of one or more virtual images for training, (ii-1) has instructed a first discriminator to determine whether the second training image is one of primary virtual images or one of secondary virtual images, wherein the primary virtual images are at least part of the virtual images for training without a transformation from at least part of the real images for training, and the secondary virtual images are at least part of the virtual images for training with the transformation from at least part of the real images for training to thereby generate a 11-st result, and (ii-2) has instructed a second transformer to transform the second training image to at least one third training image, whose one or more characteristics are same as or similar to those of the real images for training (2) the lea rn ing device (i) has instructed the second transformer to transform at least one fourth training image, which is one of the virtual images for training to at least one fi ft h training image, whose one or more characteristics are same as or similar to those of the real images for training (ii-1) has instructed a second discriminator to determine whether the fifth training image is one of primary real images or one of secondary real images, wherein the primary real images are at least part of the real images for training without a transformation from at least part of the virtual images for training and the secondary real images are at least part of the real images for training with the transformation from at least part of the virtual images for training to thereby generate a 21-st result, and (ii-2) has instructed the first transformer to transform the fi ft h training image to at least one sixth training image, whose one or more characteristics are same as or similar to those of the virtual images for training and (3) the learning device has calculated one or more losses by referring to at least part of the first training image, the second training image, the third training image, the fourth training image, the fifth training image, the sixth training image, the 11-st result, and the 21-st result, to thereby lea rn at least part of parameters of the first transformer, the second transformer, the first discriminator, and the second discriminator; configured to execute the instructions to: perform a process of instructing the first transformer to acquire at least one test image, which is one of the real images for testing, and instructing the first transformer to transform the test image into at least one transformed test image, whose one or more characteristics are same as or similar to those of the virtual images for testing.
The testing device of Claim 22, wherein the transformed test image is used for fine-tuning of parameters included in a virtual object detector.
The testing device of Claim 22, wherein the test image is one of the real images for testing acquired by a camera included in an autonomous vehicle, and a virtual object detector included in the autonomous vehicle detects at least one object included in the transformed test image to thereby support the autonomous vehicle to drive in the real world.
