A few weeks ago, Microsoft introduced its AI-powered digital assistant Copilot following Google Workspace’s adoption of generative AI function in its service, which came as quite a shock. I could actually feel that the innovation in the LLM-based natural language processing technology which began with ChatGPT is developing much faster than expected, in a form that can be used in real life. As a person in the intellectual property field who is likely to be most impacted by the LLM innovation, the recent news surely was inspiring.
By utilizing existing business applications provided by Microsoft (such as MS word, PowerPoint, Outlook, etc.) and Microsoft Graph, an API that allows data exchange between them, and LLMs for natural language processing (probably based on ChatGPT), Microsoft Copilot showed us the potential to dramatically increase work productivity by communicating with computers through natural language.
Source: Youtube, "The Future of Work With AI - Microsoft March 2023 Event”
For me, one of the most eye-catching parts was the demo which showed that that by applying Copilot to PowerPoint, it could design the first draft of PowerPoint slides in accordance with the user’s inputted intention and develop it into a higher level by reflecting the user’s revision requests which are input through natural language.
Source: Youtube, "The Future of Work With AI - Microsoft March 2023 Event”
Then, how are such functions of Copilot being protected by patents? Since patents are not disclosed to the public during the confidentiality period of a year and a half from the filing date, we cannot check the patents for the latest technology right away. However, what if a competitor has patents similar to the technologies applied to Copilot? Let us take a look at the US patent 11,570,318 which Adobe owns.
US patent 11,570,318 named “Performing global image editing using editing operations determined from natural language requests” was applied on July 13th 2021 by Adobe and got registered on January 31st 2023.
The patent discloses a technology related to receiving a natural language request and editing an image accordingly. Adobe has been continuing its research on applying deep learning into computer vision and accumulating patents for the technology. According to my column published recently, the scope of patent rights is determined by how the patent claims are written. In the column I also mentioned that the more concise the patent claims are, the more powerful the patent is. Then, let’s take a look at the claims of the patent.
|
Claim 1 |
|
1. In a digital medium environment for editing digital images, a computer-implemented method for natural language-based editing comprising: receiving a digital image and a natural language request for modifying the digital image; and modifying the digital image in accordance with the natural language request by utilizing a language-to-operation decoding cell of a language-to-operation neural network to: determine, utilizing an operation neural network layer, an image-modification operation utilizing the digital image and the natural language request; determine one or more operation parameters corresponding to the image-modification operation utilizing an operation-based neural network layer; and generate, utilizing an executor, a modified digital image by performing the image-modification operation on the digital image in accordance with the one or more operation parameters. |
The patent consists of two main steps: (1) receiving a digital image and a natural language request for modifying the digital image; and (2) modifying the digital image in accordance with the natural language request by utilizing a language-to-operation decoding cell of a language-to-operation neural network.
If the patent claim consisted “only” of these two steps, it would not have been easy to register the patent. That is because it is simply replacing the motions that humans used to do with machines. The two steps won’t be much different from me asking the designer at our company “Could you please photoshop this image and make it brighter and more colorful?”
That is why the patent describes the substeps of (2) modifying the digital image in accordance with the natural language request by utilizing a language-to-operation decoding cell of a language-to-operation neural network. Specifically, it describes how to technically modify digital images to fit the request based on natural language.
Here, Adobe describes the three sub-steps as follows: (1) determine, utilizing an operation neural network layer, an image-modification operation utilizing the digital image and the natural language request; (2) determine one or more operation parameters corresponding to the image-modification operation utilizing an operation-based neural network layer; and (3) generate, utilizing an executor, a modified digital image by performing the image-modification operation on the digital image in accordance with the one or more operation parameters.
Taking a closer look at the sub-steps, once a natural language request to modify an image is received (e.g. “Can you make this picture of my face brighter?”), a corresponding image-modification motion (segmentation for the face part + brightness enhancement) is determined. After this first-step understanding of motion is finished, it determines the motion parameters (degree of brightness, etc.) that correspond to the image-modification motion and goes through the final step of generating the modified image accordingly.
Copilot also goes through a grounding step which preprocesses natural language input request (user prompt) and passes it to the Graph API, and LLM determines the final motion and accesses data again through the Graph API and carries out the desired motion on the business application. Looking through the process, it seems that the probability of matching the above Adobe patent is quite high.
It seems that Adobe's patent is strong enough to claim rights throughout the method of interfacing with computers through natural language processing. Would it be possible for us to also register a patent like this? Many new business models based on LLM are already being proposed. Which of these BMs are patentable and which are not?
And what strategy can we use to obtain a patent?
PI IP LAW will soon organize related issues and deliver them to you through seminars and series columns. Please look forward to our upcoming series of contents, and thank you for your continued interest.
