Google DeepMind just patented reinforcement learning with pseudo-counts

Yesterday I was searching for an interesting patent to discuss in my next Software Patent Review. When I saw that Google’s DeepMind very recently received a European patent on “Reinforcement learning with pseudo-counts“, I knew the search was over:

What’s the point of the patent?

The patent covers a reinforcement learning technique which selects actions for a mechanical agent with a neural network that is trained with a pseudo-count method.

It combines bonus rewards (which are inversely proportional to pseudo-counts derived from a sequential density model) with actual rewards that resulted from the agent performing actions.

The combined rewards are then used to train the neural network to select actions to be performed by the agent.

Fig. 1 of EP 3 459 018
Fig. 1 of EP 3 459 018

What is it good for?

According to the patent, the neural network incentivizes the agent to explore the environment more thoroughly, e.g., encouraging the agent to explore observations that have not been observed frequently.

The training of the neural network with pseudo-counts is said to reduce computational time and resources required to select actions for the agent to explore the environment.

Direct link with physical reality, anyone?

A look into the prosecution history reveals an interesting detail: In the original filing, DeepMind tried to cover the reinforcement learning technique in the context of both a physical and a virtual embodiment:

  • a mechanical agent (e.g. a robot) which interacts with a real-world environment
  • a software agent which interacts with a simulated environment (e.g. a video game)

But if you’ve seen me talk about G1/19 and the EPO’s current stance on AI patents, you will have guessed that the European patent was only granted after DeepMind limited the claims to the robot embodiment – the granted claims require that “the agent is a mechanical agent and the environment is a real-world environment”.

I guess this patent is a good example of the fact that the EPO currently refuses to grant patents on core AI innovations, but only grants them when the invention is applied to a (traditional) technical context. Direct link with physical reality, anyone?

Fun fact: The US family member has been pending since 2018 without any action from the USPTO. DeepMind also filed family member patents in China and Japan.

Link to the European patent: EP 3 459 018 B1

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.