Overview

Cox’s theorem is an attempt to derive probability theory from a small, common-sense set of uncontroversial desiderata, and to demonstrate its uniqueness as an extension of two-valued (true/false) logic to degrees of belief. That’s a big deal. As today’s paper mentions, Peter Cheeseman has called Cox’s theorem the “strongest argument for the use of standard (Bayesian) probability theory”. But Cox’s theorem is non-rigorous as originally formulated, and many people have patched up the holes for use in their various fields. Often today, if someone refers to “Cox’s theorem”, they usually mean one of the fixed-up versions.

Jaynes’ version unfortunately contains a mistake, and today’s paper fixes it by replacing some of the axioms with the simple requirement that probability theory remain consistent with respect to repeated events.

It may be difficult without reading the book to see why this paper is important to AI, so perhaps in the near future I’ll discuss that at greater length. For today, however, I’ll simply be explaining each of the axioms, and setting you up to read the paper more easily. It is certainly worth a close reading, to ground your confidence in the interpretation of probability theory as a logical system that extends true-false logic to handle uncertainty, so you can reap the associated benefits.

Abstract

There are multiple proposed interpretations of probability theory: one such interpretation is true-false logic under uncertainty. Cox’s Theorem is a representation theorem that states, under a certain set of axioms describing the meaning of uncertainty, that every true-false logic under uncertainty is isomorphic to conditional probability theory. This result was used by Jaynes to develop a philosophical framework in which statistical inference under uncertainty should be conducted through the use of probability, via Bayes’ Rule. Unfortunately, most existing correct proofs of Cox’s Theorem require restrictive assumptions: for instance, many do not apply even to the simple example of rolling a pair of fair dice. We offer a new axiomatization by replacing various technical conditions with an axiom stating that our theory must be consistent with respect to repeated events. We discuss the implications of our results, both for the philosophy of probability and for the philosophy of statistics.

# AxiomsThis paper proposes a new axiomatization of probability theory, with five axioms. As a variant of Cox’s theorem, these axioms are supposed to represent a set of “common sense” desiderata for a logical system under uncertainty. That is, each of these axioms are things we naturally want to be true of any logical system under uncertainty. Cox’s original axioms were more intuitively essential to me, however, so I’ll also try to give justifications for demanding each of the following axioms, as well as explaining them technically.Remember the ultimate goal is to build probability theory up from a minimal set of absolute requirements for any logical system. The punchline is that probability theory as described historically by greats like Kolmogorov turns out to be the unique extension of true-false logic under uncertainty, and we can derive it from “common sense”.To emphasize the point that while we’re writing these axioms we haven’t yet got probability, following Jaynes I’ll refer to our measure of certainty/uncertainty as “plausibility”.

Parting thoughts

1. I hope this exposition of the axioms helps you read the paper yourself, though I realize I may not have provided sufficient motivation to do so yet. That would make it a bit like my post deriving something surprising about Boltzmann machines without first explaining what Boltzmann machines are. I intend to rectify this in the future for both posts.
2. I could make this a lot clearer for people with less set theory, group theory, or probability theory background. If that would be helpful to you, please leave me a comment on what specifically didn’t make sense so I can get a feel for my audience.
3. To memorize these and make reading the proof easier, I labeled each of the five axioms with some relevant symbol, and combined them into a mneumonic. In case that helps you too, here it is: .

1. Emergent Tool Use From Multi-Agent Autocurricula

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

https://arxiv.org/abs/1909.07528v1

I notice that OpenAI, DeepMind, and Google Brain are involved in a lot of the interesting work in reinforcement learning lately, and many of the papers that catch my eye have at least some authors from either of these organizations. I’m an aspiring Bayesian, so it wasn’t too long before I starting reading papers because they were authored by one of these organizations.

Anyway, the term “autocurriculum” seems to come from this DeepMind paper:

Here we explore the hypothesis that multi-agent systems sometimes display intrinsic dynamics arising from competition and cooperation that provide a naturally emergent curriculum, which we term an autocurriculum.

This gives me a word for something I’ve observed about my young son: The activities he’s naturally inclined to engage in at each stage of his development seem uncannily well-suited for teaching him the next thing he should learn. Wanting to put things in his mouth, plus a capacity for boredom, motivated him to develop reaching and grabbing, then crawling, then pathfinding, then complex navigation…

In the case of the OpenAI paper, putting multiple adversarial agents into complex environments and allowing them to learn causes them to learn new behaviors in phases.

We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt

Each time a team of agents learns a dominant strategy, the opposing team is pressured to develop a strategy capable of defeating it, which then pressures the first team to come up with another stretegy to defeat that one, and on and on until a truly dominant strategy emerges.

My takeaway from this is that emergent autocurricula may be another good reason for me to study multi-agent systems.

This paper comes with a nice blog post and a cute video: https://openai.com/blog/emergent-tool-use/

2. Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard “shallow” RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

https://arxiv.org/abs/1909.10618v1

The big finding here is that “most of the observed benefits of hierarchy can be attributed to improved exploration”.

This is not the first time I’ve heard that a complicated technique in RL has been studied and found to boil down to better exploration or more even coverage of the state space. Diagnosing Bottlenecks in Deep Q-learning Algorithms contained a similar revelation about replay buffer sampling, for example, and I get the impression that the maximum entropy RL framework seems to be overtaking more ad hoc trust region methods such as PPO. Anyway, this is why theory is important even to mere industry practitioners. As theory catches up to practice, we learn why things work, and the answers are often surprising and useful.

3. Brain-Inspired Hardware for Artificial Intelligence: Accelerated Learning in a Physical-Model Spiking Neural Network

Future developments in artificial intelligence will profit from the existence of novel, non-traditional substrates for brain-inspired computing. Neuromorphic computers aim to provide such a substrate that reproduces the brain’s capabilities in terms of adaptive, low-power information processing. We present results from a prototype chip of the BrainScaleS-2 mixed-signal neuromorphic system that adopts a physical-model approach with a 1000-fold acceleration of spiking neural network dynamics relative to biological real time. Using the embedded plasticity processor, we both simulate the Pong arcade video game and implement a local plasticity rule that enables reinforcement learning, allowing the on-chip neural network to learn to play the game. The experiment demonstrates key aspects of the employed approach, such as accelerated and flexible learning, high energy efficiency and resilience to noise.

https://arxiv.org/abs/1909.11145v1

This paper was presented at ICANN 2019, and published in Lecture Notes in Computer Science. In case it’s unclear what’s going on here: The authors built a small-scale prototype (32 neurons, 32 synapses each) of an apparently analog hardware simulation of a biological learning model of the brain (STDP). They then used it to a) simulate a simplified Pong (on-chip), and b) successfully learn to play using reinforcement learning (again, on-chip). Emulating their own system on an Intel i7-4771 was an order of magnitude slower, so we’re talking about a real improvement. This is an auspicious beginning, and they hint at scaled-up work to come.

I look forward to specialized neuronal hardware. I’m especially interested to hear that they simulated actual neurons to some degree, with spike-timing dependence, rather than the simplified model that I’m used to working with. I expect this means they intend to simulate actual brains at some point. Stay tuned.

4. Pre-training as Batch Meta Reinforcement Learning with tiMe

Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads to higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification to be a central challenge and motivate it with theoretical analysis. Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without fine-tuning on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of environment interactions.

https://arxiv.org/abs/1909.11373v1

This paper attempts to bring the benefits of pretraining (on some pre-recorded batch) to reinforcement learning. This is non-trivial, since Q-learning algorithms are known to be unstable on batches produced by “foreign policy” (my phrase).

The value function diverges if Q fails to accurately estimate the value of

This is mitigated by online Q-learning algorithms because the contents of the replay buffer, while produced by an off-policy algorithm, was at least produced through interaction with the environment, and so the distribution of the induced doesn’t deviate too much from the distribution in the replay buffer. Even then, this phenomenon is still a source of instability for Q-learning.

In batch learning, the problem is worse. The recorded batch was not produced by our induced policy, and perhaps not even by a single policy. Further, the environment reflected in the batch may not even have been produced by a single Markov decision process.

I’m interested in this paper because the authors apply meta RL to the problem, and claim to achieve good performance on unseen MDPs sampled from the same family as those represented by the training batch. If that’s so, it has positive implications for my own work.

5. Reinforcement Learning with Chromatic Networks

We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way. By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings. For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing >90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward. We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.

https://arxiv.org/abs/1907.06511v2

From the introduction:

The main question we tackle in this paper is the following:

Are high dimensional architectures necessary for encoding efficient policies and if not, how compact can they be in in practice?

More compact achitectures not only take less space, but also produce inferences more quickly and cheaply. This matters to me because my work is often done on cloud computing infrastructure, which incentivizes parsomony. I’m also professionally interested in neural architecture search for multi-task scaling purposes. More on this later, perhaps.

The authors find compact policies by jointly optimizing the RL objective and “the combinatorial nature of the network’s parameter sharing profile”. Inspired by two other papers, they reduce the number of distinct weights by sharing a single weight between multiple neuronal connections. The first paper from which their inspiration for this arises used Toeplitz matrices to represent the neural network, and the second randomly assigns weights (Weight-Agnostic Neural Networks, or WANNs) and then learns the connection topology to maximize an RL goal.

WANNs replace conceptually simple feedforward networks with general graph topologies using NEAT algorithm providing topological operators to build the network.

Our approach is a middle ground, where the topology is still a feedforward neural network, but the weights are partitioned into groups that are being learned in a combinatorial fashion using rein- forcement learning. While 10 shares weights randomly via hashing, we learn a good partitioning mechanisms for weight sharing.

How do they do this?

We leverage recent advances in the ENAS (Efficient Neural Architecture Search) literature and theory of pointer networks to optimize over the combinatorial component of this objective and state of the art evolution strategies (ES) methods to optimize over the RL objective.

Our key observation is that ENAS and ES can naturally be combined in a highly scalable but conceptually simple way.

Ah. So… how do they do this?

We’ll both just have to read the whole paper. In my light read, I notice this one is so full of interesting insights and pointers to important results from other research that it’s worth our time. Basically though, they alternate between neural architecture search and RL optimization, using their own ENAS variant to optimize a pointer network capable of partitioning weights to be shared.

6. Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://sites.google.com/view/dynamical-distance-learning

https://arxiv.org/abs/1907.08225v2

I picked up this paper partly because reward shaping is currently of professional interest to me, but also because I’m watching Haarnoja for his work on distributional RL.

This paper is about making reward-shaping easier by learning a more direct distance measure for the purpose. In general, if you know your distance from a goal, there are many optimization methods available to you for reducing that distance and achieving your goal. The better this distance measure, the smoother the landscape, and the more quickly you arrive.

The semi-supervised way_in which they approach this problem also strikes me as relevant to AI safety, a topic of personal interest. With the unsupervised training of their dynamical distance measure, they add “a small amount of preference supervision” to set the task goal, and this results in its achievement. A manually-specified reward function is very dangerous, and I’m interested in any novel methods that avoid their direct use (or, more to the point, I’m interested in methods of motivating AIs which more directly align holistic human flourishing with the AI’s objectives).

For a quick overview, don’t miss the link they posted at the end of the abstract.

7. Model Imitation for Model-Based Reinforcement Learning

Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizon, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return.

https://arxiv.org/abs/1909.11821v1

AGI seems likely to be model-based, rather than model-free. I think this because I (an AGI) personally reuse my own models all the time, frequently attempt near-transfer to solve some novel problem. So anything that claims progress on model-based learning is at least worth a look to me.

Earlier I blogged about Efficient Exploration with Self-Imitation Learning via Trajectory-Conditioned Policy, and looking at it now, I’m surprised I only alluded to their use of Transformers. In that paper, they learn to imitate a past trajectory by mimicking the trajectory distribution, conditioned on a past trajectory. This paper wants to create a model of the environment that similarly mimics its distribution, but they use Wasserstein GANs (WGANs) instead. GANs have been wildly successful in generative image models, and WGANs are an especially promising variant. I’ve been keeping an eye out for papers that use GANs in areas outside computer vision.

If you know what a WGAN is and you understand that the authors are trying to get a WGAN to mimic the environment’s bounded trajectory segment transition distribution, then you can imagine what they’re doing. They also provide a theoretical bound for the expected distributional error.

I’ll mention in closing that in their experiments, they also end up doing better than most other methods, using 50% fewer samples. That it works at all suggests to me that the model is sufficiently accurate to take notice. GANs are interesting.

8. Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning

To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo’s problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel which has a slip velocity with fixed intensity, Vs , but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasi-optimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process, and are therefore not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when V s is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.

https://arxiv.org/abs/1907.08591v2

This paper is a pure personal indulgence. I read James Gleick’s Chaos and got interested in dynamical systems theory. I don’t know much, but I do know turbulent flows are a pain to predict, which I assume would mean they’re a pain to navigate within. I’ve also heard that neural networks do pretty surprisingly well at predicting chaotic dynamics, so I’m interested to see it “applied”. The paper brings up several other examples of successful neural navigation and prediction:

Promising results have been obtained when applying RL algorithms to similar problems, such as the training of smart inertial particles or swimming particles navigat- ing intense vortex regions [31], Taylor Green flows [32] and ABC flows [33]. RL has also been successfully imple- mented to reproduce schooling of fishes [34, 35], soaring of birds in a turbulent environments [36, 37] and in many other applications [38–40]. Similarly, in the recent years, artificial intelligence techniques are establishing them- selves as new data driven models for fluid mechanics in general [41–46].

I can’t state their results better than they can, so here you go:

In this paper, we show that for the case of vessels that have a slip velocity with fixed intensity but variable di- rection, RL can find a set of quasi-optimal paths to efficiently navigate the flow. Moreover, RL, unlike ON, can provide a set of highly stable solutions, which are insensitive to small disturbances in the initial condition and successful even when the slip velocity is much smaller than the guiding flow. We also show how the RL protocol is able to take advantage of different features of the underlying flow in order to achieve its task, indicating that the information it learns is non-trivial.

Parting thoughts

1. Surprisingly, even this list of eight do not cover all of the papers that sounded interesting to me. It was quite a good week for announcements on the arXiv.
2. This is the format I originally had in mind for arXiv highlights, but since the abstracts tend to invite questions that I can’t answer without at least skimming the paper, I ended up reading them more thoroughly. With this format, I can cover more ground, but less deeply.

Problem setup

We model the active perception problem as a planning problem, defined by the tuple , where is the state of the world, consisting of the set of observable states and the set of partially observable states ; is the set of actions of the autonomous agent; is the set of actions of the opponent; we further assume that regardless of the intention, the opponent has the same set of observable actions. Otherwise, an intention is easily identifiable once an action that is uniquely corresponding to that type of intention is observed. $T:S A^a A^o _S $ is the transition probability, where denotes the space of probability distribution over the space . is the observation probability; is the reward function; is the prior probability of the opponent being an adversary; and is the discount factor.

Further, the opponent is assumed to be either neutral (merely self-interested, in a known way) or hostile (goal-directed, as defined by a known MDP), with bounded rationality, (it may not be able to take the optimal action) and it is likely to behave deceptively.

Notice that the actual behavior of the opponent is known if its disposition is known, which to my mind may or may not be a reasonable assumption, depending on the setting. Since I’ve had AI safety on the brain lately, it strikes me as unrealistic in a situation where your opponent is smarter than you are. It may be more realistic in settings where everyone has the same goal and it’s relatively clear how anyway would try to achieve it if they didn’t have to deal with other agents.

The authors’ adversarial model is interesting. ( is the parameter to that specifies whether the agent is neutral: , or adversarial: ):

We use the following equation to model an adversarial agent’s policy :

The thing to take away from this is that both rationality and deception are tunable parameters. The rationality of the opponent is controlled by the temperature parameter , by adjusting how well the opponent makes use of the optimal Q function. The degree to which the opponent is deceptive is controlled by , which adjusts how much the KL-divergence of the existing policy from the neutral policy affects the opponent’s search for an optimal strategy.

Bayesian filtering

We maintain a belief over the hidden variable by Bayesian filtering.

As I mentioned, I’m rather short on time today, so I must apologize again for not actually spending the time to explain this. For now, suffice it to say that the opponent is either neutral () or hostile (), and how your agent reacts to it depends very much on which one of those it believes it is playing against. Bayesian filtering will allow it to make the most of the evidence available, so it can use its best guess as it trains.

We define a hybrid belief-state dependent reward to balance exploration and safety where we use the shorthand to denote , the belief that the opponent is an adversary; and is the state dependent reward.

This reward balances exploration behavior and safety. The negative entropy reward can be interpreted as maximizing the expected logarithm of true positive rate (TPR) and true negative rate (TNR). The state-dependent reward depends both on the observable state and the partially observable intent state , as well as the action of the autonomous agent. This reward is used to ensure safety. For instance, some actions could be dangerous to the neutral [opponent], which are discouraged by a large negative reward.

Our agent is trained using Soft-Q Learning while values of are varied, with corresponding opponent behavior. Interestingly, in the case study section the authors mention that the actual adversary models were not always provided in the learning phase.

The active perception agent has to identify the hidden intent while bein grobust to this model uncertainty, which is challenging.

Parting thoughts

1. I admit to being a bit confused by this paper. The authors claim to do Bayesian filtering, but it’s not an explicit feature of the algorithm. In fact, they seem to be sampling for use in training by using only , their prior probability for their belief state. Perhaps it’s a typo.
2. They also seem to claim that the two models of the opponent behavior must be known, but then they mention they’re not available during the learning phase in their case study. Drop me a line if this makes sense to you.

This week

This week’s paper is Discovery of Useful Questions as Auxiliary Tasks from the University of Michigan and DeepMind. It was accepted to NeurIPS 2019 (which I rather hope I’ll be attending). The paper contains a very exciting concept that strikes at the heart of human learning: We learn not only by noticing statistical correlations and inferring concepts, but by actively seeking the answers to helpful questions that occur to us as we navigate the world. That’s also much of what science is about: increasing your understanding of the world by choosing particularly good questions to ask.

Useful questions as an auxiliary task

The authors formulate the problem as a reinforcement learning problem with a main task you’d like to accomplished, augmented with auxiliary tasks generated by the system itself to aid in representation learning, and ultimately to accomplish the main task more efficiently. I’ve mentioned before that this is of professional interest to me.

In this paper the questions are represented as “general value functions” (GVFs), “a fairly rich form of knowledge representation”, because

GVF-based auxiliary tasks have been shown in previous work to improve the sampling efficiency of reinforcement learning agents engaged in learning some complex task…. It was then shown that by combining gradients from learning the auxiliary GVFs with the updates from the main task, it was possible to accelerate representation learning and improve performance. It fell, however, onto the algorithm designer to design questions that were useful for the specific task.

The main insight in this paper is that the gradients induced while learning the main task contain information about what questions would aid in learning a helpful representation.

The main idea is to use meta-gradient RL to discover the questions so that answering them maximises the usefulness of the induced representation on the main task.

Auxiliary tasks

Why should learning something other than the main task help? It teaches composable fundamentals relevant to the task so that the neural network doesn’t have to learn everything from scratch all at once. The kinds of auxiliary tasks we’re talking about here are things like controlling pixel intensities and feature activations. Other examples mentioned in the paper are auxiliary tasks where the agent needed to learn to measure depth, loop-closures (e.g., the letter “C” is not closed, but the letter “O” is), observation reconstruction (which, as an aside, can be used in the construction of intrinsically-motivated, “curious” agents), reward prediction, etc. When agents were required to learn each of these tasks simultaneously with learning their own main tasks, they learned more efficiently than when they were required to learn their main task alone.

But, as we just discussed, each of these examples (see the paper for more) and were hand-crafted. The agents themselves did not attempt to add to their tasks, and careful hand-tuning was required to get the observed improvements.

Meta-learning

A meta-learner progressively improves the learning process of a learner that is attempting to solve some task.

I can hardly overstate how useful this is. In my own work, we aren’t done as soon as we’ve trained a neural network to perform well on a single task. There is an entire host of related tasks on which we’ll need to retrain it in the future. Our work involves training an agent to control the behavior of some software, which is not fixed. If our agent cannot be quickly retrained on other software (perhaps out of our direct control), then it becomes much more expensive and difficult to maintain.

This paper mentions previous work in learning better initializations for a given task, learning to explore, unsupervised learning to develop a good or compact representation, few-shot model adaptation, and learning to improve the optimizers.

The discovery of useful questions

This is Figure 1 of our paper, depicting the architecture that discovers and uses useful questions. It consists of two neural networks, a main task & answer network parametrized by , and a question network parametrized by . The main task & answer network takes the last observations in and produces two categories of output: a) decisions from the policy and b) answers to the “useful questions” . The question network takes future observations , and produces two outputs: a) cumulants , and b) discounts . Cumulants (a term from the GVF literature) are described as scalar functions of the state, the sum of which must be maximized. To me, this just sounds like an obstruse way to say “other loss function”, which makes sense because these are what are describing our auxiliary goals.

Lest you think this method requires time travel, fear not. We can see steps into the future using the time machine of Waiting, which is ok because it only happens during training.

As the authors explain, previous work with auxiliary tasks would have only had the main task & answer network on the left, because the cumulants and discounts were hand-crafted. The question network on the right, and its effective use, is the main contribution of this paper. The number of “other loss functions” is still fixed, but the components of the actual functions that compute them (cumulants and discounts) are represented by an -parametrized neural network that is itself trained on the gradients of the -parametrized main task and answer network.

In the researcher’s own words:

In their most abstract form, reinforcement learning algorithms can be described by an update procedure that modifies, on each step , the agent’s parameters . The central idea of meta-gradient RL is to parameterise the update by meta-parameters . We may then consider the consequences of changing on the -parameterised update rule by measuring the subsequent performance of the agent, in terms of a “meta-loss” function . Such meta-loss may be evaluated after one update (myopic) or updates (non-myopic). The meta-gradient is then, by the chain rule,

The actual computation of this is challenging, because changing affects updates to on all future timesteps. This is the reason training the question network requires looking steps “into the future”. Holding fixed, they compute , in order to finally compute the meta-loss evaluation .

The algorithm then alternates between normal RL training of the main task & answer network, and meta-gradient training of the question network to produce and use questions that maximize the performance of the agent on the original task. It is a very general solution, and empirically outperforms hand-designed auxiliary tasks in many cases.

Parting thoughts

1. The authors themselves note that their algorithm augments an on-policy reinforcement learning algorithm, and I look forward to their promised future work adapting these techniques to an off-policy setting.
2. I notice I take detours from the main article purposes to write about areas of RL that I want to remember to investigate further in the future (e.g., auxiliary task in general, and meta-learning in general). That’s a good habit, though I’ll need to remember to cultivate it without seeming too distracted.
3. This paper mentions that Xu et al. in 2018 tried learning the discount factor and the bootstrapping factor (using meta-gradients), which is an idea I had myself (a year later). Apparently this substantially improved performance on the Atari domain, so I feel vindicated.

This week

This (last) week’s paper is Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting. I’m interested for reasons both professional and personal.

First, I have this problem. Our recent (successful) work has gotten neural nets to do some very interesting things, but expanding will require continuous training in production. This makes catastrophic forgetting (CF) a very real problem, since most of the DRL research assumes you’re training your agent on a single task, and then enjoying it in inference mode forever after.

Second, I’m interested because I’ve got a little son, (the source of the variability in my weekend free time) and I often see him learn something mind-bogglingly fast, and then cement it over the course of a couple days. Pseudo-rehearsal is biologically plausible, and I’m interested in intelligence in its own right.

Catastrophic Forgetting and Pseudo-rehearsal

An agent trained on one task can learn to accomplish that task. If that same agent is then moved to another task, it will learn that other task, but often at the expense of “catastrophically forgetting” the neural net weights learned for the previous task. Several solutions have been proposed, (which are cited in today’s paper, and I’ll likely be reading them) but most are likely not what humans and animals do.

Researchers have proposed extensions to this method such as utilising previous examples’ gradients during learning, picking a subset of previous samples which best represents the population and using a variational auto-encoder to compress stored items. Such rehearsal methods are cognitively implausible and therefore, do not shine light on how mammal brains might efficiently solve the CF problem.

Pseudo-rehearsal trains a generative model (a GAN) to produce examples from all previous tasks, and uses this to implicitly rehearse foregoing data. Today’s paper employes this scheme and a few other tricks to build a system capable of learning multiple tasks.

The RePR model

The researchers dub their method RePR, and it works like this: They build short- and long-term memory systems, and transferring learned behaviors from short- to long-term memory while rehearsing past behavior in long-term memory.

The STM system:

The first part of our model is the short- term memory (STM) system, which serves a similar function to the hippocampus and is used to learn the current task. The STM system contains two components, a DQN that learns the current task and an experience replay containing data only from the current task.

The LTM system:

The second part is the long-term memory (LTM) system, which serves a similar function to the cortex. The LTM system also has two components, a DQN containing knowledge of all tasks learnt and a GAN which can generate sequences representative of these tasks.

They then do periodic consolidation:

During consolidation, the LTM retains previous knowledge through pseudo-rehearsal, while being taught by the STM how to respond on the current task. All of the networks’ architectures and training parameters used throughout our experiments can be found in the appendices. Transferring knowledge between these two systems is achieved through knowledge distillation, where a student network is optimised so that it outputs similar values to a teacher network.

Parting thoughts

1. This sounds brilliant, and analogous to what mammals do. I’m eager to experiment with it, and to introspect and ponder how my own brain learns, with this new model in mind.
2. I wonder very much what we do in sleep. As I’ve mentioned before, I’m quite attracted to the model described in The Miracle of the Boltzmann Machine, but off-hand, I don’t know how to reconcile that model with the concept of nightly rehearsal of the day’s activities. Perhaps the brain is doing two things during sleep? Ockam’s razor impells me to think again.

Reward tampering

I’ve heard it said that no agent will ever become more intelligent than it takes to edit its own reward function, giving itself a simpler task. This paper treats such problems seriously, with some encouraging results.

From an AI safety perspective, we must bear in mind that in any practically implemented system, agent reward may not coincide with user utility. In other words, the agent may have found a way to obtain reward without doing the task. This is sometimes called reward hacking or reward corruption. We distinguish between a few different types of reward hacking.

Results summary

One way to prevent the agent from tampering with the reward function is to isolate or encrypt the reward function, and in other ways trying to physically prevent the agent from reward tampering. However, we do not expect such solutions to scale indefinitely with our agent’s capabilities, as a sufficiently capable agent may find ways around most defenses. Instead, we have argued for design principles that prevent reward tampering incentives, while still keeping agents motivated to complete the original task. Indeed, for each type of reward tampering possibility, we described one or more design principles for removing the agent’s incentive to use it. The design principles can be combined into agent designs with no reward tampering incentive at all.

An important next step is to turn the design principles into practical and scalable RL algorithms, and to verify that they do the right thing in setups where various types of reward tampering are possible. With time, we hope that these design principles will evolve into a set of best practices for how to build capable RL agents without reward tampering incentives. We also hope that the use of causal influence diagrams that we have pioneered in this paper will contribute to a deeper understanding of many other AI safety problems and help generate new solutions.

Parting thoughts

1. I look forward to reading this paper more thoroughly, both because I understand this problem of disincentivising reward hacking is hard, and because Causal Influence Diagrams sound interesting and generally useful.
2. AI safety is important, and I rather hope that awareness of some ways your agents could cheat will help to prevent such errors from leaking out into the world before they are caught.

DRL isn’t superhuman on Atari yet

Wait, what? I was quite skeptical of this claim. Mnih et al. published the groundbreaking Playing Atari with Deep Reinforcement Learning in 2013, claiming superhuman performance. Surely someone would have noticed by now?

Apparently not, and then most DRL algorithms for the next six years used either the same human scores reported in that paper, or human beginners. It’s true that DQN significantly outperformed their own human player, but that player was not, by far, the best in the world. Other recent claims of superhuman performance have proven that claim against the best players in the world (the paper mentions AlphaGo against Lee Sedol, OpenAI Five against OG, and AlphaStar against Mana), but not for the Atari benchmark.

The most poignant detail to me in this paper involved the common “normalized human score”, where 0% is the score of a random agent, and 100% is the score of the human baseline. On this scale, the median score achieved by the world record holders across all Atari games is 4.4k%. Clearly you can’t claim superhuman performance if there are humans who beat your target by a factor of 44, unless you yourself exceed this score.

For reference, the original Rainbow algorithm achieved a median of 200% over all Atari games, and other algorithms seem to do worse. If the normalized human score is fitted to a maximum equal to the human world record for each game, and run with different time limits, a tuned IQN variant of Rainbow receives a median score of less than 4% (there were other problems with the way benchmarks were done, and correcting for them reduces performance even further).

We have a long way to go then. The paper has a useful analysis drawing on both previous and original research as to why DRL algorithms are so bad at Atari, and I encourage a careful reading. Some of them, such as reward clipping, are called out in previous research as explicitly chosen to improve performance, but (to treat this particular example), it has been mentioned that this causes the agent to prefer many small rewards over a single large reward.

I encourage anyone working with the Atari benchmark to read the paper for themselves.

Parting thoughts

1. I actually find it somewhat personally encouraging that there’s room for improvement on Atari. It’s easy to experiment, and I have some ideas myself.
2. That said, it is rather scary that we could overlook something like this for so long, as a community.
3. Anyway, someone will take this as a call to arms, and make progress. Peter Drucker said, “If you can’t measure it, you can’t improve it.” Now that we have better measurements, I predict improvements.

Considerations in a nutshell

The introduction and background are well summarized in their last paragraph:

In summary, most recent studies focus on developing effective and robust multi-agent DRL algorithms to achieve coordination among intersections. The number of intersections in those studies are usually limited, thus their results might not apply to large open network. Although the signal control is indeed a continuing problem, it has been always modeled as an episodic process. From the perspective of traffic considerations, expert knowledge has only been incorporated in down-scaling the size of the control problem or designing novel reward functions for DRL algorithm. Few studies have tested their methods given different traffic demands, or shed lights on the learning performance under different traffic conditions, especially the congestion regimes. To fill the gap, our study will treat the large-scale traffic control as a continuing problem and extend classical RL algorithm to fit it. More importantly, noticing the lack of traffic considerations on learning performance, we will train DRL policies under different density levels and explore the results from a traffic flow perspective.

Set up

def rule_184(lane):
    l = [False] + lane + [False] # pad
    return [(l[i-1] and not l[i]) or (l[i] and l[i+1])
            for i in range(1,len(l)-1)]

def show(t, lane):
    print(f't{t}:\t', ' '.join(['🚘' if i else '_' for i in lane]) )

ti = [True, True, True, True, True, False, False, True, False, False, False, False, False, False, False]

for i in range(7):
    show(i, ti)
    ti = rule_184(ti)

t0:  🚘 🚘 🚘 🚘 🚘 _ _ 🚘 _ _ _ _ _ _ _
t1:  🚘 🚘 🚘 🚘 _ 🚘 _ _ 🚘 _ _ _ _ _ _
t2:  🚘 🚘 🚘 _ 🚘 _ 🚘 _ _ 🚘 _ _ _ _ _
t3:  🚘 🚘 _ 🚘 _ 🚘 _ 🚘 _ _ 🚘 _ _ _ _
t4:  🚘 _ 🚘 _ 🚘 _ 🚘 _ 🚘 _ _ 🚘 _ _ _
t5:  _ 🚘 _ 🚘 _ 🚘 _ 🚘 _ 🚘 _ _ 🚘 _ _
t6:  _ _ 🚘 _ 🚘 _ 🚘 _ 🚘 _ 🚘 _ _ 🚘 _

Experiments

The authors use a maximum-queue-first (LQF) greedy algorithm as their baseline for comparison, which services the lane with the longest queue length at all times.

Parting thoughts

1. In the end, I’m way more interested in the experimental setup of this paper than the conclusions. As usual, I learned a ton, and I may actually use rule 184 as a model for traffic flow on something.
2. Isn’t it obvious given their problem formulation that the agents can’t learn under conditions of congestion, since it means their input is essentially whited out? I would be more impressed with the conclusion if a neural net with complete visibility had trouble learning with congestion. It also seems to me extremely suggestive that a supervised policy can learn from only two examples, and I would very much like to see if the major conclusions of this paper explode with a more realistic network topology. Queueing theory contains all sorts of counterintuitive surprises, and it seems likely to me that their results are more indicative of one of those surprises, rather than some deep fact about DRL’s ability to manage urban congestion.
3. It’s interesting that they formulate the problem as a continuing one, against the prevailing trend in the traffic signal control literature. I agree with them, that even if you get to a state where there’s no traffic, that’s a function of the demand, not of the agent’s choices. I bring this up because I too have found that it’s really quite important to recognize an infinite-horizon problem when you have one, or else your agent learns to rack up debts until the end of the artificial episode when all is “forgiven”.
4. It’s fascinating that all random policies, no matter how bad, are best around 75% congestion. I have been admonished to avoid scheduling myself at more than 70% capacity to avoid the ringing effect. I wonder if this is an empirical vindication of that…

Composable hierarchical RL

Architecture

The composite policy consists of the individual policy networks, each with its own reward function, trained to take observations in and output parameters of a conditional Gaussian. There is also a special activation vector selector network trained on the same states to produce weights corresponding to how much each constituent policy applies to the current state. All of these networks share early layers, since they all benefit from an accurate high-level state representation. Finally, some function takes all of these outputs and determines what action to actually take in the environment.

The Q function networks are similarly arranged, sharing early layers which take a state and an action to produce a Q function for each subtask, as well as a composite Q function.

 

Parting thoughts

1. In a way, the methods proposed here seem rather obvious, and I found this paper quite easy to understand given that it violated none of my expectations. I also haven’t been paying enough attention to hierarchical RL to know off-hand why training the sub-policies in parallel off of the same recorded environment interactions hasn’t been tried before (or whether it has been without my notice). Perhaps it was necessary for off-policy RL to reach a level of maturity sufficient for sub-policies to see enough relevant data to train? In any case, don’t hear me faulting the authors for trying the obvious. It is relieving a non-obvious that a straightforward formulation works so well.
2. I’d love to see this work combined with imitation learning and inverse RL to figure out what sub-policies are necessary in the first place from demonstrations. That seems like a very practical framework for real-world learning.

The problem

The main problem the authors want to solve is insufficient exploration leading to a sub-optimal policy. If you don’t explore your environment enough, you will find local rewards, but miss globally optimal rewards. In this maze (their Figure 1), you can see that an agent that fails to explore will collect two apples in the next room, but may miss acquiring the key, unlocking the door, collecting an apple, and discovering the treasure.

In the notoriously difficult Atari game (for RL agents) Montezuma’s Revenge, it is similarly extremely unlikely that random exploration suffices to explore the environment and achieve a high score. The authors report state-of-the-art performance without expert demonstrations on Montezuma’s Revenge, netting 25k points.

SOTA without demonstrations

So, more precisely, how did they achieve this, and why does it work?

The main idea of our method is to maintain a buffer of diverse trajectories collected during training and to train a trajectory-conditioned policy by leveraging reinforcement learning and supervised learning to roughly follow demonstration trajectories sampled from the trajectory buffer. Therefore, the agent is encouraged to explore beyond various visited states in the environment and gradually push its exploration frontier further… We name our method as Diverse Trajectory-conditioned Self-Imitation Learning (DTSIL).

Parting thoughts

1. I love seeing methods developed for generative language models used in another context entirely, to generate another kind of sequence. I’m overjoyed that it worked well.
2. They need a high-level embedding for two reasons: first because storing entire trajectories exactly in memory is expensive, and second because it’s quite difficult to re-execute a previously-encountered trajectory exectly, so in order for this method to work at all it’s important that an approximate re-execution be possible.

The problem

Learning from demonstrations is more difficult than it may seem at first glance. The trouble mainly stems from covariate shift: the input distribution your agent will see in production is very likely to be different than that encountered during training. Many machine learning algorithms have this problem, reinforcement learning algorithms included, but imitation learning has it especially bad, for a simple reason: the expert demonstrations you are attempting to follow necessarily explore a very small subset of the state space. The whole point of them is to stay on good trajectories, meaning bad trajectories never get explored.

This causes two issues:

1. The agent can’t in general figure out how to get back into the subset of state space where the expert demonstrations apply, even if it gets only slightly off-course, and
2. Value functions for states and actions are affected by unseen states, making it very likely that the agent will wander off as soon as it’s allowed.

Key intuition

The authors solve this problem by pre-training with supervised learning using a loss function that drives down the value of all states outside of those explored in the expert demonstrations , by an amount proportional to their Euclidean distance from the closest state in . In their own words:

Consider a state in the demonstration and its nearby state that is not in the demonstration. The key intuition is that should have a lower value than , because otherwise likely should have been visited by the demonstrations in the first place. If a value function has this property for most of the pair of this type, the corresponding policy will tend to correct its errors by driving back to the demonstration states because the demonstration states have locally higher values.

And Figure 1 is a nice visual demonstration:

Value Iteration with Negative Sampling (VINS)

Into the weeds now.

Parting thoughts

1. I thought it was quite strange that they learned and a dynamical model , and then used in the algorithm. I thought, “Why not just learn ?” The answer was given in their Section A appendix, and was quite interesting. I’m not sure it applies to our case, but it’s important. TL;DR learned from demonstrations alone is degenerate, because there’s always a that perfectly matches the demonstrations and doesn’t depend at all on .
2. One of my coworkers (and upcoming Computable author!) wondered to me if the induced policy could be made explicit, by explicitly training a policy network to bring the agent back into safe territory. It could be trained with gradient descent, because are just networks, and the technique for training deterministic policies just follows the gradient of the function. I wonder too.

Textures, not shapesTraining a CNN for object recognition typically involves only showing the algorithm many examples of images that contain or don’t contain a target object. Humans also need to see many examples of various objects to get the basic idea. Humans, however, seem to have a bias towards recognition by shape which is missing from CNNs in general.> Geirhos, Bethge and their colleagues created images that included two conflicting cues, with a shape taken from one object and a texture from another: the silhouette of a cat colored in with the cracked gray texture of elephant skin, for instance, or a bear made up of aluminum cans, or the outline of an airplane filled with overlapping clock faces. Presented with hundreds of these images, humans labeled them based on their shape — cat, bear, airplane — almost every time, as expected. Four different classification algorithms, however, leaned the other way, spitting out labels that reflected the textures of the objects: elephant, can, clock.This is a problem worth solving, since the addition of even a small amount of noise can throw off CNN-based classifiers, where humans aren’t fooled. “Adversarial examples” even do this maliciously, adding exactly the right amount of noise to cause misclassification. So how to fix this?> Geirhos wanted to see what would happen when the team forced their models to ignore texture. The team took images traditionally used to train classification algorithms and “painted” them in different styles, essentially stripping them of useful texture information. When they retrained each of the deep learning models on the new images, the systems began relying on larger, more global patterns and exhibited a shape bias much more like that of humans.(The illustration of images painted with alien textures that originally appeared here is no longer hosted by Quanta — see the original article for the imagery.)There were many other insights in this relatively short article, and I commend it to you. It enriched my understanding of what’s going on in neural networks, and how far we still need to go to reach parity with humans.

RL for open-domain dialog generation

The author’s domain is dialog generation. They want to build a better chat bot, and they have quite a few recorded conversations. RL is good at refining these processes, but has a cold-start problem, plus they would certainly prefer to make use of the data they have on-hand. For this, they need to be able to make use of offline data, hence “Way Off-Policy”. This data is so off-policy it wasn’t even generated by a policy.

So they want to train DRL from samples acquired from some other control of the system (in their case, human interaction data), much like Deep Q-learning from Demonstrations. There are a couple of reasons this is important for others such as myself:

First, since collecting real-world interaction data can be expensive and time-consuming, algorithms must be able to leverage off-policy data - collected from vastly different systems, far into the past - in order to learn.

Second, it is often necessary to carefully test a policy before deploying it to the real world; for example, to ensure its behavior is safe and appropriate for humans. Thus the algorithm must be able to learn offline first, from a static batch of data, without the ability to explore

A generative model + Q learning

The authors first pre-train a generative model on the distribution of collected trajectories, and initialize the Q networks from this model. They then sample a fixed number of actions from it, and output the one with the highest Q-value as their policy’s decision. In later reinforcement learning, they penalize their model for KL-divergence from this distribution.

To perform batch Q-learning, we first pre-train a generative model of using a set of known environment trajectories. In our case, this model is then used to generate the batch data via human interaction. The weights of the Q-network and target Q-network are initialized from the pre-trained model, which helps reduce variance in the Q-estimates and works to combat overestimation bias. To train we sample < , , , > tuples from the batch, and update the weights of the Q-network to approximate Eq. 1. This forms our baseline model, which we call Batch Q

Overestimation bias

Most deep RL algorithms fail to learn from data that is not heavily correlated with the current policy. Even models based on off-policy algorithms lik Q-learning fail to learn when the model is not able to explore during training. This is due to the fact that such algorithms are inherently optimistic in the face of uncertainty.

If you’re taking the max of something (as in Bellman-equation-based algorithms), then the higher the variance, the higher the max value. This causes an over-estimation bias. We may have seen a really high value for some state once, so now we over-value that state, despite it being atypical. It may not be immediately obvious why this is a problem, but which states are we likely to overvalue? Precisely the states we haven’t visited often. Why is that a problem? This sounds good for exploration, right? But if we’re trying to train our agent with canned data, it’s important that the live agent stick pretty close to the states where the canned data does well, and it’s counter-productive to have it believe that everywhere but the pre-explored state space is worth exploring.

A popular solution to the overestimation problem in Q-learning algorithms is to train two Q networks on the same data, put the input through both, and take the minimum value. This helps with the bias because they’ll likely disagree unless we can be really certain of the value of the input, and if they disagree we can go with the least confident. The authors of the current paper take a different tack, training a single neural net with dropout, and using the disagreement with different dropout masks as an estimate of uncertainty.

Parting thoughts

1. I didn’t talk much about their model architecture, which is “Variational Hierarchical Recurrent Encoder Decoder (VHRED)”, largely because I think if I ever tried to make use of this directly I would employ transformers instead. They do mention that transformer architectures are a “powerful alternative”, but they chose to work with hierarchical architectures so they could extend their work to hierarchical control in the future. That’s interesting. In my own work at the moment, the important thing is the “way off-policy” part, not so much the chat bot part.

2. It’s very interesting to me that both of the methods for correcting overestimation bias make use of uncertainty estimators that I’ve seen mentioned elsewhere:

• Estimating Risk and Uncertainty in Deep Reinforcement Learning

…we show that the disagreement between only two neural networks is sufficient to produce a low-variance estimate of the epistemic uncertainty on the return distribution, thus providing a simple and computationally cheap uncertainty metric.

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

…we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs

3. This article wasn’t really shorter than if I had done multiple papers, less deeply. I’ll have to practice at that, not least because it’s time-consuming, but information is valuable. How does Adrian Colyer do this every day?

This week

All of this week’s papers piqued my interest because of the sample-efficiency problem in modern DRL. Reinforcement learning algorithms need to interact with the environment quite a bit before they become good at a task, and anything that can shorten this time is of interest. My group is currently working on a learning task with a very low sample rate, so we are actively on the hunt for anything that improves sample efficiency.

• Growing Action Spaces, by Farquhar et al. at Oxford and Facebook AI Research.
• Learning to Interactively Learn and Assist, by Woodward et al. at Google Brain.
• ProLoNets: Neural-encoding Human Experts’ Domain Knowledge to Warm Start Reinforcement Learning, by Silva et al. at Georgia Institute of Technology.

Parting thoughts

1. I’m extremely pleased to have finally gotten this off the ground. Please comment on anything and everything, and we’ll drive this thing together.
2. Growing Action Spaces is immediately relevant to my group, since in the medium-term, we intend to increase our action spaces combinatorially, and will inherit all of the trouble this brings. More on this another time.
3. I wonder how often in complex real environments the “Learning to Interactively Learn and Assist” agents will learn to communicate in a way that humans find unintuitive. Since the quickest way to communicate involves some compression, would we need to add some term representing human understandability? How best to do this?
4. “Learning to Interactively Learn and Assist” seems like a relevant paper for AI safety, though as far as I could tell in my quick read, it wasn’t billed that way. If we train agents that don’t have goals of their own necessarily, but take their cues from us in real time, are we safer than if we attempted to craft the perfect reward function, or demonstrated our desires in a one-and-done fashion?
5. I’ve gotta actually read the ProLoNets paper. There was even more to it than I highlighted, and they included an ablation study which will likely tell me if I can incorporate their concepts piecemeal into my own work.

Given

1. Objective function
2. and probability of a given BM state where is the BM transition matrix, assuming

Want to show

Proof

1. Definition of expected value
2. Let
3. Chain rule
4. Expand
5. Quotient rule
6. Chain rule, and notice is everywhere except , so
7. So #5 becomes
8. Separating terms
9. Cancelling and moving factors outside sums
10. Definition of and
11. Substituting #10 into #3 and #3 into #2 we have
12. Separating into two terms
13. Definition of conditional probability
14. , combining sums, and
15. Definition of expected value