Depending on the nature of the interface mappings and

Depending on the nature of the interface mappings and whether the environ-
ment receives actions from sources other than the learner, there are four basic
types of learning from the environment. It is important to identify these types
because different learning methods may only work for different types of configu-
rations.
2.3.1 Transparent Environment
In the first type of learning from the environment, the environment has no hid-
den states and the perception interface mapping ψ is one-to-one. We call this
type of learning “learning from a transparent environment” because the learner
can clearly see the inside of the environment. Learning from a transparent envi-
ronment is not always easy. The size of the environment may be very large, and
there may be too much for the learner to see. In order to act effectively in this
type of environment, a compact, high-level model must be abstracted from the
environment.
2.3.2 Translucent Environment
In the second type of learning from the environment, the environment has hidden
states or the perception interface mapping ψ is many-to-one. We call this type of
learning “learning from a translucent environment” because the learner cannot see

2.3. THE ENVIRONMENT AND ITS TYPES 17
all the states of the environment at once (but can infer them through experience).
Clearly, learning from a translucent environment is more difficult than learning
from a transparent environment. The learner is bound to experience nondetermin-
ism and it must look into the history of its actions and observations in order to
construct appropriate hidden model states.
2.3.3 Uncertain Environment
In the third type of learning from the environment, the interface mappings are
noisy. This means that the observations of the learner do not truly reflect the
outputs of the environment, and the actions of the learner are not delivered to
the environment undisturbed. This kind of learning occurs frequently in the real
world because the sensors and the effectors (devices that carry out actions) cannot
be built so that they are completely precise. We call this type of learning “learning
from uncertain (or noisy ) environments.” To learn from such environments, the
learner must abstract models that can filter out the noise or that best estimate the
uncertainty.
2.3.4 Semicontrollable Environment
In the fourth type of learning from the environment, the environment receives ac-
tions from sources other than the learner. From the learner’s point of view, the
action interface mapping η seems to be a one-to-many mapping and the environ-
ment changes its outputs by itself. We call this type of learning “learning from
a semicontrollable environment” because the environment receives actions that
the learner cannot control. An extreme of this type of learning occurs when the
learner’s actions have no effect at all on the behavior of the environment yet the
output of the environment keeps changing. The learner cannot control the envi-
ronment but observes the changes of the outputs. We call this type of learning
“learning from an observation-only environment.” In general, we will refer to the
type of environment that changes without the actions from the learner a “ time-
variant environment.” The reasons for the changes in the environment vary, but
these reasons have no significance from the learner’s point of view. To learn from
this type of environment is even more difficult. The learner must model not only