A method for controlling a robotic device includes observing a first object associated with an object type at one or more first locations in an environment over a period of time prior to a current time. The method also includes generating a probability distribution associated with the one or more first locations based on observing the first object over the period of time. The method further includes observing, at the current time, a second object associated with the object type at a second location in the environment. The method still further includes determining a probability of the second object being at the second location based on observing the second object at the second location. The probability is based on the probability distribution associated with the one or more first locations. The method also includes controlling the robotic device to perform an action based on the probability being less than a threshold.