where m and a designate “message” (the vector containing the 11 observation variables listed in Section II-B) and “action” (one of the seven actions listed in Section II-C),respectively, the superscript t refers to the time step, while the subscript (m or a) indicates whether the function is associated with messages or with message-action pairs. The function χ is an indicator function, taking the value 1 if the message/message-action pair is visited and 0 otherwise. The function K represents the number of times a particular message/message-action pair is visited.