In a short but rather important book published in 1948, The Mathematical Theory of Communication, Claude Shannon and Warren Weaver introduced some fundamental and very useful ideas. Their conceit was to frame their analysis in terms of sending and receiving encoded messages through a communication channel, regardless of any significant 'meaning' beyond the code itself. They were conscious of sources of noise that might corrupt the successful receipt of messages. They also proposed a measure for how surprised or 'informed' an experienced observer of a communication channel might be about future observations, namely, information (also called Shannon entropy).
The fundamental unit of consideration in the theory of communication is the number of distinct states, or encodings, that the sender of a message might use, and that the receiver might also recognize. This is called the channel capacity (or bandwidth), which is how much information a channel could convey in the extreme case of no noise and a set of equally likely message states.
The idea of information loss refers to the effect of noise or other corruption of the channel between sender and receiver.
A communication channel is said to show redundancy when some of its states or encodings show up more frequently than others. Sending messages over time that are related to each other can be wasteful. Contemporary data compression techniques recognize this and substitute shorter messages for redundant longer messages. When a communication channel is faulty, or subject to noise, redundancy can repair the damage. Digital cellphone technologies and human language use redundancy on purpose for this reason.
The goal of cryptography is to obscure the contents of a transmitted message. Historically, these methods involve making messages unrecognizable to someone who does not know the encoding. Breaking cryptographic codes, called cryptanalysis, depends in part upon finding redundancies in intercepted communications that hint at the encoding method.
Perhaps not surprisingly, Shannon was working on cryptography and cryptanalysis when he developed these ideas.
For more background information, please see Jeff's original post Signals, Sensors and Data
The author's affiliation with The MITRE Corporation is provided for identification purposes only, and is not intended to convey or imply MITRE's concurrence with, or support for, the positions, opinions, or viewpoints expressed by the author.