2.       MATHEMATICAL FOUNDATIONS

 In the following, it is  shown  that the generalized information theory is based on more general mathematical concepts in comparison of Shannon’s information theory.

  2.1       Objective, Subjective and Logical Probabilities

 Let us use predictions about raining as example to explain the distinctions between three types of probabilities.

 1) Assume  a prediction “Tomorrow will be rainy” has been made by an observatory for many times. We can  derive the frequency ( or say percentage ), such as 70%, of rain  from the statistics of objective facts.  The frequency  is called  objective probability. The probability a prediction is selected is also derived from the statistics of objective facts and hence is also called objective probability. In the following,  an objective probabilities is denoted by P( ). 

 2) The probability of  rain in the future , which is forecasted by audience  according to a weather forecast is called subjective probability, namely  as is  used by the Bayesian school (Chen, 1982). It is also called possibility by some researchers. In the following, Q(x_i) and Q(x_i|A_j) denote subjective probabilities.

 3) For a certain weather or rainfall (x_i), the probability in which a  proposition such as “It is  very rainy” is judged true, is a logical probability as discussed by Reichenbach (1982) and other logical empiricists. It is often called belief-degree by researchers. In this paper, Q(A_j|x_i) and Q(A_j) denote logical probabilities.

 The membership grade in fuzzy mathematics is  but the different expression of  logical probability of a proposition. Their relationship are  examined as follows.

 Let a set of objects be  A={x₁, x₂, ..., x_m}, and a set of predicates  be  B={y₁, y₂, ..., y_n},  X and Y be random variables taking values in A and B respectively. For example, A is a set of  different ages and B is a set of predicates, such as  “ X  is about twenty ” ( X  here means a room that will be filled with a person’s age denoted by x_i, i=1, 2, ..., m). We use y_j or y_j(X) to stand for a predicate , and y_j(x_i) for a proposition. If X is given, such as X=x₁ or X=Tom’s age, then the predicate will becomes a proposition: “Tom’s age is about twenty”. If a proposition is a judgment or conjecture about uncertain event, it is also called  a prediction. 

 Following the random-set projection theory  presented by Wang (1987),  we may  treat a fuzzy set as the statistical result of a random set (see Figure 1); and thus, the membership grade is defined as follows:

 Definition 2.1.1 the  membership grade of x_i in fuzzy subset A_j  of A is

Q(A_j |x_i)=P(XÎS_j|X=x_i)=P(x_iÎS_j)= ,     (1)

where Q(A_j|x_i) Î[0,1], S_j is a random set with respect to fuzzy subset A_j of A, s_k  is a given set as a value taken by S_j, Q(s_k|x_i) Î{0,1} is the feature function of set s_k.

In this paper, the logical probability of proposition y_j(x_i), or logical condition probability of predicate y_j is defined as the same as the membership grade, i.e.

Q(y_j(x_i) is true)= Q(x_iÎA_j)=Q(A_j|x_i).  (2)

  For example, Tom’s age (x_i) is 16. The membership grade of his age in fuzzy set A_j={ages that are about x_j=20} is 0.45 as shown in Figure 1. The logical probability of the proposition “Tom’s age is about twenty” is also 0.45. In this case, those clear sets, s_k for k=1,2, ..., are judged on the meaning of “about twenty years old” by different persons and in different cases. 

Further, let the logical probability of the predicate y_j be defined as

  Q(y_j is true)=Q(XÎA_j)=Q((A_j)= Q(x_i)Q(A_j |x_i).  (3)

  Equation (3) is similar to that provided by Zedah (1986).

  It should be remembered that a predicate y_j has both an objective or selective probability P(y_j) and a logical probability Q(y_j is true)=Q(A_j). They are generally unequal. For instance, if a meteorological observatory always  tells us “There will be no rain”(y_j), then the selective probability would be P(y_j)=1; yet, Q(A_j) is related  to the meaning of the sentence and usually has nothing to do with the probability in which the sentence is selected. Our statistical experience could infer that the  logical probability of “There will be no rain.” is about 0.8 .  Similarly, P(y_j|x_i) is the condition probability in which y_j is selected for given x_i; yet Q(A_j|x_i) is the logical probability of proposition y_j(x_i). They are very different. In brief, for objective and subjective probabilities, there are

, ;

 however, for logical probability, there are

in most cases.

2.2         Set-Bayesian Formula

  Assume that A’ is a clear subset of  A, the feature function of A’ is Q(A’|x_i)Î{0,1}, and the forecasted probability of event XÎA’ is Q(A’). Then

  Q(A’)=Q(XÎA’)= Q(x_i)Q(A'|x_i),   (4)

  where Q(x_i) is subjective or forecasted probability of x_i. Let us define Q(x_i|A')=Q(X=x_i|XÎA'), hence the condition probability of x_i in A’ is

Q(x_i|A')=Q(X=x_i, XÎA')/Q(XÎA')=Q(x_i)Q(A'|x_i)/Q(A').   (5)

  The above formula is the Bayesian formula with a set as condition, and hence is called set-

Bayesian formula. Let the clear set A’ be replaced with a fuzzy set A_j. Then we have set-Bayesian formula for fuzzy set:

 Q(x_i|A_j)=Q(X=x_i|XÎAj)=Q(X=x_i|XÎS_j)

=Q(X=x_i, XÎS_j)/Q(XÎS_j)=Q(x_i)Q(A_j|x_i)/Q(A_j).   (6) 

2.3       Generalized Mapping and Generalized Entropy

 To measure Shannon’s entropy of a set, such as the set of rainfalls, ages, or colors, we need set partition. Yet, to measure its generalized entropy, we only need set covering. Two groups of mathematical concepts related to two types of entropies are shown in Table I. These concepts are very elementary in modern mathematics. For each pair of  the concepts in Table I, the left is a special case of the right.

 Table I  mathematical concepts basing entropy and generalized entropy

 Let A₁, A₂, ..., A_n be subsets of A. The set A_J ={A₁, A₂, ..., A_n} is called a covering of A if

  . (7)

  Further, A_J  is called a partition of A if A_jÇA_j’=F (empty set) for all A_j, A_j ÎA_J  and A_j ¹A_j  (which means no overlapping sets).

  We call a subset R of the product set A´B a relation from A to B; and further call R a generalized mapping if for each x_iÎA there must be at least one y_j so that (x_i, y_j)ÎR (see Figure 2). If further, there is one and only one y_j for each x_i so that (x_i, y_j)ÎR, then generalized mapping will become mapping or regular mapping.  For each pair of  (x_i, y_j)ÎR, we call y_j  the image of x_i and call x_i the inverse image of y_j. The difference between mapping and generalized mapping is that each x_i can only have one image in mapping; yet, a x_i may have more than one images in generalized mapping.

Figure 2  Illustrations of  relation, generalized mapping, and mapping

It can be proven that a generalized mapping from A to B determines a covering of A,  and a regular mapping on A´B determines a partition of A; vice versa (see Figure 2). Similarly, a generalized mapping from A to B determines a similarity relation on A´A, and a regular mapping from A to B determines a equivalence relation on A´A; and vice versa.

 Definition 2.3.1 The generalized entropy determined by the generalized mapping from A to B is defined as 

  ,  (8)

where P(y_j) is a objective probability in which the predicate y_j is selected; A_j a set formed by all inverse image of  y_j; Q(A_j ), defined by Equation (3),  is the logical probability of y_j.

If the generalized mapping becomes a regular mapping, then Q(A_j )=P(y_j ) for  j=1, 2, ..., n,  and hence the generalized entropy becomes Shannon’s entropy:

. (9)

2.4       Fuzzy mapping, Generalized Condition Entropy, and Generalized Mutual Entropy

A generalized mapping R from A to B can be  determined by a m´n (m=|A|, n=|B|) matrix with elements r(x_i , y_j)Î{0,1} for i=1, 2, ..., m;  j=1, 2, ..., n, where r(x_i , y_j)=1 means (x_i , y_j)ÎR. If r(x_i , y_j)Î[0,1], then we call R a fuzzy relation.

 Definition 2.4.1  A fuzzy relation  is called a fuzzy  mapping (or fuzzy generalized mapping) if

  r(x_i, y_j )³1, i=1, 2, ..., m.  (10)

 Definition 2.4.2  Assume R is fuzzy mapping and Q(A_j|x_i)=r(x_i, y_j) for i=1, 2, ..., m;  j=1, 2, ..., n. we call

        (11)

 the generalized condition entropy.

 In Equation (11), the condition probability P(y_j|x_i) for i=1, 2,..., m indicates the rule for selecting sentence y_j; whereas logical probability Q(A_j|x_i) for i=1, 2, ..., m embodies  the meaning of sentence y_j. The former changes from case to case and person to person.  Yet, the latter is determined by  linguistic culture. The former must be normalized; whereas  the latter needn't be. The reason is that, in set B, there are probably two sentences, such as “Tomorrow will be rainy” and “Tomorrow will be  very rainy”, that are equally true at the same time.

 Let us call  H(X;Y)=H(Y)-H(Y|X) the generalized mutual entropy. Section 3.2 will explain that H(X;Y) indicates the generalized mutual information, i.e. H(X;Y)=I(X;Y). An example of calculating the generalized mutual entropy is shown as follows.

 Example 2.4.1 Three  kinds of possible weathers are x₁=no-rain, x₂ =light-rain, x₃=heavy-rain;  A={x₁, x₂, x₃}; P(x₁)=P(x₂)=P(x₃)=Q(x₁)=Q(x₂)=Q(x₃)=1/3. There are sentences y₁=“There is no rain”,  y₂=“There is light rain”, y₃=“There is heavy rain”,  y₄=“There is rain”; B={y₁, y₂, y₃, y₄}. The selective condition-probability matrix and the logical condition-probability matrix of sentences are shown as follows: 

 {P(y_j |x_i) | i=1, 2, 3; j=1, 2, 3, 4} =   

_,

{Q(A_j|x_i)|i=1, 2, 3; j=1, 2, 3, 4} =

_.

  From the above data, we have

  P(y₁)= P(y₄)=1/3;  P(y₂)=P(y₃)=0.5/3

Q(A₁)=1/3;   Q(A₂)=Q(A₃)=1.2/3;   Q(A₄)=2/3;

H(Y)= - P(y_j)logQ(A_j)=1.16(bits)

H(Y|X)= - P(x_i, y_j)log Q(A_j|x_i)= -0.1log0.2=0.46(bits);

H(X;Y)=H(Y)-H(Y|X)=0.70(bits).   

Definition 2.4.3 The subset R of the product set A´A is called a fuzzy similarity relation if

 1) R is self-reciprocal, i.e. r_jj= r_ij for  j=1, 2, ..., m;

2) R is symmetrical, i.e. r_ij=r_ji, i, j=1, 2, ..., m;  

3) r_{ij ³}1.

  And a fuzzy similarity relation is called a fuzzy equivalence relation if r_ij=1.

Assume A is a set of different gray levels or colors. The confusion probability in which x_i and x_j are confused by  sight (or a detector) can be a similarity function r(x_i, x_j). Let represent a prediction y_j=“X is likely x_j” and A_j be a fuzzy set including all X ÎA confused with x_j.   Then Q(A_j|x_i)=r(x_i, x_j ). In this case, we also call  Q(A_j|x_i)=r(x_i, x_j ) a fuzzy discrimination function. Treating r(x_i, x_j) as logical condition probability, we can calculate sensory information, which is conveyed by sense organs or detectors, in the same way as semantic information. In practice, both logical condition probability function Q(“y_j  is true”|X) and confusion probability function r(X, x_j ) can be obtained by the set-valued statistics ( refer to Figure 1 and Equation (1)).