Desktop version

Home arrow Sociology arrow Exponential Random Graph Models for Social Networks Theory, Methods, and Applications

GOF

How Do You Know Whether You Have a Good Model?

To say a model fits our data is to argue that the combination of such network structures specified in the model is a good representation of how this particular network could have been formed. GOF therefore allows us to know whether the specified model for our observed data represents particular network structures or graph features well. The GOF procedure gives us an indication of whether our model is able to represent important graph features.

To illustrate the utility of GOF in ERGM, let us examine the results of two GOFs (models A and C) for the communication network for The Corporation. These GOFs are presented in Table 13.3. What we see is that model A does not capture many of the structural network features because it does not include parameters for network self-organization (e.g., reciprocity, transitivity). However, when we included Markov parameters (model B), we were unable to obtain convergence for the model. This lack of convergence of model B, as shown in Table 13.2, demonstrates that the model cannot find a stable solution to the data, and as such, it is pointless to conduct a GOF. Finally, model C, which includes actor- relation and higher-order structural effects, successfully converges, and we see that GOF is considerably improved. Taken together, we argue that model C is the best of the three models for this communication network.

Specifically looking at Table 13.3, there are four columns that refer to the GOF. These are the count (or statistic), which is simply the relevant statistic for our data (often, simply the count of the relevant

180 Exponential Random Graph Models for Social Networks

Table 13.3. Selected goodness-of-fit (GOF) details for communication network (n = 38) for models A and C

parameter [PNet name]

GOF

count

mean

standard

deviation

(SD)

GOF

?-ratio

Model A

Arc

146

146.14

10.54

-0.01

Reciprocity

44

12.60

3.48

9.01

2-path

559

541.68

83.38

0.21

2-in-star

313

280.92

43.49

0.74

2-out-star

283

276.30

42.44

0.16

Transitive triad [030T]

212

77.11

19.11

7.06

Popularity spread [AinS]

167.51

162.61

18.30

0.27

Activity spread [AoutS]

166.39

162.22

18.20

0.23

Path closure [AT-T]

153.97

67.72

15.23

5.67

Cyclic closure [AT-C]

120.97

63.85

17.19

3.32

Multiple 2-paths [A2P-T]

452.47

485.67

67.43

-0.49

Sender (seniority)

73

73.30

7.68

-0.04

Sender (projects)

993

994.63

96.12

-0.02

Receiver (seniority)

84

84.21

7.94

-0.03

Receiver (projects)

1005

1007.57

96.66

-0.03

Homophily (seniority) [interaction]

60

60.30

6.90

-0.04

Heterophily (projects) [difference]

584

585.91

62.66

-0.03

Homophily (office) [matching]

51

50.99

6.16

0.00

Advice entrainment [covariate arc]

24

24.17

3.57

-0.05

SD in-degree distribution

2.39

1.95

0.23

1.89

Skew in-degree distribution

0.37

0.40

0.34

-0.08

SD out-degree distribution

2.02

1.89

0.21

0.61

Skew out-degree distribution

-0.17

0.29

0.34

-1.38

Model C

Arc

146

146.60

14.80

-0.04

Reciprocity

44

44.07

6.79

-0.01

2-path

559

566.36

116.35

-0.06

2-in-star

313

310.17

66.83

0.04

2-out-star

283

301.34

62.80

-0.29

Transitive triad [030T]

212

212.42

54.18

-0.01

Popularity spread [AinS[

167.51

168.56

25.91

-0.04

Activity spread [AoutS]

166.39

167.26

25.61

-0.03

Path closure [AT-T]

153.97

154.85

31.16

-0.03

Cyclic closure [AT-C]

120.97

121.92

29.30

-0.03

Multiple 2-paths (A2P-T)

452.47

457.20

79.75

-0.06

Sender (seniority)

73

73.83

11.06

-0.08

Sender (projects)

993

994.47

142.35

-0.01

Receiver (seniority)

84

84.59

12.18

-0.05

Receiver (projects)

1005

1008.53

141.21

-0.03

Homophily (seniority) [interaction]

60

61.11

11.33

-0.10

Heterophily (projects) [difference]

584

584.80

88.84

-0.01

Homophily (office) [matching]

51

51.20

7.92

-0.03

Advice entrainment [covariate arc]

24

23.86

4.06

0.03

SD in-degree distribution

2.39

2.28

0.31

0.35

Skew in-degree distribution

0.37

0.40

0.35

-0.08

SD out-degree distribution

2.02

2.18

0.28

-0.56

Skew out-degree distribution

-0.17

0.35

0.34

-1.56

configuration). The mean is the estimated statistic produced by our model, and the next column is the associated SDs. For a model to represent a particular graph feature well, we want the observed data to be plausible (i.e., not too far away from the mean). For example, in the first row of Table 13.3 for the arc effect, we observed 146 ties, whereas the sample mean from the model was 146.14. The final column is the t-ratio of the GOF, which is a measure of fit of the model.

Recall (see also Section 12.9) that GOF is carried out on the premise that the estimation has converged; in other words, the difference between simulated statistics and observed statistics for “fitted effects,” as measured by the convergence statistic, is smaller than 0.1 in absolute value (e.g., in phase 3 of the Robbins-Monro procedure in PNet). Because the GOF also compares simulated average statistics under the fitted model with the observed statistics, the GOF t-ratios for the fitted statistics should be smaller than 0.1 (in absolute value). Of course, the estimation is a stochastic procedure, so we do not expect to obtain the exact same values for GOF and estimation for these t-ratios for fitted statistics. Thus, if the GOF statistics are a little over 0.1, there should be no need for concern. However, sometimes the GOF statistics can be substantially over 0.1. The first step is to run a much longer GOF simulation to determine whether the problem is resolved. If not, you may not have a fully converged model, and there may be a need to reestimate the model.

In detail, when the GOF statistic for a fitted effect is substantially over 0.1, the precision of either the samples in phase 3 of the Method of Moments estimation or the GOF is insufficient. Of these two, the sample with most iterations and lowest autocorrelation should be most trusted. If the number of iterations is the same, a discrepancy means that either the “number of iterations in each step” (given in the PNet output) for phase 3 is too small or the ratio of the number of iterations in the GOF to the “number of samples to be picked up” is too low (this is comparing the thinning; see Section 12.2.2). In case of the former, you will probably find that some of the autocorrelations are too big (in which case the basic premise of converged estimates is likely to be violated and the model needs refitting, with a larger multiplication factor; see Section 12.4.2). In the second case, we need to increase the number of iterations relative to the number of samples picked up for the GOF sample.

The “gof() ” routine in statnet works both as an independent convergence check (like phase 3) and GOF (but note that the SEs are calculated in the basic fitting routine “ergm() ”). The properties of the GOF sample may likewise have to be improved by increasing the burn-in, thinning, or sample length. If there is a need to do this, it is assessed using a combination of numerical criteria and graphic inspections of the sample. In general, it is a good idea to graph the sequences of statistics produced by ergm and PNet (see Section 12.2 on simulation).

For statistics not specifically included in our model, however, we are interested to see if their values are extreme. A graph statistic with a t-ratio greater than 2.0 can be regarded as extreme, in which case we infer that it is not plausible that a statistic of that value could have arisen from the model (the extreme value 2.0 is chosen with reference to the approximate critical value of a standard normal variate; for more on this, see Section 12.9). In other words, the model does not represent that graph feature well. In model A, we have a t-ratio of 0.74 for the number of 2-in-stars, which is clearly less than 2.0. Thus, the model reasonably captures the number of 2-in-stars in our observed network, even without including a parameter for it in the model. However, the number of reciprocated ties and the number of transitive triads are not well represented by the model. Because these are rather fundamental features of a directed social network, we conclude that model A is not a very good model for the data.

Table 13.3 lists only a small number of possible graph features that can be examined in a GOF as illustrations. For directed networks, Robins, Pattison, and Wang (2009) suggested examining GOF on the following graph features:

  • Graph counts and associated statistics:
    • - Dyadic statistics: numbers of arcs, reciprocated arcs
    • - Markov graph configuration counts: numbers of 2-in-stars, 2- out-stars, 3-in-stars, 3-out-stars, mixed 2-stars (2-paths), cyclic triads, and transitive triads
    • - Statistics for social circuit parameters: A-in-S, A-out-S, AT-T, AT-U, AT-D, AT-C, A2P-T, A2P-D, A2P-U
  • Degree distributions:
  • - SDs and skewness of both in- and out-degree distributions
  • - Correlation between in- and out-degrees across nodes
  • Closure:
    • - For single triangle closure, global clustering coefficients, that is, 3 x (no. of triangles)/(no. of 2-stars). For directed graphs, there are four ways to complete a single triangle: from a 2- path to complete a 3-cycle (clustering-C); from a 2-path to complete a transitive triad (clustering-T); from a 2-in-star to complete a transitive triad (clustering-U); and from a 2-out-star to complete a transitive triad (clustering-D).
    • - Clustering coefficients based on social circuit parameters: the ratio of the alternating k-triangle statistic to the analogous alternating k-2-path statistic (i.e., this coefficient is a measure of the proportion of alternating k-2-paths that have a base present to complete an alternating k-triangle). This gives an additional four higher-order clustering coefficients, AC-T, AC- D, AC-U, and AC-C, where “AC” stands for “alternating k- cluster.”
  • Geodesic distributions:
  • - Quartiles of the geodesic distribution (labeled G25, G50, G75, and G100, respectively - G50 as the median can be taken as a measure of average geodesic length for the graph). Because some geodesics may be infinite, rather than use means and SDs, and hence t-ratios, Robins, Pattison, and Wang (2009) proposed to examine the percentile of these statistics for the observed graph in the distribution of statistics from the simulated distribution.
  • Triad census:
  • - Counts of the sixteen triad types defined by Holland and Lein- hardt(1970)

These features, along with some others, are included in GOF in PNet and most of them in statnet.

Let’s now look more specifically at the GOF for models A and C in Table 13.3. Model A does not reproduce many of the features of the data. For example, there are 44 reciprocated ties in the data, yet the model suggests a mean of 12.60. The GOF t-ratio for this reciprocity count is 9.01, much higher than 2. Model A substantially underestimates reciprocated ties for our observed network, and the same occurs for transitive triads. However, the count of Markov 2-out-stars is 283, and our model produces a mean of 276.30. The associated GOF t-ratio of 0.16, which is much less than 2, suggested that our model is able to capture the presence of 2-out-stars sufficiently even without specifically including a 2-out-star parameter.

For model C, reciprocity is modeled explicitly as a parameter in the model, and we also see that model C represents the number of transitive triads well (not surprising, given that there is an alternating triangle parameter in the model).

The inclusion of effects for network self-organization (i.e., purely structural effects) alters some of the actor-relation estimates (Table 13.2). For example, the negative sender effect for seniority that was significant for model A is no longer significant in model C. However, the inclusion of the purely structural effects does not “wash away” the actor- relation effects entirely. Although estimates differ slightly, there are still homophily effects for seniority and number of projects in model C as there were in model A. The attributes of the actors and network selforganization tendencies both have a role in explaining the presence of ties in this communication network.

GOF may be helpful in choosing which network configurations to include for model specification. If a feature is not well fitted by a model, a judicious choice of an additional parameter in a new model may help. However, we stress that GOF should not in any way be seen as a replacement to theoretical motivation.

 
Source
< Prev   CONTENTS   Source   Next >

Related topics