[statnet_help] A question about AIC/BIC in ERGMs

[Carter T. Butts]

Hi, Steffen -

On 11/14/22 4:08 AM, Steffen Triebel wrote:

>

> Greetings statnet-users,

>

> I have read Hunter et al. 2008 (specifically p. 256ff.) about how AIC 

> may not be the best criterion to evaluate ERGMs and that this is even 

> more true for BIC. However, while I am also reporting and discussing 

> the statistics/visual representations estimated through the 

> gof-function, it is common in my field to report AIC/BIC values in 

> ERGMs and discuss them.

>

I think that the 2008 papers sounded an appropriately cautious note, 
given what we knew at that time.  (I would count myself heavily on the 
"cautious" end in that case - we had very little theory on the matter, 
and not much experience.)  At this point, however, we know a lot more.  
I'll give a particular shout out to Michael Schweinberger, who has done 
a lot of important theoretical work in showing that the asymptotic and 
finite-N concentration behavior we'd hope to see in ERGMs does (so far) 
seem to be present in reasonable cases.  There's a lot more to be done 
there, but we now know that the dependence problem is less of a barrier 
to using conventional approximations than might have been feared. On the 
practical side, we also by now have a lot of simulation results (done by 
various people in various papers) that again show that the frequentist 
properties of the ERGM MLE seem to be pretty good for reasonable models 
of the type that people use.  This is also encouraging.  With respect to 
the AIC and BIC, our own simulation studies have so far indicated that 
using the BIC based on nominal degrees of freedom is annoyingly and 
unreasonably good for typical ERGMs (or at least, the ones we have 
looked at).  (I say "annoyingly and unreasonably" because BIC selection 
often beats alternatives even for outcomes for which it is not 
technically designed, including alternatives lovingly crafted to be 
superior for particular model selection goals.  My experience to date 
has been that it is very hard to beat the BIC for the sorts of 
relatively low-dimensional models that we typically use in the field.)  
I'm unfortunately unaware of a good published comparison among model 
selection schemes (what I am describing above is unpublished), but that 
has been our experience so far.


> I am modeling a large two-mode network and am a bit puzzled about the 

> AIC/BIC values, as they are very large. My assumption is that the size 

> of these values is due to the large network (about 9000 “actors” and 

> 500 “groups”).

>

The AIC and BIC are both penalized deviance metrics.  Their underlying 
rationales are different, but the actual metrics differ only in the 
penalty applied to the deviance.  For the AIC, this is 2 per model 
degree of freedom, and for the BIC it is the number of model degrees of 
freedom multiplied by the log of the data degrees of freedom.  We do not 
know the effective degrees of freedom in typical ERGM settings, but can 
use the nominal degrees of freedom (i.e., the number of edge variables) 
as a proxy; one can show that this approximation is unlikely to matter 
much in practice, and indeed it seems not to in my experience with 
typical models. Since the "core" of the metric in both cases is the 
deviance, you will see the values become larger when the network is 
large. Exactly how much larger will depend on a lot of things, but at 
constant density you would usually expect to see the deviance scale 
roughly with the square of the number of vertices.  (Of course, the 
density won't be constant in real life - it will usually fall as 1/N - 
but that at least gives you a sense of why it grows.)


> My main question is what to make of the differences in AIC values. In 

> Kim et al. (2016), the AIC value of 1196 compared to 1264 is 

> interpreted as “substantially smaller”. The AIC values in my models 

> are 123352 versus 125383. I am unsure if the absolute or relative 

> difference matters: If the absolute difference matters, then a 

> difference of 2031 would also mean the AIC is “substantially smaller”. 

> If the relative difference matters, than the AIC in my models will 

> have reduced around 1/60^th versus roughly 1/20^th in the work I 

> referred in this paragraph.

>

Generally, it is difficult to talk about "big" or "small" differences 
absent some additional context.  But some heuristics are helpful.  Under 
the AIC, a deviance improvement of 2 units is needed to justify adding 
another degree of freedom to your model, so you can heuristically think 
of the (deviance change)/2 as a very rough unit of improvement - under 
appropriate assumptions, that's how many "noise predictors worth of 
improvement" you are seeing.  If I add a single parameter and it 
improves the deviance by 10, then that's about (under AIC asymptotics) 
five "minimal parameters' worth" of improvement.  For the BIC, you could 
use the log of the data degrees of freedom in a similar way.  It should 
be stressed that these are /heuristics/, and should not be taken too 
seriously, but can be helpful.  One can also consider the fractional 
improvements in the deviance (as one does in the case of the R^2), and 
some folks do...but these can be tricky to interpret in practice for 
binary models.  Metrics have been proposed for such things, but I am not 
sure that they are all that useful.  In the end, the deviance is 
important as an objective function, and penalized deviance metrics are 
very useful model /selection/ tools, but usually you'll be better off 
actually /assessing/ models by looking at how well they do at 
reproducing behaviors that are substantively important.  For ERGMs, the 
gof() function is a starting point for that (though, in any given 
application, one may want to use other tests).

Hope that helps,

-Carter


> Thanks for any advice on this,

>

> Steffen

>

>

> _______________________________________________

> statnet_help mailing list

> statnet_help at u.washington.edu

> https://urldefense.com/v3/__http://mailman13.u.washington.edu/mailman/listinfo/statnet_help__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OApk5DNxXp5qh_yjXWz1Vig3IveoxpTNy_P5SoENL-pUzh1j3G7u9cRKe7t5VTEjcYTSJ1MIGrR70_BFNVhg9wU$  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/statnet_help/attachments/20221114/69543a62/attachment.html>