<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi, Steffen -<br>
</p>
<div class="moz-cite-prefix">On 11/14/22 4:08 AM, Steffen Triebel
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:B919946C-C561-41B6-BE27-0C4412CE42DC@icloud.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">Greetings statnet-users,<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I have read Hunter et
al. 2008 (specifically p. 256ff.) about how AIC may not be
the best criterion to evaluate ERGMs and that this is even
more true for BIC. However, while I am also reporting and
discussing the statistics/visual representations estimated
through the gof-function, it is common in my field to report
AIC/BIC values in ERGMs and discuss them.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<p>I think that the 2008 papers sounded an appropriately cautious
note, given what we knew at that time. (I would count myself
heavily on the "cautious" end in that case - we had very little
theory on the matter, and not much experience.) At this point,
however, we know a lot more. I'll give a particular shout out to
Michael Schweinberger, who has done a lot of important theoretical
work in showing that the asymptotic and finite-N concentration
behavior we'd hope to see in ERGMs does (so far) seem to be
present in reasonable cases. There's a lot more to be done there,
but we now know that the dependence problem is less of a barrier
to using conventional approximations than might have been feared.
On the practical side, we also by now have a lot of simulation
results (done by various people in various papers) that again show
that the frequentist properties of the ERGM MLE seem to be pretty
good for reasonable models of the type that people use. This is
also encouraging. With respect to the AIC and BIC, our own
simulation studies have so far indicated that using the BIC based
on nominal degrees of freedom is annoyingly and unreasonably good
for typical ERGMs (or at least, the ones we have looked at). (I
say "annoyingly and unreasonably" because BIC selection often
beats alternatives even for outcomes for which it is not
technically designed, including alternatives lovingly crafted to
be superior for particular model selection goals. My experience
to date has been that it is very hard to beat the BIC for the
sorts of relatively low-dimensional models that we typically use
in the field.) I'm unfortunately unaware of a good published
comparison among model selection schemes (what I am describing
above is unpublished), but that has been our experience so far.<br>
</p>
<blockquote type="cite"
cite="mid:B919946C-C561-41B6-BE27-0C4412CE42DC@icloud.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">I am modeling a large
two-mode network and am a bit puzzled about the AIC/BIC
values, as they are very large. My assumption is that the
size of these values is due to the large network (about 9000
“actors” and 500 “groups”).</span></p>
</div>
</blockquote>
<p>The AIC and BIC are both penalized deviance metrics. Their
underlying rationales are different, but the actual metrics differ
only in the penalty applied to the deviance. For the AIC, this is
2 per model degree of freedom, and for the BIC it is the number of
model degrees of freedom multiplied by the log of the data degrees
of freedom. We do not know the effective degrees of freedom in
typical ERGM settings, but can use the nominal degrees of freedom
(i.e., the number of edge variables) as a proxy; one can show that
this approximation is unlikely to matter much in practice, and
indeed it seems not to in my experience with typical models.
Since the "core" of the metric in both cases is the deviance, you
will see the values become larger when the network is large.
Exactly how much larger will depend on a lot of things, but at
constant density you would usually expect to see the deviance
scale roughly with the square of the number of vertices. (Of
course, the density won't be constant in real life - it will
usually fall as 1/N - but that at least gives you a sense of why
it grows.)<br>
</p>
<blockquote type="cite"
cite="mid:B919946C-C561-41B6-BE27-0C4412CE42DC@icloud.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"> My main question is
what to make of the differences in AIC values. In Kim et al.
(2016), the AIC value of 1196 compared to 1264 is
interpreted as “substantially smaller”. The AIC values in my
models are 123352 versus 125383. I am unsure if the absolute
or relative difference matters: If the absolute difference
matters, then a difference of 2031 would also mean the AIC
is “substantially smaller”. If the relative difference
matters, than the AIC in my models will have reduced around
1/60<sup>th</sup> versus roughly 1/20<sup>th</sup> in the
work I referred in this paragraph.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<p>Generally, it is difficult to talk about "big" or "small"
differences absent some additional context. But some heuristics
are helpful. Under the AIC, a deviance improvement of 2 units is
needed to justify adding another degree of freedom to your model,
so you can heuristically think of the (deviance change)/2 as a
very rough unit of improvement - under appropriate assumptions,
that's how many "noise predictors worth of improvement" you are
seeing. If I add a single parameter and it improves the deviance
by 10, then that's about (under AIC asymptotics) five "minimal
parameters' worth" of improvement. For the BIC, you could use the
log of the data degrees of freedom in a similar way. It should be
stressed that these are <i>heuristics</i>, and should not be
taken too seriously, but can be helpful. One can also consider
the fractional improvements in the deviance (as one does in the
case of the R^2), and some folks do...but these can be tricky to
interpret in practice for binary models. Metrics have been
proposed for such things, but I am not sure that they are all that
useful. In the end, the deviance is important as an objective
function, and penalized deviance metrics are very useful model <i>selection</i>
tools, but usually you'll be better off actually <i>assessing</i>
models by looking at how well they do at reproducing behaviors
that are substantively important. For ERGMs, the gof() function
is a starting point for that (though, in any given application,
one may want to use other tests). <br>
</p>
<p>Hope that helps,</p>
<p>-Carter<br>
</p>
<blockquote type="cite"
cite="mid:B919946C-C561-41B6-BE27-0C4412CE42DC@icloud.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Thanks for any advice on
this,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Steffen<o:p></o:p></span></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
statnet_help mailing list
<a class="moz-txt-link-abbreviated" href="mailto:statnet_help@u.washington.edu">statnet_help@u.washington.edu</a>
<a class="moz-txt-link-freetext" href="https://urldefense.com/v3/__http://mailman13.u.washington.edu/mailman/listinfo/statnet_help__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OApk5DNxXp5qh_yjXWz1Vig3IveoxpTNy_P5SoENL-pUzh1j3G7u9cRKe7t5VTEjcYTSJ1MIGrR70_BFNVhg9wU$">https://urldefense.com/v3/__http://mailman13.u.washington.edu/mailman/listinfo/statnet_help__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OApk5DNxXp5qh_yjXWz1Vig3IveoxpTNy_P5SoENL-pUzh1j3G7u9cRKe7t5VTEjcYTSJ1MIGrR70_BFNVhg9wU$</a>
</pre>
</blockquote>
</body>
</html>