Quantcast

Anti-NetLogo bias in research funding?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Anti-NetLogo bias in research funding?

David Earnest
To the community of NetLogo researchers:

I am curious whether the community perceives that funding agencies tend not to fund projects that propose using NetLogo as a research tool. Currently my colleagues and I are preparing an NSF grant proposal responding to a CFP that specifies agent-based modeling as a desireable method. Yet one colleague asserted that NSF reviewers tend to view NetLogo models as "toy models" rather than validated, useable research models.

I have heard this criticism before--the ABM community either implies or explicitly states that "serious" ABM research uses RePast or MASON. To the degree reviewers hold this opinion, then, proposals specifying NetLogo as the toolkit of choice may not survive the process of peer review.

Has anyone had similar experiences or share these impressions? If so, what are the methodological and/or technical objections to NetLogo?

Personally, I have used NetLogo to implement some pretty advanced modeling methodologies (e.g. genetic algorithms). I have used it in my research for the last 13 years. I have published studies in which I used NetLogo to model social scientific phenomena. I have taught ABM using NetLogo on three continents. I argue that NetLogo's low learning threshold and large community of users give it considerable advantages over RePast/ReLogo or MASON in terms of the transmissibility and reproducibility of findings--which of course is the foundation of science.

Against these advantages for NetLogo, however, I have heard others argue that RePast and MASON scale more easily to very large simuations (e.g. more than 10,000 agents). I have also noticed that RePast can pass simulation data relatively easily to R, ORA, Pajek, MatLab and other analytic software.

None of this, however, convinces me that RePast is somehow a more "serious" research tool than NetLogo. Is the preference for RePast simply a conceit, then?

I sometimes wonder if the advent of NetLogo is analogous to the translation of the Bible from Latin to the vernacular: when the tool is available to the masses, the high priests lose their authority . . .

David C. Earnest
Old Dominion University
Norfolk, Virginia

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

Michael H. Agar
Hi David. Just had that experience with a proposal to NIH from a group I advise who want to use ABM to look at clinic management, medical teams and patient outcomes. An article on prior work is in the current issue of JASSS, Luci Leykum lead author. We responded like this:

First we gave a little history describing the earlier simple forms of "Logo" and then early "Starlogo" and saying many (ahem) people thought that was Netlogo now.

Second, we used Nigel Gilbert's overview

http://www.amazon.com/Agent-Based-Models-Quantitative-Applications-Sciences/dp/1412949645/ref=sr_1_1?s=books&ie=UTF8&qid=1349959882&sr=1-1&keywords=nigel+gilbert

where he evaluated Netlogo against more traditional modeling languages and it came out pretty well, then point out that it was based on version 3.X.

Third, we said that if you jumped into the present with version 5.X, it was now used as the software basis for a graduate level introduction to ABMs, as in

http://www.amazon.com/Agent-Based-Individual-Based-Modeling-Practical-Introduction/dp/0691136742/ref=sr_1_1?s=books&ie=UTF8&qid=1349959809&sr=1-1&keywords=netlogo

Don't know the response yet, but that's what we tried. Amazed me to see the comment because there was already a published model that several readers had asked the author about and the reviewer's comment was devoid of specifics and, to use a technical term from one of the younger team members, "snarky."

Hope that helps.


Mike Agar
www.ethknoworks.com





On Oct 10, 2012, at 9:52 AM, David Earnest wrote:



To the community of NetLogo researchers:

I am curious whether the community perceives that funding agencies tend not to fund projects that propose using NetLogo as a research tool. Currently my colleagues and I are preparing an NSF grant proposal responding to a CFP that specifies agent-based modeling as a desireable method. Yet one colleague asserted that NSF reviewers tend to view NetLogo models as "toy models" rather than validated, useable research models.

I have heard this criticism before--the ABM community either implies or explicitly states that "serious" ABM research uses RePast or MASON. To the degree reviewers hold this opinion, then, proposals specifying NetLogo as the toolkit of choice may not survive the process of peer review.

Has anyone had similar experiences or share these impressions? If so, what are the methodological and/or technical objections to NetLogo?

Personally, I have used NetLogo to implement some pretty advanced modeling methodologies (e.g. genetic algorithms). I have used it in my research for the last 13 years. I have published studies in which I used NetLogo to model social scientific phenomena. I have taught ABM using NetLogo on three continents. I argue that NetLogo's low learning threshold and large community of users give it considerable advantages over RePast/ReLogo or MASON in terms of the transmissibility and reproducibility of findings--which of course is the foundation of science.

Against these advantages for NetLogo, however, I have heard others argue that RePast and MASON scale more easily to very large simuations (e.g. more than 10,000 agents). I have also noticed that RePast can pass simulation data relatively easily to R, ORA, Pajek, MatLab and other analytic software.

None of this, however, convinces me that RePast is somehow a more "serious" research tool than NetLogo. Is the preference for RePast simply a conceit, then?

I sometimes wonder if the advent of NetLogo is analogous to the translation of the Bible from Latin to the vernacular: when the tool is available to the masses, the high priests lose their authority . . .

David C. Earnest
Old Dominion University
Norfolk, Virginia




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

grimm.volker
In reply to this post by David Earnest


Hi David,

I am not aware of any research proposal in Germany/Europe that was rejected because NetLogo was included as the software platform to be used. However, you are right, NetLogo still seems to have - at least in ecological modelling - the reputation of being only good for teaching, not for "serious" science. This might have been true to some degree in the early years of NetLogo, but definitely no more for at least five years (actually, since NetLogo no longer tried to mimic concurrency of processes).

The reasons for the wrong perception of NetLogo might include its history as a teaching tool, its sometimes Kindergarten-like look-and-feel ("turtles", shape editor), the focus of the Models Library on animation and fancy pattern formation ("video games"), and the lack of examples in the library which show how you actually do science using NetLogo (I admit that it would be difficult to do so).

The real potential limitations to use NetLogo for science are: it can be slow, especially if one uses, without careful consideration, primitives which look convenient, like "with" "-of", "in-radius", but can be very costly in terms of runtime (see how using "with" was avoided in the Segregation model in the Models library to save computation time). NetLogo also has no integrated debugger, which often requires to debug code like in the 1970s, using "print" statements. And, the interface is just too small for more complex models.

Nevertheless, in the projects I was involved with where NetLogo was used, these limitations where not relevant. Still, I guess it would help a lot to change the wrong perception of NetLogo in some domains, and just make it even more useful, if runtime would come closer to, e.g., C++ programs, if more advice would be given of how to speed up programs, if a debugger existed, and if one would have a tabbed interface. I am not a computer scientist, though, so I don't know if any of this is possible and how much effort it would require.

In the meantime, I suggest to include in research proposals a paragraph stating that NetLogo has matured from a tool primarily for teaching to one well-suited for science, that you are aware of runtime limitations but believe they are not relevant for your project, that the advantages, in particular much faster prototyping, and the extensive toolbox for visual debugging, are important for your project. You might cite a few scientific publications based on NetLogo programs in your field (just look for Wilensky 1999 in Web of Science). You also might include a demonstration prototype of your intended model in your proposal, including runtime information, but then definitely avoid screenshots of the NetLogo interface or View.

And, you might mention that even if NetLogo runs, say, 10 times slower than an alternative implementation, who cares in the times of computing clusters?

NetLogo has actually been linked to Mathematica and, more recently, to R (see the R-Extension and RNetLogo developed by Jan Thiele).

You might also cite the papers by Steve Railsback and Steve Lytinen and colleagues, who compared NetLogo, Swarm, Repast, Mason, etc. some years ago. Their more recent paper found that NetLogo can be 20 times faster than ReLogo, the Repast implementation of Logo (Repast itself might still be faster, of course).

I hope this helps.

Best regards,

Volker


--- In [hidden email], "David Earnest" <davidearnest44@...> wrote:

>
> To the community of NetLogo researchers:
>
> I am curious whether the community perceives that funding agencies tend not to fund projects that propose using NetLogo as a research tool. Currently my colleagues and I are preparing an NSF grant proposal responding to a CFP that specifies agent-based modeling as a desireable method. Yet one colleague asserted that NSF reviewers tend to view NetLogo models as "toy models" rather than validated, useable research models.
>
> I have heard this criticism before--the ABM community either implies or explicitly states that "serious" ABM research uses RePast or MASON. To the degree reviewers hold this opinion, then, proposals specifying NetLogo as the toolkit of choice may not survive the process of peer review.
>
> Has anyone had similar experiences or share these impressions? If so, what are the methodological and/or technical objections to NetLogo?
>
> Personally, I have used NetLogo to implement some pretty advanced modeling methodologies (e.g. genetic algorithms). I have used it in my research for the last 13 years. I have published studies in which I used NetLogo to model social scientific phenomena. I have taught ABM using NetLogo on three continents. I argue that NetLogo's low learning threshold and large community of users give it considerable advantages over RePast/ReLogo or MASON in terms of the transmissibility and reproducibility of findings--which of course is the foundation of science.
>
> Against these advantages for NetLogo, however, I have heard others argue that RePast and MASON scale more easily to very large simuations (e.g. more than 10,000 agents). I have also noticed that RePast can pass simulation data relatively easily to R, ORA, Pajek, MatLab and other analytic software.
>
> None of this, however, convinces me that RePast is somehow a more "serious" research tool than NetLogo. Is the preference for RePast simply a conceit, then?
>
> I sometimes wonder if the advent of NetLogo is analogous to the translation of the Bible from Latin to the vernacular: when the tool is available to the masses, the high priests lose their authority . . .
>
> David C. Earnest
> Old Dominion University
> Norfolk, Virginia
>


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

SethTisue
Administrator
>>>>> "volkergrimm" == volkergrimm  <[hidden email]> writes:

 volkergrimm> You might cite a few scientific publications
 volkergrimm> based on NetLogo programs in your field (just look for
 volkergrimm> Wilensky 1999 in Web of Science).

see also
http://ccl.northwestern.edu/netlogo/references.shtml

additions to this list are extremely welcome -- please send them
to [hidden email]

--
Seth Tisue | Northwestern University | http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

fsondahl-2
In reply to this post by grimm.volker
> You might cite a few scientific publications based on NetLogo programs in your field (just look for Wilensky 1999 in Web of Science).

Another good place to find published academic work that has used Netlogo is:

http://ccl.northwestern.edu/netlogo/references.shtml

(Also, if you have published something that used NetLogo that's not already on this list, please contribute a citation...)

Cheers,

~Forrest

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

Bill Rand
In reply to this post by David Earnest
Hi David,

  Sorry its taken me a couple of days to respond, but I wanted to address
this issue.  I have received funding from the NSF, DARPA, Google, and the
Marketing Science Institute for proposals that clearly mention NetLogo
during the proposal (usually they prominently feature it), and in the NSF
case this has been multiple times.  Moreover, in the context of the NSF,
these proposals have been directly to the CISE division, which is the
division that probably has the best capability of judging the quality of a
simulation framework. Even in the context of my vast number of rejected
proposals :), I personally have never had a reviewer mention to me that
they thought NetLogo was ill-suited for modeling.

  I am aware, and run into regularly, people who think that NetLogo is
ill-suited for research.  I'm not sure where this misperception comes from,
it could be the teaching background of Logo as Volker mentions, or it could
be people's incorrect belief that anything with good graphics and good
documentation can't be good for research :)  Regardless, the argument that
always wins people over in a one-on-one context is the lifecycle to
development argument.  Most people will concede that you can develop models
quicker in NetLogo than just about any other framework out there, and so
the total time to results is often less than most other approaches. And as
Volker mentions if the runtime takes 2 or even 10 times as long that's
largely irrelevant because that's computer time, which is cheap, not my
time, which is expensive :)

  Anyway, just my two cents,

Take care,
Bill

On Wed, Oct 10, 2012 at 11:52 AM, David Earnest <[hidden email]>wrote:

> **
>
>
> To the community of NetLogo researchers:
>
> I am curious whether the community perceives that funding agencies tend
> not to fund projects that propose using NetLogo as a research tool.
> Currently my colleagues and I are preparing an NSF grant proposal
> responding to a CFP that specifies agent-based modeling as a desireable
> method. Yet one colleague asserted that NSF reviewers tend to view NetLogo
> models as "toy models" rather than validated, useable research models.
>
> I have heard this criticism before--the ABM community either implies or
> explicitly states that "serious" ABM research uses RePast or MASON. To the
> degree reviewers hold this opinion, then, proposals specifying NetLogo as
> the toolkit of choice may not survive the process of peer review.
>
> Has anyone had similar experiences or share these impressions? If so, what
> are the methodological and/or technical objections to NetLogo?
>
> Personally, I have used NetLogo to implement some pretty advanced modeling
> methodologies (e.g. genetic algorithms). I have used it in my research for
> the last 13 years. I have published studies in which I used NetLogo to
> model social scientific phenomena. I have taught ABM using NetLogo on three
> continents. I argue that NetLogo's low learning threshold and large
> community of users give it considerable advantages over RePast/ReLogo or
> MASON in terms of the transmissibility and reproducibility of
> findings--which of course is the foundation of science.
>
> Against these advantages for NetLogo, however, I have heard others argue
> that RePast and MASON scale more easily to very large simuations (e.g. more
> than 10,000 agents). I have also noticed that RePast can pass simulation
> data relatively easily to R, ORA, Pajek, MatLab and other analytic
> software.
>
> None of this, however, convinces me that RePast is somehow a more
> "serious" research tool than NetLogo. Is the preference for RePast simply a
> conceit, then?
>
> I sometimes wonder if the advent of NetLogo is analogous to the
> translation of the Bible from Latin to the vernacular: when the tool is
> available to the masses, the high priests lose their authority . . .
>
> David C. Earnest
> Old Dominion University
> Norfolk, Virginia
>
>  
>



--
Asst. Prof. in Marketing, Decision, Operations & Information Technology,
and Computer Science
Director, Center for Complexity in Business, http://www.rhsmith.umd.edu/ccb/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Anti-NetLogo bias in research funding?

SethTisue
Administrator

I'm only speculating, but perhaps some people have outdated beliefs that
working in NetLogo is more limiting than it really is.  Certainly
limitations exist when working only in pure NetLogo, but many of those
limitations disappear if you're willing to code to our API's in Java or
Scala.

For example, if NetLogo's GUI for interacting with models isn't
featureful enough or customizable enough for your project, you can use
our controlling API to embed parts of the NetLogo GUI in a custom GUI.
Or, you can discard our GUI entirely and run the model headless, in the
background, with a fully custom GUI on top of it that you build
yourself.

Or if the NetLogo language is lacking in features or speed you need, in
many cases this can be remedied by writing an extension using our
extensions API.

Admittedly, using our APIs requires knowledge of Java (or Scala).  Many
NetLogo users don't have that knowledge.  But that same knowledge is
needed anyway in order to use Repast or MASON at all, so I think it's
most fair to compare those toolkits not against pure NetLogo, but
against NetLogo plus its APIs.

Finally, now that NetLogo is open source software, that removes a major
barrier that once existed to modelers making effective use of our APIs.
NetLogo hasn't been open source very long yet (just one year now, almost
to the exact day), and it takes time for perceptions to change.

--
Seth Tisue | Northwestern University | http://tisue.net
developer, NetLogo: http://ccl.northwestern.edu/netlogo/
Loading...