»Center for Innovative Learning Technologies
»Find references in our database written by Chris Hoadley
"One of the fundamental changes is that there is an increasing reliance on knowledge-on-tap as opposed to knowledge-in-the-head."
"There are an awful lot of systems that treat knowledge as equivalent to information, and so they try to maximize peoples' exposure to information."
"A key lesson that we need to remember is how much the human touch makes technology and whatever platforms we use successful or not."
"To their credit, NSF has really gone out on a limb in creating new ways to fund or support these kinds of infrastructural needs that were absolutely crucial for increasing the nimbleness, the applicability and success of the fields of research that they supported in the past through short-term studies."
"In today's world, it's not what you know it's what you have access to knowing."
"I think that our evaluation metrics still tend to focus on individuals, partly for historical and partly for logistical reasons. But if we're taking about things like trying to change society, then we really need to look at group-level outcomes."
Interview by Marti Louw :: Fall 2008
Louw: Chris, can you start by telling us about your research interests.
Hoadley: One research focus is in the area of computer-supported collaborative learning (CSCL). I also study how people learn to design environments for collaboration and for learning. I'm really interested in ways that people collaborate to share either science knowledge or personally relevant environmental knowledge, and I have a strong interest in knowledge networking and different forms of institutional knowledge management.
Louw: You seem really active online as a scholar and as an individual. Can you talk about how you use the Internet professionally?
Hoadley: One of the things that I try to invest time in is using technologies that may aid people in improving their experiences and life goals. I personally have been a pretty heavy collaborative technology user both for my own productivity and just to try things out and see what's out there. I use Facebook not only as a way to maintain awareness of friends, but also professional awareness of colleagues and what they're up to. In the past I've tried different technologies for document sharing and shared knowledge repositories. One thing that, as a scholar, I use frequently are different kinds of bibliographic tools and bibliographic sharing tools. And these days, I would say that I'm beginning to have technologies in my life that allow me to do things more portably, like using an iPhone to be able to look things up on the web, or to be able to carry information or documents that are important around with me even when I don't have a laptop. E-mail, obviously, is a really important collaborative technology for me and I also try to stay on top of other kinds of shared resource tools like GoogleDocs or Yahoo groups, or different kinds of ListServs.
Louw: Can you give us a vision of how you'd like to see academia and scholarship change given the new kinds of knowledge-sharing and social networking tools that are coming online? Do you see scholarly publication changing—what will be the same and what will be different?
Hoadley: I think one of the fundamental changes is there is an increasing reliance on knowledge-on-tap as opposed to knowledge-in-the-head. So if we look at scholarship one hundred years ago, you might see a lot of time spent becoming expert on what's in the literature and scholars subscribed to certain journals that they could read as they come out to be up-to-date and maintain cutting-edge of expertise—so a heavy train-up and then staying on top of things as they develop. These days, because of the pace of knowledge development, the size of scholarship in general and because people have less clear domains of expertise or disciplinary bailiwicks, they need to be able to surf into the global knowledge ocean, get what they want, and return to their work. And this is a pretty dramatic change.
Even within the relatively short span of my career, the ways in which people use research literature is changing. It used to be that journals and libraries were the cornerstone of academic discourse. Increasingly conferences and face-to-face interactions are becoming a bigger and bigger part of that interchange. People are using a lot of non-traditional formats to get things around—whether it's taking old formats like print journals and uploading them into searchable databases or blogs and discussion boards where people can query experts to non-traditional forms of publishing like white papers, websites or wikis.
There's a real shift in terms of the ways in which people can extract expertise from other people — it's more on-demand in a socially contextualized way. But, it also means that there are lot of challenges that didn't exist in the past with respect to setting, quality control and personal responsibility for knowledge. It's not that it will necessarily replace the old scholarship model entirely, it's that there are different models cropping up that support these more ad-hoc, in-demand kinds of scholarship needs.
Louw: Are there drawbacks to having knowledge created and distributed in this way? You hear terms like "fragmentation," "dispersion" and the "modularization of knowledge," these kind of concerns...
Hoadley: One thing I've noticed personally is cases where different communities concern themselves with the same thing, but may not be aware of each other's activities in the scholarly world. There are entire fields that basically study the same thing but don't know about each other—and it's fascinating because to a large extent these kinds of schisms happen for old-fashioned sociological reasons that have been with us since the beginning of time. They also happen for more modern reasons, like key words: if a community decides to label themselves using certain words, and people outside the community are googling using other words, they may not stumble across each other. And so the information and library science aspect of it actually have a lot more relevance than people might think because we don't have necessarily perfect ways of determining who out there knows something that we care about.
I think many scholars are very nervous about the idea of 'ad-hocness' being a way that quality crumbles in the scholarly enterprise and that we might lose the kind of cumulativity that research disciplines have attempted to build over the last few centuries. I would say that 'cumulativity' is a bit of an artifice and not necessarily an achievement that we have reached as much as we would like to believe we have. The idea that things are less cumulative, that we do reinvent the wheel, is really just acknowledging something that's always been true which is that knowledge that has been figured out isn't of any use if it's all on shelves and not in heads of people to apply.
So the notion that people can be empowered to become temporary experts or quasi-experts in something that matters to what they're doing actually is very encouraging to me, it's not necessarily scary or disappointing. But it does mean that we have to recognize that it's not some sort of linear march of progress that leads us to a final truth, but rather that people are solving problems and they use different kinds of knowledge to do so. It means that when we think about cumulativity, we need to think about how people are organized to solve problems together. For instance, what does it mean to improve K-12 science education or to make better museum exhibits or even to improve our environmental impact in terms of the energy that we generate and consume. Those kinds of problems can be worked on if they're cumulative or fragmented, and whether the knowledge is published in journals or not almost seems irrelevant to whether those efforts are haphazard or not.
Louw: We have so many terms for describing online communities—learning communities, knowledge-building communities, social networks, communities of practice or purpose or affinity; are there categories or useful dimensions to describe different types of online communities and their characteristics? Is a typology or ecology of online communities emerging?
Hoadley: I would say wholeheartedly yes. In early days of Internet technology, people were very enamored with the idea of somehow 'virtualizing' the ways that people work together. And some of the books from the early '90s really had this gee whiz quality to them that said, "Look! All these things that people do in real life, they're starting to happen online. Who would have thought that people might fall in love or grieve or invent or self-promote or lie online?" And after a while, I think people grew to use online technology in ways that embedded online technologies into communities they had already participated in or allowed them to participate in virtual communities that they hadn't participated in before. But we haven't been very careful about how we describe the nuances of these kinds of online associations.
There's a presumption that if you build it, people will come; if you slap a web board online, then that means everyone's a community and that great things will happen. And I think that having realistic expectations about what is a community — a collection of people, a project team, an ad hoc association, an affinity group — is a very important question. And within the concept of community, are people a community with a shared interest, a shared practice, a shared knowledge and learning goal? Those different kinds of communities have distinct properties, and they may overlap — there may be communities that fit many of these definitions, but they also need to be considered as different from each other.
It's important in for both the research literature and the designers of online communities to keep in mind very clearly what it is we're after when we're trying to foster a certain kind of collaboration or a certain kind of relationship among a group of people. Do we really want people to be the kind of community where they get invited to each other's weddings and funerals? Do we want it to be the kind of community where people can casually get what they need in some very effective way while conducting their regular work practices? And those kinds of questions can help distinguish different sorts of online communities and the different kinds of technologies for supporting communities, whether online or offline.
Louw: Let's talk about an online community you were behind, the Center for Innovative Learning Technologies Knowledge Network (CILTKN) that launched in 1999. What kind of online community was that trying to be and what was the impetus for the site?
Hoadley: In the late '90s, the National Science Foundation was very interested in improving coherence and cumulativity in educational technology research, and in particular they were concerned about different groups, such as industry and academia, not necessarily communicating with each other as much as they would like. As a result, they funded a large project under the Knowledge and Distributed Intelligence program called the Center for Innovative Learning Technologies (CILT). That program conducted many activities that were designed to bring people together. There was a strong interest in making sure there was a way for people to participate in this community — essentially a community of interest or a community of scholarship — that didn't involve face-to-face meeting. Initially, that meant starting with things like online discussion boards. Later, it moved to things that we would now call "knowledge management tools," essentially tools to help organize, store, and retrieve large digital libraries of documents such as might be used within a corporation. And both of those attempts really fell flat. As a result, two different strategies were born out of that failure: one was a sort of quick-and-dirty news feed, called LT-Seek, which was a highly edited series of links related to learning technology (curated by John Rakestraw) that were sent out at regular intervals, and it proved quite successful. And the second one was the CILT Knowledge Network, which I designed. The goals for this online system were to permit the more easy exchange of information that was already of interest and existing in the communities that were stakeholder groups being served by CILT.
We started by doing an information sharing study and looked at what kinds of categories of information scholars need from each other. Some of the key types of information included things like bibliographic references and syllabi for courses. We worked pretty hard at not trying to set up yet another place for people to communicate with each other, because we felt there were already plenty of tools for that and our prior failures had really shown that people didn't want yet another place to discuss things. So instead what we tried to do was create a system that would allow people to access information that other people had in as seamless a way as possible, with as little barrier to locating and utilizing that information as possible.
We worked on two major design principles: one was that all these pieces of information should be collected and disseminated as transparently as possible. The second design principle was, rather than trying to create an artificial social space for people to talk to each other, we should just make sure that, whenever a piece of information was shared, there was some sort of pointer to a human being and that could be used to follow up. So if a bibliographic entry is of interest to me, then that may indicate that the person who suggested that paper is of interest to me, or it may suggest that the author of that paper is of interest to me. And so we tried to make it as easy as possible for people to share information that they were already typing into their own computers, but also for them to do so in a way that other people could contact them about it if needed. It resulted in a pretty successful system.
At its peak, the CILT-KN supported about ten thousand registered users and many more guests and was a pretty successful way for increasing the visibility of the learning technologies community, not just within the stakeholder groups that we initially identified but also to the larger public. This was one of the unexpected outcomes — although the system was initially designed to serve the information needs of scholars first, we found a lot of interest and participation from professional educators and teachers who wanted this kind of knowledge, but didn't have an easy way to locate the relevant information, no way to contextualize it, or attach it to individuals and communities.
Louw: Do you have any general caveats about online communities that try to bridge or support research and practice, which in some ways mirror the academia to industry relationship you mentioned above.
Hoadley: One challenge I faced in other projects has been the degree to which academics in particular forget how what they write is not easily read by people who haven't been trained in the same way as they have. One of the biggest barriers to the world's research on medicine being of use to patients is that the patients can't read the research papers. And there are important differences in the way that things are communicated — there are important skills in being able to convey ideas to a general audience that need to be kept in mind. Another think to watch out for is that people will not, in general, use any tool that doesn't integrate with their practices. So for a tool to be so useful that it really changes peoples' workflow is very rare, and as a result, the tools that seem to be most successful are the ones that really fit into the ways that people live their lives already. Creating a special purpose tool and a special purpose website and a special purpose activity that doesn't fit in with peoples' schedules can be done and it can be 'incentivized' — special purpose conferences or training institutes have been used in the past to do that. But by in large, the things with staying power are the ones that people use every day and the only way you can use something every day is if it fits into your life. So when designing, I think that's a really key constraint or design criterion that needs to be kept in mind.
Louw: An article you co-authored "Using Technology to Transform Communities of Practice" talks about the C4P Framework. Can you tell us about this model and how we might use it to discover evidence of learning?
Hoadley: There were a couple things that were really key about that framework. And I can't take credit for it myself — it's really a product of my student, Pete Kilner, and his colleagues who developed an online community that served tens of thousands of members. In his case, they were professional soldiers. It came to a lot of hard-won knowledge about what it is that makes content valuable. One of the key contributions of that framework is to reconceptualize knowledge as not just stuff that gets written down that's some sort of fact or useful sentence. There are an awful lot of systems that treat knowledge as equivalent to information, and so they try to maximize peoples' exposure to information. And that's true of e-learning systems, knowledge management systems, and online community systems. The C4P Framework identifies that content is meaningless except in relation to the other components of the framework, whether it's the information context, whether it's conversations and discourse practices. At the center of the framework is purpose — people don't talk about things for no reason in particular, they talk about things because they have an agenda and that agenda may be social or personal, or it may be some sort of business purpose or to just have a pleasant interaction with somebody. But purpose is key in figuring out how to interpret what people are talking about. So by taking content and making that just one element of this relational framework, it reminds us that when we're trying to support the creation of meaning and value and learning, that we need to think about all of those components of what makes content valuable. And we need to either directly support those, as Pete and his colleagues did in their system, or we need to think about how those elements are supported outside the system, as we did in CILT-KN. So trying to find a way to ensure that good ideas don't get written down somewhere and then forgotten, or that information really gets transformed into knowledge is key when you're designing systems like these.
Louw: For the CAISE Inquiry Group "Assessing the Impacts of Professional Online Communities for Informal Science Education" we spent two days reviewing and talking about different NSF-funded professional online community websites, many aimed as the ISE community. In one way or another these sites were all designed to support professional development, knowledge sharing, and networking in a field. Any reflections on what you heard?
Hoadley: One thing that comes to mind is how far we've come. We take for granted the sophistication of the tools that we use. It's hard to even remember a time before blogs, much less a time before the web. It was not so long ago that we didn't have these tools at our disposal. Our conversations reminded me of the enormous wealth of tools at our disposal to help us achieve our goals as communities. That doesn't necessarily dilute some of the very important functions that existed long before technology did, like facilitation, leadership, trust building, and some of the human dimensions of communities are still just as much a factor in the success of online communities as they were when we did them face-to-face.
For instance, I remember from the discussion of MSPnet how sophisticated the tools were in supporting certain kinds of information-sharing, but also how a lot of that was made more successful by careful and energetic facilitation by human beings who took the time to get to know who was in the community and who knew what and try to elicit certain kinds of participation that would be beneficial to the group as a whole. A key lesson that we need to remember is how much the human touch makes technology and whatever platforms we use successful or not.
Louw: The NSF seems to be spending a tremendous amount of money on these different kinds of web infrastructures and community-building efforts. Can we advance fields and achieve all of the grand visions these different sites have? Is the right way to build and sustain these online knowledge-sharing communities through top-down federal funding?
Hoadley: My gut-level reaction is it's absolutely necessary if NSF is to achieve their goals, which is broadly speaking the advancement of knowledge. It has been historically difficult for them to support because the NSF was founded on a model that didn't involve these kinds of infrastructural support projects to the degree that they do now. In some ways the technologies themselves are a 'Trojan horse' to allow support for things like professional facilitation and development. If NSF has a model where they issue grants for a limited amount of time to work on particular kinds of research problems, that model is not well mapped to the need for ongoing facilitation, community building and knowledge synthesis.
So, historically, NSF didn't need to support people who wanted to do literature review. They might have supported infrastructure in terms of some large instrument, but they have not had to consider ongoing infrastructural support as a key component of research or of knowledge advancement. And that suited them well for many years, especially given annual budgeting cycles in the federal government. But I do believe that, for their mission to be achieved, we very much need long-term support for synthetic venues — places where people can bring together what they need to be able to figure out what has gone before and how it matters for what comes next.
To their credit, NSF has really gone out on a limb in creating new ways to fund or support these kinds of infrastructural needs that were absolutely crucial for increasing the nimbleness, the applicability and success of the fields of research that they supported in the past through short-term studies. So they're really working on it, but it's still a big disconnect between the traditional model of what it means to support research and what is really required to do what is necessary to improve various fields.
Louw: The NSF/ISE program recently released a new missive in the Framework for Evaluating Impacts of Informal Science Projects to help potential grantees think about how to measure and evaluate project impacts. The framework outlines five impact categories - Knowledge, Engagement, Attitudes, Behavior, Skills, and provides an "Other" category. From a research standpoint, do you have any thoughts about the merits of this framework — how it's good and what it might be missing when thinking about web infrastructure and online community projects?
Hoadley: I think that in general the framework is great, in the way that evaluation research frameworks are great, because it's explicit about saying that different projects have different goals and they need to both be clear about those goals and how they plan to measure progress towards them. Historically, informal science education as a field has had a lot of extra burdens that, for instance K-12 science education didn't have. For one, ISE projects tend to have an interest in outcomes other than factual recall. They may very well have had methodological challenges with collecting data because it's difficult to assess whether a science demonstration in a high school in 1972 had an impact in someone's career choice to become a scientist in 1992. There are a lot of face-valid kinds of outcomes that have allowed the Informal Science Education program area to continue to receive funding despite a lack of hard data proving that the impact that they have had was as substantial as it really has been. Not everything needs to be proven via research — some things can be followed by relatively uncontroversial beliefs such as "museums are important," or "improving attitudes about science in general is important for the national wealth and security." And the framework that we discussed is a good step in that it's asking people to be more explicit about what they care about and it asks them to try to measure it to the degree possible. Where I think it could be dangerous is if it's used as a bludgeon to say, "If you haven't proven something by measuring it through careful research, then you either are failing as an evaluator or failing as an implementer or intervention designer." And one of the things we've seen in K-12 research in that space is that many of the things that are hardest to measure are the things that we value the most, but if the system is geared towards rewarding that which is easily measured, then people will focus on those things that are easy to measure. And I don't think we want to go down that road in Informal Science Education to the degree we already have in K-12 Science Education or in K-12 education in general. It's important that we stick to what our values are even if they're things that are really difficult to measure. But I like the idea of just trying to be data-driven to the degree it's practical.
To some extent, the political will has to be created to say "We believe that the measurement effort, the feedback loops, are worth x percent of our efforts" because obviously somebody who's measuring isn't implementing. So if there's a certain amount of federal funding for the Informal Science Education program, somebody really needs to say, "We think it's a wise use of our money to spend 20% of it on project evaluation or assessment" or "We think it's wise to spend 2% of it on program evaluation and assessment" — how much of it needs to be for the measuring versus the implementing is to a large degree a political decision that I hope everybody participates in.
Louw: What are some interesting ways of thinking about measuring impact in these different categories (knowledge, engagement, attitudes, behavior and skills)? You've talked about social network and document analysis, bibliometrics — our field may not be thinking about some of these methods for evaluating the impacts of online community building and professional development projects.
Hoadley: The ones you just mentioned are some of my favorites that are perhaps overlooked. One of the things that has really helped us in the last fifteen or twenty years in educational research is that we have many more social science research tools at our disposal than we had in the past. And where maybe a hundred years ago the way you might think about measuring impact of some educational intervention would be to give someone a recall-based test, we now know a lot more than we used to about things like performance assessments. We also know a lot more about things like measuring attitudes, motivational research, and studying culture and how people are configured socially. That can include things like looking at collaboration skills or the use of skills to work with other people. It can also include sociometrics, social network analysis - who's talking to whom about what and how much. We can look at the documents created by people, for example research literature, which is the traditional target of bibliometrics. But we can also look at things like document analysis of children's homework papers. We have a lot of tools at our disposal methodologically that we didn't twenty-five years ago, and those tools can be used to help us think creatively about where evidence for outcomes that we care about might crop up. And those outcomes are sometimes elusive when we're talking about things like changes in cultural practice, or popular attitudes toward science, or personal career choice.
We have a lot of things that are tough to measure — they may take a long time, they may occur in places where we don't have ready access to people, they may just be internal to the individual. But we can also use the tools of the trade to think creatively about where there might be evidence to show when something really great is happening.
One thing that has really improved the situation methodologically is that we're getting better at understanding how to integrate research with the designers and designers' sense of success, and I would include teachers as designers in that case — the jazzed-up feeling that someone gets when they're running an activity in a classroom and they know that it's working or the instinctive reaction that someone gets when they create an exhibit and they just know that it's really achieved some design goals that they didn't really know would be there until the exhibit was built and they got to play with it themselves. We're a lot better at integrating those kinds of instincts or personal reactions to interventions with formal research activities, whether it's teacher-research type activities, or design-based research, or even just trying to do really good qualitative research on peoples' experiences through phenomenology. So we have a lot of ways to try to take what we know from the non-formal scholarly ways of knowing and carry them over into the formal, scholarly ways of knowing, through evaluation research and other kinds of studies.
Louw: Is there a case study or a project you could point to as an example of looking at ways of integrating the practitioners, or designers, or teachers' insights into formal research?
Hoadley: I'd have to think about it. One example I might point to is the SIM-CALC project that did a very long trajectory of growing from essentially a design project where interventions were created and vetted and studied very much in the micro, heavily informed by designers' experiences, and then the designers' conclusions about what was working and what wasn't working were vetted and tested in macro through large clinical trials. Even that description doesn't do it justice because it makes it sound like first SimCalc designed, then tested, when in fact the design process was ongoing, the feedback loops included both formal and informal research from day one until the present. And the emphasis has always been on understanding why and how, rather than whether, something is working in teaching math.
Louw: What valuable aspects of online communities might be lost or overlooked in defining impacts in these broad categories of knowledge, attitude, skills, etc.? What might that 'other' category be?
Hoadley: One is social capital. In today's world, it's not what you know it's what you have access to knowing. If you have some really stellar online community where people all of a sudden may not know a lot individually, but they know who to ask when they need some piece of information or some knowledge. That's a very strong gain or impact that may not show up on anything that you can look at individually. Another thing that is really crucial is people's habits and practices — they're one of the hardest things to measure and online communities do have the power to shape an individual's practice but it's very difficult to capture, especially in the timescales in which we usually conduct evaluation research. So being able to identify what the community provides to members of that community in the large is a real challenge for people who are trying to document the impact of various kinds of online interventions.
Louw: During the CAISE Inquiry Group you brought up a set of terms for thinking about knowledge impacts as "know-how," "know-who," and "know-what" and you linked those to different kinds of assessment methods. Can you tell us a more about your thinking on this in terms of an online experience?
Hoadley: Traditionally, we've thought a lot about knowledge-in-the-head in terms of recall and "know-what" is a very important aspect of learning. Recall-based tests often tap this kind of knowledge. But I think we also are beginning to understand what people are capable of doing, and that kind of "know-how" is very important. Sometimes it may be very tacit, sometimes it may include skills that are difficult to measure, like being able to drive a car — skills-based assessment can address that kind of know-how. You could also think about "know how" as something much larger, as in someone designing an investigation or whether someone has a sense of empowerment in their life in terms of career choice. Performance assessments might help here, as might more longitudinal methods. And finally the "know-who" is really related to this issue of twenty-first century scholarship — if it becomes increasingly important for people not to have that knowledge in their head but to have access to it in the social world, then they very much need to know who has what kinds of knowledge, what that knowledge is good for, how to access it, and some subtler things too like how to contextualize that knowledge. It may be that I know exactly the right textbook to go read to locate the answer to a question I have, but it may also be that reading that textbook is useless unless I understand who produced it, who their audience was, and what their community takes granted. I may not be able to interpret that book unless I have a sense of those things. We're really only just beginning to explore how to assess that. But I believe these three types of knowledge are absolutely crucial for understanding the impact of systemic interventions.
A tacit but really important cultural change is that we've moved from the idea that individuals either succeed or fail at school, to a society in which groups of people are measured on their ability to achieve collective goals. For instance, a work team in a company might share in a bonus based on accomplishments none of the team members could have produced individually. And school increasingly is being allowed to make room for some of those skills that allow people to collaborate and work as a team as opposed to skills that involve individual performance on some artificial exam. I think that our evaluation metrics still tend to focus on individuals, partly for historical and partly for logistical reasons. But if we're taking about things like trying to change society, then we really need to look at group-level outcomes.
Louw: What would be some innovative ways to capture these kinds of outcomes and their impact?
Hoadley: One thing that's always fun is the technique of backward design where, even before you start designing some intervention, you clarify what goals you have for the intervention by saying "Well what would it look like if people were really successful at what I care about here? What's the final proof in the pudding that I'd love to see?" And with online communities, you can start brainstorming pretty easily some of the things that you would love to see in the end —that a certain group of people are engaged in certain ways, that they have practices that allow them to draw on each others' strengths, that they are able to draw upon the knowledge of others to accomplish goals that they have that are authentic and important, and that they grow as human beings in a culture that values some of the things that the designer has in mind.
For example, the community might be exhibit designers and we could think about, "What would a motivated and vibrant community of exhibit designers look like if they had the best sort of social supports to be fully empowered and enabled?" And in the same way, we could think about "What would the delighted and engaged version of a family that had just come home from a science museum look like? And what would they look like over time, over the year following their first encountered with the museum?" It opens you up to thinking about outcomes that are relational, not contained within each individual. So we could think about things like attitudinal measures, but we can also think about things like performance assessments, or ethnographies that help measure what people are doing in their everyday lives. And those are all important potential outcomes — some of them may not be practical for a given evaluation study, but at least it's nice to be able to think about them because they will guide the design.
Louw: If you were writing a grant proposal to build an online community for Informal Science Education, what impacts might you try to go after?
Hoadley: I would start with all the easy ones that everybody uses just because they're easy and people would expect them to be in there, like: how many people join the community, how often they log on, their ratings when given a simple "Do you like it or not?" survey. But then I would also think carefully about the purposes of the community—if the purpose of the community is to improve cumulativity among a group of people, then I would think about what cumulativity looks like to that group of people. Do I need to, for instance, start measuring whether they cite each others' papers? Do I need to look at the degree to which they cite research literature at all? Do I need to look at the degree to which they refer to each others' work in informal venues? I might be interested in whether there's increased "know who," so I might start looking at social networks to see whether people are increasingly connected to each other. If there are particular practices I'm hoping to foster through this community, I would try to gauge those practices out in the wild — among members of the audience for this intervention. And I might for instance try to do some kind of baseline study with a sample of people for some activities that I thought were of high value. And then after my intervention, or maybe during the implementation, I'd try to update that baseline to see whether there's progress. I would also think carefully whenever implementing something like that about the timescale of the funding, because if I am hoping that it's a ten-year effort and I'm applying for a three-year grant, I'm going to have to consider outcomes that might actually show up within the first three years.
But I'd also want to consider outcomes that, in ten years, I'll be glad that I started measuring now. If I think it's a three-year grant that will end in three years and not necessarily continue for ten, then I might choose different things to throw into the proposal. For practical reasons it's nice to have a defensive stance in which you can say, "I have enough experience through the literature or through my team or through my personal knowledge that I know that certain things will take place," and measure those things. It's easy as a researcher to think there's no reason to document things that we already know. But I also, in the evaluation space, understand that it's important to have evidence to back up what we already know. So even if it feels redundant, I'd probably collect some things that would show evidence that what I was doing was having at least some of the intended effects — and that might include things like surveys or a simple analysis of certain kinds of participation.
Louw: Log server analysis is a rather blunt tool for understanding why people are coming to a site, what they are doing and how it might be changing their practice. Are there some interesting uses or ways of augmenting log servers and building them into an evaluation plan?
Hoadley: I really wish somebody would write a book on this. I don't feel like an expert on that topic, but I would say the things that I have seen that are most compelling are aggregate statistics that can be used to make generalizations of insights that were documented in other ways. So if someone does some qualitative analysis of a few users and shows that they have a pattern of interacting with the site and then someone else does a very blunt log file analysis and says, "Yes, 62% of the users followed something like that pattern," then that is often very convincing to outsiders. That's the persuasive side of log files. But in terms of insight, I find that log files can be used as prompts for individuals, when you're interviewing them or serving them later, or debriefing them. Another thing that log files can be useful for is to help orient you towards major patterns that wouldn't necessarily be obvious any other way, whether it's where people are spending their time or how people are using the site — what time of day, what location they're using it from — there's a certain amount of exploratory data analysis of log files. But it's difficult to write that into an evaluation plan because that kind of insight is not easy to predict ahead of time. So I love log file analysis, but I also never put it in an evaluation plan.
Louw: On a completely different tack, you've taken leave on a Fulbright that will be looking at communities, technology and sustainability education in developing areas. Can you tell us about that?
Hoadley: I have what's called a South Asia Regional Research Fulbright. Traditionally, Fulbright scholarships allow scholars to spend a school year or maybe less in a foreign country as a way to improve mutual understanding between the peoples of the United States and the peoples of the world. There's a new, small program that allows people to visit multiple countries in a case where that overlaps with the goals of the program. So for the next year I'm going to be in India and Nepal working with rural communities. The goal is environmental sustainability and education related to sustainability. In the developing world, a lot of issues related to sustainability hinge on community-level action, decision-making, or knowledge. So at the village level in some of these places, things like how land is used or what kinds of careers are available for people, what kinds of agriculture are practiced — these things are really pretty much under the community's control more so than under an individual's or a government's. My research is on whether technology can be used to allow different communities, maybe in different regions, or in my case in different countries, to swap stories and learn from each other's sustainability habits and knowledge and challenges in a way that allows people to enhance their own ability to act as a community towards their goals. So we'll be doing some field work with students, children, but also with community members and adults on what the major environmental challenges they're facing in their communities, what options they have or what kinds of actions they've taken in the past to deal with those challenges, and then having them document it and use different kinds of technology to share that experience with people who might be facing similar environmental challenges but in a different social or political or religious or historical context. We're hoping that will be a high leverage way to increase peoples' ability to act together as a community towards their goals, sustainably.