Monthly Archives: May 2012

Badge System Design: what we talk about when we talk about validity

Every day we conduct conversations with folks new to the idea of Open Badges. Each of these conversations is steeped in inquisitiveness. Questions abound. Curiosity spills out. Thought waves feel palpable. Sometimes we’re lucky enough to share the moment when the light goes on. That time feels magical, full of promise. That moment illuminates the room with the thousand-watt possibilities of the Open Badge initiative. The “what if” moment is something that should be experienced by everyone.

In some ways, the Open Badges team is continually explaining what we do—and not only what we do but why we do the things the way we do them. This constant questioning could affect us in a few ways: we could hunker down and stick to our prepared statements, we could challenge the folks who question our work, or we could listen closely to these queries and try to parse the spaces between the words, the silent places where no content yet exists. These interregnums help us to interrogate our own understanding of Open Badges. It’s probably fair to say that we do all of these, yet we try to keep focused on the last of these techniques because they provide the greatest opportunity for growth—not only for us but for our conversationalists as well.

The question of validity is posed fairly commonly.* It goes something like this, “How can we ensure that the badges have a sense of validity?” or “Who will vet them?” or “How will we know that they’re worthwhile badges issued from reputable sources?”

There is a good deal of subtext embedded in these seemingly simple questions. And bound into that subtext is an unwitting/unacknowledged acceptance of the sociocultural status quo. That tacit acceptance should be unpacked and considered. How does any organization achieve validity? How do standards become standards? When the landscape is unknown, how do you learn to trust anything?

Validity
Validity addresses the question of representational accuracy: does something perfectly represent the thing it’s allegedly designed to represent? In the case of Open Badges, the question of validity quickly becomes multifaceted. Questions that have arisen include the following:

  • Does a particular badge represent appropriate learning?
  • To whom is the badge meaningful?
  • Does the issuer have the authority to issue a particular badge?
  • Does the earning of a badge indicate that the learner has learned?
  • Does the earning of a badge indicate that the earner has been accurately assessed?

To bring a different perspective to these questions, let’s replace the word badge with the word class. This should provide some insight into how much our unquestioning acceptance of the status quo affects our acceptance of learning validity.

  • Does the taking of a particular class represent appropriate learning?
  • To whom is the completion of a class meaningful?
  • Does anyone have the authority to teach a particular class?
  • Does the completion of a class indicate that the learner has learned what they were supposed to?
  • Does the completion of a class indicate that the learner has been accurately assessed?

This simple exercise exposes our somewhat complicated relationship with understandings of validity with regards to existing institutions. Taking this a step further, imagine if the rules in an educational system were entirely reconsidered, to wit:

To demonstrate the power of rules, I like to ask my students to imagine different ones for a college. Suppose the students graded the teachers, or each other. Suppose there were no degrees: you come to college when you want to learn something, and you leave when you have learned it. Suppose tenure were awarded to professors according to their ability to solve real-world problems, rather than publishing academic papers. Suppose a class was graded as a group, instead of as individuals. (Meadows, 1999, p. 14)

Did those questions shift your perspective? I know that they resonated deeply within my consciousness. They’re the equivalent of “what if everything you knew about education were turned on its head?” questions. They’re disorienting in the best possible way. And they lead me to humbly suggest that in place of questions about validity—or at least hand in hand with these sorts of questions—we might consider asking questions about credibility and reliability, too. These areas seem to be more readily delineated and a tad more easily unpacked.

Credibility 
Credibility inspires belief and is derived from perceptions of trustworthiness and expertise. These things can be assessed through personal means but quite often are accepted tacitly. How so? Through the cultural shorthand of pre-existing standards. We countenance many sociocultural values with little to no deep consideration, i.e., everyone was doing it, I just followed the crowd, etc. Let’s consider some ways that we might be able to classify what we mean when we talk about credibility.

Expanding the idea into a taxonomy, B.J. Fogg proposes the following four types of of credibility: presumed, surface, reputed, and earned (Fogg, 2003). Presumed credibility arises from “general assumptions in the mind of the perceiver,” Surface credibility from “simple inspection or initial firsthand experience,” Reputed credibility occurs through “third-party endorsements, reports or referrals,” and Earned credibility, perhaps the most important in a new system, stems from “firsthand experience that extends over time” (Fogg, 2003, p. 131).

While we can negotiate the definitions, this basic structure brings order to the chaos of credibility, and it helps to elucidate our complicated understanding of validity. This categorization also allows us to interrogate the credibility of existing systems, and in particular, the formal system of education currently found within the United States. Here’s where Open Badges provides us an opportunity to intervene in a significant system.

Reliability
Reliability might be considered the replicability quotient of an event, idea, performance, etc. Something that can be consistently measured is considered reliable. The Open Badge Infrastructure is most certainly a reliable tool: it will produce badges that hew to its standards. However, the badge systems that are produced with that tool or housed in that tool may prove to be reliable, but then again, they may not. And yet, this dichotomy is true of any tool. In the right hands, bad tools can produce good results and in the wrong hands good tools can produce bad results. Skill is necessary and happily, it can be learned. From a systems standpoint, the US education system is also just such a tool, producing “products” of varying completeness and quality. This perceptual double standard should inform our questioning of new systems, especially one as reconstructive as Open Badges might prove to be.

The known and the unknown
There are many questions that the Open Badges initiative seeks to answer and many more that its implementation raises. Right now, we’re completely comfortable operating in the liminal space between the known and the yet-to-be-discovered, the present and the future, the understood and the ambiguous. The design of the Open Badge Infrastructure offers solutions to a number of questions regarding validity. For other questions we should ask ourselves, what is the purpose of this query? Am I expecting an answer that will only serve to reinforce a complicated and difficult but familiar system? We do ourselves a disfavor if we accept the current state of affairs without asking ourselves the following: “What exists here now?”, “What is worth keeping?”, and “What can be improved?” Those are precisely the questions that Open Badges Issuers, Earners and Displayers seek to answer themselves.

- – -

More soon.

references
Fogg, B. J. (2003). Persuasive technology. San Francisco, CA: Morgan Kaufmann.
Meadows, D. (2008). Thinking in systems. White River Junction, VT: Chelsea Green Publishing

For now let’s skip the reasons as to why this question arises from some audiences more than it does others. Although I’m happy to discuss it if you would like—you simply have to ask and away we’ll go.

Badge System Design: seven ways of looking at a badge system

Badge system design can be considered in a variety of ways. I tried to come up with thirteen ways to discuss them  so I could write a poem riffing on one of my favorite poems, Thirteen Ways of Looking at a Blackbird (Thirteen Ways of Looking at a Badge System) but I’ve had to settle for seven eight (see addendum below).

Below you’ll find the seven different possible categorizations listed with a few representations of each type of thinking. This is not an exhaustive list by any means: it’s simply an opportunity to unpack our influences and perceptions as we begin the process of designing badge systems.

The methods outlined below include philosophical, conceptual, pedagogical, visual (aesthetic), technical, categorical, and ownership. The last one, ownership, feels a bit odd because it’s not quite parallel to the rest of the bunch. I like a system that has a nice balance and this one has a slight imbalance. Happily, this slightly odd fit serves to emphasize the importance of allowing for an outlier. The outlier will cause you to reconsider your system every time—and that’s a good thing. The outlier is the thing that keeps your badge system honest, keeps it moving and evolving. Because if you’re designing a system so as to keep everyone within a certain range, you’re trying too hard. And you’re deep in the midst of a lush forest.

In any case, I’m curious to hear your reaction to these potential sorting efforts. No doubt these groupings can intermixed and most certainly they can be layered, possibly interleaved with one another.

philosophical

  • representation: understood vs. hidden
  • social acceptance vs. formal acceptance
  • intellectual property vs. copyright free
  • cognitive surplus vs. waste of time
  • extrinsic vs. intrinsic
  • carrot vs. stick
  • top down vs. bottom up

conceptual

  • possession
  • systems design vs. emergence
  • corporate vs. academic
  • amateur vs. professional
  • rhythmic vs. erratic

pedagogical

  • education vs. learning
  • assessment
  • teaching vs. perceiving/absorbing/
  • injection vs. osmosis
  • project based vs standards based
  • expert-taught vs. peer learned & assessed

visual/aesthetic

  • representational vs. abstract
  • categorical vs. individual

technical

  • siloed vs. shared
  • open vs. proprietary
  • system vs. single

categorical

  • formalized vs. free for all
  • few categories vs. many

ownership

  • organizational vs. personal
  • owned vs. shared

Are there additional ways to consider the design of badge systems? Do any of these seem innate? Far-fetched? What do we gain by sorting through systems in this way? I continue working on questions like these and look for your feedback (which, according to Donella Meadows, is a good way to ensure that your system is running smoothly).

Still, I have to try it.
With apologies to Wallace Stevens

VIII
I know noble accents
And lucid, inescapable rhythms;
But I know, too,
That the Open Badge is involved
In what I know.

- – -

More soon.

May 23, 2012 addendum: Recent thinking points to the fact that these categories exclude content. So now there are 8 ways to sort through badge system design. Some possible representations of that categorization include: language choice; content-driven vs. context-driven; formal vs. informal; system vs. one-off; single language vs. multiple languages; alliterative vs. rhyming vs. allusion-based, etc. 

Badge System Design: beyond a binary approval system

For those who labor long and hard to craft good and just standards, as well as those who have suffered from their absence. On the one hand, the fight against the tyranny of structurelessness. On the other, the fallacy of one size fits all  (Lampland & Starr, 2009).

This book dedication found in Standards and Their Stories captures the inherent paradox of badge system design. By seeking to standardize the process we risk the introduction of systemic rigidity. And yet by developing badges without a plan we risk the possibility of ideological entropy. In my writing about this topic I’m attempting to walk the middle path: somewhere in between fanatical dictums and a mad free-for-all. I wish I could say that it was easier than this, but then I’d be lying.

The status quo
Even while we’re in the midst of talking about a potentially reconstructive idea like Mozilla Open Badges, I still rather rotely refer to my own typically conventional educational route with “my undergrad degree this” or “my grad degree that.” Perhaps this is to be expected. It certainly hearkens to one of the issues that the open badges in the wild will have to confront: the seeming intractability of the status quo. In the Open Badges world this desire for stability echoes within the repeated request for a standard method of validation; it’s mated to a deep concern about badge quality. In unfamiliar situations such as these we tend to rely on current cultural understandings and touchstones. In this case, degrees and certificates, accreditation systems and educational rankings.

The status quo of our formal academic system has transmogrified into a sort of binary approval system. You pass or you fail. You go to a respected school or you go to a second-tier school. You graduate or you don’t. It all seems pretty inexorable. We gravitate toward that which is customary. The familiar often appears to be less threatening than the entirely unknown. Indeed, there is a robust academic research field that studies this tendency, especially with regards to our proclivities toward risk and reward: behavioral economics. (For a deep and delightful dive on this read Dan Ariely’s Predictably Irrational.)

Resonance
I’m hoping that some day people will refer to not only their formal schooling but their non-traditional learned experiences as well (hopefully badged in the open way) without speaking of one of them as second-rate or less than the other. That noted, I’ll return to my rather classical undergraduate education to make a point. I double-majored in graphic design and writing. The classes I took in design inform a significant amount of the way that I think. This is not to say that every design class I took made sense or built on every preceding design class so that one day I had taken enough of them to—ta-dah!—be called a designer. On the contrary, I gleaned information from a variety of sources. My deep learning occurred in many different venues, a bit of it very much outside the realm of what typically would be called design. Nevertheless, some aspects of design that I learned in those college classes continue to reverberate within me.

One of the most resonant aspects of those years pertains to users and audiences and owners and consumers and interested parties and even uninterested parties. The idea of multiple audiences pulses within me at the root. Akin to that concept, another: juxtaposition. What is there versus what is not there; what has been asked versus what has not been asked; the solid versus the void. Good designers are problem solvers, not stylists or skinners. They interrogate situations and ask why? They poke around in seemingly unrelated categories. They consider the complicating factors of temporality and fickle end users while acknowledging that a problem owner requires resolutions. They know that solutions can have many audiences and that things that seem simple and straightforward can be damn complex. (Massimo Vignelli has spoken eloquently on this subject in Massimo Vignelli on Rational Design.” Actually, read all the interviews on Steven Heller’s Design Dialogues site.)

Hard questions
Why do I mention all of this? Because as you begin the process of badge system design, you, too, will be delving into these areas. You, too, will be learning to act as a designer. You’ll be gathering information from many sources—no doubt a few of them entirely unexpected. And most likely you’ll find yourself asking deep and sometimes existential questions. I encourage you to remain open to the idea that periodically, like the question, the answer will prove to be both complex and difficult and very much not binary. Sometimes you will have to try something to know if it works because there will be no answer until you do. Accept this. Your badge system will benefit from this sideways approach. That is, believe it or not, the middle path.

- – -

Much more soon.

references
Ariely, D. (2008). Predictably Irrational. New York, NY: Harper Collins.
Lampland, M. & Starr, S. L. (2009). Standards and their stories. (p. dedication). Ithaca, NY: Cornell University Press.
Vignelli, M. (1998). Massimo Vignelli on Rational Design. In Heller, S. (Ed.), Design dialogues (pp. 3-8). New York, NY: Allworth Press.

Learning, coding, systems of power, and Mozilla

Starting this summer, we’re aiming to help create a group of webmakers. Building on Mozilla’s Manifesto—to promote openness, innovation and opportunity on the web—we’re rationalizing a set of core skills, developing learning objectives and outcomes associated with those skills and offering opportunities to try them out. This effort aligns extremely well with the development and promotion of #5 in our mission list: “Individuals must have the ability to shape their own experiences on the Internet.”

What’s a webmaker?
What do we mean by webmaker? Someone who contributes to the web but also someone who understands the web and its inherent power. Our focus is on moving people toward doing rather than perceiving but both are required. Experimentation is where we’re headed. Guiding people toward understanding the software that constitutes the web so that they can make more informed and educated decisions about not only how they interact with the web, but how they interact with the systems that lead to the power of the web. Yes, systems as we’ve been discussing in previous posts. (Avoiding the complex discussion of Foucault’s systems of power for now, thanks.)

Code is political
Code is political. While that may seem to be a polemical statement, it’s one that serves to inform the currently omnipresent drive to teach people to code. Code is enveloped in systems of power—systems of power that will increasingly play large roles in people’s lives. Understanding that you can create as well as consume seems a fair balance. More people having a literacy is something to be desired, not shunned or disdained. (More info here: Lawrence Lessig’s Code is Law)

What do we mean by literacy?
Traditional literacy lifts people out of poverty, modifies their worldviews, opens up new vistas and provides new opportunities for further enrichment, whether they be social, political, professional, or ideological. If you want your own proof, just search with this combination of terms “literacy and poverty.” Who’s to say that digital literacy won’t accomplish similar things? In the vein of the scientific method, why not test it out?

Literacy itself is a complex term that encompasses a broad spectrum. In our case, literacy is a basic communication skill, akin to numeracy or traditional language literacy. We’re not aiming to make everyone into Joycean code experimenters pushing the boundaries of language and comprehension, nor are we aiming to move everyone toward Hemingway-esque brevity and conciseness, but if some of you decide those pathways are for you, all the better. At least you’ll be moving forward with a broader understanding of what’s possible. And you will be making the decision for yourself, not having it handed to you by some faceless mega-corporation.

Our initial take on web literacy skills is bouncing along as an ongoing experiment (sounds familiar, right?). In the same vein as iterate often, we’re out there trying things on, seeing what feels right. Working with other organizations to leverage their understandings of web literacy and expand upon our own.

What we’re interested in doing with webmaking is shining a light into a place you may not have considered looking before. Showing you that that place is not full of monsters, is not incomprehensible, but is instead simply the exact same world you’ve been experiencing all along just translated into another language. Learning to code is a deciphering of sorts—a decoding of symbols. It offers a different lens through which to view the world.

Opportunity
This new knowledge lens may significantly alter the way you perceive the world; it’s hard to say how it will affect you. Perhaps that unknown quantity is precisely why Mozilla believes learning to code is something everyone should be afforded the opportunity to learn how to do. The operative word in that sentence is opportunity.

Knock, knock, knock.

Open Badges Lexicon: Earners and Issuers

We’ve leapt into Badge System Design in some earlier posts (1, 2, 3) and we’ll be returning to it shortly, In the interim, I’d like to step back to consider a small number of basic Open Badges tenets. In this edition, we’ll address our evolving lexicon and in particular the nomenclature of Earners and Issuers.

A common language
In addition to their ability to transcend physical boundaries, badges introduce many potential languages, e.g., visual, verbal, cultural, pedagogical, etc. Badges will activate these languages, sometimes one at a time, sometimes all at once. Each of these languages may speak to different audiences, and often to many audiences at once. As simple as we try to make our badges, they will be deeply influenced by our worldviews: imbued with our community’s understandings, desires, and values—and those will be intertwined with the earner’s understandings, desires, and values. In turn, those perceptual strands will be woven through the general public’s social assumptions and cultural fibers. Teasing out a strand (or badge) will not reveal the germ of the process but it may help point toward some of what has influenced it. In short, badges can stand alone, but will remain bound into a complex sociocultural system.

Consequently, flexibility in our system design is key. As we attempt to build and rationalize an open badges lexicon, we recognize that a need for individuation, modification, or personalization will always exist. This is built into the OBI system. By designing an extremely flexible product, we’ve accommodated many different potentials.

What does all of this flexibility get us? For one thing, it opens the door to cultural interoperability. The ability to have the Open Badges system accommodate many different cultures, communities, and values. Given that badges exist as forms of cultural representation that interoperability is essential to a robust system. (We will, no doubt, revisit this concept in a later post.)

Along these lines, we began a document for people to share their ideas about Open Badges definitions of terms. In a nice turn of events, this open approach has lead to some fascinating questions about intent and prescriptiveness. Some questions raised in that document have yet to be answered: it’s an ongoing discussion, one that requires back and forth, give and take. We anticipate that it will continue to raise questions, too. And we’re excited about these provocations because they’ll help us to better understand the ecosystem and improve upon our Open Badges system.

Earner vs. holder vs. owner
One question in that open google document queried our choice of the word, “earner.” As with all things Open Badges, we arrived here after considerable thought—along with the aid of some legal help. (You can read more about our legal considerations here.)

A bit of background: we started with “learner” and ended up at “earner.” Believe it or not, dropping the initial consonant involved quite a bit of in-depth thought. We wended our way around to that term after close consideration of the people who might come into possession of a badge. Even the term “earner” presents some weaknesses. Badges can be used to show affiliation, skills, competencies, associations, etc. Some of the folks we’ve spoken with have suggested that badges can and should be earned by organizations themselves. In point of fact, we don’t know all the ways badges can be used, yet. That’s the beauty of a flexible system.

Earner
We chose earner for fairly prescriptive reasons: because we’d like to suggest that badges must be earned, not simply received. However, as badge meaning is initially defined by the issuer, this moniker may change. The earner can be referred to in the way that makes sense to your group. It’s worth remembering, though, that your earner/holder/recipient/whatever will be interacting in a broad ecosystem along with many Issuers, Displayers & other earner/holder/recipient/whatevers. They’ll have an opportunity to speak for and about themselves and may choose their own sobriquet.

Because the earner exists as the hub of their own personal Open Badge ecosystem they wield quite a bit of power: power of self-representation, power of social contracts with Issuers, power of control with Displayers. Earners define their association with the entire ecosystem: what to earn, where to earn it and with whom, and then, ultimately, how to display what they’ve earned. As Erin Knight has said so eloquently about a personal collection of badges housed in a badge backpack, they can act as “a living transcript.”

Issuer
This one is pretty obvious as to why we chose it: these groups, organizations, individuals, institutions, corporations, etc., do the hard work of issuing badges. Not only do they create badges and badge system designs that transmit their values to badge earners, and a variety of additional publics (cf., Michael Warner’s Publics and Counterpublics, much more on this in later posts)—they also build the criteria for those badges, develop badge progressions, create scaffolding opportunities, and undertake the difficult problem of assessment. Plus, they make the commitment to civic participation in the broader Open Badge ecosystem.

- – -

In a future post I’ll address Mozilla’s approach to privacy, as well as explain our rationale for naming Displayers and Endorsers. Much more soon.

references
Warner, M. (2005). Publics and Counterpublics. Boston, MA: MIT Press