Processing, Categorizing, and Acting on Extensive Feedback: My Method

I already wrote a bit on the feedback I received from my upgrading session the other day at the Institute of Education. The whole process was incredibly productive and I am still in that enjoyable phase of euphoria where you are processing feedback that you know will make the final thesis all the better. It is a real rarity in this life (and in my professional iterations of years past) to receive this type of constructive feedback. It is most welcome, especially as I wade through my incoming pilot project data and begin my analysis on whether or not the design methodology is holding fast.

So I thought a quick post as to my process for dealing with large amounts of feedback might be useful to some of you out there who might be going through the same thing at some point. I am not saying my way is correct, but I have been a big believer in transparency in my process and journey through this PhD so I would be a bit of hypocrite if I didn’t make this all available publicly. I can and will improve my method in each iteration, but this is how I process feedback at this stage of my life.

So, I received two separate (amazingly insightful) reports from my two upgraders, both of whom are distinguished academics and researchers (both of whose work I have used in other projects, mostly on mobile learning field activities). These reports list the questions asked and considered, the feedback and suggestions given, etc. So I start with these reports, read them both twice (at a minimum) and list them all together in a note as one gigantic piece of feedback. So I assemble before I disassemble. At this stage, the mind is most prone to overload so I tread lightly, not considering in depth too many of the details, but rather just scanning for patterns and potential categorizations.

So below is the assembled feedback:

Feedback

This submission represents a very clearly-designed research study, which focuses on tightly on specific aspects of mobile learning in South Korean universities. The definition of mobile learning (p6) is well-constructed – sophisticated and complex, but conceptually tight. Despite the word limit of the IOE upgrade submission, the package manages to give a good sense of the research questions, the contents of the literature review, the research design and the theoretical and methodological frameworks. These are outlined clearly and succinctly, in a confident and accessible academic voice.

Points of advice after the interview:

1. consider the cultural context further
2. explore the concept of habitus more fully
3. take into account different uses of ‘participatory’ in new media studies
4. be explicit about disciplinary location – the work is productively inter-disciplinary, and this could be more explicitly stated
5. develop a critical voice to keep the claims of mobile learning theory, multimodality and communities of practice theory balanced and moderate.

The upgrade document represents a focused study drawing on relevant literature within the context of mobile media and learning. The research design is thorough and proposes a strong set of questions to be investigated and potentially answered. There is a strong emphasis on methodology and process although these could change through the pilot study. In terms of the document structure, I realize the candidate is limited in terms of word count, but emphasizing other aspects, especially positioning the wider context of the work would have been useful. At times the documents feels repetitive in discussing research design and questioning, and these areas could be tighter in the final submission. In light of this there are some questions to be posed in the viva as follows:

1. How did you come up with your research questions?
2. Does the how in the first question enable enough of a discussion?
3. Is a why relevant here and if so, how does this fit into the research design?
4. Who are your main theorists within the field of community practice? And which aspects of multimodality are you drawing on specifically? Will this change through the results of the pilot study?
5. How have mobile technologies been used for participation so far? Do we need to situate the act of participation in terms on online communities, are the mobile technologies offering online connectivity or creating participation through offline social spaces?
6. What examples would you have given in Language, literature, composition and new media and history and cultural heritage to situate this work? Would these help to define a wider context?
7. How do you define mobile media? What artefacts or applications does this term encompass and how are they relevant to your study? (thinking about Hjorth’s work in relation to this, and the nature of drawing on different disciplines)
8. Who are your key theorists in terms of mobile media and learning and also in terms of mobile media practices (how are these related if at all?)? Are there areas you are deliberately excluding from your study? What was your process in undertaking a literature review in this area?
9. Are there comparable artefacts that have been created that you will focus on as case studies?
10. Are there patterns present across these disciplines in the humanities within Korea? How does this link to UK/US definitions of humanities (is this comparison important?)

Aggregated Questions (from both reviewers)

1. RQs are clear except for 3 – “How do graduate students engage in the participatory process in the humanities in South Korean universities?” Explain further?
2. P5 – Kress and Pachler – note that ‘habitus’ isn’t their concept, but derived from Bourdieu. Explain further how you are employing this concept?
3. P6/7. The definitions of mobile learning. To play Devil’s advocate – could this be achieved through use of a series of desktop computers at school and home? What are the claims being made for particular technologies? And are cultural distinctions being made between ‘educational’ technologies and those employed in people’s everyday lives? Is this distinction significant?
4. Informal uses (p7) – the socialized/individualized distinction you make is obviously important. What about cultural aspects of these technologies and how they’re used?
5. “The use of Multimodality is in keeping with the general broadening in the humanities of what constitutes accepted practice and evidence, thereby loosening the traditional authority of text (Bowen & Kinnear) and shifting towards the visual (Kress, 2000).” p 10. ‘Text’ here should probably be ‘language’, as visual communication would also be considered text in semiotics. Might you critique the emphasis on the visual here? what about the auditory, for example?
6. Methodologies. Why not observation? Is there a danger of losing the context in only collecting the ‘submitted artifacts’?
7. Give us a sense of how a multimodal analysis of a submitted artifact might be undertaken. Why confine to Bezemer and Kress’s model? How to synthesise this with analysis of interview data?
8. Twenty graduate students. What’s the rationale for this sample and sample size? How is the sample made up and why?
9. Is there a danger of over-optimistic theoretical stances? Mobile learning, Communities of Practice, Multimodality all have built-in biases towards positive outcomes. What about the limits or constraints of mobile technologies; the dangers of disintegrating or unstable CoPs; the risk of incoherence or unstable meaning in multimodal communication? Allow for some sceptical or critical element? Cf Buckingham’s Beyond Technology; Martin Oliver on CoPs in World of Warcraft.

See what I mean? Excellent feedback. Accessible and relatable to a particular action item. That is what immediately jumped out as I scanned for patterns. Everything was pragmatically based. So I broadly sketched some initial takeaways from a few scans of the feedback.

Takeaways

1. Define Korean context. Situate the Korean context early and often.
2. Define culture. Position it within the larger fields of study.
3. Define Korean Humanities and how it compares with Western model.
4. Define habitus and situate my use of it in mobile learning in the broader sociological realm: which aspects am I using and why
5. Specify which aspects of multimodality I will use. Be specific and critical.
6. Define which parts of CoP I am beginning to use (establish Legitimate Peripheral Participation earlier on as the focus). Be critical early and often.
7. Define mobile media. Situate it within media studies or at least explain how other disciplines are using that term.
8. Need baseline example or definition of multimodal artifact. Look to Korean context.
9. Give rationale for sample size. Broaden explanation in my methods section.
10. Be critical throughout of the gaps in the theory. Develop critical stance of all my literature.

From there I began to extrapolate higher level categorizations, which basically amounted to this:

1. Critical Stance
2. Defining terms
3. Contextualizing (Korea)
4. Acknowledging/diverting from origins/other fields/other discourses

Slotting and Sequencing

I then slotted each piece of feedback and each question in one of these broader categorizations. From there, I merely prioritized each one in terms of which I would approach first, second, etc. I then schedule time for reflection/writing on my calendar and that is that. This process all happened within the last two days. So within 48 hours I feel like I had quickly and painlessly moved from the vastness of extensive feedback to generalization and then to action. I never let the vastness of the feedback overwhelm me in its scope and precision. I acted on it and moved forward with it. For me, this is critical to any success I have had anywhere. I have found no merit in my life in sitting and stewing. Act, categorize, be open to iterations, process, refine and repeat. But always do something.

In case you are curious, I am going to start with the Defining Terms set of feedback points. I think with precise working definitions of each of these terms, I will be a long way farther down the line of writing the other sections. Without those precisely defined terms, I suspect the writing of the Critical Stance and Contextualization sections will suffer from a lack of focus. So my sequence is as follows:

1. Defining Terms
2. Contextualizing (Korea)
3. Acknowledging/Diverting from Other Fields ( I need to acknowledge the sociological roots of a lot of my terms and their accepted uses in other fields, such as Cultural Studies, Media Studies, Korean Studies even)
4. Critical Stance (once a precise context is established, the appropriateness of my choice of theories will become clearer)

So there you have it. My method for dealing with feedback. Writing has already begun and paired with my pilot data, I am seeing a clear path ahead (clear as in recognizable path, not free from obstacles). All is well to end 2013.