Performance Management and Compensation Questionnaire Performance Management and Compensation Questionnaire

The intent of this paper is to generate discussion around this broad framework, and revisit the existing tools and methods that support this kind of faceted multi-method approach to researching teaching and learning through social media. This paper starts with a review of relevant literature that informs the landscape of social media and learning, provides the theoretical underpinning of significant methods and approaches that help form our proposed analytic framework, and integrates concepts from other knowledge domains into a learning analytics perspective. Next, this paper provides a case study that further explains and demonstrates our analytic framework. We apply our framework to a dataset collected from a Connectivist Massive Online Open Course (cMOOC), illustrate several analysis methods we rely on, and show how they can be used in a combinatory, complementary fashion to generate new insights. We then discuss our framework in relation to a number of contexts: from how it might be employed in formal educational to evaluate and optimize learning design, to how it can help detect and understand learner behaviours in informal, self-regulated learning contexts. The paper concludes with a reflection of our work in relation to the ongoing development in learning analytics research and tool development, a discussion of limitations and potential issues surrounding our framework, and a look ahead to directions for future work. 2 THE LANDSCAPE OF RESEARCH ON SOCIAL MEDIA AND LEARNING 2.1 Formal and Informal Learning Contexts Higher education faculty recognize the value that social media can leverage in their curriculum, with over one-third of teaching faculty in the US using some form of social media in their courses, and adoption rates of social media as high as 80% in university classrooms in the US (Moran, Seaman, & Tinti-Kane, 2012). A recent EDUCAUSE study (Dahlstrom, Walker, & Dziuban, 2013; Smith & Caruso, 2010) indicates that social media are being formally integrated into institutional academic learning experiences, and being informally used by students to supplement their learning experiences. This allows students to reach wider social networks via social media while simultaneously “meeting the student population where it lives: i.e., online, in social networking sites and in the microforms of communication adopted in Twitter” and other popular online platforms (Gruzd, Haythornthwaite, Paulin, Absar, & Huggett, 2014, p. 254). ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 48 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 Learners use various forms of social media to bridge the gap between in-school and out-of-school learning by enabling the discovery of connections between their traditional curricula, their personal interests, and online communities that can support and further their engagement and learning (Ito et al., 2013). Traditional learning contexts and online platforms such as learning management systems (LMSs) do not often expose students to the learning opportunities afforded by social media in terms of enabling connections to peers, communities, and resources across time and space (Dabbagh & Kitsantas, 2012). To this end, learners use social media to expand their learning opportunities beyond the classroom and the LMS in a self-directed manner, enabling the personalization of their learning experiences to their own interests, their own learning goals, and their own preferences in terms of participation, online communities, and social media platforms (Mcloughlin & Lee, 2010; Siemens, 2008). As learners progress through school and towards professional life, formal learning plays an increasingly smaller role in lifelong learning experiences while informal learning becomes integral to developing knowledge and skills (Banks et al., 2007; Chen & Bryer, 2012). Informal learning opportunities are afforded through connections and interactions with networks of peers, and with the ideas and resources made available through those networks. In this way, informal learning supports involvement in a knowledge-creating culture: developing knowledge-building competencies, understanding one’s own learning in relation to, and in contribution to, a larger knowledge-building community (Scardamalia & Bereiter, 2006), shaping the (online) community of practice (Lave & Wenger, 1991; Wenger, 1998; Haythornthwaite & Andrews, 2011). Social media enable learners to pursue this kind of social, groupbased learning by providing the means to create, find, organize, and share resources, and participate in networks and communities with a shared learning focus or interest (e.g., see Gruzd & Haythornthwaite, 2013). Thus, social media amplify and expand the informal learning opportunities available to learners. Ziegler, Paulus, and Woodside (2014) note that research on informal learning has largely relied on retrospective accounts of learning from the learners themselves, through interview or survey data. However, asking people what they have learned, and how they have learned it, can be problematic as respondents often lack awareness of their own learning, and regard it as part of their own general capability rather than something learned (Eraut, 2010). While self-reported data provides an account of the lived experiences of individuals, dialogue and textual language that occurs during social activity provide an account of the social reality constructed by those engaged in conversation. Rorty (1992) argues that language creates, rather than represents, lived experiences. The language that comprises the exchanges and interactions on social media is a valuable source of data that can be analyzed to understand how informal learning occurs. 2.2 Text-based Content Analysis Social media creates a vast quantity of textual data that record the history of group interaction as networks and communities form, grow, and decline. Content analysis is a method for examining patterns of text and language. Content analysis relies on systematic techniques that compress large amounts of text into fewer coded categories, enabling researchers to discover and explore the focus of ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 49 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 attention in text or dialogue (Krippendorff, 1980; 2012). Within educational and learning contexts, content analysis has been used to investigate asynchronous discussion to identify markers of collaboration and co-operation (De Wever, Schellens, Valcke, & Van Keer, 2006); to detect cognitive presence in online discussions (Kovanović et al., 2016); to conduct sentiment analysis to understand the relationship between sentiment expressed in discussion forums and attrition rates in a MOOC (Wen, Yang, & Rosé, 2014). Analyses can give insight into the characteristics, interests, and priorities of a learning network, and reveal patterns of language and interaction that characterize a community and foster learning (Haythornthwaite & Gruzd, 2007). Analysis of online discussions can uncover underlying mechanisms of group interaction, and identify unique language patterns that demonstrate instances of thinking, collaboration, or learning (Strijbos, Martens, Prins, & Jochems, 2006). Further, text analysis can provide insight into concepts central to discussion or to generating interest within a learning community, the nature of exchanges occurring (i.e., informational, socially oriented, and so on), or the semantic or affective weight of language used in discussions. In determining which social processes and concepts should be examined through content analysis, researchers are led by theories and perspectives that guide understanding of learning (see De Wever et al., 2006, for a review of common concepts and processes studied, along with corresponding theories; see also Rogers, Dawson, & Gašević, 2016; Eynon, Schroeder, & Fry, 2016; Wise & Shaffer, 2015). While there are many perspectives on what social processes and concepts are most appropriate for studying learning, most content analysis work relies on the development of categories that define the processes and concepts under investigation, coding them to identify and interpret text that falls under one or more categories (see Krippendorff, 1980; 2012). Research choices include the definition of categories and the selection of units of analyses — words, symbols, or phrases — within the text that represent or indicate a category. For example, if a category was defined by emotive expression, words such as “love” or “hate” are likely to be useful units that identify discussion contributions that can be categorized as emotive expression. Content analysis often relies on manually finding, labelling, and interpreting categories in text. While this is manageable for smaller corpora, manual content analysis is not practical for larger datasets such as those found in social media or MOOCs. While teams could be formed to distribute the burden of manual coding and analysis, the resulting lack of consistency and agreement on interpretation of categories introduces the problem of consistency and reliability. Automated text analysis offers an alternative for such datasets. 2.2.1 Automated Text Analysis The field of computational linguistics has developed many Natural Language Processing (NLP) algorithms and techniques to automate the analysis and representation of text. Many of these techniques provide analysis in the form of finding meaningful patterns in text through word counting, key phrase matching, or visualization of patterns of categories (Rosé et al., 2008). Tools such as Linguistic Inquiry and Word ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 50 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 Count (LIWC) rely on dictionary-based methods to identify organizations of words and phrases that indicate specific mental states or emotions (Pennebaker, 2003). NLP relies on lexical analysis to identify word classes (i.e., nouns, verbs, etc.) and syntactic analysis to reveal grammatical structures in text (Liddy, 1998; Rubin, Stanton, & Liddy, 2004). This allows nouns and noun phrases — considered to be the most informative elements of text (Boguraev & Kennedy, 1999; Carley & Palmquist, 1992; Carley, 1997; Corman, Kuhn, McPhee, & Dooley, 2002) — to be identified, and visualized in topic maps or world clouds (Haythornthwaite & Gruzd, 2007). Using machine learning approaches towards NLP, semantic analysis allows for automatic analysis of text beyond dictionary-based categorization and frequency counts of words. Through a process of training a program on massive textual data sets and focusing on frequency, proximity, and many other linguistic factors, a program can learn and assign context to language. This goes beyond understanding meanings and categorizations of words towards understanding relationships between words, phrases, and ideas akin to human-like, common-sense knowledge about the world through language. Semantic analysis enables complex tasks such as word-sense disambiguation for words with multiple meanings, building systems capable of answering questions posed in plain language, or translating across languages. Table 1 presents a list of examples of currently available content analysis tools and their key features. Tool name Netlytic LIWC Atlas.ti NVivo LightSIDE RapidMiner Weka 2.3 Table 1. Examples of content analysis tools and key features Key features A cloud-based text and social network analysis tool that allows users to capture and import online conversational data, and find, explore, and visualize emerging themes of discussions. A dictionary-based text analysis program that categorizes words that reflect different emotions, cognitive styles, social and psychological states. Software that aids qualitative analysis of unstructured data (text, multimedia, etc.) through coding, annotation, and visual structuring. A qualitative data analysis software package that allows users to classify, sort, and arrange unstructured data, and examine relationships within data. A text mining tool bench that leverages machine learning to enable automated analysis of conversational interactions and social aspects of text (e.g., perspective modelling, sentiment analysis, opinion mining). An analytics software platform that offers text analysis and sentiment analysis tools. A collection of machine learning algorithms for data mining tasks, including semantic analysis and sentiment analysis. Social Network Analysis The emergence and growth of social media — networked tools, platforms, and their associated practices — has inspired rethinking of how we might learn in today’s highly connected environment (Siemens, 2005). This line of thinking has led to the conceptualization of a personalized learning network — a collection of interoperating applications that form an ecology of social media and networks through ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 51 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 which individuals explore and learn (Fiedler & Väljataga, 2011). An ecosystem approach leads to a particular network-based pedagogy where learning is supported through practice, reflection, and participation in communities, and engaging in a distributed environment consisting of networks of people, services, and resources that provide learning opportunities (Downes, 2006). While learning networks provide opportunities for the learner, the distributed, interconnected nature of the model provides challenges for educators, learning designers, and researchers interested in understanding how people learn and the effectiveness of their learning experience. Social Network Analysis (SNA) provides knowledge, perspectives, and tools that can be applied to the interpretation and design of networked learning (Haythornthwaite & de Laat, 2010; Haythornthwaite, de Laat, & Schreurs, 2016). SNA can help in understanding how and why learners in a network are connected, how they seek each other out, and how their connections, configurations, and interaction patterns support information and

The intent of this paper is to generate discussion around this broad framework, and revisit the existing tools and methods that support this kind of faceted multi-method approach to researching teaching and learning through social media. This paper starts with a review of relevant literature that informs the landscape of social media and learning, provides the theoretical underpinning of significant methods and approaches that help form our proposed analytic framework, and integrates concepts from other knowledge domains into a learning analytics perspective. Next, this paper provides a case study that further explains and demonstrates our analytic framework. We apply our framework to a dataset collected from a Connectivist Massive Online Open Course (cMOOC), illustrate several analysis methods we rely on, and show how they can be used in a combinatory, complementary fashion to generate new insights. We then discuss our framework in relation to a number of contexts: from how it might be employed in formal educational to evaluate and optimize learning design, to how it can help detect and understand learner behaviours in informal, self-regulated learning contexts. The paper concludes with a reflection of our work in relation to the ongoing development in learning analytics research and tool development, a discussion of limitations and potential issues surrounding our framework, and a look ahead to directions for future work. 2 THE LANDSCAPE OF RESEARCH ON SOCIAL MEDIA AND LEARNING 2.1 Formal and Informal Learning Contexts Higher education faculty recognize the value that social media can leverage in their curriculum, with over one-third of teaching faculty in the US using some form of social media in their courses, and adoption rates of social media as high as 80% in university classrooms in the US (Moran, Seaman, & Tinti-Kane, 2012). A recent EDUCAUSE study (Dahlstrom, Walker, & Dziuban, 2013; Smith & Caruso, 2010) indicates that social media are being formally integrated into institutional academic learning experiences, and being informally used by students to supplement their learning experiences. This allows students to reach wider social networks via social media while simultaneously “meeting the student population where it lives: i.e., online, in social networking sites and in the microforms of communication adopted in Twitter” and other popular online platforms (Gruzd, Haythornthwaite, Paulin, Absar, & Huggett, 2014, p. 254). ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 48 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 Learners use various forms of social media to bridge the gap between in-school and out-of-school learning by enabling the discovery of connections between their traditional curricula, their personal interests, and online communities that can support and further their engagement and learning (Ito et al., 2013). Traditional learning contexts and online platforms such as learning management systems (LMSs) do not often expose students to the learning opportunities afforded by social media in terms of enabling connections to peers, communities, and resources across time and space (Dabbagh & Kitsantas, 2012). To this end, learners use social media to expand their learning opportunities beyond the classroom and the LMS in a self-directed manner, enabling the personalization of their learning experiences to their own interests, their own learning goals, and their own preferences in terms of participation, online communities, and social media platforms (Mcloughlin & Lee, 2010; Siemens, 2008). As learners progress through school and towards professional life, formal learning plays an increasingly smaller role in lifelong learning experiences while informal learning becomes integral to developing knowledge and skills (Banks et al., 2007; Chen & Bryer, 2012). Informal learning opportunities are afforded through connections and interactions with networks of peers, and with the ideas and resources made available through those networks. In this way, informal learning supports involvement in a knowledge-creating culture: developing knowledge-building competencies, understanding one’s own learning in relation to, and in contribution to, a larger knowledge-building community (Scardamalia & Bereiter, 2006), shaping the (online) community of practice (Lave & Wenger, 1991; Wenger, 1998; Haythornthwaite & Andrews, 2011). Social media enable learners to pursue this kind of social, groupbased learning by providing the means to create, find, organize, and share resources, and participate in networks and communities with a shared learning focus or interest (e.g., see Gruzd & Haythornthwaite, 2013). Thus, social media amplify and expand the informal learning opportunities available to learners. Ziegler, Paulus, and Woodside (2014) note that research on informal learning has largely relied on retrospective accounts of learning from the learners themselves, through interview or survey data. However, asking people what they have learned, and how they have learned it, can be problematic as respondents often lack awareness of their own learning, and regard it as part of their own general capability rather than something learned (Eraut, 2010). While self-reported data provides an account of the lived experiences of individuals, dialogue and textual language that occurs during social activity provide an account of the social reality constructed by those engaged in conversation. Rorty (1992) argues that language creates, rather than represents, lived experiences. The language that comprises the exchanges and interactions on social media is a valuable source of data that can be analyzed to understand how informal learning occurs. 2.2 Text-based Content Analysis Social media creates a vast quantity of textual data that record the history of group interaction as networks and communities form, grow, and decline. Content analysis is a method for examining patterns of text and language. Content analysis relies on systematic techniques that compress large amounts of text into fewer coded categories, enabling researchers to discover and explore the focus of ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 49 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 attention in text or dialogue (Krippendorff, 1980; 2012). Within educational and learning contexts, content analysis has been used to investigate asynchronous discussion to identify markers of collaboration and co-operation (De Wever, Schellens, Valcke, & Van Keer, 2006); to detect cognitive presence in online discussions (Kovanović et al., 2016); to conduct sentiment analysis to understand the relationship between sentiment expressed in discussion forums and attrition rates in a MOOC (Wen, Yang, & Rosé, 2014). Analyses can give insight into the characteristics, interests, and priorities of a learning network, and reveal patterns of language and interaction that characterize a community and foster learning (Haythornthwaite & Gruzd, 2007). Analysis of online discussions can uncover underlying mechanisms of group interaction, and identify unique language patterns that demonstrate instances of thinking, collaboration, or learning (Strijbos, Martens, Prins, & Jochems, 2006). Further, text analysis can provide insight into concepts central to discussion or to generating interest within a learning community, the nature of exchanges occurring (i.e., informational, socially oriented, and so on), or the semantic or affective weight of language used in discussions. In determining which social processes and concepts should be examined through content analysis, researchers are led by theories and perspectives that guide understanding of learning (see De Wever et al., 2006, for a review of common concepts and processes studied, along with corresponding theories; see also Rogers, Dawson, & Gašević, 2016; Eynon, Schroeder, & Fry, 2016; Wise & Shaffer, 2015). While there are many perspectives on what social processes and concepts are most appropriate for studying learning, most content analysis work relies on the development of categories that define the processes and concepts under investigation, coding them to identify and interpret text that falls under one or more categories (see Krippendorff, 1980; 2012). Research choices include the definition of categories and the selection of units of analyses — words, symbols, or phrases — within the text that represent or indicate a category. For example, if a category was defined by emotive expression, words such as “love” or “hate” are likely to be useful units that identify discussion contributions that can be categorized as emotive expression. Content analysis often relies on manually finding, labelling, and interpreting categories in text. While this is manageable for smaller corpora, manual content analysis is not practical for larger datasets such as those found in social media or MOOCs. While teams could be formed to distribute the burden of manual coding and analysis, the resulting lack of consistency and agreement on interpretation of categories introduces the problem of consistency and reliability. Automated text analysis offers an alternative for such datasets. 2.2.1 Automated Text Analysis The field of computational linguistics has developed many Natural Language Processing (NLP) algorithms and techniques to automate the analysis and representation of text. Many of these techniques provide analysis in the form of finding meaningful patterns in text through word counting, key phrase matching, or visualization of patterns of categories (Rosé et al., 2008). Tools such as Linguistic Inquiry and Word ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 50 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 Count (LIWC) rely on dictionary-based methods to identify organizations of words and phrases that indicate specific mental states or emotions (Pennebaker, 2003). NLP relies on lexical analysis to identify word classes (i.e., nouns, verbs, etc.) and syntactic analysis to reveal grammatical structures in text (Liddy, 1998; Rubin, Stanton, & Liddy, 2004). This allows nouns and noun phrases — considered to be the most informative elements of text (Boguraev & Kennedy, 1999; Carley & Palmquist, 1992; Carley, 1997; Corman, Kuhn, McPhee, & Dooley, 2002) — to be identified, and visualized in topic maps or world clouds (Haythornthwaite & Gruzd, 2007). Using machine learning approaches towards NLP, semantic analysis allows for automatic analysis of text beyond dictionary-based categorization and frequency counts of words. Through a process of training a program on massive textual data sets and focusing on frequency, proximity, and many other linguistic factors, a program can learn and assign context to language. This goes beyond understanding meanings and categorizations of words towards understanding relationships between words, phrases, and ideas akin to human-like, common-sense knowledge about the world through language. Semantic analysis enables complex tasks such as word-sense disambiguation for words with multiple meanings, building systems capable of answering questions posed in plain language, or translating across languages. Table 1 presents a list of examples of currently available content analysis tools and their key features. Tool name Netlytic LIWC Atlas.ti NVivo LightSIDE RapidMiner Weka 2.3 Table 1. Examples of content analysis tools and key features Key features A cloud-based text and social network analysis tool that allows users to capture and import online conversational data, and find, explore, and visualize emerging themes of discussions. A dictionary-based text analysis program that categorizes words that reflect different emotions, cognitive styles, social and psychological states. Software that aids qualitative analysis of unstructured data (text, multimedia, etc.) through coding, annotation, and visual structuring. A qualitative data analysis software package that allows users to classify, sort, and arrange unstructured data, and examine relationships within data. A text mining tool bench that leverages machine learning to enable automated analysis of conversational interactions and social aspects of text (e.g., perspective modelling, sentiment analysis, opinion mining). An analytics software platform that offers text analysis and sentiment analysis tools. A collection of machine learning algorithms for data mining tasks, including semantic analysis and sentiment analysis. Social Network Analysis The emergence and growth of social media — networked tools, platforms, and their associated practices — has inspired rethinking of how we might learn in today’s highly connected environment (Siemens, 2005). This line of thinking has led to the conceptualization of a personalized learning network — a collection of interoperating applications that form an ecology of social media and networks through ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution – NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 51 (2016). Analyzing social media and learning through content and social network analysis: A faceted methodological approach. Journal of Learning Analytics, 3(3), 46–71. http://dx.doi.org/10.18608/jla.2016.33.4 which individuals explore and learn (Fiedler & Väljataga, 2011). An ecosystem approach leads to a particular network-based pedagogy where learning is supported through practice, reflection, and participation in communities, and engaging in a distributed environment consisting of networks of people, services, and resources that provide learning opportunities (Downes, 2006). While learning networks provide opportunities for the learner, the distributed, interconnected nature of the model provides challenges for educators, learning designers, and researchers interested in understanding how people learn and the effectiveness of their learning experience. Social Network Analysis (SNA) provides knowledge, perspectives, and tools that can be applied to the interpretation and design of networked learning (Haythornthwaite & de Laat, 2010; Haythornthwaite, de Laat, & Schreurs, 2016). SNA can help in understanding how and why learners in a network are connected, how they seek each other out, and how their connections, configurations, and interaction patterns support information and