Assessment

Career Connect’s evidence-based assessment is designed to evaluate and showcase UNT students’ marketable skills and guide UNT’s actions for improving them.

These marketable skills (that is, written and oral communication, teamwork, and critical thinking) are among the top marketable skills that employers want and are targeted by Career Connect assessment.  Career Connect’s assessments benefits UNT faculty, staff, and students by allowing them to:

  • provide UNT students with opportunities to receive marketable skills badges and credentials and showcase them in their ePortfolios and through their UNT transcripts
  • develop a database of student artifacts and evaluation ratings—specific to UNT faculty or staff courses or programs—that can be used to help demonstrate teaching or program effectiveness
  • use Career Connect student data for UNT faculty or staff research projects or publication submissions
  • integrate students’ ePortfolio use and reflection into UNT courses or programs
  • join the national conversation on university-level credentialing by helping to address a key concern about the pervasive lack of individual-level student evidence and rigor for student-awarded credentials
  • help UNT identify and establish benchmarks for its core skills (that is, communication, teamwork, and critical thinking) among UNT students and departments that can be compared to other universities across the country

More information is provided below about Career Connect’s assessment, how to integrate it into your UNT course or activity, and how it benefits UNT faculty, staff, and students.

What marketable skills does Career Connect measure and with what rubrics?

Career Connect uses the following definitions and rubrics for oral and written communication, team work, and critical thinking. These definitions were developed by the Association of American Colleges and Universities and their work with faculty and experts from different disciplines and universities across the country.

Definitions

Oral Communication. A prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners' attitudes, values, beliefs, or behaviors.

Written Communication. The expression of ideas in writing that develop through iterative experiences across curricula and can involve learning to work in different genres, styles, writing technologies, and mixing texts, data, and images.

Teamwork. Behaviors that represent the collective of individual contributions towards team tasks and discussions and the manner of interactions among team members. Contributions are measured in both quantity and quality.[1]

Critical Thinking. A habit of the mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusions.

Rubrics

Career Connect’s skill rubrics were developed by the American Association of Colleges and Universities through the work of multiple faculty and experts nationwide. Career Connect adapted these rubrics to make them more user-friendly. Below is the abbreviated version of the rubrics and a hyperlink to the long version--that maps to the abbreviated version--is included here.

Rating Scale: 1=beginning, 2=developing, 3 = proficient, and 4=distinguished, NA=not applicable

Indicate the degree to which the student's written communication demonstrates the criteria listed below.

Written Communication

1

2

3

4

NA

  1. Main idea is well-developed and thought provoking.
 

 

 

 

 

  1. Organization includes topic sentence, intentional transition, and content cohesion.

 

 

 

 

 

  1. Content is thoroughly developed.

 

 

 

 

 

  1. Disciplinary conventions were applied with no errors.

 

 

 

 

 

  1. Evidence is high quality, credible, and relevant to main idea.

 

 

 

 

 

Indicate the degree to which the student's oral communication demonstrates the criteria listed below.

Oral Communication

1

2

3

4

NA

  1. Central message is compelling.

 

 

 

 

 

  1. Posture, gestures, and vocal expressiveness enhance the message.

 

 

 

 

 

  1. Language (e.g., vocabulary, sentence structure) supports effectiveness of the message.

 

 

 

 

 

  1. Explanation and examples support the clarity and credibility of message.

 

 

 

 

 

Indicate the degree to which the student's teamwork demonstrates the criteria listed below.

Teamwork

1

2

3

4

NA

  1. Benefits the team by contributing ideas and work that advance the project.

 

 

 

 

 

  1. Contributes to project planning and management.

 

 

 

 

 

  1. Fosters collaboration by valuing the contributions of all team members.

 

 

 

 

 

  1. Supports and motivates the team and helps to create a constructive team climate.

 

 

 

 

 

  1. Responds to feedback or conflict in a way that reflects mutual respect.

 

 

 

 

 

Indicate the degree to which the student’s critical thinking demonstrates the criteria listed below.

Critical Thinking Skills

1

2

3

4

NA

  1. Describes the problem, question, or issue comprehensively.

 

 

 

 

 

  1.  Examines and questions the accuracy, relevance, and completeness of evidence.

 

 

 

 

 

  1. Considers context, assumptions, other perspectives, and credibility and authority of sources.

 

 

 

 

 

  1. Justifies own argument, position, or hypothesis.

 

 

 

 

 

  1. Presents conclusions and outcomes that are logical.

 

 

 

 

 

Why and how does Career Connect marketable skills?

Results from national employer surveys for the past 5 years (for example, the National Association of Colleges and Employers Job Outlook survey) indicate that written and oral communication, teamwork, and critical thinking are reported by employers to be among the top 5 or 10 marketable skills that they want college students to have upon graduation.

These four marketable skills:

  • represent the key student learning outcomes “that educators, employers, and policy makers agree are essential for student success in the workplace and in life.” (American Association of Colleges & Universities, 2013a, 2013b, 2017; Burger & Starbird, 2012; Nunez, 2013)
  • are aligned with UNT core rubrics
  • can be measured (at least one of the four skills) in every UNT department and therefore allow for student learning outcome comparisons across UNT
  • were developed by the Association of American Colleges and Universities with the help of multiple faculty and experts from many different disciplines and colleges around the country

How Career Connect measures these skills is also important because Jessie Brown and Martin Kurzweil (2017) argue that while student competency achievement and credentialing programs (like Career Connect) have proliferated across the country, there is not much evidence to support a high-level of rigor for measuring skills in these programs.

Most other university credentialing programs do not use individual-level evidence, coupled with longitudinal proficiency standards, to award credentials for skills that students demonstrate. Many university credentialing programs award credentials on the basis of student participation. Career Connect addresses this concern through the use of a badge-based, credentialing system that uses psychometrically valid and reliable measures or rubrics of student skills. Career Connect is one of the few credentialing programs across the county that collects individual-level evidence of student learning outcomes, assesses it on the basis of long-term proficiency standards, and is working to validate all of its rubrics.

How do I receive training on how to assess the four skills with Career Connect rubrics?

Career Connect offers multiple rubric trainings each semester. For step-by-step instructions (with screen shots) on how to assess the skills in the ePortfolio, please click here.

Career Connect hosts multiple rubric trainings each semester. These trainings show UNT faculty and staff how to complete the rubrics by using appropriate evidence to assess the four Career Connect skills. For the Spring 2018 semester, Career Connect hosted four trainings in February, March, and April and also offered online rubric trainings that can be accessed here for UNT faculty and staff. UNT faculty and staff can assign the training to a teaching assistant or a community partner who is also involved in the Career Connect Activity. Career Connect rubric trainings for the Fall 2018 semester will be scheduled soon.

How do I assess these four skills with Career Connect rubrics in the ePortfolio?

UNT faculty and staff members will assess one or more of the four Career Connect skills using rubrics developed by Career Connect. UNT faculty or staff will log in to their ePortfolio communities to access the rubrics or receive an email with a hyperlink to the rubrics. Each rubric contains four to five questions and provides a rating for each question on a scale of 1 (beginning) to 4 (distinguished). For step-by-step instructions (with screen shots) on how to assess the skills in the ePortfolio, please click here.

What are Career Connect’s plans for assessing (1) program implementation and (2) student learning outcomes?

On September 2016, the University of North Texas (UNT) selected UNT Career Connect as the Quality Enhancement Plan (QEP) for the 2016 to 2020 school years. Career Connect’s five goals are to:

  1. Develop a transparent, useful pathway for students to participate in Career Connect.
  2. Develop a robust ePortfolio system.
  3. Provide valid and reliable evidence of learning that is valuable to students who participate in UNT Career Connect course work and co-curricular activities.
  4. Recognize and incentivize curricular and co-curricular community activities in a badge based or other recognition structures.
  5. Develop enhanced academic credentials tied to the QEP framework (e.g. extended transcripts).

In addition to describing the Career Connect program, the 2016 QEP outlined the key evaluation components and indicators for assessing Career Connect’s goals. Career Connect expanded these evaluation components to include the following methodological details for its program implementation and outcome assessments.

Program implementation assessment methodology

Career Connect:

  • created evaluation questions that address Career Connect’s five goals and key implementation or milestone activities for UNT faculty and students
  • collects data from UNT faculty, staff, students, and community partners through Career Connect’s ePortfolio and AirTable systems and UNT’s learning management system or CANVAS
  • analyzes these data to answer evaluation questions regarding Career Connect’s implementation and whether program improvements can be made
  • analyzes the psychometric properties of the four Career Connect rubrics
  • reports key QEP implementation findings and metrics (see Table 1 below) that focus on the progress, support, opportunities and challenges, and the validation of Career Connect’s reflection and skills measures

Table 1. QEP Evaluation Metrics

University and Community Engagement

UNT Student’s ePortfolio Use

Career Connect’s Contribution to Scholarship

University Support

Stakeholder Preference Surveys

 

  • Number of QEP activities
  • Number of participating students
  • Number of participating community members
  • Number of students using the ePortfolio
  • Number of tools in active use within the ePortfolio
  • Number of conference presentations provided
  • Number of publications
  • Administrative support and feedback
  • Program support from the Career Center, CLEAR, and other UNT entities listed in QEP
  • Administration of stakeholder feedback survey
  • Administration of focus groups
 
 
 

Program outcome assessment methodology

Career Connect:

  • created an evaluation question (see below) to test Career Connect’s theory of change using a quasi-experimental design
  • collects student learning outcome data from (1) UNT faculty, staff, and community partners’ who rate UNT students’ assignments in their ePortfolios, and (2) UNT’s Office of Research and Effectiveness
  • will analyze the potential impact of the Career Connect program on UNT students’ skills using research standards developed by the What Works Clearinghouse[2], U.S. Department of Education
  • will report key outcome findings that address the below evaluation question

Do UNT students who participate in Career Connect Activities through UNT courses receive significantly higher UNT faculty, staff, or community ratings (on average) in written and oral communication, teamwork, or critical thinking than UNT students ho participate in similar UNT courses that do not include a Career Connect Activity?

Therefore, key findings of the report will focus on whether:

  • baseline equivalence was established between the treatment and control groups
  • there are significant mean outcome differences between the treatment and control groups
  • the magnitude of the effect is undetectable, small, medium, or large[3]

It should be noted that it can take years for a university-wide initiative to be adopted and Career Connect staff are working hard to meet this goal.  Once we were are close to accomplishing this goal, Career Connect will begin working with UNT departments that would be a good fit for implementing a quasi-experimental design.

Exploratory design of student learning outcomes

Compared to the myriad of other university-level, credentialing programs across the country, Career Connect is unique because it awards credentials on the basis of multiple UNT faculty or staff assessments of individual students’ marketable skills that are targeted through experiential learning activities.

Career Connect’s uniqueness allows its evaluation to focus on student-level, marketable skills outcomes across time within quasi-experimental designs (as outlined in the preceding section), and within-time through cross-sectional, exploratory designs. Career Connect’s exploratory design will include (for each semester) separate “end of the semester” analysis of student outcomes in the areas of written and oral communication, teamwork, critical thinking, and reflection. The analysis will include:

  • Descriptive (frequency and mean) analyses of student demographic characteristics and outcomes for total sample
    • Outcome means and demographic frequencies. For example, examining differences across various demographic categories and between students who are and are not:
      • using their ePortfolios
      • participating in Career Connect Activities
      • rated as proficient in one or more of Career Connect’s marketable skills
  • Correlational and ANOVA analysis
    • Correlational analysis of outcome variables for total population and within subgroups
    • ANOVA analysis to assess mean outcome differences among different subgroups
  • Focus groups—UNT students, faculty, and staff
    • Career Connect administered a motivational survey to representative samples of UNT students who did and did not participate in a Career Connect experience during spring 2018. The survey assessed their motivation to use and confidence in using ePortfolios.
    • This survey will be repeated every year to assess for mean-level changes in motivation to use and confidence to use ePortfolio
    • Based on results from this survey, we will conduct student focus groups among specific student subgroups to delve deeper into general findings from the motivational survey
      • Based on the results of implementation assessment, Career Connect will conduct focus groups with UNT faculty and staff to delve deeper into implementation challenges and opportunities
  • Content analysis of students’ reflection answers—Career Connect plans to conduct content analysis of students’ reflection answers using NVivo
 

[1] These definitions are taken directly or paraphrased from the AAC&U website at https://www.aacu.org/value-rubrics.

[3] Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159 (see http://www.bwgriffin.com/workshop/Sampling%20A%20Cohen%20tables.pdf).