Project Overview
My teammates Edward Cody, Nick Menough, Lenet Ron, and I worked together on our final project for IE 6810: Cognitive Engineering, where we analyzed teamwork dynamics by measuring interaction metrics like trust, cohesion, and response time during collaborative gameplay.
Introduction
For our final project in IE 6810 (Cognitive Engineering), we were tasked to conduct a study to evaluate interaction metrics.
Our team decided to examine and evaluate the interaction metrics of playing the game Unrailed!
Goals of the study
- Determine what elements cause team success
- Predict "success" in terms of Unrailed! gameplay
My Role
This project was a collaborative effort with Edward Cody, Nick Menough, and Lenet Ron as part of our final project in IE 6810: Cognitive Engineering. My primary contributions included:
Literature Review: Researched and summarized prior work on team cohesion, communication frequency, task distribution, and trust to inform our study design.
Data Collection & Analysis: Collected and analyzed gameplay data from Team B (YouTube players) to compare against our team’s (Team A) performance, helping identify differences in communication and task allocation strategies.
Synthesis & Reporting: Collaborated with the team to integrate findings into the final report and presentation, ensuring clear connections between literature, data, and results.
What is "Unrailed!"?
Unrailed! is a fast-paced cooperative multiplayer game that allows up to four players to work together in real time. The game challenges players to lay down train tracks across procedurally generated worlds that vary in terrain, obstacles, and resource availability.
The goal of the game
The primary objective is simple but intense: keep the train moving and prevent it from derailing for as long as possible.
To achieve this, players must gather resources, craft tracks, and coordinate roles under constant time pressure. The train never stops moving, which creates a dynamic environment where collaboration, planning, and quick decision-making are essential for success.
Key Game Elements
- Water Bucket – Used to cool down the train’s engine before it overheats.
- Pickaxe – Allows players to mine rocks for crafting new tracks.
- Axe – Used to chop trees for wooden resources.
- Crafting Wagon – Converts gathered resources into usable train tracks.
- Bridges – Placed over rivers so tracks can be laid across them.
- Bolts – Special collectibles that can be used between rounds to upgrade the train.
- Train Wagons – Each has a unique function (engine, crafting, storage) that players must maintain and protect.
Literature Review & Related Works
Our study was informed by key findings from research on team performance and trust:
Salas et al. (2008) – On Teams, Teamwork, and Team Performance
- This study synthesized 50 years of research on team communication, cognition, and effectiveness.
- The authors found that factors such as team composition, work structure, and task characteristics play a critical role in performance outcomes.
- These insights directly informed how we measured and interpreted team cohesion in our own study.
Feitosa et al. (2020) – Measuring Team Trust: A Critical and Meta-Analytical Review
- This meta-analysis examined how trust impacts team performance and reviewed the methods used to measure this relationship.
- Their coding framework guided how we calculated and categorized team trust metrics within our research.
Methodology
The goal of the study was to evaluate a comprehensive framework for measuring and understanding teamwork in a cooperative gaming environment.
Definition of Success:
Success was measured using six key metrics:
- Task Distribution – how evenly work was shared among players
- Response Time – how quickly teams reacted to critical events
- Team Trust – how confident members were in each other’s actions
- Team Cohesion – the quality of coordination and collaboration
- Team Skill Comparison – relative balance of player abilities
By analyzing team dynamics in Unrailed!, we aimed to build a replicable methodology that captures not only team performance but also the specific behaviors and interactions that drive success.
This research also sought to address a broader question:
How can cooperative tasks be optimized in dynamic, high-pressure scenarios?
Insights from this study are intended to inform the design of collaboration frameworks relevant to real-world settings, from workplace teams to emergency response units.
Data Collection
1. Gameplay Recording & Event Logging
- All sessions were recorded to generate event logs capturing player actions, communication, and critical events.
- Actions such as Mining Resources (MR), Track Placement (PT), and Resource Retrieval (RR) were manually annotated using Kinovea.
- Critical Events (CE) — including train fires or resource shortages — were tagged with precise timestamps to calculate response times.
- Communication was transcribed and categorized (frequency, context, and collaborative effectiveness).
2. Team Setup & Group Selection
We compared two distinct groups under the same methodology:
Team A - our research Team
- This group consisted of the four researchers (including myself) who played and recorded multiple gameplay sessions.
- Direct involvement allowed for detailed observation and firsthand insight into decision-making and coordination.
Team B - Stumpt Team
- This group was selected from the YouTube channel Stumpt and included four players with a mix of novice and experienced gamers.
- This team was chosen as a natural benchmark because they had played together for over four hours across several sessions and identified the analyzed session as their highest-scoring gameplay.
Both teams were evaluated across multiple sessions to ensure consistency and reliability of data. The same annotation process was applied to both groups to allow for a fair comparison.
Analysis: Metrics of Measurement
After collecting event logs and transcripts, we transformed the raw data into measurable indicators of team performance. Four primary categories were defined: Team Trust, Team Cohesion, Team Skill Comparison, and Overall Team Success.
Each category was derived from a combination of quantitative metrics (task distribution, communication frequency, response times) and qualitative insights (dialog analysis, collaboration patterns). The following subsections describe how each metric was calculated and why it was important for evaluating team performance.
Team Trust
Team trust was derived from three core indicators:
- Task Distribution: How evenly players shared responsibilities (Task Distribution Score).
- Communication Frequency: Number of messages or verbal exchanges per minute.
- Response Time: Speed at which players reacted to critical events.
We used a weighted formula to calculate a Trust Metric:
Team Trust Metric=w1(Task Distribution Score)+w2(Communication Frequency)−w3(Response Time)
Weights (w₁, w₂, w₃) were assigned based on their relative importance, with communication frequency given the highest weight due to its strong predictive value for trust.
For example, in one session:
- Task Distribution Score = 0.7
- Communication Frequency = 5.2 messages/minute
- Response Time = 3 seconds
Plugging into the formula:
0.4(0.7)+0.4(5.2)−0.2(3)=2.08
This metric allowed us to compare trust across teams and sessions using a consistent scale.
Team Cohesion
Cohesion was evaluated by analyzing the tone, content, and alignment of player dialogue. Key indicators included:
- Presence or absence of trust (e.g., “You got it?” “Thanks!” show reliance and mutual respect)
- Level of agreement or disagreement in decision-making
- Motivation and encouragement expressed during play
- Overall alignment toward shared goals
This qualitative analysis helped capture the “soft factors” of teamwork that are not always visible in quantitative metrics.
Team Skill Comparison
We examined the relative skill level of players within each team by observing:
- Frequency of errors or missteps
- Ability to recover from mistakes
- Efficiency in task completion
Skill comparisons helped us understand whether differences in performance were due to strategy/communication or simply individual skill gaps.
Team Success
Team Success was defined as the ability to complete tasks efficiently and adapt effectively under dynamic, high-pressure scenarios.
We computed a weighted performance index that combined all previous metrics (trust, cohesion, communication frequency, and response time).
Our process included:
- Assessing annotated transcripts and in-game events
- Applying weights to each factor based on its contribution to team performance
- Comparing scores across Team A and Team B to determine which group exhibited stronger teamwork overall
Key Insight: Teams with higher trust, quicker responses, and more balanced task distribution consistently achieved higher success scores.
Results: Overall Team Success
Using the weighted performance index, we compared Team A (our research team) and Team B (Stumpt).
After calculating the metrics for both groups, we visualized the results to highlight key differences in trust, cohesion, and overall success scores.
Key Findings
Team A Outperformed Team B Overall
- Team A achieved a significantly higher success index (2.48) compared to Team B (0.80), driven by strong communication frequency, high trust, and faster response times.
Communication Was the Strongest Predictor of Success
- Communication frequency accounted for 60% of the variability in team success. Team A averaged 8.53 messages/minute, which supported faster decision-making and task execution.
Trust and Task Distribution Influenced Adaptability
- Team A’s higher trust score (2.54) contributed to efficient delegation, though their uneven task distribution (0.63) highlighted room for workload balancing.
Team B Showed Strength in Cohesion and Task Balance
- Team B distributed tasks more evenly (0.72) and demonstrated stronger cohesion, but their lower communication frequency (1.83 messages/minute) and slower response times (6.0s) limited adaptability in high-pressure moments.
Cohesion Alone Was Not Enough
Despite stronger collaboration indicators, Team B’s lower trust and communication made it harder to recover during critical events.
Implications
Our findings highlight that communication frequency was the strongest predictor of team success, accounting for 60% of performance variability. Teams that communicated more frequently adapted faster and completed tasks more efficiently. Trust and task distribution also played key roles, though they were secondary to communication.
Interestingly, Team B’s strong cohesion did not offset their lower communication rate, suggesting that collaboration alone is insufficient without frequent, clear exchanges of information.
These insights can guide the design of collaborative training programs and teamwork frameworks, emphasizing the need to foster real-time communication and trust to improve performance in dynamic, high-pressure environments.
Conclusion
Team A outperformed Team B in overall success, driven by significantly higher communication frequency, trust, and responsiveness.
These strengths allowed them to adapt quickly to dynamic in-game challenges, completing tasks efficiently under pressure.
While Team B excelled in task distribution and cohesion, their slower response times and lower communication limited their ability to succeed in a fast-paced environment, which underscores the critical role of communication and trust in cooperative teamwork.
Limitations
Our study faced several challenges that shaped the interpretation of results:
Transcript Accuracy:
WebEx transcripts contained frequent errors and often missed overlapping speech. Manual review was necessary, introducing potential human error.
Reliance on Verbal-Only Communication:
Without nonverbal cues (body language, gestures), teamwork relied solely on tone and word choice, which can slow decision-making and reduce clarity compared to real-world settings.
Self-Selection Bias:
We analyzed our own gameplay (Team A), which may have introduced bias when comparing to Team B’s footage.
Pre-Existing Rapport:
Three of four Team A players had collaborated previously, which may have boosted communication and trust relative to Team B.
Horizontal Team Structure:
Lack of formal roles may have influenced team cohesion and collaboration, potentially skewing performance metrics.
Future Work
Future iterations of this study could address these limitations by:
Capturing Nonverbal Cues:
Using video recordings to analyze gestures, facial expressions, and body language alongside speech.
Improving Transcription:
Leveraging advanced speech-to-text tools that handle overlapping dialogue more accurately.
Measuring Rapport:
Implementing pre- and post-game surveys to understand how prior relationships impact trust and teamwork.
Analyzing Silence:
Investigating the role of “silent periods” as a possible indicator of alignment, disengagement, or focus during gameplay.
Thank you for Reading!
Credits:
Created with images by teerawit - "Abstract background. Molecules technology with polygonal shapes, connecting dots and lines. Connection structure. Big data visualization." • Studio Romantic - "Concept of partnership creative work startup teamwork team business people." • khwanchai - "Assistance, teamwork and achievement concept. silhouette of man helping friend to success together" • adrian_ilie825 - "Human resources management or increase teamwork performance and leadership concept" • aFotostock - "Together Group diversity partner business meeting trust in businessman and team talking together asian Teamwork Collaboration Team Meeting Communication concept with Business people Working Together" • Monster Ztudio - "Conceptual of creative idea and innovation. Hand picked wooden cube block with head human symbol and light bulb icon and question mark symbol" • Andrii Yalanskyi - "Opening a padlock breaks the chain. Using the correct key or combination. Overcoming barriers or limitations. Security breach. Failure to fulfill a promise."