The Build-Measure-Learn of Player Feedback

Where, when, and how often developers choose to integrate player feedback has been a topic of focus for the gaming community ever since its inception 50+ years ago. There are dozens of ways that this process can take place (if it does at all) but the basic steps are, 1) sourcing of potential players, 2) distributing the game to those players, 3) obtaining feedback on the player experience, and 4) integrating that feedback into the game. The process, when implemented well, closely aligns with the Build-Measure-Learn startup methodology coined by Eric Reis.

While it sounds simple, leveraging player feedback to improve gameplay is fraught with potential pitfalls and missteps, including:

1. Sourcing: In today’s environment, user acquisition is difficult across the board. Sourcing players for feedback can range in complexity and scale, from leveraging third-parties with global workforces of dedicated video game players across different regions and backgrounds, to simply asking Discord followers to try out the new game. Sourcing has seen improvements in recent years; Solsten (Konvoy portfolio company), for example, utilizes player psychology to identify relevant audiences for games. ‍
2. Distribution: While distributing games to players can be as easy as sending a downloadable version of the game to a group of players, as games grow more sophisticated and as security becomes more important, it is critical to distribute the game in a safe and secure environment. One of the recent use cases of cloud gaming has been to distribute playable game tests in controlled environments so that no download is required, reducing the risk of leaks or theft.‍
3. Gathering Feedback: Player feedback can come in many forms from in-game telemetry data to a simple post-game interview or survey. Accurately collecting feedback through an organized and efficient process has largely eluded the industry. We believe that there is room to further streamline and standardize player feedback. ‍
4. Integrating Feedback: Integrating this feedback into games is a complex process. For small developers, this may be as simple as making the adjustment in-game after seeing a complaint on Twitter. For AAA developers, this process can be complex: aggregating bug reports, telemetry data, and player feedback into a single ticket that is funneled from quality assurance (QA) teams to developers.

While companies like Solsten have innovated by introducing a player psychology lens to each phase, we believe another unlock for the industry is an evolution of the type of data that can be captured and evaluated.

Player feedback can be objective, contextual, or somewhere in between. Objective data is quantifiable and measurable, while contextual data contains additional information about the player experience. Consider this chart of player feedback methodologies across the spectrum of objective to contextual:

Each of these methodologies have pros and cons, but we believe that supervised playtesting and telemetry data are the most prone for disruption:

1) Supervised Playtesting: Supervised playtesting consists of a member of the development team being present either physically or via a streaming service to walk through the game live with a player. The benefits of this can be impactful in ways that surveys and data cannot because the developer can understand the context of the situation and the type of player then ask targeted questions in real-time.

However, this approach is largely unscalable from a cost and logistics perspective given the requirements for a developer and player to be present at the same time.

2) Telemetry Data: This type of data is the most objective source of information and can be collected via code while the game is being played. This can be anything from levels completed to time spent on a certain interface, however, this data often lacks context. For example, you may find that a player spends 20 minutes in their inventory, however, it is unclear if the player is struggling to understand how it works or if they are admiring their collection of items. The type of data that can be captured is often limited to telemetry data and fails to capture how the player is feeling or experiencing your game.

Combining Data And Contextual Information:

A major unlock for the player feedback process could be the ability to combine telemetry data (objective, ongoing, scalable) and the benefits of supervised playtesting (contextual, real-time) to incorporate physiological data into the game feedback loop. Physiological data is anything that can be captured by monitoring the body, including heart rate and stress levels. Widespread access to this data, which is inherently contextual, will allow for more clarity in early stage metrics and enable developers to make more informed decisions as they continue to improve their games.

Today, game companies can hire labs to do a comprehensive analysis around consumer behavior. These tests can include monitoring heart rate, stress levels, facial coding, electroencephalograms (EEGs) and response latency. While we are excited to see this technology advance in testing labs, including more advanced options like Functional MRI (fMRI) or more detailed brain monitoring, we are most excited about the ability to take existing technology outside of the lab. This would enable large scale data capture of physiological information allowing for the standardization of player feedback.

This could come in many forms including:

1) Gloves or Wearable Devices: There are many physiological indicators that can be tracked through contact with the body including heart rate, oxygen and Galvanic Skin Response (measuring stress). This option could potentially be made available through existing devices like the Apple Watch or Whoop.

2) Brain Monitoring: One of the most complex approaches would be to monitor players’ brain activity during a session. This approach is particularly difficult given the quality of data and ability to effectively distribute devices.

Physiological data can help bridge the gap between objective data and playtesting by allowing players to provide real-time contextual data in the form of a physiological response, forgoing the need for someone to be in the room (physically or remotely). Now, when a developer sees a player in their inventory system for 20 minutes and has the data to prove they are highly engaged, the developer knows that they have created an enjoyable system without the need for a post-game interview.

Layering on psychological data would then help developers understand the reasons why and more accurately hypothesize on how to continue to keep these players engaged. While this is likely to be an unlock for the industry, there are multiple problems prohibiting this from being widely applied in games:

1. Data Quality / Accuracy: Some data sources, specifically brain monitoring, are prone to noise (data that is difficult to understand or draw meaning from), which makes it difficult to directly correlate the data to key game improvement learnings.
2. Data Integration: Combining physiological data with other data sources including telemetry, gameplay, and surveys is time consuming and manual today.
3. Distribution: Broadly distributing devices specifically made to capture this data is expensive and in some cases not commercially viable.
4. Privacy: Physiological data requires an extra level of privacy.

While we believe that these technologies could significantly impact gaming by allowing developers clearer insights into their games, we believe that the broad distribution of this technology is likely to have an even greater impact beyond gaming. When widely distributed, these new technologies could be applied to an array of consumer applications, capturing a more contextual form of user feedback for movies, books, art, social platforms, products, advertisements, and even politics.

The ability for a company to understand the type of physical response that a product or image is likely to instill in a user could radically change the way we approach advertising and marketing all together.

Takeaway: The integration of player feedback within the gaming industry stands at the doorstep of innovation / reinvention, with the potential for significant advancement through technological advancements. The balance between objective data and contextual insights from playtesting could be harmonized by creating new tools and widely distributing existing physiological data capture technology. Advancements in player feedback could unlock a more nuanced understanding of player experiences, enabling developers to refine their games more rapidly and effectively (leading to more retention, higher engagement, and more revenue).

We believe that gaming will continue to lead the charge of setting the standard for accurate and technical product feedback for not only gaming but enterprises across other industries (repeating the proliferation we saw with “gamification”).