Supplemental Playtesting Report 4/6/2021
This document is extra work I did for a school project. It is also the data we worked with to make our changes, discussed later. The game we were playtesting was not created by us, it was the FPS Microgame made by Unity. (https://learn.unity.com/project/fps-template)
Our versions were uploaded to itch.io:
Unmodified: (https://oplinger.itch.io/fpstesting)
Modified: (https://oplinger.itch.io/fpstesting2)
All work shown here was done by me.
Our initial report was not sufficient to cover the measurables and outcomes we had initially planned to cover. I will be using the data we initially collected to try and find workable solutions, if any, to the initial hypothesized problems, or find if they are problems at all.
MEASURABLES
Our stated measurables were to be:
1. How long a player is in a certain section of a level?
2. How long a player is using a weapon?
3. What mechanics the player used the most?
4. Where the player died the most?
5. The average time it took a player to progress through each section?
We also discussed measuring play time, readability, and game speed.
THE EASY STUFF
There are some things we did not specifically gather data for, but information can be gleaned from the survey data.
GAME SPEED
“Game speed” is more perceptual than it is quantitative, but we actually put in a couple great questions to learn about it.
“How much time did you feel like you were playing for?”
This question specifically asks for how long participants felt like they were playing.
Figure 1 units in minutes
While the plurality of players felt they were playing for around 5 minutes, the majority actually said it felt like longer. With one feeling like they played for 20 minutes! This normally would not be a problem, the responses did not feel negative, however when we look at actual play session data
Figure 2
These durations are slightly misaligned. The overwhelming majority of sessions were under 4m 25s. Only 3 sessions could be above 10 minutes, where the majority of our players felt like they were. This could point towards the game being too boring, or too slow. Therefore gameflow would need to be adjusted. For example changing the level to be more linear, or smaller to decrease down time where time perceptions tend to lengthen.
“Did you feel the guns reload speed accurately matched the amount of damage they did?”
“How did you feel about the movement- did you feel fast enough? Was it clear where you would and would not be able to move to with your movement abilities?”
I want to lump these two questions together, as they sort of cover similar things. How fast does the game feel, “Game speed” may be a little ambiguous as a term, so I want to cover another possible way to understand it. Luckily we have questions for that. First, reload speed vs damage output isn’t exactly what we need, but since our responses were not multiple choice, there may be some information in them. Namely these responses:
· Shotgun was the only one that seemed a little too long in-between shots.
· The shotgun's reload speed took too long
· Pistol felt a bit slow to reload
· The burst rifle felt slow
· Felt a little slow
This is 27% of respondents feeling some weapons reload too slowly, which would impact “Game Speed” and could lead to our game feeling sluggish or boring to more than a quarter of our players. Some fixes for that would be to speed up reloads, or have a way to get around it (active reload from Gears of War, or weapon switching like Doom Eternal) or make the game more methodical which would require large amounts of effort.
The second question I think has a more direct correlation to “Game Speed” as if the player moves too slowly you absolutely feel it as a player, and it can easily become frustrating. The second half of the question is irrelevant for this. We have these responses that are usable:
· I hate saying this, but movement is too fast. I love the movement. It feels fast, fluid, and being able to sprint sideways while firing is fantastic. But from the perspective of the typical person playing, think movement speed is too fast, especially while sprinting.
· Pretty fast, I didn't really use sprint
· Movement felt fine. I did forget mid game that I could sprint so wasnt able to test that out much. And jetpack movement also felt fine.
· The speed was fast, like Quake, but there was no 'bhopping', which I guess is a niche thing. The jetpack thing was a nice surprise
· It was clear, and I felt that I could outpace the bullets of the larger enemies, but I had to be moving at all times, so it was a good tension of mixing movement and gunplay.
· I thought that the normal speed was enough. Running was unnecessary
· The movement was good, a bit overwhelming indoors when you move so fast
Oh.
Okay so that is clearly not a problem, movement may even be too fast.
All of this combined, it feels like participants maybe overestimated playtime due to how fast they were zipping through the game, and tacked on some extra time to meet with what they expected their time should be. So it is likely that “Game Speed” is not a problem.
READABILITY
We asked a couple of questions relating to readability.
“Were you able to tell what you were shooting and if you hit it?”
100% of respondents said yes.
“Was it apparent you could change guns?”
95% said yes.
“Was the game properly readable- Were you able to distinguish between enemies and scenery during fast-paced moments?”
100% said yes again.
I don’t feel like readability was an issue. Although we do not have any in-game metrics to measure this, how players felt may be more important in this case.
THE HARD STUFF
DEATHS
One of our actual measurables was to see where players were dying the most. The majority of our players did, however, die from falling. 12/22 deaths were from this. I did a very quick and possibly error prone calculation of…adding up the X Y and Z values of where the player died for the non-falling deaths and looked at any that were close. There was a single grouping around the -50 area. It seems the highest death count is with Hoverbot (2) near the beginning, and this bot is near to turret 1 and hoverbot 1. Meaning this area had the mosts deaths, I assume from players not knowing the game.
Figure 3
Figure 4
This heatmap shows us roughly where people have died (the 10 is fall deaths, as they report at 0,0,0) I wouldn’t say there is any specific spot here where players are dying very often.
SECTION LENGTH
This is something we can very easily see with a heatmap.
Figure 5
The start, and slightly left of start are where most of the time is spent. Ignoring those however, there are a few interesting spots. For example the bottom area has a very high total duration. Almost as much as the larger top area. Time spikes almost entirely coincide with weapon pickups, like (60,20), but the spikes are way higher in the bottom area. I’m curious if that has to do with shooting the wall decorations or testing out weapons.
The other high duration areas seem to be doorways where enemies are located on the other side. This correlated to playstyles I saw personally and some of the survey responses. (0,10) and (10,10), for example, are the doorway to the top area, where 3 hoverbots are located. I don’t think this data shows any significant issues with how long players are spending in a section, or that any section needs to be redesigned. Especially when considering most players went to the bottom area first.
TIME USING EACH WEAPON
Figure 6
On average players vastly preferred using the blaster. This average however may be somewhat skewed as the blaster is also the starting weapon. The shotgun is interesting here, participants reported not using it much due to its low range, since enemies can be safely sniped with basically all other weapons.
I also looked at number of kills per weapon:
Figure 7
Still the blaster seems to be most preferred. The blaster may be overpowered compared to the other weapons, or the other weapons may not feel useful. Feedback has some comments about the usefulness of the other weapons. It does get interesting though when we combine these two:
Figure 8
The number of kills per minute used shows that the disc launcher can maybe do the highest overall damage output, or players were not holding it for very long.
CONCLUSION
Things we wanted to look at with this playtest include:
· Game Speed
· Player Deaths
· Readability
· Play Time
· Length of Time In Level Sections
· How Long Players Used A Weapon
Based on this data, I think it is safe to say we do not have a game flow problem. However there is a perception of our game being longer than it is (Fig.1&Fig.2.) There may be some things in the game that break players out of a flow state and cause them to perceive time slower, or their estimates may just be how long they think it should have taken them. The game being as quick as it is, I believe players were not frustrated at not knowing where to go. They seem to be most frustrated about falling off/through the level.
Outside of game flow, players did not seem to die in any one place very often. In fact, 11 of the 26 sessions did not die at all. Players seemed drawn to the bottom area, so there is a gathering of deaths there(Fig.4), but I believe that is due to learning the game, not necessarily difficulty. Though we may want to reduce the number of bots there to 1 to help ease players into the game.
Readability was not an issue according to survey data. Though some data gathered points towards the wall decorations in the bottom area where the disc launcher is causing undue stress. So around those spots we see a spike in time taken.
Play time was lumped into game speed, as the game is very short. With players taking about 4 minutes to end the session.(Fig.2)
As for how long players spent in each section of the level, as described above players spent more time in the disc launcher section of the map, and it appears that many players were drawn there first(Fig.5.) Players also may have been distracted by the wall decorations. Otherwise players did not spend a significant amount of time anywhere.
Lastly we wanted to know how long players used each weapon, with the blaster being the overwhelming favorite.(Fig.6, Fig.7) In looking at this data however, the amount of kills for the time used showed that the disc launcher is quite the powerhouse. (Fig.8) This could point to our weapons being out of balance and may need to be looked at.
THINGS NOT LOOKED AT
I did not look at what mechanic players used most. That would have invariably been shooting. Survey results show that many people would have liked the jetpack to be more useful.
I also did not look at the average time per player per section, as….i don’t know how to do that in excel and I can’t just manually pull that from the data. I think the total section time is a good indicator for this question.
RECOMMENDATIONS
My recommendations for changes we should focus on is the disc launcher area. It seems that this area slows many players down and causes a large amount of deaths for new players, suggesting that it may be too difficult for those starting off.
A secondary recommendation would be to look at the weapons, but that might be linked to enemy typings and end up too complex for a small change to test later. Though we could make some bots move faster to get closer to the player, making the shotgun more useful. We could also make the turrets immune or resistant to normal bullets, making the disc launcher more useful. The blaster and the Burst Rifle might just amount to play style, but maybe if we slowed the blaster down, it could make the Burst Rifle seem more interesting.
And my last recommendation would be to add colliders to level edges, and see about fixing players falling through the floor. This causes players a lot of frustration, and this could make the game feel better overall.
Changes
This section is the one I was responsible for in a larger document, but it discusses the changes we made to the above game and the data collected to support our conclusions.
We ended up reducing the range of the starting pistol, and removing the jetpack
TELEMETRY:
In our last iteration we had a lot of players taking their time. This led to a lot of deaths near doors, and an amount of sniping we were not happy with. We changed the distance with which bullets would travel, this resulted in more deaths within rooms and more deaths overall.
This is a heatmap of our previous version. The 10 in the center is fall deaths, as those register at origin. Doorways seemed to be a problem, and I assume this means players were sniping from doorways and getting caught.
After our changes we had an increase in player deaths, but many more deaths were now located inside rooms rather than huddled around doorways. We solved one problem of players hanging out around doorways and sniping, but we may have made the game too difficult with the spike in deaths! More tweaking is probably needed.
In our old session players took their time in some sections
We had a lot of players taking their time in the bottom area. Overall almost half an hour for such a small area.
In our modified game, playtesting sessions went much faster. Average session length this time was only 2:38. Though we also had fewer victories; only 10. We did seem to increase our problem spot, the bottom area.
For our new session, we again see it taking much longer than anticipated. Again almost as much as the far larger top area. This might indicate that this area’s difficulty has been unchanged, and was not resolved by a more aggressive playstyle or the shorter range of the guns.
Comments