Message from Matt

An update on RIOT, the Engine Upgrade, and Customer Support

AN UPDATE ON RIOT BETA

Listening to feedback and short-term plans.

First, many players have asked why we "wasted" time doing RIOT, and I think that's a valid question.

The mechanics for RIOT were specifically designed to give my team a chance to learn how the inner workings of APB fit together. I felt that we could implement this mode without taking a lot of dev resources and we could keep our focus on the Engine Upgrade. Not only that, the experience would give us an excellent sandbox to make mistakes. If we messed up RIOT, we could simply set it aside with no lasting damage. If we mess up the main game, then it becomes much more difficult to fix.

The design for RIOT also included a number of underlying systems that we need. Some systems are focused on design elements such as cleaning up the district blocks or tracking new aspects of play. Other systems focused on replacing Joker Mystery Boxes. It's clear that legislation will eventually make these illegal, so we needed new ways of building value that our players can spend money on.

Next, here are some interesting stats for the first week:

• RIOT was played 4x more on Citadel than on Jericho due to low population issues starting the mode

• Fight Club ran 2.2x more matches than RIOT during the same week on Jericho, but 92%+ of accounts playing FC also played RIOT

• Fight Club ran 1.2x more matches than RIOT during the same week on Citadel, but we saw 1.5x more accounts in RIOT than FC

• ~40% of the active player base during the week played RIOT at least once

We are spending a lot of time going through all the feedback here on the forums to see what is feasible. There have been lots of great suggestions.

There are clear issues with the mode, and the team is working on a relatively light weight pass of tweaks and fixes to make it more fun. Areas that need work are the end-game, sound, and respawns.

Right now, the big decision I need to make is whether to:

(1) take the mode down until it has seen major work or

(2) add some more tiers along with another reward that will enable players to continue grinding Joker Tickets while we make changes.

I'm not interested in clinging to or defending something that isn't working, but I still see some value in the mode itself - just perhaps not in the original way we intended.

We are not moving forward with any Battle Pass or Seasonal content until the kinks are worked out.

3.5 ENGINE UPGRADE UPDATE

For PC, everything is now done except for migrating Unreal package content.

To give specifics, there are 1498 packages used in the game:

• 919 regular packages
• 537 map packages including dynamic layers to action districts for events and missions

So far we have converted about ~83% of them:

• 706 regular packages
• 542 map packages

That leaves 249 packages to migrate, but these are all somewhat more difficult.

• Some of them wont be migrated because G1 or LO already made changes that weren't ever in 3.0. This includes reworked Financial and Waterfront textures and geometry.
• Some of the packages are for User Interfaces. These are tricky because we have a chunk of UIs in 3.5 already that have been glued up for console. And we have new UIs in 3.0 for the trading system and other areas. Anything we migrate from 3.0 will step on aspects of the 3.5 screens and work will need to be redone. I'll have a better sense of this effort once we get a little further.
• Some of the remaining packages have changes in both 3.5 and 3.0 and those will need to be hand merged. Again, I'll have a better sense of this effort once we get a little further.

We still have plenty of bugs and crashes that need to be fixed, but we now have all the code, systems, and server work completed.

Performance Benchmarking

Performance testing of the Engine Upgrade is very important. Not many people would trade better looking graphics for worse performance. In APB's case, there are currently disk-based stutters and bad support for multiple CPU or GPU cores. One of our primary goals with the Engine Upgrade is to fully utilize modern hardware.

DISCLAIMER: It is important to point out that the Engine Upgrade has an entirely different rendering pipeline - some good and some bad. Many players have seen what the console builds look like, and the performance on those devices is terrible. The starting point of the Engine Upgrade was these exact console versions just running on PC, so there has been a lot of work to do since we started. Unfortunately most of the work is very specialized and cannot be done by a huge team without tripping over each other. We have kept as many people dedicated to the upgrade as possible, but this work is filled with rabbit holes and pitfalls. Sometimes areas that appear simple to optimize end up using global data that touches many other areas of the code.

I want to thank @SKay for all of his hard work leading the benchmarking effort to help us gauge progress in these areas.

To make sure the test runs were valid, they had to be able to easily reproduceable as well as being able to give a proportionate representation of what a user might see during their time in APB. The game environments vary from claustrophobic to wide open, and the benchmarks needed to account for both. In the end, he chose a short car drive that would force the game to move quickly through as many different areas as possible under consistent conditions.

The main goals of the benchmark are to insure that:

• The 3.5 performance would at least reach the same speeds (on average) as LIVE
• The main issues (stutters, low 1%' and .1% frame rates) were at least looked at and/or addressed
• New visual upgrades or changes didn't affect the overall experience

The route above shows the benchmark run that was used in all cases. The test run as outlined above has open areas and enclosed areas in a single run that could be repeatedly reproduced for apples-to-apples comparisons to help identify progress with the Engine Upgrade.

Starting at Darryl Kent, we take a left out and stay on the road passing right of Suji. Taking the next right and keeping the City Council building to our left, we take a right at the crossroads outside of Violet Prentiss and Wilson LeBoyce, following the road as it sweeps towards Javez and passing him on the right. We hold that road and turn right when it meets the river, coming back to end in front of Darryl Kent.

Stats were captured with MSI Afterburner using RivaTuner Statistics Server running on Ultra settings 4K resolution.

Here are the benchmark results:

To explain the numbers for the graph:

• Max FPS is the highest framerate detected during the test. This in conjunction with the other metrics can determine if the game is able to push good framerates on the current hardware.
• Avg FPS is the average of all the frames that were taken during a run. This is used as a base metric to compare the other metrics in this list with.
• Min FPS is the lowest framerate detected. This is not necessarily the lowest framerate outright, rather the lowest framerate that could be normally detected through normal gameplay.
• 1% FPS is the average of the lowest 1% of frames that were taken during a run. This metric, when compared to the average, is used to determine if the PC is struggling to run the game, or if the game is struggling to drive a consistent frame rate.
• .1% FPS is the average of the lowest .1% of frames that were taken during a run. This metric is used to detect for stutters or freezes (if any)

NOTE: The benchmarks do not included the non Multi Threaded renderer results. The state of the renderer was bad enough when we started that it was clearly well below LIVE and not acceptable.

Here is a quick summary of our development timeline on the Engine Upgrade:

• May 2018, we chose to focus on infrastructure which took us a couple months to clear the cobwebs
• End of July I posted my first update on the Engine Upgrade with details around bringing on more devs for the Unification project
• November we rebuilt the Music Editor for 3.5 and the majority of external libraries had been upgraded just to get the build running
• Early February we had finally sorted out a number of crash bugs that allowed SPCT to get their first play tests on 3.5
• March we had realized how bad the renderer was and had implemented most of Multi Threaded rendering

And here is a walk through of the results working up from the bottom:

• The bottom shows our LIVE comparison which was taken in the March / April time frame.
• The first set of results above the bottom was from the initial Multi Threaded renderer implementation back in April. These results were very encouraging, however in-game lighting was completely broken in this build, so the results were skewed higher than any other build that had lighting working correctly.
• 806538 shows what the initial finished implementation of Multi Threaded rendering from May. This was our first good look at performance, but it revealed several areas in the renderer that needed work - mostly in the Culling and Level of Detail systems. Culling is what decides which elements on screen to render and which to ignore. Level of Detail (LOD) determines if we can show a lower resolution model based on distance.
• 809566 shows some early Culling work from late May, but the routine had severe graphical issues with objects not getting rendered properly.
• 810873 shows our current build that is rendering properly with Culling and LOD fixes.

CHECKING IN ON CUSTOMER SUPPORT

Addressing frustrations and extended response time.

This has been an ongoing issue. Clearly my staff was too small, and our trouble ticket queues ballooned as soon as we announced unbanning players. Since then we have been hiring staff and improving tools to help speed things up. As of today, there are 819 total tickets (unsolved) which continues to come down week by week, and there are 523 total tickets that have not been touched (contacted). Today we are answering tickets from June 5th, which means we are now at a 23 day response time.

23 days to get a response is still far too long for support turnaround, but these are the lowest numbers since we took over last year.

I try to update the CS stats on Friday. You can see progress here:
https://forums.gamersfirst.com/topic/810-tracking-customer-support/page/2/

Thank you,
Matt

Got any questions? Want to chat with other players? Then discuss this article in our Forums! You can also follow us on Twitter and Facebook.