Retrieving Data. Wait a Few Seconds and Try to Cut or Copy Again.

GTA Online. Infamous for its slow loading times. Having picked up the game again to terminate some of the newer heists I was shocked (/s) to detect that it still loads just equally dull every bit the day it was released seven years ago.

It was fourth dimension. Time to get to the bottom of this.

Recon

First I wanted to check if someone had already solved this problem. Virtually of the results I plant pointed towards anecdata about how the game is so sophisticated that information technology needs to load so long, stories on how the p2p network architecture is rubbish (not saying that information technology isn't), some elaborate ways of loading into story mode and a solo session after that and a couple of mods that allowed skipping the startup R* logo video. Some more reading told me we could save a whopping 10-30 seconds with these combined!

Meanwhile on my PC…

Criterion

1 2 3 4 five half dozen 7 viii

Story way load fourth dimension: ~1m 10s Online fashion load time: ~6m flat Startup menu disabled, time from R* logo until in-game (social club login fourth dimension isn't counted). Old but decent CPU: AMD FX-8350 Inexpensive-o SSD: KINGSTON SA400S37120G We take to have RAM: 2x Kingston 8192 MB (DDR3-1337) 99U5471 Good-ish GPU: NVIDIA GeForce GTX 1070

I know my setup is dated simply what on earth could accept 6x longer to load into online fashion? I couldn't measure out any divergence using the story-to-online loading technique as others have found before me. Fifty-fifty if it did piece of work the results would exist down in the noise.

I Am (Not) Lonely

If this poll is to be trusted then the consequence is widespread enough to mildly annoy more than eighty% of the player base. It's been 7 years R*!

Looking around a bit to find who are the lucky ~20% that go sub three minute load times I came across a few benchmarks with high-end gaming PCs and an online mode load fourth dimension of about 2 minutes. I would kill hack for a 2 minute load time! It does seem to be hardware-dependent but something doesn't add up here…

How come up their story mode still takes virtually a minute to load? (The M.2 one didn't count the startup logos btw.) Too, loading story to online takes them only a minute more while I'm getting almost five more. I know that their hardware specs are a lot better but surely not 5x better.

Highly accurate measurements

Armed with such powerful tools as the Chore Director I began to investigate what resources could be the bottleneck.

Afterwards taking a infinitesimal to load the common resource used for both story and online modes (which is near on par with high-end PCs) GTA decides to max out a single core on my auto for four minutes and do zippo else.

Disk usage? None! Network usage? In that location'southward a flake, just information technology drops basically to nix afterwards a few seconds (autonomously from loading the rotating info banners). GPU usage? Zero. Retentiveness usage? Completely flat…

What, is information technology mining crypto or something? I smell code. Really bad lawmaking.

Single thread-jump

While my old AMD CPU has 8 cores and it does pack a punch, it was made in the olden days. Back when AMD'south single-thread performance was mode behind Intel's. This might not explicate all of the load time differences only information technology should explain most of it.

What's odd is that it'due south using up just the CPU. I was expecting vast amounts of deejay reads loading up resources or loads of network requests trying to negotiate a session in the p2p network. Only this? This is probably a problems.

Profiling

Profilers are a keen fashion of finding CPU bottlenecks. In that location'south just one problem - most of them rely on instrumenting the source lawmaking to get a perfect pic of what'southward happening in the process. And I don't have the source code. Nor do I need microsecond-perfect readings - I have 4 minutes' worth of a clogging.

Enter stack sampling: for closed source applications there's only one option. Dump the running procedure' stack and current instruction pointer'due south location to build a calling tree in gear up intervals. So add them up to get statistics on what's going on. At that place's merely one profiler that I know of (might exist ignorant here) that tin practice this on Windows. And information technology hasn't been updated in over 10 years. Information technology'south Luke Stackwalker! Someone, please requite this project some dearest :)

Normally Luke would group the same functions together just since I don't have debugging symbols I had to eyeball nearby addresses to gauge if information technology's the same place. And what do we see? Not one bottleneck but two of them!

Down the rabbit hole

Having borrowed my friend's completely legitimate copy of the manufacture-standard disassembler (no, I actually can't beget the matter… gonna learn to ghidra one of these days) I went to take GTA autonomously.

That doesn't look right at all. Near loftier-profile games come with built-in protection against reverse engineering to keep away pirates, cheaters, and modders. Not that it has ever stopped them.

There seems to exist some sort of an obfuscation/encryption at play here that has replaced nearly instructions with gibberish. Not to worry, we merely need to dump the game's memory while it's executing the role we want to look at. The instructions accept to exist de-obfuscated earlier running one mode or some other. I had Process Dump lying effectually, so I used that, but there are plenty of other tools available to do this sort of thing.

Problem one: It'south… strlen?!

Disassembling the now-less-obfuscated dump reveals that one of the addresses has a label pulled out of somewhere! It'south strlen? Going down the phone call stack the side by side ane is labeled vscan_fn and after that the labels terminate, tho I'm fairly confident information technology'south sscanf.

Information technology's parsing something. Parsing what? Untangling the disassembly would take forever so I decided to dump some samples from the running process using x64dbg. Some debug-stepping later information technology turns out it's… JSON! They're parsing JSON. A whopping 10 megabytes worth of JSON with some 63k particular entries.

1 ii iii 4 v 6 7 8 9 10 11

..., { "cardinal": "WP_WCT_TINT_21_t2_v9_n2", "toll": 45000, "statName": "CHAR_KIT_FM_PURCHASE20", "storageType": "BITFIELD", "bitShift": 7, "bitSize": 1, "category": ["CATEGORY_WEAPON_MOD"] }, ...

What is information technology? It appears to be data for a "net shop catalog" co-ordinate to some references. I assume it contains a list of all the possible items and upgrades you tin buy in GTA Online.

Clearing upwards some confusion: I beleive these are in-game money purchasable items, not directly linked with microtransactions.

But 10 megs? That's nothing! And using sscanf may not be optimal but surely it's not that bad? Well…

Yeah, that's gonna take a while… To be fair I had no thought near sscanf implementations called strlen then I can't blame the developer who wrote this. I would assume information technology but scanned byte by byte and could stop on a NULL.

Problem two: Let's utilize a Hash- … Assortment?

Turns out the second offender is called right next to the first i. They're both even called in the same if statement equally seen in this ugly decompilation:

All labels are mine, no idea what the functions/parameters are really called.

The second problem? Right after parsing an item, it'southward stored in an array (or an inlined C++ list? not sure). Each entry looks something similar this:

1 2 3 4

struct { uint64_t *hash; item_t *item; } entry;

But before it's stored? Information technology checks the entire array, one by one, comparison the hash of the item to meet if it's in the list or not. With ~63k entries that's (n^2+north)/two = (63000^2+63000)/2 = 1984531500 checks if my math is correct. Virtually of them useless. You lot have unique hashes why not apply a hash map.

I named it hashmap while reversing but it's clearly not_a_hashmap. And it gets even meliorate. The hash-assortment-list-affair is empty before loading the JSON. And all of the items in the JSON are unique! They don't even need to check if information technology's in the list or not! They even have a function to straight insert the items! But use that! Srsly, WAT!?

PoC

At present that's overnice and all, but no one is going to accept me seriously unless I test this then I can write a clickbait title for the mail.

The program? Write a .dll, inject information technology in GTA, hook some functions, ???, profit.

The JSON problem is hairy, I can't realistically supersede their parser. Replacing sscanf with 1 that doesn't depend on strlen would be more realistic. But at that place's an even easier manner.

• claw strlen
• await for a long string
• "cache" the showtime and length of it
• if it'southward called once again within the string's range, return buried value

Something like:

i ii three iv 5 six 7 8 ix 10 11 12 xiii xiv 15 16 17 eighteen 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

size_t strlen_cacher (char* str) { static char* start; static char* end; size_t len; const size_t cap = 20000; if (start && str >= start && str <= end) { len = finish - str; if (len < cap / 2) MH_DisableHook((LPVOID)strlen_addr); render len; } len = builtin_strlen(str); if (len > cap) { start = str; end = str + len; } render len; }

And every bit for the hash-array problem, it's more straightforward - only skip the indistinguishable checks entirely and insert the items straight since we know the values are unique.

1 2 3 4 5 vi seven 8 ix x 11 12 13 14 15 16 17 18 nineteen 20 21

char __fastcall netcat_insert_dedupe_hooked (uint64_t catalog, uint64_t* cardinal, uint64_t* item) { uint64_t not_a_hashmap = itemize + 88; if (!(*(uint8_t(__fastcall**)(uint64_t*))(*item + 48))(item)) return 0; netcat_insert_direct(not_a_hashmap, key, &item); if (*key == 0x7FFFD6BE) { MH_DisableHook((LPVOID)netcat_insert_dedupe_addr); unload(); } render 1; }

Full source of PoC here.

Results

Well, did it work and so?

1 2 3 iv 5 6

Original online style load time: ~6m flat Time with only duplication check patch: 4m 30s Fourth dimension with only JSON parser patch: 2m 50s Time with both issues patched: 1m 50s (six*60 - (1*60+50)) / (6*60) = 69.4% load fourth dimension comeback (nice!)

Hell yep, information technology did! :))

About likely, this won't solve everyone's load times - in that location might be other bottlenecks on different systems, but information technology'south such a gaping hole that I have no idea how R* has missed information technology all these years.

tl;dr

• There'southward a single thread CPU bottleneck while starting up GTA Online
• It turns out GTA struggles to parse a 10MB JSON file
• The JSON parser itself is poorly built / naive and
• After parsing there'south a deadening detail de-duplication routine

R* please set

If this somehow reaches Rockstar: the problems shouldn't take more than than a day for a single dev to solve. Delight do something about information technology :<

You could either switch to a hashmap for the de-duplication or completely skip it on startup as a faster prepare. For the JSON parser - simply swap out the library for a more than performant one. I don't recall in that location's whatsoever easier manner out.

ty <3

Small update

I was expecting to get some attention but nowhere near this much! Subsequently reaching the acme of HN this mail service has spread similar wildfire! Thanks for the overwhelming response :)

I'll practice more than writing if something interesting comes forth, only don't expect anything of this scale soon - there was a lot of luck involved.

A few people suggested spamming this post to Rockstar'south back up - please don't! I'chiliad sure they've seen this by now. Standing would only bog down support tickets for everyone else. Social media is fair game in my volume tho.

Several HN comments suggested I add a donate push button, as they would like to buy me a beer (thank you!) and so I'm placing a link in the footer.

Cheers for reading and all the support :)

Update 2021-03-15

• Got confirmation from R* that this is getting a fix soon
• Just got awarded $10k through their H1 in-game bounty equally an exception :)) (usually only for security issues)
• Trying to figure out what's a W8 and how to fill information technology (lol)
• I did endeavor asking for more technical details only they couldn't say anything
• Will do some other benchmark on my aforementioned old setup as presently as the update is out, I'yard sure their engineers won't disappoint :)

Update 2021-03-xvi

R* released the update! Downloaded it and got my offset run results - same hardware, same measurement - from R* logo to fully online.

Fully fixed! t0st approves!

Thanks again for all the coffees, and thanks to R* for taking the fourth dimension to expect into this and the generous bounty!

rogertheept.blogspot.com

Source: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Share this post