Skip to content

Functions to control the streaming memory/buffer sizes [Fixes #3084]#3086

Merged
Pirulax merged 10 commits into
multitheftauto:masterfrom
Pirulax:streaming-mem-set
Jul 2, 2023
Merged

Functions to control the streaming memory/buffer sizes [Fixes #3084]#3086
Pirulax merged 10 commits into
multitheftauto:masterfrom
Pirulax:streaming-mem-set

Conversation

@Pirulax

@Pirulax Pirulax commented Jun 25, 2023

Copy link
Copy Markdown
Contributor

This PR aims to implement the function as discussed here

  • engineStreamingSetMemorySize
    • All positive values set the streaming size [=the maximum memory that can be used for models] to the value given [in Bytes]
  • engineStreamingGetMemorySize
    • Get the current streaming memory size [in Bytes]
  • engineStreamingRestoreMemorySize
    • Restore the streaming memory size to the cvar setting [Usually 256 MB]
  • engineStreamingSetBufferSize
    • Set the streaming buffer size - The larger it is, the more models can be loaded in one go [Though, the limiting factor is still the GTA streamer] - Can help a bit with custom IMGs loading speed [reduces pop-in]
  • engineStreamingGetBufferSize
    • Getter for the above...

Also added a new class EngineStreaming for all streaming-related functions, and 2 new class variables:

  • memorySize - The streaming memory size [in bytes] - Can also be set [uses engineStreamingSetMemorySize interally]
  • usedMemory - Amount of memory used right now for streaming

Example usage [and unit test]:

local initialSz = EngineStreaming.memorySize

EngineStreaming.memorySize = 1024 * 1024 * 1024
assert(EngineStreaming.memorySize == 1024 * 1024 * 1024)

EngineStreaming.restoreMemorySize()
assert(EngineStreaming.memorySize == initialSz)

assert(not EngineStreaming.setBufferSize(0)) -- Should fail, as the buffer size is too small
assert(EngineStreaming.setBufferSize(50 * 1024 * 1024)) -- Should not fail, this is a big enough buffer for the default gta3.img files

@TheNormalnij TheNormalnij added bug Something isn't working enhancement New feature or request labels Jun 26, 2023
@TheNormalnij

Copy link
Copy Markdown
Member

Some notices:

  • There is no error/waring when you call
engineStreamingSetMemorySize(9000 * 1024 * 1024)
-- or
engineStreamingSetMemorySize(-1)
  • GTA will not free memory immediately after call engineStreamingSetMemorySize (should be in doc)

@Pirulax

Pirulax commented Jun 27, 2023

Copy link
Copy Markdown
Contributor Author

That it something the argparser should do, not me.
I did make a [PR] (#2114), but nobody cared to review it for 2 years :D
Also, the streamer will [or, well, should anyways] free up memory, as it won't be allowed to load a new model before freeing up memory [in case it doesn't have enough memory everything will start flickering]

@CrosRoad95

Copy link
Copy Markdown
Contributor

"That it something the argparser should do, not me."
sounds like a bug, size_t is unsigned but probably it hasn't been checked while converting from lua_Number to size_t, but still you have variable upper bound, if someone have X GB of vram, you probably should not be able to set max streaming memory to more than that, i don't know what can happen ( maybe crash ) if you do this, probably gta itself care of it.

@Pirulax

Pirulax commented Jun 27, 2023

Copy link
Copy Markdown
Contributor Author

sounds like a bug, size_t is unsigned but probably it hasn't been checked while converting from lua_Number to size_t

That's exactly what #2114 aims to fix.

...but still you have variable upper bound...

You might say it's the amount of RAM the PC has - But it isn't as the OS can just use the swap file [slow, but would still work].
Also, this is an advanced feature, I expect scripters who use it to be smart about it [and not try to set it to a trillion gigabytes]

@CrosRoad95

Copy link
Copy Markdown
Contributor

wonder what happen if you set max memory stream to 0 😂

@Pirulax Pirulax changed the title Functions to control the streaming memory size [Fixes #3084] Functions to control the streaming memory/buffer sizes [Fixes #3084] Jun 27, 2023
@Pirulax

Pirulax commented Jun 27, 2023

Copy link
Copy Markdown
Contributor Author

Whole map starts flickering, nothing interesting

@TheNormalnij

Copy link
Copy Markdown
Member

Some notices:

  • There is no error/waring when you call
engineStreamingSetMemorySize(9000 * 1024 * 1024)
-- or
engineStreamingSetMemorySize(-1)
  • GTA will not free memory immediately after call engineStreamingSetMemorySize (should be in doc)

This is MTA issue. We don't perform this checks in new argument parser now. Can be fixed in next backward incompatible release via #2114. So this should not block this PR.

@Pirulax

Pirulax commented Jul 2, 2023

Copy link
Copy Markdown
Contributor Author

NOTE: Wiki pages need to be added

@Pirulax Pirulax merged commit 7ffc312 into multitheftauto:master Jul 2, 2023
@Dutchman101

Dutchman101 commented Aug 11, 2023

Copy link
Copy Markdown
Member

NOTE: Wiki pages need to be added

@Pirulax I hope you can add these functions to the wiki this soon, it's actually a responsibility of the feature's contributor and should have been checked off (or with a clear plan to work on it immediately) around merge time.

If not done, it will be a waste as usage of these functions will be minimal. There is even a chance of revert if this requirement can't be checked off on.

@Pirulax

Pirulax commented Aug 25, 2023

Copy link
Copy Markdown
Contributor Author

I've added the pages. Please check them out, and let me know if there are any errors:
engineStreamingSetMemorySize
engineStreamingGetMemorySize
engineStreamingRestoreMemorySize
engineStreamingGetBufferSize
engineStreamingSetBufferSize

Also, it seems like I didn't add a engineStreamingRestoreBufferSize function [nor did I restore it on disconnect].
I'll add it tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Increase streaming memory Using IMG feature causes map loading issue

4 participants