Group: Member
Posts: 9
Type: None
RM Skill: Skilled
Hello everyone, I've been scripting for RPG maker VX for quite awhile now, and just recently got into a wall I can't seem to figure out. It regards RPG maker's memory management for bitmap caching.
The given, unedited, algorithm was throw the bitmap into a hash then if the caller with the path as argument is in the hash, it returns the address of that bitmap, if that bitmap was disposed by the caller, it would've been disposed in the hash as well. That proved to be a problem for my relatively huge bitmaps, since it would freeze the game for 1~2 seconds until the bitmap was loaded every time an animation was requested. So to 'fix' that, I made the algorithm return a copy of the object, instead of the reference, so whenever any method would make a request for that animation bitmap, it wouldn't re-load it again freezing up the game and dispose of it whilst keeping the original.
Now, problem with that is that somehow more and more memory gets used up, about 6MB per animation request, so on a really long play through you might find yourself with no more RAM left.
My animation bitmaps are 80+ framed and about 3000x5000 on avg size, so they're quite big. Each bitmap takes on avg 80~130MB worth of RAM upon loading. Previously to my fix, RAM usage would go as high as 350MB but would go back down to normal 50MB when the animations playing finished playing, but you'd get the 1~2 second game freeze for it to load per call. After my 'fix', RAM usage is consistent with animation usage ranging from 450~830MB, no lag/freeze and resource would be freed upon animation ending; however, even if i dispose of the cloned bitmaps, the RAM freed doesn't equal the RAM used.
Sorry for my redundancy, just wanted to make it clearer, but my question is; is there any way to avoid the garbage stacking of memory, either with or without cloning/duping the bitmap without having the lag/freeze issue?
This post has been edited by korodo: Dec 30 2011, 03:28 PM
Group: +Gold Member
Posts: 4,136
Type: Scripter
RM Skill: Undisclosed
I can't be too specific without looking at your code, but you could try to force the garbage collection when disposing the clone by setting the variable handling the clone to nil and then call GC.start.
__________________________
FRACTURE - a SMT inspired game (demo) made by Rhyme, Karsuman and me. Weep and ragequit.
Group: Member
Posts: 9
Type: None
RM Skill: Skilled
QUOTE (Kread-EX @ Dec 31 2011, 01:57 AM)
I can't be too specific without looking at your code, but you could try to force the garbage collection when disposing the clone by setting the variable handling the clone to nil and then call GC.start.
By forcing garbage collection when disposing of the clone doesn't really free up more memory than before, I just run some tests with garbage collection upon disposition, setting variable to nil and animation completion, it lags the game by 0.5 seconds per animation finish and it doesn't free up more memory than it used to load ;
After all animations loaded : 480MB Animation dimensions: 3200x3840 Animation size: 671kb Animation format: JPEG
So after every animation request ( same file, so hash would only return a copy of it, not allocate memory ) you'd be getting +25MB (with GC ) or +33MB( w/o GC) upon animation finish, that would translate under a normal battle of 3 minutes (using the same animation file every turn ) every 20 seconds to +297MB ~ +225MB of garbage.
Now, normally I set garbage collection to run at every map transition, when you move from map to map to load that area's animations, that's the only time when RAM used equals the RAM freed but also the only time the hash gets reconstructed.
So my hypothesis is, the hash builds up memory as it gets requests, I think that's wrong but I don't know what else :"<
I'd be happy to show the code if that would help.
All this seems to happen during method call for an animation to play, So memory would all be handled during animation processing, so I highly doubt it would be memory leaking elsewhere.
This post has been edited by korodo: Dec 31 2011, 01:53 PM
Group: Member
Posts: 9
Type: None
RM Skill: Skilled
QUOTE (Kread-EX @ Jan 1 2012, 02:03 AM)
Yeah, the memory leak must be related to something else than the bitmap then... It would be helpful to look at your code in order to find it.
K, I'm using a heavily edited version of tankentai, all revisions and modifications done by me; some things I've added was multiple instances of simultaneous animations, and multiple instances of reflected magic animations via reverse pathing.
This is the Sprite_Base code block which is what runs the animation sequences
Sprite_Base
CODE
#============================================================================== # ** Sprite_Base #------------------------------------------------------------------------------ # A sprite class with animation display processing added. #==============================================================================
class Sprite_Base < Sprite #-------------------------------------------------------------------------- # * Class Variable #-------------------------------------------------------------------------- @@animations = [] @@_reference_count = {} #-------------------------------------------------------------------------- # * Object Initialization # viewport : viewport #-------------------------------------------------------------------------- def initialize(viewport = nil) super(viewport) @original1 = false @original2 = false @use_sprite = true # Sprite use flag @animation_duration = 0 # Remaining animation time @division = 3 end #-------------------------------------------------------------------------- # * Dispose #-------------------------------------------------------------------------- def dispose super dispose_animation end #-------------------------------------------------------------------------- # * Frame Update #-------------------------------------------------------------------------- def update super if @animation != nil @animation_duration -= 1 update_animation end @@animations.clear end #-------------------------------------------------------------------------- # * Determine if animation is being displayed #-------------------------------------------------------------------------- def animation? return @animation != nil end #-------------------------------------------------------------------------- # * Start Animation #-------------------------------------------------------------------------- def start_animation(animation, mirror = false) dispose_animation @animation = animation return if @animation == nil @animation_mirror = mirror @animation_duration = @animation.frame_max * 3 + 1
load_animation_bitmap @animation_sprites = [] if @animation.position != 3 or not @@animations.include?(animation) if @use_sprite for i in 0..15 sprite = ::Sprite.new(viewport) sprite.visible = false @animation_sprites.push(sprite) end unless @@animations.include?(animation) @@animations.push(animation) end end end if @animation.position == 3 if viewport == nil @animation_ox = 640 / 2 @animation_oy = 480 / 2 else @animation_ox = viewport.rect.width / 2 @animation_oy = viewport.rect.height / 2 end else @animation_ox = x - ox + width / 2 @animation_oy = y - oy + height / 2 if @animation.position == 0 @animation_oy -= height / 2 elsif @animation.position == 2 @animation_oy += height / 2 end end end #-------------------------------------------------------------------------- # * Read (Load) Animation Graphics #-------------------------------------------------------------------------- def load_animation_bitmap animation1_name = @animation.animation1_name @original1 = true if animation1_name.include?("Anim") @original1 = false if !animation1_name.include?("Anim") animation1_hue = @animation.animation1_hue animation2_name = @animation.animation2_name @original2 = true if animation2_name.include?("Anim") @original2 = false if !animation2_name.include?("Anim") animation2_hue = @animation.animation2_hue @animation_bitmap1 = Cache.animation(animation1_name, animation1_hue).clone @animation_bitmap2 = Cache.animation(animation2_name, animation2_hue).clone if @original1 or @original2 @animation_duration = @animation.frame_max * 2 + 1 @division = 2 else @animation_duration = @animation.frame_max * 3 + 1 @division = 3 end Graphics.frame_reset end #-------------------------------------------------------------------------- # * Dispose of Animation #-------------------------------------------------------------------------- def dispose_animation if @animation_bitmap1 != nil @animation_bitmap1.clear @animation_bitmap1.dispose end if @animation_bitmap2 != nil @animation_bitmap2.clear @animation_bitmap2.dispose end if @animation_sprites != nil for sprite in @animation_sprites sprite.dispose end @animation_sprites = nil @animation = nil end @animation_bitmap1 = nil @animation_bitmap2 = nil end #-------------------------------------------------------------------------- # * Update Animation #-------------------------------------------------------------------------- def update_animation if @animation_duration > 0 frame_index = @animation.frame_max - (@animation_duration + 3) / @division animation_set_sprites(@animation.frames[frame_index]) if @animation_duration % @division == 0 for timing in @animation.timings if timing.frame == frame_index animation_process_timing(timing) end end end else dispose_animation end end #-------------------------------------------------------------------------- # * Set Animation Sprite # frame : Frame data (RPG::Animation::Frame) #-------------------------------------------------------------------------- def animation_set_sprites(frame) cell_data = frame.cell_data for i in 0..15 sprite = @animation_sprites[i] next if sprite == nil pattern = cell_data[i, 0] if pattern == nil or pattern == -1 sprite.visible = false next end if pattern < 100 sprite.bitmap = @animation_bitmap1 else sprite.bitmap = @animation_bitmap2 end sprite.visible = true
if @original1 and (sprite.bitmap == @animation_bitmap1) anim_height = 240 anim_width = @animation_bitmap1.width / 5 elsif @original2 and (sprite.bitmap == @animation_bitmap2) anim_height = 240 anim_width = @animation_bitmap2.width / 5 else anim_height = 192 anim_width = 192 end
sprite.src_rect.set(pattern % 5 * anim_width, pattern % 100 / 5 * anim_height, anim_width, anim_height) if @animation_mirror sprite.x = @animation_ox - cell_data[i, 1] sprite.y = @animation_oy + cell_data[i, 2] sprite.angle = (360 - cell_data[i, 4]) sprite.mirror = (cell_data[i, 5] == 0) else sprite.x = @animation_ox + cell_data[i, 1] sprite.y = @animation_oy + cell_data[i, 2] sprite.angle = cell_data[i, 4] sprite.mirror = (cell_data[i, 5] == 1) end sprite.z = self.z + 300 + i sprite.ox = anim_width / 2 sprite.oy = anim_height / 2 sprite.zoom_x = cell_data[i, 3] / 100.0 sprite.zoom_y = cell_data[i, 3] / 100.0 sprite.opacity = cell_data[i, 6] * self.opacity / 255.0 sprite.blend_type = cell_data[i, 7] end end #-------------------------------------------------------------------------- # * SE and Flash Timing Processing # timing : timing data (RPG::Animation::Timing) #-------------------------------------------------------------------------- def animation_process_timing(timing) timing.se.play case timing.flash_scope when 1 self.flash(timing.flash_color, timing.flash_duration * 4) when 2 if viewport != nil viewport.flash(timing.flash_color, timing.flash_duration * 4) end when 3 self.flash(nil, timing.flash_duration * 4) end end end
This next part deals with the update method for animations in the Sprite_Battler, used during combat
Sprite_Battler
CODE
Sprite_Battler < Sprite_Base #-------------------------------------------------------------------------- # â—Sprite battler update aliased #-------------------------------------------------------------------------- alias update_original update def update update_original # updates moving animations update_move_anime if @move_anime.size > 0 # updates any reflected animations update_reflected_anime if @reflection_anime.size > 0 end #-------------------------------------------------------------------------- # â—Update method for moving animations #-------------------------------------------------------------------------- def update_move_anime for i in 0...@move_anime.size @move_anime[i].update if @move_anime[i] != nil and !@move_anime[i].disposed? end for i in 0...@move_anime.size if @move_anime[i] != nil if @move_anime[i].finish? @move_anime[i].action_reset @move_anime.delete_at(i) @anime_id -= 1 end end end end #-------------------------------------------------------------------------- # â— Update method for reflected moving animations #-------------------------------------------------------------------------- def update_reflected_anime for i in 0...@reflection_anime.size @reflection_anime[i].update if @reflection_anime[i] != nil and !@reflection_anime[i].disposed? end for i in 0...@reflection_anime.size if @reflection_anime[i] != nil if @reflection_anime[i].finish? @reflection_anime[i].action_reset @reflection_anime.delete_at(i) @reflection_id -= 1 end end end end
#-------------------------------------------------------------------------- # ◠解放 #-------------------------------------------------------------------------- def dispose self.bitmap.dispose if self.bitmap != nil @weapon_R.dispose if @weapon_R != nil if @move_anime != nil for i in 0...@move_anime.size @move_anime[i].dispose if @move_anime[i] != nil and !@move_anime[i].disposed? end end if @reflection_anime != nil for i in 0...@reflection_anime.size @reflection_anime[i].dispose if @reflection_anime[i] != nil and !@reflection_anime[i].disposed? end end @picture.dispose if @picture != nil @shadow.dispose if @shadow != nil @counter.dispose if @counter != nil if @damage != nil for i in 0...@damage.size @damage[i].dispose if @damage[i] != nil end end #@counter = 0 @hit_c = 0 @hits = 0 @balloon.dispose if @balloon != nil mirage_off # ç”»åƒå¤‰æ›´ãƒªã‚»ãƒƒãƒˆ @battler.graphic_change(@before_graphic) if @before_graphic != nil
Now this is all the methods that use the animation files, the extra garbage of memory becomes apparent during the play through of animations. So there is a chance that it is not my rewritten methods for animation handling but tankentai's use of bitmaps, I'm not sure.
I don't have the original to see if the original had garbage stacking.
Another thing I'm not sure of is the difference between a bitmap's "clear" and "dispose" methods; does clear only clear the bitmap but not free the resource?...does dispose clear and free the resource?
Could it be that RPG Maker is allocating more memory than it needs for the hash? For example, you dispose the map, but the memory reserved for the hash map isn't freed?
In nearly any other language this is the case (and thats totally understandable when you keep in mind how hash mapping works). So maybe there is a problem with the allocation of the hash table itself?
This post has been edited by -dah0rst-: Jan 4 2012, 02:21 PM
__________________________
You want Next Gen graphic algorithms in RPG VX? Ask the horst :P
Group: Member
Posts: 9
Type: None
RM Skill: Skilled
QUOTE (-dah0rst- @ Jan 4 2012, 02:18 PM)
Could it be that RPG Maker is allocating more memory than it needs for the hash? For example, you dispose the map, but the memory reserved for the hash map isn't freed?
In nearly any other language this is the case (and thats totally understandable when you keep in mind how hash mapping works). So maybe there is a problem with the allocation of the hash table itself?
Yes that might be the case; another thing I noticed when I run my C# debugger (self made, much like RPG maker VX Ace console ) to see how many bitmaps were being created and their sizes, I found it made over 100 bitmaps of whom none of them were freed after a battle. So a quick fix might be to free the bitmaps that don't take more than 300ms to load and keep the big bitmaps, so the number of created bitmaps doesn't keep on increasing.
[FIXED]
I added a method within the Cache module that gets rid of an entry from a finished animation
CODE
def self.free(file name) @cache.delete("Graphics/Animations/"+file name) GC.start end
and rewrote Sprite_Base 'load_animation_bitmap' to differentiate between big files and small files, big files are requested as copies, while small files are requested as the originals, so when a small file is finished, it is freed, while the big file remains in the hash.
Which made me realize, RPG maker VX in itself never in it's execution releases it's hash entries, it will only keep adding.
This post has been edited by korodo: Jan 4 2012, 03:57 PM
Group: +Gold Member
Posts: 4,136
Type: Scripter
RM Skill: Undisclosed
I unfortunately can't test right now, but I was going to suggest to call Cache.clear after a battle. Truthfully, since you're loading your clones via the cache module for performance instead of the Bitmap.new method the cache could grows exponentially faster than normal, and thus, calling Cache.clear to complete erase it after the battle could solve the problem.
__________________________
FRACTURE - a SMT inspired game (demo) made by Rhyme, Karsuman and me. Weep and ragequit.
Group: Member
Posts: 9
Type: None
RM Skill: Skilled
QUOTE (Kread-EX @ Jan 5 2012, 10:17 AM)
I unfortunately can't test right now, but I was going to suggest to call Cache.clear after a battle. Truthfully, since you're loading your clones via the cache module for performance instead of the Bitmap.new method the cache could grows exponentially faster than normal, and thus, calling Cache.clear to complete erase it after the battle could solve the problem.
Ya, that is true. However, if I do that then you'd be having the "loading screen" at the start of every new battle, and loading 500MB worth of data takes good 3+ seconds, doubt any player would find that appealing.
But, we could use your solution if I segment the animation files needed on a per battle basis, instead of a per map basis; but what's better? a once per map loading screen for 3 or so seconds? or a loading screen for 0.5~1.5 seconds every battle?
Or like someone once told me ".....why are your animations so fudging big?!" guess I got greedy on the HD graphics :<
Group: +Gold Member
Posts: 4,136
Type: Scripter
RM Skill: Undisclosed
The problem is that RMVX is a terribly inefficient engine when it comes to graphics processing. So I'm tempted to say that loading the animations on a per-battle basis would be a better option.
__________________________
FRACTURE - a SMT inspired game (demo) made by Rhyme, Karsuman and me. Weep and ragequit.
Group: Member
Posts: 9
Type: None
RM Skill: Skilled
QUOTE (Kread-EX @ Jan 5 2012, 01:54 PM)
The problem is that RMVX is a terribly inefficient engine when it comes to graphics processing. So I'm tempted to say that loading the animations on a per-battle basis would be a better option.
Is it because it's using a virtual machine running Ruby written in C++? or because it's using the old version of GDI? :<