View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000136Dwarf FortressTechnical -- Generalpublic2010-04-02 08:172018-01-13 10:27
ReporterNimrod Assigned To 
PriorityhighSeveritycrashReproducibilityalways
Status newResolutionopen 
PlatformPCOSWindows VistaOS VersionUltimate 64 bit
Summary0000136: When embarking on large area, DF hits 2GB memory limit and crashes
DescriptionGraphics YES
Mayday version
some minor changes to init.txt




Steps To Reproducetried every compatibility mode - no change
windowed/full screen - no change
Additional InformationAdventurer mode works without problems
Tagscrash, embark, memory, modding

Relationships

has duplicate 0000188 closeduser6 Crashed on Embark 
has duplicate 0000758 closeduser6 Load out crash 
has duplicate 0000822 closeduser6 cannot embark 
has duplicate 0001065 closeduser6 Cannot embark 
has duplicate 0001194 closeduser6 Runtime Error on Embark 
has duplicate 0001648 closeduser6 Max embark size causes crash 
has duplicate 0002436 resolveduser6 Memory Leak leads to Runtime error 
has duplicate 0003814 resolveduser6 Runtime Error game crash 
has duplicate 0004270 resolvedLogical2u Dwarf Fortress freezes forever 
has duplicate 0005284 resolveduser6 Crash on Embark 
has duplicate 0005622 resolvedLogical2u Crash on large embark - memory related 
has duplicate 0003659 resolveduser6 Cannot embark 
related to 0002715 new Crash during huge, lengthy world gen (out-of-memory) 
related to 0004833 resolveduser6 When embarking, crashes when memory use reaches 2GB. 
related to 0005283 resolvedToady One Crash upon accepting/saving a generated world when old-version saves are present 

Activities

Kurik

2010-04-02 09:10

reporter   ~0000194

I am getting this error whenever I try to embark with an embark area larger than 10x10 squares selected on the embark screen.

Nimrod

2010-04-02 09:40

reporter   ~0000203

i tried 13x13 every single time.
checking something smaller now
brb

Nimrod

2010-04-02 09:46

reporter   ~0000207

CONFIRMED!

It does not crash with an embark area of 9x9.

jbrown6982

2010-04-02 16:15

reporter   ~0000320

I confirm as well. 9x9 is the largest embark area I can select without a CTD. Major for me as I like huge embark areas. :|

DoctorZuber

2010-04-02 16:43

reporter   ~0000333

Last edited: 2010-04-02 16:54

 okay, I genned up a quick pocket region simply to test this.

9x9 works
2x10 works
2x16 works

10x10 works
16x16 crashes. eventually, hung for a good couple minutes before throwing a C++ exception at me and finally closing.

5x16 works
6x16 works
7x16 works...

16x2 works

My theory at this exact moment leads me to believe that this is a hardware limitation on your end. Not specifically a bug with the embark size. Insufficient memory most likely. The fact that I'm running this test on a pocket region probably does lower the memory requirements.

bbgun06

2010-04-02 20:44

reporter   ~0000405

I'm not sure this can be blamed entirely on insufficient memory. I have 6GB of ram, but I still can not embark on anything larger than 10x10. While the game is running in a 9x9 map, it uses about 1.8GB of memory, with about 28% free.
I am running Windows 7 64 bit.

DoctorZuber

2010-04-02 20:49

reporter   ~0000407

Last edited: 2010-04-02 20:49

 not sure what exactly to blame it on, but if I'm able to embark on 10x10 and you are not.... it does sort of imply some sort of hardware issue.

Still, you might try again on a pocket region and see if it lets you embark on a larger area that way.

Nimrod

2010-04-02 23:29

reporter   ~0000438

I run DF on a Quadcore Intel i7 920 with 6GB RAM
OS is Windows Vista Ultimate 64bit

the hardware theory cant be right as i see it

maybe it got something to do with memory allocation in a 64bit OS?
What are your specs DoctorZuber

Halconnen

2010-04-03 00:55

reporter   ~0000452

@bbgun06 above.
DF is a 32-Bit application. It -cannot- use more than 2GB at once. So at 1.8GB it's already running near the limit. Hit 2GB and it dies horribly. Increasing from 9x9 to 10x10 increases the amount of embark tiles from 81 to 100, an increase of almost 25%.

I guess that that's a hard limit at the moment.

Nimrod

2010-04-03 02:52

reporter   ~0000465

Okay, seems this crash is nailed down.

I did some testing and you are right Halconnen.
As soon as DF reaches 2Gig Memory usage it dies witha runtime error.

I set GRAPHICS:NO and did a 13x13 build while watching the memory usage - it went up to 1.8 Gig on embark and worked perefectly.
Same with GRAPHICS:YES hits 2 Gig after a few seconds and dies.

So: its a memory usage problem with graphics enabled

Kurik

2010-04-03 04:46

reporter   ~0000475

I've had this issue occur even with GRAPHICS:NO. AFAIK the game may allocate you a different number of Z-levels depending on the location you're embarking on, which of course affects memory usage, so that's another factor that can affect whether you get this crash.

bicker x 2

2010-04-03 12:51

reporter   ~0000627

I found this issue to ocour with or without graphics being on as well

bicker x 2

2010-04-09 10:08

reporter   ~0002349

This seems to again ocour with vertion 02

SirPenguin

2010-04-09 10:20

reporter   ~0002351

This isn't so much a crash as it is your system not having enough memory. What I mean by that is that I doubt this will ever be fixed - the fix is that your embark size is directly proportionate to your memory size. Too much and it crashes.

DoctorZuber

2010-04-09 12:32

reporter   ~0002384

Intel(R) Core(TM)2 Duo CPU T7100 @ 1.8 GHz 1.0 GHz
2 GB Ram
32-Bit Operating System
Vista

- I did my testing on a pocket region with various embark sizes.

- you never specified what size region you did your tests on
- larger regions very probably require more memory.

PetePetePete

2010-04-10 08:53

reporter   ~0002578

Confirming the 2GB Limit: just wanted to embark on a 16x16 (before checking here) and it crashed the moment DF hit 2GB RAM Usage.

Worked in earlier versions, so I blame it on a) the increased amount of z-levels and features or b) an unwanted memory leak somewhere down the line.

DoctorZuber

2010-04-10 09:00

reporter   ~0002580

Last edited: 2010-04-10 09:00

 on a personal note, I think you guys are at somewhat insane trying to embark on such a large area, I've never wanted or needed larger than 5x5.

However, I do agree that it shouldn't simply crash on you. If you've got the hardware to run it, go for it, have fun.

Greyhawk

2010-04-11 17:30

reporter   ~0002913

32-bit operating systems has 2 GB limit for accessed memory.

64-bit operating systems doesn't have that limit, but 32-bit applications (like Dwarf Fortress) are still limited by the 2 GB limit (even though you could have 3gb memory and 3gb virtual memory).

Two possibilities to go beyond 2GB:

- Release a 64-bit version separately.
- Run special 32-bit processes that hold parts of the data.

In the mean time, the game should stop at 95% of total memory, 95% of available memory or 95% of 2GB (whichever comes first) to prevent the locking up before the crash.

James.Denholm

2010-04-12 06:44

reporter   ~0003034

Of course, 32-bit processors can't run 64-bit applications, so to release a separate 64-bit program could cause confusion and issues for 32-bit users.

But, of course, it would be brilliant for 64-bit users.

jgoodwin

2010-04-13 19:32

reporter   ~0003365

With Visual Studio, if you specify /LARGEADDRESSAWARE on the link command line you can get 3 to 3.5 GB of address space on 32-bit windows.

See http://msdn.microsoft.com/en-us/library/wz223b1z%28v=VS.80%29.aspx

(This page is for VS 2005, but the switch also works for VS 2008 and VS 2010)

Arkaaito

2010-04-13 19:51

reporter   ~0003369

Are there any workarounds for this at the moment that would allow embarking on 16x16? I've tried turning graphics off but I still hit the memory threshold.

Logical2u

2010-04-14 19:50

manager   ~0003596

I have only one question - why do you want to embark on a 16x16 area?

That being said, no - since DF is compiled as an x86 or 32 bit program, it has the maximum memory usage limit of 2gb as mentioned above, which caps your playing area to one in which that memory limit is not achieved.

Turn graphics off - turn off temperature, economy, invaders, set a low population cap - disable all cavern layers/find an embark site with very few underground layers - find a very flat embark site - find an embark site without running water.

Those tricks should lower your memory consumption.

Langdon

2010-04-14 22:32

reporter   ~0003623

Isn't this the same as 0000002? (which Toady is fixing in .04?)

Gauphastus

2010-04-14 23:03

reporter   ~0003626

Last edited: 2010-04-14 23:04

 I got this problem when I embarked on a 4x14 or so site.
I was trying to build a bridge between two continents, something I've really been dying to do for awhile now.
Apparently that's a bit too big.

Oh well.

EDIT: But you know, the game locks up like clockwork after only a few minutes of me not doing anything really.
I uh, have a save here if it's ever requested.
I'll start up a new fort on a different save.

user6

2010-04-15 12:34

  ~0003743

Last edited: 2010-04-15 12:35

 @Langdon they're both related to excess memory usage, but I doubt the fix for 0000002 was one that allowed the program to break the 2 GB limit. More likely he optimized the site finder to use less memory. This one will probably require restricting the size of the embark area -- I don't see any way around that.

Firehound

2010-04-25 14:10

reporter   ~0005034

@Jodoodwin: Reminds me of having to download a new executable for ETW when it came out, and that the only difference was something like that /LARGEADDRESSAWARE so it wouldn't crash and freeze mid-game for no reason.

hyndis

2010-06-25 09:28

reporter   ~0009087

Sadly, ETW is still, even after being released over a year ago, far buggier and far more prone to crashing than Dwarf Fortress.

Chalk one up for Toady! He beats a multimillion dollar studio and dev team of dozens when it comes to squashing bugs.

smjjames

2010-06-30 09:36

reporter   ~0009377

Huh? How did this turn into something about ETW?

Anyways, for .08, I don't crash when I do a large area, but it does take a while for it to go through the embarkation screens and it is pretty laggy when I get to the spot sometimes. DF did hang a few times for one particular spot on the map I'm using when I was looking at a 16X13 area, but I eventually got through and I think that spot was just extrenely memory taxing for various reasons.

smjjames

2010-07-19 13:38

reporter   ~0010669

Last edited: 2010-07-26 20:51

 Sorry for double post, but:

I'm getting this with the maximum possible on 31.10 and getting the runtime error as well. When I have the taskmanager open, I often see the load going past 50% and nearly maxing out the CPU.

I don't know if it was the exact error, but its definetly a runtime error.

Sometimes it will pull through with a very large area (on the order of more than 10x10), sometimes not. It seems to help when the embark area is totally flat. It handles 16x5 or 16x6 (and vice versa) just fine however.

Edit: Just a little update, it seems to handle large embark zones better in .12, haven't tried maximum possible yet though.

user6

2010-11-17 10:10

  ~0013960

Reminder sent to: Nimrod

Does this still occur in 31.18?

SirPenguin

2010-12-17 22:05

reporter   ~0014619

I can confirm this issue still occurs on my system with 31.18

Egodeus

2011-06-10 07:21

reporter   ~0017968

I can confirm this in .25. I'm running a 64-bit windows 7 with a quad-core and 8 gigs of ram, but as DF hits the 2 gb line it crashes with the following error:

Problem Event Name: APPCRASH
  Application Name: Dwarf Fortress.exe
  Application Version: 0.0.0.0
  Application Timestamp: 4d90764f
  Fault Module Name: MSVCR100.dll
  Fault Module Version: 10.0.30319.1
  Fault Module Timestamp: 4ba1dbbe
  Exception Code: 40000015
  Exception Offset: 0008d635
  OS Version: 6.1.7600.2.0.0.256.1
  Locale ID: 1035
  Additional Information 1: 53ab
  Additional Information 2: 53ab78575e4e4ce741bf82bee235390b
  Additional Information 3: 3b1c
  Additional Information 4: 3b1c7dab8afb70c763830f7ffc3fdc79

Tried it once more to be sure but then df didn't even give me an error. Just crashed.

Making a separate 64-bit version would be sweet and I'd gladly pay good money for it, but I can imagine that the shuffle wouldn't be too straightforward for Toady. Still some sort of fail-switch to shut the game down gently if the memory allocation limit is reached would be a nice addition, especially if it came with a warning screen that "By the bye, you're trying to embark on a too large area as I cannot reserve enough memory for it."

Halconnen

2012-03-23 09:03

reporter   ~0021678

To give this a little bump for testing:

I currently do not have a system available to properly test this on myself (my main computer is borked, going to take a while to replace), but for users running DF on 64-Bit windows and 4GB or more of RAM:

Have you tried setting the LAA Flag on the Dwarf Fortress exe to see if it still behaves and -does- allocate more than 2GB of RAM?

Toady clearly didn't set it, but also didn't do anything to keep the game from hard crashing when it reaches that limit, so it might do fine unless he uses the spare bit in the pointers as a flag.

Should make no difference on a 32-Bit system since it'd take registry changes for windows to actually react to that, but for 64-Bit windows it should then proceed to allow DF up to 4GB of memory. Which is twice what it gets now.

May not work, may work. It certainly worked for Skyrim before Bethesda went and set the flag themselves.

A program like the CFF Explorer, or the Skyrim/Fallout 3 LAA tools or whatever should be able to do the job.

Echo51

2012-04-04 06:00

reporter   ~0022073

Using this LAA tool i was able to get DF to hog up all availble ram(~3gb) and it should be able to use even more, but i didn't have anymore

It seems for 64bit users that just setting the LAA flag would work wonders for huge embarks.

proof pic: http://i.imgur.com/r6UNK.png

http://www.techpowerup.com/forums/showthread.php?t=112556

MightyRavenDark

2012-09-01 08:18

reporter   ~0023514

I can confirm this as well for version 0.34.11, but I don't understand it since I used to embark on 16x16 about a year ago on a machine with only 2GB of RAM and much inferior specs overall. Using the LAA on a 32 bit machine allows the process to run fully but the game is beyond laggy, with massive lag spikes every 10 seconds while the game is unpaused.

Kipi

2012-09-01 11:12

reporter   ~0023515

MightRavenDark:

It totally depends on the land features. I think you had very few z-levels in total when you made that 16*16 embark.

What version were you using when you made that embark?

MightyRavenDark

2012-09-01 15:34

reporter   ~0023517

It was probably version 0.31.25

Rayanth

2012-11-26 23:22

reporter   ~0023747

Just to add an update, in 34.11, with no mods and no changes to any settings from download state, this limitation still exists. anything over 10x10 cannot be embarked upon, as soon as it hits about 2 gigs of ram it just crashes.

running the LAA tool to force Large Address Aware allows a 16x16 embark of 198 z-levels (that's a total of almost 117 million tiles) and upon finally drawing the map after embark was taking up 3.6 gigs of RAM. This means I wouldn't be able to actually *play* for very long before hitting the LAA limit of 4 gigs.

System specs are not at play here (quadcore i7, 24 gigs of ram, win 7 x64) , it's the simple fact that 32 bit programs can only use a maximum of 2 gigs of ram (without being linked to LargeAddressAware, or on 32bit systems) or 4 gigs of ram on 64 bit systems with Large Address Aware linked.

With all modern processors capable of running 64 bit or 32 bit mode, and a vast majority of pre-packaged systems going 64 bit, I would like to re-request that future versions be compiled for both x86 and x64. For those of us insane enough to have a Megaproject in mind that requires a full 16x16 embark, it would certainly be nice to be able to play the game to the full capabilities that it proclaims to have. (Why let us *choose* 16x16 if it's physically impossible?) =)

Rayanth

2012-12-01 23:20

reporter   ~0023757

I did quite a bit more playing around. it is plausible to get a larger embark in smaller memory if you decrease the z-levels.

DF 34.11 with MayDay and dfhack, Large custom world, cut the cave levels down to 1, and embarked on a 16x16 area where a volcano meets the beginning of a stream. there's lots of cliff, but total z-levels is 110, and total RAM usage at embark is 1.8 gigs. This is within the realm of plausibility. If one were to embark on a flatter area, with no cave levels, it could very well be playable at 16x16 without even LAA. Further z-level reduction could be had from other world gen options.

For those questioning 'why 16x16' - it's the only way to guarantee, 100%, a certain 'feature'. And imagine the megaprojects!

SirPenguin

2014-07-11 19:42

reporter   ~0025909

Not surprising, but behavior still exists in 40.0x

Talvieno

2014-07-16 05:28

manager   ~0026618

There's a tool out there that fixes it. Can't recall what it is off the top of my head, though.

ptb_ptb

2014-08-14 06:27

reporter   ~0029016

LAA Large Address Aware
http://www.techpowerup.com/forums/threads/large-address-aware.112556/

handofcode

2014-08-15 14:00

reporter   ~0029109

Fair warning: If you do this you will get a LOT of pathing bugs if you embark on large sites. The most common sign is that when you try to build something in clearly pathable areas it will say "no access to materials". Other things include dwarves being stuck in one spot and slowly starving to death.

This can be resolved by saving and reloading.

I haven't reported this particular issue as a bug since the exe is being modified but I'm pretty sure it might pop up once Toady starts the 64-bit conversion so... conflicted.

Loci

2016-12-15 14:46

viewer   ~0036086

v0.43.05: The 64-bit version removes the 2GB addressing limit; the limitations of the legacy 32-bit version are unlikely to change. It would, however, be nice if the game could proactively reject problematic embarks without crashing.

Any tangential bugs with the 64-bit version (pathing, etc.) should be reported as separate issues after confirming them against an official 64-bit release.

Add Note

Note

Issue History

Date Modified Username Field Change
2010-04-02 08:17 Nimrod New Issue
2010-04-02 09:10 Kurik Note Added: 0000194
2010-04-02 09:40 Nimrod Note Added: 0000203
2010-04-02 09:46 Nimrod Note Added: 0000207
2010-04-02 09:47 Nimrod Tag Attached: embark
2010-04-02 09:47 Nimrod Tag Attached: crash
2010-04-02 16:15 jbrown6982 Note Added: 0000320
2010-04-02 16:43 DoctorZuber Note Added: 0000333
2010-04-02 16:45 DoctorZuber Note Edited: 0000333
2010-04-02 16:47 DoctorZuber Note Edited: 0000333
2010-04-02 16:48 DoctorZuber Note Edited: 0000333
2010-04-02 16:52 DoctorZuber Note Edited: 0000333
2010-04-02 16:54 DoctorZuber Note Edited: 0000333
2010-04-02 20:44 bbgun06 Note Added: 0000405
2010-04-02 20:49 DoctorZuber Note Added: 0000407
2010-04-02 20:49 DoctorZuber Note Edited: 0000407
2010-04-02 23:29 Nimrod Note Added: 0000438
2010-04-03 00:55 Halconnen Note Added: 0000452
2010-04-03 02:52 Nimrod Note Added: 0000465
2010-04-03 04:46 Kurik Note Added: 0000475
2010-04-03 11:56 user6 Relationship added has duplicate 0000188
2010-04-03 12:01 Aquillion Tag Attached: modding
2010-04-03 12:51 bicker x 2 Note Added: 0000627
2010-04-07 19:29 user6 Summary CTD runtime error on hitting 'embark' in Fortress mod => Crash when embarking on large area
2010-04-07 19:30 user6 Relationship added has duplicate 0000758
2010-04-08 14:15 user6 Relationship added has duplicate 0000822
2010-04-09 10:08 bicker x 2 Note Added: 0002349
2010-04-09 10:20 SirPenguin Note Added: 0002351
2010-04-09 12:32 DoctorZuber Note Added: 0002384
2010-04-10 08:53 PetePetePete Note Added: 0002578
2010-04-10 09:00 DoctorZuber Note Added: 0002580
2010-04-10 09:00 DoctorZuber Note Edited: 0002580
2010-04-11 17:30 Greyhawk Note Added: 0002913
2010-04-12 06:44 James.Denholm Note Added: 0003034
2010-04-12 23:40 user6 Relationship added has duplicate 0001065
2010-04-13 19:32 jgoodwin Note Added: 0003365
2010-04-13 19:51 Arkaaito Note Added: 0003369
2010-04-13 20:01 Arkaaito Tag Attached: memory
2010-04-14 19:50 Logical2u Note Added: 0003596
2010-04-14 22:32 Langdon Note Added: 0003623
2010-04-14 23:03 Gauphastus Note Added: 0003626
2010-04-14 23:04 Gauphastus Note Edited: 0003626
2010-04-15 08:57 user6 Relationship added has duplicate 0001194
2010-04-15 12:34 user6 Note Added: 0003743
2010-04-15 12:35 user6 Note Edited: 0003743
2010-04-15 17:15 jbrown6982 Tag Attached: RESOLVED
2010-04-25 14:10 Firehound Note Added: 0005034
2010-04-26 17:26 user6 Summary Crash when embarking on large area => Crash when embarking on large area, possibly dependent on graphics
2010-04-26 17:26 user6 Tag Detached: RESOLVED
2010-04-28 14:05 user6 Category General => Technical
2010-04-29 17:50 user6 Relationship added has duplicate 0001648
2010-06-22 18:13 user6 Sticky Issue No => Yes
2010-06-22 18:13 user6 Summary Crash when embarking on large area, possibly dependent on graphics => Crash when embarking on large area
2010-06-22 18:13 user6 Relationship added has duplicate 0002436
2010-06-25 09:28 hyndis Note Added: 0009087
2010-06-29 07:38 user6 Category Technical => Technical -- General
2010-06-30 09:36 smjjames Note Added: 0009377
2010-07-16 04:08 Logical2u Relationship added parent of 0002715
2010-07-19 13:38 smjjames Note Added: 0010669
2010-07-19 13:50 smjjames Note Edited: 0010669
2010-07-26 20:51 smjjames Note Edited: 0010669
2010-11-17 10:10 user6 Note Added: 0013960
2010-12-13 14:00 user6 Relationship added has duplicate 0003814
2010-12-17 22:05 SirPenguin Note Added: 0014619
2011-03-21 06:19 Logical2u Relationship added has duplicate 0004270
2011-06-10 07:21 Egodeus Note Added: 0017968
2011-08-16 13:00 user6 Summary Crash when embarking on large area => When embarking on large area, DF hits 2GB memory limit and crashes
2012-02-19 08:19 user6 Relationship added parent of 0004833
2012-02-21 06:49 user6 Relationship added has duplicate 0005284
2012-02-21 06:51 user6 Relationship replaced related to 0002715
2012-02-21 06:52 user6 Relationship replaced related to 0004833
2012-03-12 10:08 user6 Relationship added has duplicate 0005622
2012-03-12 10:10 user6 Relationship added has duplicate 0003659
2012-03-12 12:19 user6 Relationship added related to 0005283
2012-03-23 09:03 Halconnen Note Added: 0021678
2012-04-04 06:00 Echo51 Note Added: 0022073
2012-09-01 08:18 MightyRavenDark Note Added: 0023514
2012-09-01 11:12 Kipi Note Added: 0023515
2012-09-01 15:34 MightyRavenDark Note Added: 0023517
2012-11-26 23:22 Rayanth Note Added: 0023747
2012-12-01 23:20 Rayanth Note Added: 0023757
2014-07-11 19:42 SirPenguin Note Added: 0025909
2014-07-16 05:28 Talvieno Note Added: 0026618
2014-08-14 06:27 ptb_ptb Note Added: 0029016
2014-08-15 14:00 handofcode Note Added: 0029109
2016-12-15 14:46 Loci Note Added: 0036086
2018-01-11 21:03 Loci Sticky Issue Yes => No
2018-01-11 21:21 BlueManedHawk Tag Attached: Windows
2018-01-13 10:27 lethosor Tag Detached: Windows