View Issue Details

IDProjectCategoryView StatusLast Update
0007526Dwarf FortressAdventure Mode -- Sitespublic2018-12-20 09:46
ReporterRobsoie Assigned Touser11 
PrioritynormalSeverityminorReproducibilityalways
Status confirmedResolutionopen 
Product Version0.40.04 
Summary0007526: Dark towers contain thousands of goblins and trolls, causing lag
Descriptionon a Pocket World generated with 0.40.04, with only 50 years of history to make sure i still had population low (even despite the bug fixes about population, i wanted to keep things low for my new version test), i ran into a severe slowdown in adventure mode while inside of a goblin site.

At first when i entered the goblin territory (as Travel can't go in) , game was perfectly smooth and fast, i moved next to a couple of goblins tower that had some goblins on its top, still no slowdown, still fast.

Then as i was starting to approach the location of the main dark fortress, suddenly the game slowed down very noticably.
Here is the save just before it happens :
http://dffd.wimbli.com/file.php?id=9093

Load the save, at that point,the game is still fast and smooth, walk to the south , after several steps you'll encounter a very noticable slowdown.
It's painful but still you can relatively walk.
Continue to the south, there's a river, go past the river (jump/swim/etc..)

You'll arrive in front of a goblin tower you'll see the goblin line of sight signaling it as you're in stealth mode), on the top of it there are only a dozen of goblins.
Still walk to the south past the tower, and you'll arrive near the main dark fortress, and you'll notice the slowdown is really severe at that point
Additional InformationSaves:
http://dffd.bay12games.com/file.php?id=9093 (0.40.04)
http://dffd.bay12games.com/file.php?id=9094 (0.40.04)
http://dffd.bay12games.com/file.php?id=9184 (0.40.05)
http://dffd.bay12games.com/file.php?id=9324 (0.40.06)
http://dffd.bay12games.com/file.php?id=9513 (0.40.10)
http://dffd.bay12games.com/file.php?id=9558 (0.40.10)
http://dffd.bay12games.com/file.php?id=9715 (0.40.12)
http://dffd.bay12games.com/file.php?id=10842 (0.40.24)
http://dffd.bay12games.com/file.php?id=11867 (0.42.06, 0009637, very large)
TagsNo tags attached.

Relationships

related to 0007698 new Excessive numbers of underground critters 
related to 0007547 assigneduser6 TOTAL_CIV_POPULATION only affects historical figures, not abstracted populations 
related to 0007461 new Unretired fort gets 1000+ goblin invaders 
has duplicate 0007578 resolveduser6 Serious FPS dropdown on enemies' sites 
has duplicate 0007878 resolveduser6 Dark pit population sizes make game unplayable 
has duplicate 0009177 resolveduser1294 FPS death when approaching dark tower in adv mode 
has duplicate 0009170 resolveduser1294 unplayable lag on some sites in adventure mode 
has duplicate 0009676 resolveduser11 Troll population grows uncontrollably 
has duplicate 0009637 resolvedlethosor assault on goblin city abruptly plagued by minutes-long processing time per turn 
related to 0008334 new Trolls in goblin sites respawning constantly 
related to 0007009 new Important slowdown near a goblin site 

Activities

Robsoie

2014-07-20 17:21

reporter   ~0027082

Additional saved game , after i very very slowly and painfully managed to enter the fortress and explore it a bit to find where was the population , on the surface of the dark fortress only few trolls and few goblins, couldn't be what was the source of the slowdown, then after more exploring underground of that fortress i see a giant room with hundred of trolls there :

http://dffd.wimbli.com/file.php?id=9094

Robsoie

2014-07-28 06:23

reporter   ~0027756

Last edited: 2014-07-28 06:39

Still happening in 40.05 , i thought it would be fixed with the overpopulation fixes, but in that save :
http://dffd.wimbli.com/file.php?id=9184

I left the fortress and moved toward the dark fortress on the south east of the world, only to have the game being simply unplayable in term of slow and lag at that point.

I opened that 40.05 pocket world with World Viewer and it told me at the dark fortress site :
1 Dwarf (probably the guy i retired there)
1 Mouse Monster
4951 goblins !
5254 trolls !

No surprise unfortunately it is unplayable (and that's a pocket world)

Robsoie

2014-07-28 15:37

reporter   ~0027820

In 40.05 i experimented today a lot with
MAX_STARTING_CIV_NUMBER
MAX_POP_NUMBER
MAX_SITE_POP_NUMBER
in the entity_default.txt
and
TOTAL_CIV_NUMBER
TOTAL_CIV_POPULATION
SITE_CAP
from worlgen.txt

And from my experimentations, i ended noticing that the goblins' dark fortresses always end having, in each of them, more than 1000 individuals of goblins (and as much trolls) inside, while every other civs (dwarves, humans, elves) have much more reasonnable 100/150 individuals for their main bigger site.
This when i set the exact same settings for each civs in the entity_default.txt

Even with only 20 years of history and no war (from what i see in World Viewer) , there's always that huge amount of goblins+trolls in a dark fortress, overwhelming numbers that just make the game unplayable when you're adventuring close to the site.

user11

2014-07-28 15:47

  ~0027823

Thanks for the saves/follow-ups.

Borschwanger

2014-08-07 06:37

reporter   ~0028518

Last edited: 2014-08-07 06:39

The three most populous sites on my current world are actually elves this time. Elves total 8-11k with various animal populations ranging from a few thousand up to 15300 just for Tigers+Jaguars+Gorillas. Several Dark Fortresses follow up just behind. Year 250.

http://dffd.wimbli.com/file.php?id=9324

Sunsivy, Oaksilvers, Conflictfogs, Wheeledmalice, Plaguefield, Hexfrost, Balanceplague, Realmsterror, Waddlehate, and Beachedseduced are worst. A couple more in the low thousands and then it drops off to below 1000 in mostly caves and then quickly down to under 500. Seems about 90-95% of sites have roughly 200 or less residents.

In addition to the things Robsoie tried, I have also tried making worlds where Goblins and Elves were almost exclusively homosexual species by adding [orientation] tags to no effect (not in this world). Either it doesn't work like I imagine or I did it wrong (never did try 100% homosexual, though I probably should have...). Only mod is Phoebus and used some advanced worldgen parameters (savagery, good/evil, (semi)megabeasts, probably a few others).

smjjames

2014-08-07 22:06

reporter   ~0028570

Sorry for offtopic, but is [orientation] tags really a thing?

Borschwanger

2014-08-07 22:52

reporter   ~0028571

I dunno. It is in the wiki.

http://dwarffortresswiki.org/index.php/DF2014:Creature_token#ORIENTATION

http://dwarffortresswiki.org/index.php/DF2014:Marriage

user6

2014-08-08 09:00

  ~0028582

0007905 pertains to ORIENTATION. Please keep the discussion here on topic.

Robsoie

2014-08-25 07:37

reporter   ~0029610

Last edited: 2014-08-25 07:43

world generated on 40.10 at only +/- 50 years , no raws edited out of the gob/elf/human/dwarves names files to remove accentuated characters
save there :
http://dffd.wimbli.com/file.php?id=9513

An elf bowman adventurer left his companion waiting at the entrance of the goblin site, all was well, until i started to approach 2 goblins dark towers on my way to their dark fortress, at such point it slowdown heavily.

But from the point of the save continue directly to the North , moving between the 2 dark towers, toward the dark fortress, it's going to be slower and slower into simply unplayable if you don't sprint (in which it's slow but at least you can move).

according to legends viewer, the dark fortress should have
1310 Goblins
802 Trolls
798 Ogres
5 Goblin Outcasts
1 Lizard Demons
1 Elf Outcasts

smjjames

2014-08-29 12:19

reporter   ~0029777

Last edited: 2014-08-29 12:25

I don't know if Toady One needs more saves to help solve this or not, but heres one of a tower site that is barely playable. I'm stubborn enough to just try and plow my way through it.

http://dffd.wimbli.com/file.php?id=9558

Nothing special about the save other than to demonstrate the laggyness. In the save, I'm just trying to wait for one of the rescued prisoners to snap out of it. Right now, I wish I could just pick up the kid as I move on or give him something to cheer him up and snap out of it because it's taking too long as is.

According to Legendsviewer, there should be roughly:

10154 Goblins
5100 Trolls
131 Humans
78 Human Prisoners
1 Skunk Fiends

I already killed some trolls and goblins, so it'll be a little less.

Also, my computer is a pretty good one, so, with computers not that good, it's unplayable.

smjjames

2014-08-31 07:43

reporter   ~0029820

Need more savegames?

samanato

2014-09-03 08:48

reporter   ~0029913

This seems to be related to the goblins' lack of need to eat or drink. Putting [NO_EAT] and [NO_DRINK] on kobolds, for example, will swell up cave populations to thousands in a similar way as in dark fortresses.

smjjames

2014-09-03 09:15

reporter   ~0029915

Last edited: 2014-09-03 09:44

They can still have thousands in a cave without those tags. I'll try removing those and see how things go.

Okay, in the world I tested it in, the largest populated tower has 2267 Goblins and 1850 Trolls, about what you would get with some of the most populated human towns/cities. Most of them have goblin and troll populations that range between 1k to 2k each, seeming to average between 1,200 and 1,500. Theres also several with each troll and goblin population less than a thousand.

Seems MUCH more reasonable to me.

To note, I did make them mortal with the same lifespans as dwarves (elves are still immortal because elves), but that didn't affect the populations.

Toady One, I'm sure you're reading this, what was the origional reason for putting the [NO_EAT] and [NO_DRINK] tags on goblins? Might not be needed anymore.

Quietust

2014-09-03 09:53

reporter   ~0029916

Last edited: 2014-09-03 09:59

In the original save linked in the bug report itself, there are 174 units in play at first, and after walking south and triggering the lag, there are 461 units in play. If I walk even further south to load more of the fortress (across the river and over the next trench), it increases further to 611.

smjjames's savegame has a whopping 1,606 units in play, which is clearly excessive.

smjjames

2014-09-03 10:07

reporter   ~0029917

Last edited: 2014-09-03 10:39

In this new world I generated for adv mode, the three largest towers (and the only three to breach 3k) have populations of 3198 Goblins 2250 Trolls and 3578 Goblins 2550 Trolls and 5497 Goblins 3100 Trolls. Which is MUCH better than the 5k+, sometimes up to 10k for each population. Most towers have less than 2k for each population and some have less than 1k for each.

Removing the [NO_EAT] and [NO_DRINK] tags help, but the way that the goblins are spread out, or rather, not spread out, causes lag problems and then theres the trolls doing crazy pathing which only adds to the issues.

Edit: Over a thousand in play, wow. I'll check out one of the towers in this modified save and see how thing are.

smjjames

2014-09-03 15:46

reporter   ~0029921

Last edited: 2014-09-03 17:30

Now testing a low pop tower with populations of
1275 Goblins
99 Humans
67 Human Prisoners
12 Dwarves
4 Dwarf Prisoners
1 Lemur Monsters

This goblin civ doesn't have trolls at that site for some reason, other sites of that civ have trolls though. I split off a compressed file from the spot I did the first loaded unit count if you guys request it.

Going eastward directly towards the spire. The first small tower I get to has 139 loaded units, 69 goblins, 36 crundles and the rest various cavern wildlife. No significant lag.

Second count, near the river before the spire, loaded unit number is 122. Mild conversation spam related lag and a bit of pathing lag.

Third count, approx center of spire, loaded unit number is 182, including cave wildlife. Slight amount of lag.

Second test with a Dark Tower population of:
3578 Goblins
2550 Trolls
104 Humans
25 Human Prisoners
3 Kobolds (I gave kobolds normal speech, which is why they are there)
3 Dwarves
1 Llama Fiends

First small tower reached and the lag is starting. Units loaded (or otherwise seen in DFhacks reveal+exterminate) is 396 with 226 goblins and 130 crundles, the rest, various cave critters plus a few humans.

Second reading, a couple tiles outside the tower and the lag has ramped up, loaded unit count is 690, with 468 trolls and 159 goblins. And um, when I did a wait one hour to clear out the reveal data, DF crashed. I'll check if that was a DFhack related thing or something else.

Also, I've been getting fairly regular brief freezes with DF as I move forward. Earlier I thought it was because of me using reveal all, but this time I didn't use it and I'm getting the brief lagspike pause. Also, I tested the wait one hour and while it didn't crash, it did briefly freeze, and I replicated what I did earlier when it crashed and it didn't that time, so, I don't know what that crash was about.

smjjames

2014-09-16 08:47

reporter   ~0030216

Still happens in a 40.12 save which is modded to remove [NOEAT] and {NODRINK] from goblins plus givng them the lifespan of dwarves.

According to legendsviewer, the site pop is almost 1,600.

http://dffd.wimbli.com/file.php?id=9715

Witty

2015-05-10 06:18

reporter   ~0032666

Still very much an issue in 40.24.

http://dffd.bay12games.com/file.php?id=10842

Vanilla goblins, only raw mod is allowing them to be adventure-playable.
The site lag makes the region entirely unplayable

user1294

2015-12-03 03:22

  ~0033331

Robsoie posted a save at 0009170.

Boatsniper

2015-12-25 11:57

reporter   ~0034192

Still occurs in 42.03, most prevalent within Dark Goblin Fortresses.

I don't think it's normal for enormous Troll civilizations to exist in the dug out underground caverns (as in taking occupation titles, like weaver, administrator, animal trainer, etc.). To add onto the problem, the goblins and trolls will start killing each other once they get line of sight. It makes absolutely no sense.

This issue is easily reproducible, modded or otherwise; generate a world with a high site cap and decent history, adventure your way into a Dark Goblin Pit/Fortress, and watch the game become disastrously unplayable.

FantasticDorf

2016-07-30 03:59

reporter   ~0035724

Last edited: 2016-07-30 07:05

A quick check with a third party tool (legends reader found here: http://www.bay12forums.com/smf/index.php?topic=72702.msg6658295#msg6658295 ) reveals that since 43.05 site populations are still exploding into enormous numbers. On my unmodded world there are globally 95200 goblins for a world with only 25347 historical figures to have ever lived, not to mention about ten thousand elves and 8000 humans.

Only 24375 goblins can be supported in the 195 dark pits that have been built to accommodate the sprawl which stays within the rules of 125 per site. With in my most prominent tower, it has 10851 goblins and 2526 trolls & 2526 beak dogs. The oldest and first tower but just 1 tower out of a network of towers with large similar statistics, and this trends between the 3 goblin civilisations to reach the big 95200 global number.

The wars that occur, such as my largest and latest war 'The War of Murders' has killed 23394 people, more than the 14491 historical figures that are supposed to be dead and the war still hasn't ended or either side feeling inclination to stop for more than 60 years (declared in 167, ongoing in 250)

Auxiliary populations are crashing the game and making it strenuous on systems in adventure mode to approach sites because livestock is spinning out of control to keep up with the numbers in order to 'feed' those who even don't eat, all these people and the subsequent pathing causes as described lag spikes.

Murders and war cannot keep on top of the goblins and i cannot fathom where or why all these unaccounted non-historical people are coming from.

@boatsniper trolls have always (intentionally or not) been like that, since they are civilians but also livestock they get the best of both worlds with automatic job assignment and population scaling. However they cannot commit to a job because they are slow learners without 3rd party tool exploitation.

Murphy

2016-07-31 03:10

reporter   ~0035728

I thought they wouldn't ALL load at the same time? 1000+ goblins is the population pool, and individual non-historical goblins are pulled out when you visit the tower, and then they offload when you leave. You shouldn't actually ever see all 1000 present at the same time.

Might not be the case with trolls, however, since they count as animals.

I've noticed that the structure of dark fortress can link cavern layers together, causing water from the above layer to fall through the holes into the layer below. Could that be the real reason of the lag?

LordKnows

2016-08-02 03:10

reporter   ~0035735

Last edited: 2016-08-02 10:16

My 43.05 world seems affected by a similar, but somewhat different issue of overpopulation, and I was advised to take my save here.

http://dffd.bay12games.com/file.php?id=12322

Notable about this is that this only happened AFTER world-gen had finished and I started playing fortress and adventurers. In 850 there were around 2500 fly-men in the world, now there are over half a million largely located in just two sites, Brassdark and Pointrose. I had two consecutive crashes on attempting to enter pointrose earlier, but just now was able to enter with only extreme performance issues (unsurprisingly, it's full of fly-men, 250 thousand apparently.)

I don't have a backup save before the population started growing unfortunately, but do have the xml dump from 850 which I read with legendsviewer to see the huge difference over the two years.

EDIT: I just noticed that the world and fly-man population had changed in the two weeks or so of in-game time that had passed since I last looked at the legends. Pointrose now only has 14629 fly men living there, which explains why it stopped crashing for me probably.

It seems that fly-man outcasts, of which the vast majority of Pointrose consisted of have simply disappeared or something leaving only the ca. 15k that were "normal" fly-men in Pointrose, and brassdark still has hundreds of thousands... Really strange.

Here is a backed up save before the outcasts disappeared, this is only two weeks before the save with a quarter-million less fly-man outcasts:
http://dffd.bay12games.com/file.php?id=12324

FantasticDorf

2018-03-16 12:42

reporter   ~0037948

A exerpt from the public domain of Reddit to highlight how this bug's persistence into 44.06 - 7 and that it breaks dark tower raids (which are a primary target for children rescues)

Reddit thread: ( https://www.reddit.com/r/dwarffortress/comments/83cu59/i_think_the_new_versions_off_site_pillaging_could/ )

And subject image: ( https://i.redd.it/pkt14h002vk01.png )

Any amount of squads cannot hope to face 2526 trolls in a single encounter even with the most trained and well equipped soldiers.

FantasticDorf

2018-04-05 09:22

reporter   ~0038130

Last edited: 2018-04-05 14:06

A link to the use of [USE_EVIL_ANIMALS] has been established in my save, where overpopulation and a poor allocation of non-creature tags like [PET] make them buggy because use_evil_animals applies its own uses without respecting existing creature tags.

Use_evil_animals in itself is a AI only allocator for trolls to exhibit non-creature raw roles, when they are simulated in fortress mode, the trolls are actually civilian peasants & 'other animals' contributing towards population without further tags tacked on like [PET] and don't function as pure livestock at all.

Trolls in themselves only have numbers culled by attacks on the civilization, and would contribute to food (troll-cannibalism & beak dogs) on goblin sites while being immortal also extends their lifespan. Troll peasants when abstracted and added to other civs as the save demonstrates causes the populations to spiral out of control even if the entity is incapable of directly eating them or using their materials like dwarves.

Save: http://dffd.bay12games.com/file.php?id=13651 (fixed, i put in the link while it was still loading oops)

FantasticDorf

2018-11-09 23:49

reporter   ~0038927

Ive managed to further improve my methods and deliver results, it turns out that trolls were stressing the food populations by cannibalising each other which became apparent when i changed troll RAWS (in accordance with [ANIMAL] to prevent outsourcing, local populations only)

A average fully grown Troll yields 30-40 meat from muscle alone, remove that muscalture's edibilty with something like a 'tough meat' material definition with no edible parts and the populations drop to expected levels when goblins (with a raw change) & trolls are eating.

This means that all civs must be prioritising animals with the most meat per capita, including fat horses as part of their food simulation over other animals like pigs or cows. Addressing this could signficantly improve the stability of animal populations by making civs value them for something else or deliberately fatten up animals by diverting crops to feeding livestock encouraging them to eat grazers and be more selective.

bluebird9281

2018-12-19 23:57

reporter   ~0039030

removing [NO_EAT][NO_DRINK] tag from the goblins might help.(a workaround) they have such a high population in their capitals cuz they don't eat/drink and the capitals ignore population cap!

FantasticDorf

2018-12-20 09:46

reporter   ~0039031

Sometimes my fixes don't always work and it still happens inevitably, anecdotally from running my recommendation for a long time and keeping a eye on the populations.

Something more concrete might still be needed.

Add Note

Note

Issue History

Date Modified Username Field Change
2014-07-20 17:12 Robsoie New Issue
2014-07-20 17:21 Robsoie Note Added: 0027082
2014-07-28 06:23 Robsoie Note Added: 0027756
2014-07-28 06:24 Robsoie Note Edited: 0027756
2014-07-28 06:39 Robsoie Note Edited: 0027756
2014-07-28 15:37 Robsoie Note Added: 0027820
2014-07-28 15:46 user11 Relationship added related to 0007698
2014-07-28 15:47 user11 Note Added: 0027823
2014-07-28 15:47 user11 Assigned To => user11
2014-07-28 15:47 user11 Status new => acknowledged
2014-07-28 15:48 user11 Relationship added related to 0007547
2014-07-28 15:51 user11 Relationship added related to 0007461
2014-08-06 18:03 user6 Summary heavy slowdown case near dark fortress => Dark towers contain thousands of goblins and trolls, causing lag
2014-08-06 18:04 user6 Relationship added has duplicate 0007578
2014-08-06 18:04 user6 Relationship added has duplicate 0007878
2014-08-07 06:37 Borschwanger Note Added: 0028518
2014-08-07 06:38 Borschwanger Note Edited: 0028518
2014-08-07 06:39 Borschwanger Note Edited: 0028518
2014-08-07 21:57 user6 Status acknowledged => confirmed
2014-08-07 22:06 smjjames Note Added: 0028570
2014-08-07 22:52 Borschwanger Note Added: 0028571
2014-08-08 09:00 user6 Note Added: 0028582
2014-08-25 07:37 Robsoie Note Added: 0029610
2014-08-25 07:43 Robsoie Note Edited: 0029610
2014-08-29 12:19 smjjames Note Added: 0029777
2014-08-29 12:25 smjjames Note Edited: 0029777
2014-08-29 12:25 smjjames Note Edited: 0029777
2014-08-31 07:43 smjjames Note Added: 0029820
2014-09-03 08:48 samanato Note Added: 0029913
2014-09-03 09:15 smjjames Note Added: 0029915
2014-09-03 09:36 smjjames Note Edited: 0029915
2014-09-03 09:38 smjjames Note Edited: 0029915
2014-09-03 09:44 smjjames Note Edited: 0029915
2014-09-03 09:53 Quietust Note Added: 0029916
2014-09-03 09:54 Quietust Note Edited: 0029916
2014-09-03 09:59 Quietust Note Edited: 0029916
2014-09-03 10:07 smjjames Note Added: 0029917
2014-09-03 10:08 smjjames Note Edited: 0029917
2014-09-03 10:39 smjjames Note Edited: 0029917
2014-09-03 15:46 smjjames Note Added: 0029921
2014-09-03 15:56 smjjames Note Edited: 0029921
2014-09-03 16:03 smjjames Note Edited: 0029921
2014-09-03 16:27 smjjames Note Edited: 0029921
2014-09-03 16:53 smjjames Note Edited: 0029921
2014-09-03 17:03 smjjames Note Edited: 0029921
2014-09-03 17:27 smjjames Note Edited: 0029921
2014-09-03 17:30 smjjames Note Edited: 0029921
2014-09-16 08:47 smjjames Note Added: 0030216
2014-09-22 09:50 user6 Relationship added related to 0008334
2015-05-10 06:18 Witty Note Added: 0032666
2015-05-10 10:15 lethosor Additional Information Updated
2015-05-10 10:16 lethosor Sticky Issue No => Yes
2015-12-03 03:19 user1294 Relationship added has duplicate 0009177
2015-12-03 03:20 user1294 Relationship added has duplicate 0009170
2015-12-03 03:20 user1294 Relationship added related to 0007009
2015-12-03 03:22 user1294 Note Added: 0033331
2015-12-25 11:57 Boatsniper Note Added: 0034192
2016-05-08 17:08 user11 Relationship added has duplicate 0009676
2016-07-28 19:11 lethosor Relationship added has duplicate 0009637
2016-07-28 19:12 lethosor Additional Information Updated
2016-07-30 03:59 FantasticDorf Note Added: 0035724
2016-07-30 07:05 FantasticDorf Note Edited: 0035724
2016-07-31 03:10 Murphy Note Added: 0035728
2016-08-02 03:10 LordKnows Note Added: 0035735
2016-08-02 10:16 LordKnows Note Edited: 0035735
2018-03-16 12:42 FantasticDorf Note Added: 0037948
2018-04-05 09:22 FantasticDorf Note Added: 0038130
2018-04-05 14:06 FantasticDorf Note Edited: 0038130
2018-11-09 23:49 FantasticDorf Note Added: 0038927
2018-12-19 23:57 bluebird9281 Note Added: 0039030
2018-12-20 09:46 FantasticDorf Note Added: 0039031