View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0007526 | Dwarf Fortress | Adventure Mode -- Sites | public | 2014-07-20 17:12 | 2018-12-20 09:46 |
| Reporter | Robsoie | Assigned To | |||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | confirmed | Resolution | open | ||
| Product Version | 0.40.04 | ||||
| Summary | 0007526: Dark towers contain thousands of goblins and trolls, causing lag | ||||
| Description | on a Pocket World generated with 0.40.04, with only 50 years of history to make sure i still had population low (even despite the bug fixes about population, i wanted to keep things low for my new version test), i ran into a severe slowdown in adventure mode while inside of a goblin site. At first when i entered the goblin territory (as Travel can't go in) , game was perfectly smooth and fast, i moved next to a couple of goblins tower that had some goblins on its top, still no slowdown, still fast. Then as i was starting to approach the location of the main dark fortress, suddenly the game slowed down very noticably. Here is the save just before it happens : http://dffd.wimbli.com/file.php?id=9093 Load the save, at that point,the game is still fast and smooth, walk to the south , after several steps you'll encounter a very noticable slowdown. It's painful but still you can relatively walk. Continue to the south, there's a river, go past the river (jump/swim/etc..) You'll arrive in front of a goblin tower you'll see the goblin line of sight signaling it as you're in stealth mode), on the top of it there are only a dozen of goblins. Still walk to the south past the tower, and you'll arrive near the main dark fortress, and you'll notice the slowdown is really severe at that point | ||||
| Additional Information | Saves: http://dffd.bay12games.com/file.php?id=9093 (0.40.04) http://dffd.bay12games.com/file.php?id=9094 (0.40.04) http://dffd.bay12games.com/file.php?id=9184 (0.40.05) http://dffd.bay12games.com/file.php?id=9324 (0.40.06) http://dffd.bay12games.com/file.php?id=9513 (0.40.10) http://dffd.bay12games.com/file.php?id=9558 (0.40.10) http://dffd.bay12games.com/file.php?id=9715 (0.40.12) http://dffd.bay12games.com/file.php?id=10842 (0.40.24) http://dffd.bay12games.com/file.php?id=11867 (0.42.06, 0009637, very large) | ||||
| Tags | No tags attached. | ||||
| related to | 0007698 | new | Excessive numbers of underground critters | |
| related to | 0007547 | assigned | TOTAL_CIV_POPULATION only affects historical figures, not abstracted populations | |
| related to | 0007461 | new | Unretired fort gets 1000+ goblin invaders | |
| has duplicate | 0007578 | resolved | Serious FPS dropdown on enemies' sites | |
| has duplicate | 0007878 | resolved | Dark pit population sizes make game unplayable | |
| has duplicate | 0009177 | resolved | FPS death when approaching dark tower in adv mode | |
| has duplicate | 0009170 | resolved | unplayable lag on some sites in adventure mode | |
| has duplicate | 0009676 | resolved | Troll population grows uncontrollably | |
| has duplicate | 0009637 | resolved | lethosor | assault on goblin city abruptly plagued by minutes-long processing time per turn |
| related to | 0008334 | new | Trolls in goblin sites respawning constantly | |
| related to | 0007009 | new | Important slowdown near a goblin site |
|
|
Additional saved game , after i very very slowly and painfully managed to enter the fortress and explore it a bit to find where was the population , on the surface of the dark fortress only few trolls and few goblins, couldn't be what was the source of the slowdown, then after more exploring underground of that fortress i see a giant room with hundred of trolls there : http://dffd.wimbli.com/file.php?id=9094 |
|
|
Still happening in 40.05 , i thought it would be fixed with the overpopulation fixes, but in that save : http://dffd.wimbli.com/file.php?id=9184 I left the fortress and moved toward the dark fortress on the south east of the world, only to have the game being simply unplayable in term of slow and lag at that point. I opened that 40.05 pocket world with World Viewer and it told me at the dark fortress site : 1 Dwarf (probably the guy i retired there) 1 Mouse Monster 4951 goblins ! 5254 trolls ! No surprise unfortunately it is unplayable (and that's a pocket world) |
|
|
In 40.05 i experimented today a lot with MAX_STARTING_CIV_NUMBER MAX_POP_NUMBER MAX_SITE_POP_NUMBER in the entity_default.txt and TOTAL_CIV_NUMBER TOTAL_CIV_POPULATION SITE_CAP from worlgen.txt And from my experimentations, i ended noticing that the goblins' dark fortresses always end having, in each of them, more than 1000 individuals of goblins (and as much trolls) inside, while every other civs (dwarves, humans, elves) have much more reasonnable 100/150 individuals for their main bigger site. This when i set the exact same settings for each civs in the entity_default.txt Even with only 20 years of history and no war (from what i see in World Viewer) , there's always that huge amount of goblins+trolls in a dark fortress, overwhelming numbers that just make the game unplayable when you're adventuring close to the site. |
|
|
Thanks for the saves/follow-ups. |
|
|
The three most populous sites on my current world are actually elves this time. Elves total 8-11k with various animal populations ranging from a few thousand up to 15300 just for Tigers+Jaguars+Gorillas. Several Dark Fortresses follow up just behind. Year 250. http://dffd.wimbli.com/file.php?id=9324 Sunsivy, Oaksilvers, Conflictfogs, Wheeledmalice, Plaguefield, Hexfrost, Balanceplague, Realmsterror, Waddlehate, and Beachedseduced are worst. A couple more in the low thousands and then it drops off to below 1000 in mostly caves and then quickly down to under 500. Seems about 90-95% of sites have roughly 200 or less residents. In addition to the things Robsoie tried, I have also tried making worlds where Goblins and Elves were almost exclusively homosexual species by adding [orientation] tags to no effect (not in this world). Either it doesn't work like I imagine or I did it wrong (never did try 100% homosexual, though I probably should have...). Only mod is Phoebus and used some advanced worldgen parameters (savagery, good/evil, (semi)megabeasts, probably a few others). |
|
|
Sorry for offtopic, but is [orientation] tags really a thing? |
|
|
I dunno. It is in the wiki. http://dwarffortresswiki.org/index.php/DF2014:Creature_token#ORIENTATION http://dwarffortresswiki.org/index.php/DF2014:Marriage |
|
|
0007905 pertains to ORIENTATION. Please keep the discussion here on topic. |
|
|
world generated on 40.10 at only +/- 50 years , no raws edited out of the gob/elf/human/dwarves names files to remove accentuated characters save there : http://dffd.wimbli.com/file.php?id=9513 An elf bowman adventurer left his companion waiting at the entrance of the goblin site, all was well, until i started to approach 2 goblins dark towers on my way to their dark fortress, at such point it slowdown heavily. But from the point of the save continue directly to the North , moving between the 2 dark towers, toward the dark fortress, it's going to be slower and slower into simply unplayable if you don't sprint (in which it's slow but at least you can move). according to legends viewer, the dark fortress should have 1310 Goblins 802 Trolls 798 Ogres 5 Goblin Outcasts 1 Lizard Demons 1 Elf Outcasts |
|
|
I don't know if Toady One needs more saves to help solve this or not, but heres one of a tower site that is barely playable. I'm stubborn enough to just try and plow my way through it. http://dffd.wimbli.com/file.php?id=9558 Nothing special about the save other than to demonstrate the laggyness. In the save, I'm just trying to wait for one of the rescued prisoners to snap out of it. Right now, I wish I could just pick up the kid as I move on or give him something to cheer him up and snap out of it because it's taking too long as is. According to Legendsviewer, there should be roughly: 10154 Goblins 5100 Trolls 131 Humans 78 Human Prisoners 1 Skunk Fiends I already killed some trolls and goblins, so it'll be a little less. Also, my computer is a pretty good one, so, with computers not that good, it's unplayable. |
|
|
Need more savegames? |
|
|
This seems to be related to the goblins' lack of need to eat or drink. Putting [NO_EAT] and [NO_DRINK] on kobolds, for example, will swell up cave populations to thousands in a similar way as in dark fortresses. |
|
|
They can still have thousands in a cave without those tags. I'll try removing those and see how things go. Okay, in the world I tested it in, the largest populated tower has 2267 Goblins and 1850 Trolls, about what you would get with some of the most populated human towns/cities. Most of them have goblin and troll populations that range between 1k to 2k each, seeming to average between 1,200 and 1,500. Theres also several with each troll and goblin population less than a thousand. Seems MUCH more reasonable to me. To note, I did make them mortal with the same lifespans as dwarves (elves are still immortal because elves), but that didn't affect the populations. Toady One, I'm sure you're reading this, what was the origional reason for putting the [NO_EAT] and [NO_DRINK] tags on goblins? Might not be needed anymore. |
|
|
In the original save linked in the bug report itself, there are 174 units in play at first, and after walking south and triggering the lag, there are 461 units in play. If I walk even further south to load more of the fortress (across the river and over the next trench), it increases further to 611. smjjames's savegame has a whopping 1,606 units in play, which is clearly excessive. |
|
|
In this new world I generated for adv mode, the three largest towers (and the only three to breach 3k) have populations of 3198 Goblins 2250 Trolls and 3578 Goblins 2550 Trolls and 5497 Goblins 3100 Trolls. Which is MUCH better than the 5k+, sometimes up to 10k for each population. Most towers have less than 2k for each population and some have less than 1k for each. Removing the [NO_EAT] and [NO_DRINK] tags help, but the way that the goblins are spread out, or rather, not spread out, causes lag problems and then theres the trolls doing crazy pathing which only adds to the issues. Edit: Over a thousand in play, wow. I'll check out one of the towers in this modified save and see how thing are. |
|
|
Now testing a low pop tower with populations of 1275 Goblins 99 Humans 67 Human Prisoners 12 Dwarves 4 Dwarf Prisoners 1 Lemur Monsters This goblin civ doesn't have trolls at that site for some reason, other sites of that civ have trolls though. I split off a compressed file from the spot I did the first loaded unit count if you guys request it. Going eastward directly towards the spire. The first small tower I get to has 139 loaded units, 69 goblins, 36 crundles and the rest various cavern wildlife. No significant lag. Second count, near the river before the spire, loaded unit number is 122. Mild conversation spam related lag and a bit of pathing lag. Third count, approx center of spire, loaded unit number is 182, including cave wildlife. Slight amount of lag. Second test with a Dark Tower population of: 3578 Goblins 2550 Trolls 104 Humans 25 Human Prisoners 3 Kobolds (I gave kobolds normal speech, which is why they are there) 3 Dwarves 1 Llama Fiends First small tower reached and the lag is starting. Units loaded (or otherwise seen in DFhacks reveal+exterminate) is 396 with 226 goblins and 130 crundles, the rest, various cave critters plus a few humans. Second reading, a couple tiles outside the tower and the lag has ramped up, loaded unit count is 690, with 468 trolls and 159 goblins. And um, when I did a wait one hour to clear out the reveal data, DF crashed. I'll check if that was a DFhack related thing or something else. Also, I've been getting fairly regular brief freezes with DF as I move forward. Earlier I thought it was because of me using reveal all, but this time I didn't use it and I'm getting the brief lagspike pause. Also, I tested the wait one hour and while it didn't crash, it did briefly freeze, and I replicated what I did earlier when it crashed and it didn't that time, so, I don't know what that crash was about. |
|
|
Still happens in a 40.12 save which is modded to remove [NOEAT] and {NODRINK] from goblins plus givng them the lifespan of dwarves. According to legendsviewer, the site pop is almost 1,600. http://dffd.wimbli.com/file.php?id=9715 |
|
|
Still very much an issue in 40.24. http://dffd.bay12games.com/file.php?id=10842 Vanilla goblins, only raw mod is allowing them to be adventure-playable. The site lag makes the region entirely unplayable |
|
|
Robsoie posted a save at 0009170. |
|
|
Still occurs in 42.03, most prevalent within Dark Goblin Fortresses. I don't think it's normal for enormous Troll civilizations to exist in the dug out underground caverns (as in taking occupation titles, like weaver, administrator, animal trainer, etc.). To add onto the problem, the goblins and trolls will start killing each other once they get line of sight. It makes absolutely no sense. This issue is easily reproducible, modded or otherwise; generate a world with a high site cap and decent history, adventure your way into a Dark Goblin Pit/Fortress, and watch the game become disastrously unplayable. |
|
|
A quick check with a third party tool (legends reader found here: http://www.bay12forums.com/smf/index.php?topic=72702.msg6658295#msg6658295 ) reveals that since 43.05 site populations are still exploding into enormous numbers. On my unmodded world there are globally 95200 goblins for a world with only 25347 historical figures to have ever lived, not to mention about ten thousand elves and 8000 humans. Only 24375 goblins can be supported in the 195 dark pits that have been built to accommodate the sprawl which stays within the rules of 125 per site. With in my most prominent tower, it has 10851 goblins and 2526 trolls & 2526 beak dogs. The oldest and first tower but just 1 tower out of a network of towers with large similar statistics, and this trends between the 3 goblin civilisations to reach the big 95200 global number. The wars that occur, such as my largest and latest war 'The War of Murders' has killed 23394 people, more than the 14491 historical figures that are supposed to be dead and the war still hasn't ended or either side feeling inclination to stop for more than 60 years (declared in 167, ongoing in 250) Auxiliary populations are crashing the game and making it strenuous on systems in adventure mode to approach sites because livestock is spinning out of control to keep up with the numbers in order to 'feed' those who even don't eat, all these people and the subsequent pathing causes as described lag spikes. Murders and war cannot keep on top of the goblins and i cannot fathom where or why all these unaccounted non-historical people are coming from. @boatsniper trolls have always (intentionally or not) been like that, since they are civilians but also livestock they get the best of both worlds with automatic job assignment and population scaling. However they cannot commit to a job because they are slow learners without 3rd party tool exploitation. |
|
|
I thought they wouldn't ALL load at the same time? 1000+ goblins is the population pool, and individual non-historical goblins are pulled out when you visit the tower, and then they offload when you leave. You shouldn't actually ever see all 1000 present at the same time. Might not be the case with trolls, however, since they count as animals. I've noticed that the structure of dark fortress can link cavern layers together, causing water from the above layer to fall through the holes into the layer below. Could that be the real reason of the lag? |
|
|
My 43.05 world seems affected by a similar, but somewhat different issue of overpopulation, and I was advised to take my save here. http://dffd.bay12games.com/file.php?id=12322 Notable about this is that this only happened AFTER world-gen had finished and I started playing fortress and adventurers. In 850 there were around 2500 fly-men in the world, now there are over half a million largely located in just two sites, Brassdark and Pointrose. I had two consecutive crashes on attempting to enter pointrose earlier, but just now was able to enter with only extreme performance issues (unsurprisingly, it's full of fly-men, 250 thousand apparently.) I don't have a backup save before the population started growing unfortunately, but do have the xml dump from 850 which I read with legendsviewer to see the huge difference over the two years. EDIT: I just noticed that the world and fly-man population had changed in the two weeks or so of in-game time that had passed since I last looked at the legends. Pointrose now only has 14629 fly men living there, which explains why it stopped crashing for me probably. It seems that fly-man outcasts, of which the vast majority of Pointrose consisted of have simply disappeared or something leaving only the ca. 15k that were "normal" fly-men in Pointrose, and brassdark still has hundreds of thousands... Really strange. Here is a backed up save before the outcasts disappeared, this is only two weeks before the save with a quarter-million less fly-man outcasts: http://dffd.bay12games.com/file.php?id=12324 |
|
|
A exerpt from the public domain of Reddit to highlight how this bug's persistence into 44.06 - 7 and that it breaks dark tower raids (which are a primary target for children rescues) Reddit thread: ( https://www.reddit.com/r/dwarffortress/comments/83cu59/i_think_the_new_versions_off_site_pillaging_could/ ) And subject image: ( https://i.redd.it/pkt14h002vk01.png ) Any amount of squads cannot hope to face 2526 trolls in a single encounter even with the most trained and well equipped soldiers. |
|
|
A link to the use of [USE_EVIL_ANIMALS] has been established in my save, where overpopulation and a poor allocation of non-creature tags like [PET] make them buggy because use_evil_animals applies its own uses without respecting existing creature tags. Use_evil_animals in itself is a AI only allocator for trolls to exhibit non-creature raw roles, when they are simulated in fortress mode, the trolls are actually civilian peasants & 'other animals' contributing towards population without further tags tacked on like [PET] and don't function as pure livestock at all. Trolls in themselves only have numbers culled by attacks on the civilization, and would contribute to food (troll-cannibalism & beak dogs) on goblin sites while being immortal also extends their lifespan. Troll peasants when abstracted and added to other civs as the save demonstrates causes the populations to spiral out of control even if the entity is incapable of directly eating them or using their materials like dwarves. Save: http://dffd.bay12games.com/file.php?id=13651 (fixed, i put in the link while it was still loading oops) |
|
|
Ive managed to further improve my methods and deliver results, it turns out that trolls were stressing the food populations by cannibalising each other which became apparent when i changed troll RAWS (in accordance with [ANIMAL] to prevent outsourcing, local populations only) A average fully grown Troll yields 30-40 meat from muscle alone, remove that muscalture's edibilty with something like a 'tough meat' material definition with no edible parts and the populations drop to expected levels when goblins (with a raw change) & trolls are eating. This means that all civs must be prioritising animals with the most meat per capita, including fat horses as part of their food simulation over other animals like pigs or cows. Addressing this could signficantly improve the stability of animal populations by making civs value them for something else or deliberately fatten up animals by diverting crops to feeding livestock encouraging them to eat grazers and be more selective. |
|
|
removing [NO_EAT][NO_DRINK] tag from the goblins might help.(a workaround) they have such a high population in their capitals cuz they don't eat/drink and the capitals ignore population cap! |
|
|
Sometimes my fixes don't always work and it still happens inevitably, anecdotally from running my recommendation for a long time and keeping a eye on the populations. Something more concrete might still be needed. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2014-07-20 17:12 | Robsoie | New Issue | |
| 2014-07-20 17:21 | Robsoie | Note Added: 0027082 | |
| 2014-07-28 06:23 | Robsoie | Note Added: 0027756 | |
| 2014-07-28 06:24 | Robsoie | Note Edited: 0027756 | |
| 2014-07-28 06:39 | Robsoie | Note Edited: 0027756 | |
| 2014-07-28 15:37 | Robsoie | Note Added: 0027820 | |
| 2014-07-28 15:46 |
|
Relationship added | related to 0007698 |
| 2014-07-28 15:47 |
|
Note Added: 0027823 | |
| 2014-07-28 15:47 |
|
Assigned To | => user11 |
| 2014-07-28 15:47 |
|
Status | new => acknowledged |
| 2014-07-28 15:48 |
|
Relationship added | related to 0007547 |
| 2014-07-28 15:51 |
|
Relationship added | related to 0007461 |
| 2014-08-06 18:03 |
|
Summary | heavy slowdown case near dark fortress => Dark towers contain thousands of goblins and trolls, causing lag |
| 2014-08-06 18:04 |
|
Relationship added | has duplicate 0007578 |
| 2014-08-06 18:04 |
|
Relationship added | has duplicate 0007878 |
| 2014-08-07 06:37 | Borschwanger | Note Added: 0028518 | |
| 2014-08-07 06:38 | Borschwanger | Note Edited: 0028518 | |
| 2014-08-07 06:39 | Borschwanger | Note Edited: 0028518 | |
| 2014-08-07 21:57 |
|
Status | acknowledged => confirmed |
| 2014-08-07 22:06 | smjjames | Note Added: 0028570 | |
| 2014-08-07 22:52 | Borschwanger | Note Added: 0028571 | |
| 2014-08-08 09:00 |
|
Note Added: 0028582 | |
| 2014-08-25 07:37 | Robsoie | Note Added: 0029610 | |
| 2014-08-25 07:43 | Robsoie | Note Edited: 0029610 | |
| 2014-08-29 12:19 | smjjames | Note Added: 0029777 | |
| 2014-08-29 12:25 | smjjames | Note Edited: 0029777 | |
| 2014-08-29 12:25 | smjjames | Note Edited: 0029777 | |
| 2014-08-31 07:43 | smjjames | Note Added: 0029820 | |
| 2014-09-03 08:48 | samanato | Note Added: 0029913 | |
| 2014-09-03 09:15 | smjjames | Note Added: 0029915 | |
| 2014-09-03 09:36 | smjjames | Note Edited: 0029915 | |
| 2014-09-03 09:38 | smjjames | Note Edited: 0029915 | |
| 2014-09-03 09:44 | smjjames | Note Edited: 0029915 | |
| 2014-09-03 09:53 | Quietust | Note Added: 0029916 | |
| 2014-09-03 09:54 | Quietust | Note Edited: 0029916 | |
| 2014-09-03 09:59 | Quietust | Note Edited: 0029916 | |
| 2014-09-03 10:07 | smjjames | Note Added: 0029917 | |
| 2014-09-03 10:08 | smjjames | Note Edited: 0029917 | |
| 2014-09-03 10:39 | smjjames | Note Edited: 0029917 | |
| 2014-09-03 15:46 | smjjames | Note Added: 0029921 | |
| 2014-09-03 15:56 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 16:03 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 16:27 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 16:53 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 17:03 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 17:27 | smjjames | Note Edited: 0029921 | |
| 2014-09-03 17:30 | smjjames | Note Edited: 0029921 | |
| 2014-09-16 08:47 | smjjames | Note Added: 0030216 | |
| 2014-09-22 09:50 |
|
Relationship added | related to 0008334 |
| 2015-05-10 06:18 | Witty | Note Added: 0032666 | |
| 2015-05-10 10:15 | lethosor | Additional Information Updated | |
| 2015-05-10 10:16 | lethosor | Sticky Issue | No => Yes |
| 2015-12-03 03:19 |
|
Relationship added | has duplicate 0009177 |
| 2015-12-03 03:20 |
|
Relationship added | has duplicate 0009170 |
| 2015-12-03 03:20 |
|
Relationship added | related to 0007009 |
| 2015-12-03 03:22 |
|
Note Added: 0033331 | |
| 2015-12-25 11:57 | Boatsniper | Note Added: 0034192 | |
| 2016-05-08 17:08 |
|
Relationship added | has duplicate 0009676 |
| 2016-07-28 19:11 | lethosor | Relationship added | has duplicate 0009637 |
| 2016-07-28 19:12 | lethosor | Additional Information Updated | |
| 2016-07-30 03:59 | FantasticDorf | Note Added: 0035724 | |
| 2016-07-30 07:05 | FantasticDorf | Note Edited: 0035724 | |
| 2016-07-31 03:10 | Murphy | Note Added: 0035728 | |
| 2016-08-02 03:10 | LordKnows | Note Added: 0035735 | |
| 2016-08-02 10:16 | LordKnows | Note Edited: 0035735 | |
| 2018-03-16 12:42 | FantasticDorf | Note Added: 0037948 | |
| 2018-04-05 09:22 | FantasticDorf | Note Added: 0038130 | |
| 2018-04-05 14:06 | FantasticDorf | Note Edited: 0038130 | |
| 2018-11-09 23:49 | FantasticDorf | Note Added: 0038927 | |
| 2018-12-19 23:57 | bluebird9281 | Note Added: 0039030 | |
| 2018-12-20 09:46 | FantasticDorf | Note Added: 0039031 |
