Mission percentages bugged

[HoreTore]

You can't ever have a probability over 100%.

Sounds like someone at CA missed the lecture on conditional probability.

The probability of two spies with a 75% chance of opening the gate woould be:

p(spy one succeeds) + p(spy two succeeds: given spy one failed)

this would be (75%) + (75%)*(25%) = 93.75%

Add a third spy with a 75% chance and it's:

75% + (75%*25%)+(75%*6.25) = 98.43%

You can never get all the way to 100%.

Ever seen public order over 100%? Yes, you have. You have to play by the rules of the game, not the rules of the physical world, because, well, this is not a physical thing. If you look at the mechanics for things, you'll notice that most things give things in batches of 5%. For example, the amount of unrest in a city caused by spies, is skill level of the spy multiplied by that 5%. So, it would seem logical too me that this logic in carried over to this section of the game too, and then you'll easily get over 100%...

[Foz]

Some modder here should try to take a look at the files regarding mission success percentage and try to figure out if they are working properly or needs to be fixed

I have heard way too much anectdotal envidence to believe that people are just simply getting a bit lucky or has no luck

The only anecdotal evidence I've seen to that effect is "I reloaded the game but I can't make the outcome come out how I want, waaaaaah." No one has said anything close to "spies really fail 70% when they have 95% success rates" that I have seen yet. People complain a lot, but the only thing that I've seen frequently in their reports is the behavior after loading a saved game, and none of the comments at all have any actual testing behind them that should make anyone question anything.

I'm not concerned at all because my game hasn't done anything that would indicate a broken random number system, so I'm probably going to remain unconvinced of any problem here unless someone posts multiple results from 95% success tests where they obtain far more failures than 5%, or something similar.

[Whacker]

The only anecdotal evidence I've seen to that effect is "I reloaded the game but I can't make the outcome come out how I want, waaaaaah." No one has said anything close to "spies really fail 70% when they have 95% success rates" that I have seen yet. People complain a lot, but the only thing that I've seen frequently in their reports is the behavior after loading a saved game, and none of the comments at all have any actual testing behind them that should make anyone question anything.

I'm not concerned at all because my game hasn't done anything that would indicate a broken random number system, so I'm probably going to remain unconvinced of any problem here unless someone posts multiple results from 95% success tests where they obtain far more failures than 5%, or something similar.

I've ran several tests that you can find if you dig around in old threads. Suffice to say that my findings didn't even come close to the percentages given for the actual mission.

[Werner]

Another factor to look at is that the skill levels for spies and assassins go well above "10/10" eyes. I just counted up all the skills, traits, and retinue for one of my assassins and it came out to be 15 points of skill. So my question is how does the game account for this?

NOTE: Foz, its been established that the numbers system is broken. If you dont use assassins or spies very much it wont hurt your game and you can turn a blind eye to it. What we're trying to do is fix it.

[HoreTore]

It accounts for the extra levels just like any other levels. A level above ten increases your chances.

[dismal]

Ever seen public order over 100%? Yes, you have. You have to play by the rules of the game, not the rules of the physical world, because, well, this is not a physical thing. If you look at the mechanics for things, you'll notice that most things give things in batches of 5%. For example, the amount of unrest in a city caused by spies, is skill level of the spy multiplied by that 5%. So, it would seem logical too me that this logic in carried over to this section of the game too, and then you'll easily get over 100%...

Public order is not a probability. The probability of an event occuring can't be more than 100%. This is inherent inthe definition of probability.

The game may calculate whether a gate opens however it wants, but it's nonsensical to say something has a probability outside the range 0 to 100%.

[Foz]

I've ran several tests that you can find if you dig around in old threads. Suffice to say that my findings didn't even come close to the percentages given for the actual mission.

Another factor to look at is that the skill levels for spies and assassins go well above "10/10" eyes. I just counted up all the skills, traits, and retinue for one of my assassins and it came out to be 15 points of skill. So my question is how does the game account for this?

NOTE: Foz, its been established that the numbers system is broken. If you dont use assassins or spies very much it wont hurt your game and you can turn a blind eye to it. What we're trying to do is fix it.

Who established that? And how? I said I haven't seen anything, because I haven't, lol. Point me in the right direction so I can read and contribute accordingly.

[Werner]

Well at this point I suppose it would be good to know how the code is written and take a look at the "random seed" generator that another fellow was talking about. Ultimately we need a patch but it might not hurt to understand how it determines the success/failure rate.

[HoreTore]

Public order is not a probability. The probability of an event occuring can't be more than 100%. This is inherent inthe definition of probability.

The game may calculate whether a gate opens however it wants, but it's nonsensical to say something has a probability outside the range 0 to 100%.

In the real world, yes, in a computer program, no, it is not nonsensical.

[Whacker]

Use the search function until you find them, I don't even remember the thread names anymore as it's been well over a month or two. What I can tell you is my testing methods.

Basically I would set it up so my assassin is right next to the target. I would then attempt the mission, note success or failure, then reload. No other actions were taken or performed so I could limit this down as much as possible so no other variables (such as the supposed "reseeding") could skew the results. What I noted in every case was that the chance for success doesn't remotely match what's given, a random example would be a 3 rank assassin having a 25% chance to kill a 2 star general. My findings would be something like success 4 times out of 50 or so. The bottom line:

1. The probability displayed is not accurate given my and a few others tests.
2. The game may or may not have a 'seed' that affects the outcome.
3. The game may or may not 'roll' exactly when the action is attempted. This is what I suspect happens, FWIW.
3. The outcome is definitely not fixed, as the successes on reloading are evidence thereof.

Others have suggesed two things as well. One, it appears that proximity of other faction's spies (even those not of your target's faction) can negatively impact your chances for success given an assassin's mission. Two, apparently skill/ancils/traits that total up to and over 10 are indeed counted towards success. So you really can have a 20 skill monster roaming around, and the probabilities of successful mission completion will be given based on the assassin being at 20 skill, not 10. I did not do any testing in RTW/BI/A to see if the game would let you have more than 10 skill and if so if that was accounted for, but my gut reaction is no it did not. It is unknown if this 10+ skill mechanic is intentional or a bug, but it's generally accepted to be true.



Edit:

I just thought of this. Before you jump all over that example I made Foz, I understand that in general a 25% chance means that 25 times out of 100 an event will succeed. I don't remember ever doing 100 tests, but in the example I gave 4/50 is 8%, which is far less than 25. Sure I could run 50 more tests after that and there's a snowballs chance in hell that the last 21 would succeed giving me 25%, but it's unlikely. My successes were very random and pretty evenly distributed when I did these. Also as a human I can only run so many tests before I get bored or tired, so I didn't run more. In my made-up example with a 25% success rate I should still see at least roughly 12 successes out of 50. The fact that all of my test runs consistently showed success rates far less than what was displayed is sufficient for me.

[Whacker]

In the real world, yes, in a computer program, no, it is not nonsensical.

He's right HoreTore. A game may do the math on a certain mechanic so that the outcome is guaranteed to happen given the inputs, but the probability of something happening cannot exceed 100% by definition. For example, say that spy skill is cumulative for opening a gate, and you have 3 spys that each have a 50% chance of opening the gate on assault. You add that up and get 150%, but in reality the probability is 100% because it cannot be more than this. Make sense? Good.

Also public order isn't a probablity function, it's a cumulative statistical percentage. Your 80% garrison + your charming 20% commander + your 20% fun buildings will all add up to 120% order.

[Foz]

Basically I would set it up so my assassin is right next to the target. I would then attempt the mission, note success or failure, then reload.

This is exactly why I asked. It's been suggested multiple times (more than anything else) that reloading is the primary cause of these skewed random results, so reloading says nothing in general about the number generator, it only tells us what happens if you reload to try the same agent mission again. A general test of the system would have to avoid doing this, and more than likely means a bunch of consecutive campaign turns where you keep doing different things with agents so that you actually test the random number stream. Random number generators typically use a seed value to determine how to generate a semi-random stream of numbers... so while any given stream looks pretty random, you can actually get the exact same stream back if you seed the generator with the same number. It simulates random well enough for most purposes, without actually being random.

Anyway, the point is that the random system may be entirely fine even if it gives horrible results when you reload the game - all it needs to do is generate a sufficiently random number stream so that each new event is ~random. If it does that, then it's working flawlessly, because it's not even clear that reloading the game should give you a different result the next time, nor does it matter since the same exact event never happens multiple times in the same game. You may try the same thing again the next turn, but it's not the same event because the turn has changed, the random number stream may have advanced, and any number of other conditions will be different. Loading the game may be removing all those different factors, which means the failure of the random system here is insignificant unless it can also be shown to fail under the typical operating conditions of an ongoing campaign. Only then would it have a problem that will actually affect gameplay, where the current results indicate only an obscure quirk, unrelated to ongoing gameplay, and not even as yet shown to be unintended...

[JCoyote]

Well it's easy to test this. All someone has to do is keep track through a campaign... write the percentage for each strategic character event, and whether it succeeds or not. Then afterwards organize them by the particular percentage... all the 35% together, etc. Then you could get a pretty clear idea how close it's getting.

[dismal]

Well it's easy to test this. All someone has to do is keep track through a campaign... write the percentage for each strategic character event, and whether it succeeds or not. Then afterwards organize them by the particular percentage... all the 35% together, etc. Then you could get a pretty clear idea how close it's getting.

Yep, although it can take large number of readings for samples to converge on their underlying means.

Pulling a random number and checking it versus a value is pretty rudimentary stuff in the grand scheme of computer programming, so it would be rather surprising to me if it were bugged. But you never know.

[Foz]

Well it's easy to test this. All someone has to do is keep track through a campaign... write the percentage for each strategic character event, and whether it succeeds or not. Then afterwards organize them by the particular percentage... all the 35% together, etc. Then you could get a pretty clear idea how close it's getting.

Yep, although it can take large number of readings for samples to converge on their underlying means.

Yes indeed. IIRC the normal approximation of binomial (that is, a 2-option trial, like success/failure) only holds with sample size at least 30, so that would be the least you'd want for the testing. It also requires p (the probability of success) be [.05, .95], which I believe were determined to be the limits the game typically allows for agent success or failure: 5%, and 95%. Perhaps those bounds are not only to ensure a chance for either option always, but more statistically motivated: it's far easier to test the agent missions if you know you can approximate the situation with the normal distribution.

If anyone does intend to test this in a campaign, it's probably simplest to just cheat (via console) some agents up to godlike status, then run them around doing 95/5 missions and record a ton of results quickly. It has the benefit of making all the tests have the same odds, which piles up useful results far faster than any other way likely would.

[JCoyote]

Well Foz, couldn't wrap around become a problem with that idea?

[HoreTore]

He's right HoreTore. A game may do the math on a certain mechanic so that the outcome is guaranteed to happen given the inputs, but the probability of something happening cannot exceed 100% by definition. For example, say that spy skill is cumulative for opening a gate, and you have 3 spys that each have a 50% chance of opening the gate on assault. You add that up and get 150%, but in reality the probability is 100% because it cannot be more than this. Make sense? Good.

Also public order isn't a probablity function, it's a cumulative statistical percentage. Your 80% garrison + your charming 20% commander + your 20% fun buildings will all add up to 120% order.

Yup, if the game handles probability like the real world, then it's true. However, that might not be the case. Consider a board game rpg, say D&D. Now, you have a 20 sided dice, and you have make the thing you attempt on any outcome. However, a rule says that a roll of a one is always a failure. Now, even if you have 100% chance of success, you can still fail it. However, if you have say 150% chance of success, then you are entitled another roll if you roll a 1, and will have to roll more than 10 on that roll.

I have no idea if that's the case in this game, however, it COULD be, and thus you will see 150% "probability".

And yes, I know that PO isn't a probability, it was just representing the game stacking percentages.

[JCoyote]

Public order is mechanically different. It isn't stacking. It simply stats a point at which no disorder can happen as 100%, and how far above that you are is the margin you have against things like spies, etc. It is a probability issue below 100%, but above it it isn't anymore because the events it is measuring against can't happen anymore. 130% public order is 0% chance of riots/rebellion with a 30% safety margin that has to drop before it can happen.

[HoreTore]

Yes yes, I know that, but that was quite besides my point...

A bad example, nothing more.

[Foz]

Yup, if the game handles probability like the real world, then it's true. However, that might not be the case. Consider a board game rpg, say D&D. Now, you have a 20 sided dice, and you have make the thing you attempt on any outcome. However, a rule says that a roll of a one is always a failure. Now, even if you have 100% chance of success, you can still fail it. However, if you have say 150% chance of success, then you are entitled another roll if you roll a 1, and will have to roll more than 10 on that roll.

I have no idea if that's the case in this game, however, it COULD be, and thus you will see 150% "probability".

I think the point is that the game should be clear that the number it is displaying is not an actual probability in any case where it is not. If they want to use percentages to mean something other than the actual chance that the event happens, they could do so, but it requires very clear explanation, as percentages as commonly used by anyone always indicate the chance of an event happening with 100% being absolutely will happen no question and 0% never ever cannot possibly happen.

Even in the situation you point out above, the probability of your success is still not above 100%. You actually have a (0.05)(0.5) = 0.025 chance to fail, and thus a 97.5% success rate...

I think the biggest issue here is that probability and statistics are very well defined things with absolutely clear meanings, and the developers should not be using statistical terms if they do not intend to invoke their statistical meanings as a result.

[Whacker]

I think the point is that the game should be clear that the number it is displaying is not an actual probability in any case where it is not. If they want to use percentages to mean something other than the actual chance that the event happens, they could do so, but it requires very clear explanation, as percentages as commonly used by anyone always indicate the chance of an event happening with 100% being absolutely will happen no question and 0% never ever cannot possibly happen.

Even in the situation you point out above, the probability of your success is still not above 100%. You actually have a (0.05)(0.5) = 0.025 chance to fail, and thus a 97.5% success rate...

I think the biggest issue here is that probability and statistics are very well defined things with absolutely clear meanings, and the developers should not be using statistical terms if they do not intend to invoke their statistical meanings as a result.

I'm still not convinced that there's always a 5% pad on the high and low end of the mechanics here. I'm also not convinced that there's a random "seed" that's generated each turn for events, it'd be much easier and less memory intensive to generate a random number and just calculate the result when the action is performed. And as for my testing, it's the best we can do given what we've got to work with, it's as close to a "control" in test as is possible. Someone just playing the game normally and recording results would be possibly useful information provided they recorded actual skill and target strength (command skill, merchant skill, etc depending on target), as well as probability given for success. There's still too many variables introduced when you do this, and you're still assuming that the game actually does a random seed.

The bottom line is we all can't know what's going on for certain what's going on and how this game handles the mechanics, the best we can do is guess. CA is the only source that can give us a definitive answer.

All I know is that what's displayed to me in the game does not remotely reflect what my actual chances are in practice.

[HoreTore]

I think the point is that the game should be clear that the number it is displaying is not an actual probability in any case where it is not. If they want to use percentages to mean something other than the actual chance that the event happens, they could do so, but it requires very clear explanation, as percentages as commonly used by anyone always indicate the chance of an event happening with 100% being absolutely will happen no question and 0% never ever cannot possibly happen.

Even in the situation you point out above, the probability of your success is still not above 100%. You actually have a (0.05)(0.5) = 0.025 chance to fail, and thus a 97.5% success rate...

I think the biggest issue here is that probability and statistics are very well defined things with absolutely clear meanings, and the developers should not be using statistical terms if they do not intend to invoke their statistical meanings as a result.

I know of course that something cannot have over 100% chance of happening. My argument, is that in a GAME, you CAN see those odds represented, event though you still don't have over 100% probability. It might be an explanation as to why you see spies having over 100% to open the gates. Statistics may be clear, but you have a lot of games representing stuff in ways that break it.

[Foz]

I'm also not convinced that there's a random "seed" that's generated each turn for events, it'd be much easier and less memory intensive to generate a random number and just calculate the result when the action is performed.

There in lies the problem. You cannot simply "generate" a random number. Where would it come from? Computers deal in rules, 1s and 0s, and calculations - random is anything but that. Thus the process of generating numbers that are seemingly random yet dispersed uniformly throughout the range [0, 1] is not a trivial task on a computer. The way it's typically done is to define a process that gives the general uniform dispersal you want, with a suitable period (since it's all function-based, the sequences always repeat at some point - the idea is to make that a LONG time). Then you use the seed to set the starting state of the system - you start at a state that isn't ever the same, in order to avoid giving the same set of numbers every time from the generator. So the seed doesn't really determine the result, it only mixes up the result some so that the generator does not become predictable, while still being able to generate numbers to the given specs. Outside of using a physical random number generator, the pseudo-random number generator I've attempted to describe is the only way I know of to get "random" numbers for computing purposes. Coincidentally a generator does calculate the next result on demand, it's only the internal state of the process that is needed to do so, which typically does not take up much ram at all (could easily be ~100 bits, and would never need to be much larger). It's this internal state I've suggested is saved in the savegame, thus giving the duplicated results some people have noted when loading to attempt something again.

I realize that may not be the easiest thing to digest... so for the moment I guess I'll let people try to do that, and see what if any remarks follow as a result.

[Whacker]

There in lies the problem. You cannot simply "generate" a random number. Where would it come from? Computers deal in rules, 1s and 0s, and calculations - random is anything but that. Thus the process of generating numbers that are seemingly random yet dispersed uniformly throughout the range [0, 1] is not a trivial task on a computer. The way it's typically done is to define a process that gives the general uniform dispersal you want, with a suitable period (since it's all function-based, the sequences always repeat at some point - the idea is to make that a LONG time). Then you use the seed to set the starting state of the system - you start at a state that isn't ever the same, in order to avoid giving the same set of numbers every time from the generator. So the seed doesn't really determine the result, it only mixes up the result some so that the generator does not become predictable, while still being able to generate numbers to the given specs. Outside of using a physical random number generator, the pseudo-random number generator I've attempted to describe is the only way I know of to get "random" numbers for computing purposes. Coincidentally a generator does calculate the next result on demand, it's only the internal state of the process that is needed to do so, which typically does not take up much ram at all (could easily be ~100 bits, and would never need to be much larger). It's this internal state I've suggested is saved in the savegame, thus giving the duplicated results some people have noted when loading to attempt something again.

I realize that may not be the easiest thing to digest... so for the moment I guess I'll let people try to do that, and see what if any remarks follow as a result.

I just realized I was misunderstanding what I *think* people were meaning by the 'random seed' in terms of how these events are calculated. Perhaps I am wrong and I did understand, but I'll elaborate. Before I get into this, I did check with two codemonkey friends before I posted this just to make sure some of my understandings were still correct. It's been a long time since I've banged out code for a living.

First in terms of random number generation, the C/C++ rand() and srand() functions will work absolutely fine, with the srand() seed simply being the system time in most all cases. Some software shops, I think Bioware being one, have made it publically known that they've improved on these base functions to get "better" results/"rolls" for their various D20 based games. As you pointed out it's only pseudorandom, esp. if one goes with the base C/C++ reference functions, but it's still been found to be sufficient in most cases. Someone on my floor in my dorm back in the day had to write a paper on this subject for a 200 level course, the overall observation (which is in line with the general viewpoint) is that it's not perfect and can vary somewhat by platform, but generally does give a good, consistent, full spread of results given a particular range set to work with.

The point where I was both misunderstanding and making an assumption was; as a matter of good programming practice, one should ALWAYS reinitialize the srand() seed before running a rand() to ensure that a new random number is generated. Thus in my thoughts, reload or no reload, the game should be performing the srand() first immediately by the actual rand() to decide of the event is a success or not. In my view (and in buddy's views) running reseeding only at the start of the campaign map turn or on reloading the game is "bad programming". This is of course subjective and depending on the particular needs, reseeding once may be a legitimate strategy to use, but again * can only guess and assume based on my limited previous experiences, education, and input from more knowledgeable sources. In terms of modern pc power and even the minimum specs required to run this game, running a simple srand() to reseed would require a minimum of cycles and should quite honestly be absolutely zero performance hit whatsoever on the cpu. In terms of memory again this is going to be minimal, seeing how much is required simply to run the game, much less good ol' Windows. Storing a 'seed' of some kind in a savegame in my view would be completely pointless based on the above. Given how frequently this needs to run, what maybe 10, 20, 40 times per turn towards the start of a campaign?, it's not like this is something that needs to be constantly recalculated and would put more strain on the system resources.

Now I don't know how CA is doing this, they could be doing something completely different for all we know. /shrug Just some thoughts, and it's late and I'm really tired. Good discussion though.