Hey,

I am unsure if this has been discussed before (I couldnt find any mention of it), however I seem to be having an issue where i am loosing some precision loss.

The main place I have seen this is when I have my program increment a variable (Data type: double) by 0.1. I want to stop the variable from being increased above 0.7 or below 0.1. My issue was that it appear that instead of increasing/decreasing my variable by 0.1, it is actually doing a value similar to 0.0999999999998. Or something similar to that (I havent been able to work out the exact error). So when I have the value displayed on the screen through a text gauge it will round 0.69999999 to 0.7 however my catch "if(variable > = 0.7, 0.7, variable + 0.1)" doesnt catch the issue until it gets to 0.8.

I have gotten around this issue by adjusting the calculation event to this "if(variable > = 0.7 - 0.01, 0.7, variable + 0.1)", which works for the time being, but I just thought I would bring this up with you as I dont remember seeing the issue pre the patch updates to Power Vision 2.7

Power Vision Version: 2.7.10456.
Screed Model: PV450.
Operating system: Windows 7 SP1 64bit

Hello ADE,

Well, I had to be reminded about float behavior to get an answer for you.

The short answer is that your variable, which is a double, can't be represented as precisely as the constant (0.7) so the equals test will almost always fail. Not really a bug, it is just how floats in a Base 2 world work.

There are several ways to accomplish what you need, including what you are already doing, which changes the test so that you don't use =. Also, you could convert to integers and then the = would work for your comparison. There are probably others that will suit your situation.

Hope this helps!

Hey Sara,

Thanks for your reply,

I thought that double data types were accurate to 15 digits, and I would only get rounding issues if I used more digits than that? But with you discussion on the nature of floating points and then thinking how my incrementing value (0.1) would be represented in binary I see that it would require an infinite number of bits. Darn here I was thinking 0.1 would be an easy number.

As you mentioned there are plenty of ways around this so I'm fine just thought it was a bug at first