Skip to main content

Accuracy problem with System.Double and System.Single

So, what do I do while my class is working on labs?  Well, if I’m not answer emails from the Navy, I’m working on some side projects.  Right now, I’m working on a project to grab meta data via geographical coordinates. When I develop code, I brake the objective down into manageable parts.  I then look at each part and ask myself if I know how to do it.  For each part that I do not know how, I start writing code to discover the path to success.

Well, while working on a bit of code, I needed to increment my value for latitude by .05.  Something odd happened so I wrote out a small bit of code to focus in on the issue.  Here is the code.

$Increment = .05

$Start = 0

$Value2 = $Start

 

While ($Value2 -lt 10)

{

    Write-host $Value2

    $Value2 += $Increment

}

The data type of $Value2 is Double.  Everything starts out fine.

0

0.05

0.1

0.15

0.2

Until…

3.45

3.5

3.55

3.6

3.65

3.69999999999999

3.74999999999999

3.79999999999999

3.84999999999999

 

Why System.Double start incrementing in .049999999999999 and then back to .05 later on, I do not know.  OK, let’s force the variable to be System.Single

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5000001

0.5500001

0.6000001

0.6500001

 

OK, now it increments by 0.050001 and then back to .05. 

Luckily, I found a data type that does not loose it’s accuracy under these conditions, System.Decimal.

$Increment = .05

$Start = 0

[Decimal]$Value4 = $Start

 

While ($Value4 -lt 10)

{

    Write-host $Value4

    $Value4 += $Increment

 

 

}

$Value4 | GM

 

Below is a sample of the desired output.

0

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

 

I took a look at System.Double on MSDN. According to Microsoft, System.Double coheres to the standard for binary floating-point arithmetic. (See the Remarks section).  Since some fractional values cannot be represented precisely.  Look at System.Decimal.  In the Remarks section, you will see that it dose not have a round off error.  At least not at the precision level that I need.

I now need to decide between the smaller memory footprint of Single or Double data types as compared to the much larger storage requirement of Decimal.  It looks like that I will be computing the Longitude and Latitude values in Decimal, But using the –as operator in PowerShell to convert them into Single for storage since I do not need to go beyond 2 decimal places.

[Decimal]$Decimal = 5.5

$Single = $Decimal -as [Single]

$Single

 

The result will be a Single value that will have the precision that I need with a smaller memory footprint.

Comments

Popular posts from this blog

Adding a Comment to a GPO with PowerShell

As I'm writing this article, I'm also writing a customization for a PowerShell course I'm teaching next week in Phoenix.  This customization deals with Group Policy and PowerShell.  For those of you who attend my classes may already know this, but I sit their and try to ask the questions to myself that others may ask as I present the material.  I finished up my customization a few hours ago and then I realized that I did not add in how to put a comment on a GPO.  This is a feature that many Group Policy Administrators may not be aware of. This past summer I attended a presentation at TechEd on Group Policy.  One organization in the crowd had over 5,000 Group Policies.  In an environment like that, the comment section can be priceless.  I always like to write in the comment section why I created the policy so I know its purpose next week after I've completed 50 other tasks and can't remember what I did 5 minutes ago. In the Group Policy module for PowerShell V3, th

Return duplicate values from a collection with PowerShell

If you have a collection of objects and you want to remove any duplicate items, it is fairly simple. # Create a collection with duplicate values $Set1 = 1 , 1 , 2 , 2 , 3 , 4 , 5 , 6 , 7 , 1 , 2   # Remove the duplicate values. $Set1 | Select-Object -Unique 1 2 3 4 5 6 7 What if you want only the duplicate values and nothing else? # Create a collection with duplicate values $Set1 = 1 , 1 , 2 , 2 , 3 , 4 , 5 , 6 , 7 , 1 , 2   #Create a second collection with duplicate values removed. $Set2 = $Set1 | Select-Object -Unique   # Return only the duplicate values. ( Compare-Object -ReferenceObject $Set2 -DifferenceObject $Set1 ) . InputObject | Select-Object – Unique 1 2 This works with objects as well as numbers.  The first command creates a collection with 2 duplicates of both 1 and 2.   The second command creates another collection with the duplicates filtered out.  The Compare-Object cmdlet will first find items that are diffe

How to list all the AD LDS instances on a server

AD LDS allows you to provide directory services to applications that are free of the confines of Active Directory.  To list all the AD LDS instances on a server, follow this procedure: Log into the server in question Open a command prompt. Type dsdbutil and press Enter Type List Instances and press Enter . You will receive a list of the instance name, both the LDAP and SSL port numbers, the location of the database, and its status.