Wednesday, August 03, 2011
Using double or float values in Element schemas
Q: I have a schema object type defined with a float value e.g.
loss1 instance="1:1" type="DOUBLE"
and set the object with a decimal value e.g. 8.5 . The object doesn't seem to return normal results in cases like an XML validation (CfgValidate) where it fails to validate the config. This config looks okay to me.
A: The reason it fails is due to a (not documented) restriction that floats and doubles in Element require IEEE-754 hex type formatting in their value field. E.g. if you invoke the type with a value of
loss1 type="DOUBLE"
followed by 4021000000000000, then the object will behave like a double valued object, be correctly validated, etc. Same applies to objects of type "FLOAT". There are many applets online that will convert a constant value for you, for example http://babbage.cs.qc.edu/IEEE-754/Decimal.html will convert it to 64-bit hexadecimal.
Labels: double, Element, float, schema
free invisible web counter
