| 词条 | Bfloat16 floating-point format | |||||||||||||||
| 释义 |
The bfloat16 floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a truncated (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing.[1] It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. The bfloat16 format is utilized in upcoming Intel AI processors, such as Nervana NNP-L1000, Xeon processors, and Intel FPGAs,[2][3][4] Google Cloud TPUs,[5][6][7] and TensorFlow.[7][8] bfloat16 floating-point formatbfloat16 has the following format:
The bfloat16 format, being a truncated IEEE 754 single-precision 32-bit float, allows for fast conversion to and from an IEEE 754 single-precision 32-bit float, and preserves the exponent bits while reducing the significand. Preserving the exponent bits maintains the 32-bit float's range of ≈ 10−38 to ≈ 3 × 1038.[9] The bits are laid out as follows: Contrast to an IEEE 754 single-precision 32-bit float: Exponent encodingThe bfloat16 binary floating-point exponent is encoded using an offset-binary representation, with the zero offset being 127; also known as exponent bias in the IEEE 754 standard.
Thus, in order to get the true exponent as defined by the offset-binary representation, the offset of 127 has to be subtracted from the value of the exponent field. The minimum and maximum values of the exponent field (00H and FFH) are interpreted specially, like in the IEEE 754 standard formats.
The minimum positive normal value is 2−126 ≈ 1.18 × 10−38 and the minimum positive (subnormal) value is 2−126−7 = 2−133 ≈ 9.2 × 10−41. Encoding of special valuesPositive and negative infinityJust as in IEEE 754, positive and negative infinity are represented with their corresponding sign bits, all 8 exponent bits set (FFhex) and all significand bits zero. Explicitly, val s_exponent_signcnd +inf = 0_11111111_0000000 -inf = 1_11111111_0000000NaNJust as in IEEE 754, NaN values are represented with either sign bit, all 8 exponent bits set (FFhex) and not all significand bits zero. Explicitly, val s_exponent_signcnd +NaN = 0_11111111_klmnopq -NaN = 1_11111111_klmonpqwhere at least one of k, l, m, n, o, p, or q is 1. As with IEEE 754, NaN values can be quiet or signaling, although there are no known uses of signaling bfloat16 NaNs as of September 2018. Range and precisionBfloat16 is designed to maintain the number range from the 32-bit IEEE 754 single-precision floating-point format (binary32), while reducing the precision from a 24 bits to a 8 bits. Rounding modesAccording to this [https://software.intel.com/sites/default/files/managed/40/8b/bf16-hardware-numerics-definition-white-paper.pdf white paper], 'round to the nearest; ties to even' (RNE) mode is the default and only supported rounding mode. ExamplesThese examples are given in bit representation, in hexadecimal and binary, of the floating-point value. This includes the sign, (biased) exponent, and significand. 3f80 = 0 01111111 0000000 = 1 c000 = 1 10000000 0000000 = −2 7f7f = 0 11111110 1111111 = (28 − 1) × 2−7 × 2127 ≈ 3.38953139 × 1038 (max finite positive value in bfloat16 precision) 0080 = 0 00000001 0000000 = 2−126 ≈ 1.175494351 × 10−38 (min normalized positive value in bfloat16 precision and single-precision floating point) The maximum positive finite value of a normal bfloat16 number is 3.38953139 × 1038, slightly below (224 − 1) × 2−23 × 2127 = 3.402823466 × 1038, the max finite positive value representable in single precision. Zeros and infinities0000 = 0 00000000 0000000 = 0 8000 = 1 00000000 0000000 = −0 7f80 = 0 11111111 0000000 = infinity ff80 = 1 11111111 0000000 = −infinity Special values4049 = 0 10000000 1001001 = 3.140625 ≈ π ( pi ) 3eab = 0 01111101 0101011 = 0.333984375 ≈ 1/3 NaNsffc1 = x 11111111 1000001 => qNaN ff81 = x 11111111 0000001 => sNaN See also
References1. ^{{Cite book |doi=10.23919/DATE.2018.8342167|chapter=A transprecision floating-point platform for ultra-low power computing|title=2018 Design, Automation & Test in Europe Conference & Exhibition (DATE)|pages=1051–1056|year=2018|last1=Tagliavini|first1=Giuseppe|last2=Mach|first2=Stefan|last3=Rossi|first3=Davide|last4=Marongiu|first4=Andrea|last5=Benin|first5=Luca|isbn=978-3-9819263-0-9|arxiv=1711.10374}} {{data types}}{{DEFAULTSORT:bfloat16 floating-point format}}2. ^{{Cite web | title = Intel unveils Nervana Neural Net L-1000 for accelerated AI training | author = Khari Johnson | work = VentureBeat | date = 2018-05-23 | accessdate = 2018-05-23 | url = https://venturebeat.com/2018/05/23/intel-unveils-nervana-neural-net-l-1000-for-accelerated-ai-training/ |quote = ...Intel will be extending bfloat16 support across our AI product lines, including Intel Xeon processors and Intel FPGAs. }} 3. ^{{Cite web | title = Intel Lays Out New Roadmap for AI Portfolio | author = Michael Feldman | work = TOP500 Supercomputer Sites | date = 2018-05-23 | accessdate = 2018-05-23 | url = https://www.top500.org/news/intel-lays-out-new-roadmap-for-ai-portfolio/ | quote = Intel plans to support this format across all their AI products, including the Xeon and FPGA lines }} 4. ^{{Cite web | title = Intel To Launch Spring Crest, Its First Neural Network Processor, In 2019 | author = Lucian Armasu | work = Tom's Hardware | date = 2018-05-23 | accessdate = 2018-05-23 | url = https://www.tomshardware.com/news/intel-neural-network-processor-lake-crest,37105.html | quote = Intel said that the NNP-L1000 would also support bfloat16, a numerical format that’s being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019. }} 5. ^{{Cite web | title = Available TensorFlow Ops {{!}} Cloud TPU {{!}} Google Cloud | author = | work = Google Cloud | date = | accessdate = 2018-05-23 | url = https://cloud.google.com/tpu/docs/tensorflow-ops | quote = This page lists the TensorFlow Python APIs and graph operators available on Cloud TPU. }} 6. ^{{Cite web | title = Comparing Google's TPUv2 against Nvidia's V100 on ResNet-50 | author = Elmar Haußmann | work = RiseML Blog | date = 2018-04-26 | accessdate = 2018-05-23 | url = https://blog.riseml.com/comparing-google-tpuv2-against-nvidia-v100-on-resnet-50-c2bbb6a51e5e | language = | quote = For the Cloud TPU, Google recommended we use the bfloat16 implementation from the official TPU repository with TensorFlow 1.7.0. Both the TPU and GPU implementations make use of mixed-precision computation on the respective architecture and store most tensors with half-precision. }} 7. ^1 {{Cite web | title = ResNet-50 using BFloat16 on TPU | author = Tensorflow Authors | work = Google | date = 2018-07-23 | accessdate = 2018-11-06 | url = https://github.com/tensorflow/tpu/tree/0ece10f6f4e523eab79aba0247b513fe57d38ae6/models/experimental/resnet_bfloat16 | quote = }} 8. ^{{cite report |title= TensorFlow Distributions |author= Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, Rif A. Saurous |date= 2017-11-28 |id= Accessed 2018-05-23 |arxiv= 1711.10604 |quote= All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts |bibcode= 2017arXiv171110604D }} 9. ^{{Cite web | title = Livestream Day 1: Stage 8 (Google I/O '18) - YouTube | author = | work = Google | date = 2018-05-08 | accessdate = 2018-05-23 | url = https://www.youtube.com/watch?v=vm67WcLzfvc&t=2555 | language = | quote = In many models this is a drop-in replacement for float-32 }} 2 : Binary arithmetic|Floating point types |
|||||||||||||||
| 随便看 |
|
|||||||||||||||
开放百科全书收录14589846条英语、德语、日语等多语种百科知识，基本涵盖了大多数领域的百科知识，是一部内容自由、开放的电子版国际百科全书。