Hehe, funny, Google started to use their own bfloat datatype in TPU gen 2 and gen 3 for neural networks,
https://en.wikipedia.org/wiki/Bfloat16_ ... int_format
https://www.nextplatform.com/2018/05/10 ... processor/
and now Intel starts to implement it in their hardware. That's when you know you are a big player
https://venturebeat.com/2018/05/23/inte ... -training/
Wonder if Nvidia or AMD will join.
--
Srdja
Google's bfloat for neural networks
Moderators: hgm, Rebel, chrisw
-
- Posts: 2644
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
-
- Posts: 2554
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: Google's bfloat for neural networks
I misread as "Google's bloat...", thought that Google open sourced yet another masterpiece
So this bfloat16 is basically float where you throw away 16 bits worth of mantissa.
Packing/unpacking from 32-bit float should be trivial, so probably clever, but hey only 7 bits of mantissa, is it really enough?
So this bfloat16 is basically float where you throw away 16 bits worth of mantissa.
Packing/unpacking from 32-bit float should be trivial, so probably clever, but hey only 7 bits of mantissa, is it really enough?
Martin Sedlak
-
- Posts: 2644
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
Re: Google's bfloat for neural networks
Dunnomar wrote: ↑Tue Apr 16, 2019 11:57 am I misread as "Google's bloat...", thought that Google open sourced yet another masterpiece
So this bfloat16 is basically float where you throw away 16 bits worth of mantissa.
Packing/unpacking from 32-bit float should be trivial, so probably clever, but hey only 7 bits of mantissa, is it really enough?
https://www.hpcwire.com/2019/04/15/bsc- ... -training/
"As training progresses and it hones the value of the weights, then greater precision becomes important in order to optimize the solution."
“We believe dynamic numerical precision approaches offer the best benefit to training and inferencing,”
--
Srdja