google/trax

📋 Changes

• Load checkpoint for fine-tuning in https://github.com/google/trax/commit/0349c3f2c32ed83d0ba54a0f6652766ea5467cca by @henrykmichalewski !
• Added generic input pipeline for GLUE tasks in https://github.com/google/trax/commit/2f490e83a9be1b802dccf866c52c8908645e10d6 by @henrykmichalewski - thanks a lot!
• Some nascent support for bfloat16s!
• Performer's Favor and CausalFavor - https://github.com/google/trax/commit/77db199392ff967e49156ccb24c916b270b47eca thanks @lukaszkaiser !
• Funnel-Transformer in https://github.com/google/trax/pull/1156 thanks a lot @mvxxx !
• BERT for Trax in https://github.com/google/trax/pull/1254 , https://github.com/google/trax/pull/1223, etc by @piotrekp1 - thanks a lot!
• Residual Exchange Network by @kkanska in https://github.com/google/trax/commit/3a8f402b722a0d9ae934ed399e6532dba8401f9e ! Thanks!
• Fixing broken example links in https://github.com/google/trax/pull/1263 thanks @amtagrwl !
• + 5 more

📋 Changes

• Rename `gumbel_sample` to `logsoftmax_sample` in a2497cbd8477a11f2b96e87a4d53ce46b845ffa8
• A fix to storing checkpoints in Loop in 88b033c804a4da9021505f2ce8fbb5c7d6500574

📋 Changes

• Data pipeline combinators in 17d44710bee74e8bb6e3b34baa4114b0c78160af
• Multi-host training in the Loop api 524321e0f77afa58a9fba470e5bccb19ae6bdb92
• Auto-regressive sampling in fe0fa7886ee33bfac831a969a82b351ccb02941c
• T2T Tokenizers 6306231ace01ab55c5df0c401b1f3cf4c7b6a32f
• Early work on multi-task training in 423d664b470612ebbc0e5041e8288eb1a47c30ef thanks to @koz4k
• IMDB dataset in 08bdb50db41a089b6e12c61a8a29981225bf3566
• Machine Translation en-pl in 5fb8aa8c5cb86dabb2338938c745996d5d87d996

📋 Changes

• `tl.Embedding` now has the same signature as pytorch/tf
• `train.lr_schedule` (function object) -> `train.lr_schedule_fn` (function)
• Report loss back to `training.Loop`

📋 Changes

• flat saved model/checkpoint representation
• lr schedule simplified, now they just take step number.
• configs are now in supervised/configs and rl/configs.
• RL obsolete code cleanup.

📋 Changes

• #459 by @w4-sjcho - adding names to layers, aiding debuggability thanks a lot!
• #256 and #300 by @stephenjfox and @satyarohith refining the README.md language, thanks a lot folks!
• #313 #312 #436 #396 from @pkol with lots of bugfixes, thanks a lot @pkol !
• #409 by @pkol -- a special shoutout to this PR, this fixes a long standing issue that prevented termination of the process by tracking the zombie threads -- thanks a lot for this @pkol specially !
• #386 another shoutout to @pkol for an amazing speedup in the RL code -- thanks a lot again !
• #344 a psum bugfix with tf backend from @fsx950223 - thanks a lot !
• #335 a bugfix from @friesel - thanks a lot Phillip !
• #315 better exception handling by @cool-RR - thanks a lot !
• + 4 more

📦 Reformer

• Reversible Transformer model for machine translation and other encoder-decoder tasks
• Add code for beam search, sampling, and greedy decoding (see `trax.models.beam_search.Search`)
• Memory-efficient attention classes have been re-written to use even less memory and to support faster decoding (see the new `SelfAttention`, `LSHSelfAttention` and `EncDecAttention` classes)

📦 RL

• Implemented the [Advantage-Weighted Regression](https://arxiv.org/abs/1910.00177) algorithm, a simple off-policy reinforcement learning algorithm.
• Extracted out a `PolicyBasedTrainer`, so `ppo_trainer.PPO` and `awr_trainer.AwrTrainer` now both inherit from it.
• Refactoring of the serialization code in the RL part, thanks to @koz4k !

📦 Framework

• A lot of code cleanup and refactoring of the core abstractions by Jonni, thanks Jonni!

📦 TF Numpy

• More ops added by @wangpengmit !

📋 Changes

• More bugfixes related to scan/remat.

📋 Changes

• Minor bugfix to GRU and scan.

✨ New Models

• Reformer Implementation - https://arxiv.org/abs/2001.04451 Thanks Nikita Kitaev, @lukaszkaiser and @levskaya !

📦 Colabs

• Reformer Colabs https://github.com/google/trax/commit/9dbf8636a34cffbd421bedb2a1e3d7fe006346c0
• Transformer Colab https://github.com/google/trax/commit/d2c5b84b8db53888013c56980ccbafba077ee8f2

📋 Framework Changes

• Ongoing cleanups and API simplifications.
• Optimization by @jekbradbury - thanks James!

📦 PRs

• Consistent logging `absl.logging` and setup.py fixes thanks to @lkhphuc in #198
• Code cleanups by @cclauss in #196
• Code cleanup by @pkol in #134 - thanks!
• Bug fix by @pzielinski-nyc in #151 - thanks!