Azure/aztk

📋 Changes

• Pin pipenv dependencies in setup_host.sh (#709) ([5be9e87](https://github.com/Azure/aztk/commit/5be9e87)), closes [#710](https://github.com/Azure/aztk/issues/710)

📋 Changes

• Fix: Storage table never deleted (#690) ([723995c](https://github.com/Azure/aztk/commit/723995c)), closes [#690](https://github.com/Azure/aztk/issues/690)
• Internal: Migrate off preview vsts queues (#685) ([a91690b](https://github.com/Azure/aztk/commit/a91690b)), closes [#685](https://github.com/Azure/aztk/issues/685)

📋 Changes

• Include cluster creation time in cluster list and cluster get output (#678) ([a0bc2f0](https://github.com/Azure/aztk/commit/a0bc2f0)), closes [#678](https://github.com/Azure/aztk/issues/678)
• Verify Batch task has node_info (#681) ([385040d](https://github.com/Azure/aztk/commit/385040d)), closes [#681](https://github.com/Azure/aztk/issues/681)

📋 Changes

• Remove deprecated SDK API code (#671) ([fc50536](https://github.com/Azure/aztk/commit/fc50536)), closes [#671](https://github.com/Azure/aztk/issues/671)
• Remove custom scripts (#673) ([9e32b4b](https://github.com/Azure/aztk/commit/9e32b4b)), closes [#673](https://github.com/Azure/aztk/issues/673)
• Replaced states with Enums for ClusterState, JobState, ApplicationState (#677) ([e486536](https://github.com/Azure/aztk/commit/e486536)), closes [#677](https://github.com/Azure/aztk/issues/677)
• Spark retry docker pull (#672) ([18b74e4](https://github.com/Azure/aztk/commit/18b74e4)), closes [#672](https://github.com/Azure/aztk/issues/672)
• Spark scheduling target (#661) ([4408c4f](https://github.com/Azure/aztk/commit/4408c4f)), closes [#661](https://github.com/Azure/aztk/issues/661)
• Spark submit scheduling internal (#674) ([8c2bf0c](https://github.com/Azure/aztk/commit/8c2bf0c)), closes [#674](https://github.com/Azure/aztk/issues/674)

📋 Changes

• Fix: pin all node dependencies not in Pipfile (#667) ([0606598](https://github.com/Azure/aztk/commit/0606598)), closes [#667](https://github.com/Azure/aztk/issues/667)
• Fix: vsts integration tests block (#657) ([4a60c8a](https://github.com/Azure/aztk/commit/4a60c8a)), closes [#657](https://github.com/Azure/aztk/issues/657)
• Fix: vsts mutliline secrets (#668) ([cb62207](https://github.com/Azure/aztk/commit/cb62207)), closes [#668](https://github.com/Azure/aztk/issues/668)

📋 Changes

• spark roll back scheduling disable (#653) ([93615d9](https://github.com/Azure/aztk/commit/93615d9)), closes [#653](https://github.com/Azure/aztk/issues/653)
• remove custom scripts (#650) ([442228a](https://github.com/Azure/aztk/commit/442228a)), closes [#650](https://github.com/Azure/aztk/issues/650)
• 0.9.0 deprecated code removal (#645) ([eef36dc](https://github.com/Azure/aztk/commit/eef36dc)), closes [#645](https://github.com/Azure/aztk/issues/645)
• SDK refactor (#622) ([b18eb69](https://github.com/Azure/aztk/commit/b18eb69)), closes [#622](https://github.com/Azure/aztk/issues/622)
• Add ability to specify docker run options in toolkit config (#613) ([9d554c3](https://github.com/Azure/aztk/commit/9d554c3)), closes [#613](https://github.com/Azure/aztk/issues/613) [#3](https://github.com/Azure/aztk/issues/3)
• add brief flag to debug tool (#634) ([b7bdd8c](https://github.com/Azure/aztk/commit/b7bdd8c)), closes [#634](https://github.com/Azure/aztk/issues/634)
• first run docs update (#644) ([9098533](https://github.com/Azure/aztk/commit/9098533)), closes [#644](https://github.com/Azure/aztk/issues/644)
• SDK refactor (#622) ([b18eb69](https://github.com/Azure/aztk/commit/b18eb69)), closes [#622](https://github.com/Azure/aztk/issues/622)
• + 8 more

📋 Changes

• docs links version (#614) ([a8f8e92](https://github.com/Azure/aztk/commit/a8f8e92)), closes [#614](https://github.com/Azure/aztk/issues/614)
• set defaults for SparkConfiguration, add tests (#606) ([5306a2a](https://github.com/Azure/aztk/commit/5306a2a)), closes [#606](https://github.com/Azure/aztk/issues/606)
• spark debug tool filter out .venv, make debug tool testable (#612) ([4e0b1ec](https://github.com/Azure/aztk/commit/4e0b1ec)), closes [#612](https://github.com/Azure/aztk/issues/612)
• Suppress msrest warnings (#611) ([883980d](https://github.com/Azure/aztk/commit/883980d)), closes [#611](https://github.com/Azure/aztk/issues/611)

📦 0.8.0 (2018-06-12)

• Deprecated Features
• ClusterConfiguration fields vm_count and vm_count_low_pri have been renamed to size and size_low_priority
• command line flag `--size-low-pri` for `aztk spark cluster create` has been replaced with `--size-low-priority`
• `default` secrets.yaml block has been deprecated, place all child parameters directly at the root
• Spark version 1.6 has been deprecated
• Added Features
• add cluster list quiet flag, ability to compose with delete (#581) ([88d0419](https://github.com/Azure/aztk/commit/88d0419)), closes [#581](https://github.com/Azure/aztk/issues/581)
• add node run command (#572) ([af449dc](https://github.com/Azure/aztk/commit/af449dc)), closes [#572](https://github.com/Azure/aztk/issues/572)
• + 22 more

📋 Changes

• Fix: create virtual environment even if container exists ([2db7b00](https://github.com/jafreck/aztk/commit/2db7b00))
• Fix: gitattributes for jar files (#548) ([a18660b](https://github.com/jafreck/aztk/commit/a18660b)), closes [#548](https://github.com/jafreck/aztk/issues/548)
• Fix: pass docker repo command back to the cluster config (#538) ([a99bbe1](https://github.com/jafreck/aztk/commit/a99bbe1)), closes [#538](https://github.com/jafreck/aztk/issues/538)

📋 Changes

• Moved `docker_repo` under a new `toolkit` key. `docker_repo` is now only used for custom Docker images. Use toolkit for supported images.
• Docker images have been refactored and moved to a different Dockerhub repository. The new supported images are not backwards compatible. See [the documentation on configuration files.](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml)
• Custom scripts have been removed in favor of Plugins, which are more robust. See, [the documenation on Plugins.](https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html)
• add internal flag to node commands (#482) ([1eaa1b6](https://github.com/Azure/aztk/commit/1eaa1b6)), closes [#482](https://github.com/Azure/aztk/issues/482)
• Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) (#517 ([c98df7d](https://github.com/Azure/aztk/commit/c98df7d)), closes [#517](https://github.com/Azure/aztk/issues/517)
• disable msrestazure keyring log (#509) ([3cc43c3](https://github.com/Azure/aztk/commit/3cc43c3)), closes [#509](https://github.com/Azure/aztk/issues/509)
• enable mixed mode for jobs (#442) ([8d00a2c](https://github.com/Azure/aztk/commit/8d00a2c)), closes [#442](https://github.com/Azure/aztk/issues/442)
• getting started script (#475) ([7ef721f](https://github.com/Azure/aztk/commit/7ef721f)), closes [#475](https://github.com/Azure/aztk/issues/475)
• + 45 more

📦 0.6.0 Mixed Mode, Cluster Run & Copy

• __Features:__
• `aztk spark init` customization flags
• `aztk spark cluster run` command added
• `aztk spark cluster copy` command added
• enable Spark dynamic allocation by default
• add SDK support for file-like objects
• add Spark integration tests
• add flag `worker_on_master` option for cluster and job submission mode
• + 9 more

📋 Changes

• [Job Submission](docs/70-jobs.md)
• AAD Support, see [Getting Started](docs/00-getting-started.md)
• VNET Support
• `SecretsConfiguration` inputs changed. Check in [SDK](docs/50-sdk.md) for the new format

📋 Changes

• spark submit command bug
• ssh command for linux/macos

📋 Changes

• container support
• custom script on master / worker / all-nodes
• configuration files
• renamed to aztk / Azure Distributed Data Engineering Toolkit