With the rise of DevOps, low-cost cloud computing, and container technologies, the way Java developers approach development today has changed dramatically. This practical guide helps you take advantage of microservices, serverless, and cloud native technologies using the latest DevOps techniques to simplify your build process and create hyperproductive teams.
Stephen Chin, Melissa McKay, Ixchel Ruiz, and Baruch Sadogursky help you evaluate an array of options. The list includes source control with Git, build declaration with Maven and Gradle, CI/CD with CircleCI, package management with Artifactory, containerization with Docker and Kubernetes, and much more. Whether you’re building applications with Jakarta EE, Spring Boot, Dropwizard, MicroProfile, Micronaut, or Quarkus, this comprehensive guide has you covered.
Explore software lifecycle best practices
Use DevSecOps methodologies to facilitate software development and delivery
Understand the business value of DevSecOps best practices
Manage and secure software dependencies
Develop and deploy applications using containers and cloud native technologies
Manage and administrate source control repositories and development processes
Use automation to set up and administer build pipelines
Identify common deployment patterns and antipatterns
Maintain and monitor software after deployment
About the Author
Stephen Chin is Head of Developer Relations at JFrog and author of The Definitive Guide to Modern Client Development, Raspberry Pi with Java, and Pro JavaFX Platform. He has keynoted numerous Java conferences around the world including Devoxx, JNation, JavaOne, Joker, and Open Source India. Stephen is an avid motorcyclist who has done evangelism tours in Europe, Japan, and Brazil, interviewing hackers in their natural habitat. When he is not traveling, he enjoys teaching kids how to do embedded and robot programming together with his teenage daughter. You can follow his hacking adventures at: http://steveonjava.com/.
Melissa McKay is currently a Developer Advocate with the JFrog Developer Relations team. She has been active in the software industry 20 years and her background and experience spans a slew of technologies and tools used in the development and operation of enterprise products and services. Melissa is a mom, software developer, Java geek, huge promoter of Java UNconferences, and is always on the lookout for ways to grow, learn, and improve development processes. She is active in the developer community, has spoken at CodeOne, Java Dev Day Mexico and assists with organizing the JCrete and JAlba Unconferences as well as Devoxx4Kids events.
Ixchel Ruiz has developed software applications and tools since 2000. Her research interests include Java, dynamic languages, client-side technologies, and testing. She is a Java Champion, Groundbreaker Ambassador, Hackergarten enthusiast, open source advocate, JUG leader, public speaker, and mentor.
Baruch Sadogursky (a.k.a JBaruch) is the Chief Sticker Officer @JFrog (also, Head of DevOps Advocacy) at JFrog. His passion is speaking about technology. Well, speaking in general, but doing it about technology makes him look smart, and 19 years of hi-tech experience sure helps. When he’s not on stage (or on a plane to get there), he learns about technology, people and how they work, or more precisely, don’t work together.
He is a co-author of the Liquid Software book, a CNCF ambassador and a passionate conference speaker on DevOps, DevSecOps, digital transformation, containers and cloud-native, artifact management and other topics, and is a regular at the industry’s most prestigious events including DockerCon, Devoxx, DevOps Days, OSCON, Qcon, JavaOne and many others. You can see some of his talks at jfrog.com/shownotes
See: Continuous Delivery for Java Apps: Build a CD Pipeline Step by Step Using Kubernetes, Docker, Vagrant, Jenkins, Spring, Maven and Artifactory, Publisher : Leanpub (December 14, 2017)
This book will guide you through the implementation of the real-world Continuous Delivery using top-notch technologies. Instead of finishing this book thinking “I know what Continuous Delivery is, but I have no idea how to implement it”, you will end up with your machine set up with a Kubernetes cluster running Jenkins Pipelines in a distributed and scalable fashion (each Pipeline run on a new Jenkins slave dynamically allocated as a Kubernetes pod) to test (unit, integration, acceptance, performance and smoke tests), build (with Maven), release (to Artifactory), distribute (to Docker Hub) and deploy (on Kubernetes) a Spring Boot app to testing, staging and production environments implementing the Canary Release deployment pattern.
TABLE OF CONTENTS:
INTRODUCTION Agile Scrum Scrum and Continuous Integration Deployed vs Released Scrum and Continuous Delivery XP and Continuous Delivery Automated Tests Continuous Integration Feature Branch Continuous Delivery Continuous Delivery Pipeline Continuous Delivery vs Continuous Deployment Canary Release A/B Tests Feature Flags
NOTEPAD APP: AUTOMATED TESTS, MAVEN AND FLYWAY Pre-Requisites The Notepad Application Automated Tests Unit Tests Integration Tests Acceptance Tests Page Object Distributed Acceptance Tests with Selenium-Grid Smoke Tests Performance Tests with Gatling.io Apache Maven Maven Snapshot vs Release The Default Lifecycle and its Phases Maven Repositories Repository Manager (Artifactory) Maven Plugins: Surefire and Failsafe Maven Profile Running Unit Tests Running Integration Tests Running Acceptance Tests Running Smoke Tests Running Performance Tests Publish Artifacts to Artifactory with Maven Publish a Snapshot to Artifactory Publish a Release to Artifactory The release:prepare Goal The release:perform Goal Flyway
DOCKER Introduction to Docker Difference Between Container and Image Docker Hub Create your Account Official Docker Repositories Image Tags Non-Official Docker Images Create a Repository, an Image and Push it to Docker Hub Running Containers on Docker Running Containers as Daemons Container Clean Up Naming Containers Exposing Ports Persistent Data with Volumes Environment Variables Docker Networking Create a Bridge Network Container Static IP Address Linking Containers Most Used Docker Commands Images Containers Misc Building Docker Images: Dockerfile
JENKINS: PIPELINE AS CODE AND CHATOPS Jenkins Overview Jenkins Concepts Job (or Project) Build Artifact Workspace Executor Plugin Node, Master, and Agent (or Slave) ChatOps Create a Slack Workspace Integrate Slack with Jenkins Slack Notification Plugin Use Hubot to Interact with Jenkins Jenkins Pipeline Declarative Pipeline vs Scripted Pipeline Scripted Pipeline Using Docker with Jenkins Pipelines Running Docker from Within the Jenkins Container Scaling Jenkins with Slaves
KUBERNETES Why Kubernetes? Set up a Kubernetes Cluster using Vagrant Hands-on Introduction to Kubernetes Kubernetes Concepts Namespaces Pods Labels Replica Sets Services Service Discovery using DNS Service Discovery using Namespaces Volumes Handling External Configurations Config Maps Changing Logback Log Level at Runtime Secrets Using Secrets as Environment Variables Using Secrets as Files from a Pod Deployments Readiness Probes Liveness Probes Canary Release Kubernetes Architecture Kubernetes Master Components Etcd API Server Controller Manager Scheduler Kubernetes Node Components Service Proxy Kubelet cAdvisor Kubernetes Add-ons Web UI (Dashboard) Monitoring Kubernetes with Heapster, InfluxDB and Grafana Web UI Overview DNS
“A DevOps toolchain is a set or combination of tools that aid in the delivery, development, and management of software applications throughout the systems development life cycle, as coordinated by an organization that uses DevOps practices.
“In software, a toolchain is the set of programming tools that is used to perform a complex software development task or to create a software product, which is typically another computer program or a set of related programs. In general, the tools forming a toolchain are executed consecutively so the output or resulting environment state of each tool becomes the input or starting environment for the next one, but the term is also used when referring to a set of related tools that are not necessarily executed consecutively.[3][4][5]
As DevOps is a set of practices that emphasizes the collaboration and communication of both software developers and other information technology (IT) professionals, while automating the process of software delivery and infrastructure changes, its implementation can include the definition of the series of tools used at various stages of the lifecycle; because DevOps is a cultural shift and collaboration between development and operations, there is no one product that can be considered a single DevOps tool. Instead a collection of tools, potentially from a variety of vendors, are used in one or more stages of the lifecycle.[6][7]” (WP)
Plan is composed of two things: “define” and “plan”.[8] This activity refers to the business value and application requirements. Specifically “Plan” activities include:
Tools and vendors in this category often overlap with other categories. Because DevOps is about breaking down silos, this is reflective in the activities and product solutions.[clarification needed]
Verify
Verify is directly associated with ensuring the quality of the software release; activities designed to ensure code quality is maintained and the highest quality is deployed to production.[8] The main activities in this are:
Packaging refers to the activities involved once the release is ready for deployment, often also referred to as staging or Preproduction / “preprod”.[8] This often includes tasks and activities such as:
Approval/preapprovals
Package configuration
Triggered releases
Release staging and holding
Release
Release related activities include schedule, orchestration, provisioning and deploying software into production and targeted environment.[9] The specific Release activities include:
Configure activities fall under the operation side of DevOps. Once software is deployed, there may be additional IT infrastructure provisioning and configuration activities required.[8] Specific activities including:
Infrastructure storage, database and network provisioning and configuring
Monitoring is an important link in a DevOps toolchain. It allows IT organization to identify specific issues of specific releases and to understand the impact on end-users.[8] A summary of Monitor related activities are:
Information from monitoring activities often impacts Plan activities required for changes and for new release cycles.
Version Control
Version Control is an important link in a DevOps toolchain and a component of software configuration management. Version Control is the management of changes to documents, computer programs, large web sites, and other collections of information.[8] A summary of Version Control related activities are:
Non-linear development
Distributed development
Compatibility with existent systems and protocols
Toolkit-based design
Information from Version Control often supports Release activities required for changes and for new release cycles.
^ Garner Market Trends: DevOps – Not a Market, but Tool-Centric Philosophy That supports a Continuous Delivery Value Chain (Report). Gartner. 18 February 2015.
^ abcdefg Avoid Failure by Developing a Toolchain that Enables DevOps (Report). Gartner. 16 March 2016.
^ Best Practices in Change, Configuration and Release Management (Report). Gartner. 14 July 2010.
^ Roger S. Pressman (2009). Software Engineering: A Practitioner’s Approach (7th International ed.). New York: McGraw-Hill.
A package manager or package-management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer‘s operating system in a consistent manner.[1]
A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software’s name, description of its purpose, version number, vendor, checksum (preferably a cryptographic hash function), and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.
Package managers are designed to eliminate the need for manual installs and updates. This can be particularly useful for large enterprises whose operating systems are typically consisting of hundreds or even tens of thousands of distinct software packages.[2]
Functions
Illustration of a package manager being used to download new software. Manual actions can include accepting a license agreement or selecting some package-specific configuration options.
A software package is an archive file containing a computer program as well as necessary metadata for its deployment. The computer program can be in source code that has to be compiled and built first.[3] Package metadata include package description, package version, and dependencies (other packages that need to be installed beforehand).
Package managers are charged with the task of finding, installing, maintaining or uninstalling software packages upon the user’s command. Typical functions of a package management system include:
Grouping packages by function to reduce user confusion
Managing dependencies to ensure a package is installed with all packages it requires, thus avoiding “dependency hell“
Challenges with shared libraries
Computer systems that rely on dynamic library linking, instead of static library linking, share executable libraries of machine instructions across packages and applications. In these systems, complex relationships between different packages requiring different versions of libraries results in a challenge colloquially known as “dependency hell“. On Microsoft Windows systems, this is also called “DLL hell” when working with dynamically linked libraries. Good package management is vital on these systems.[4] The Framework system from OPENSTEP was an attempt at solving this issue, by allowing multiple versions of libraries to be installed simultaneously, and for software packages to specify which version they were linked against.
Front-ends for locally compiled packages
System administrators may install and maintain software using tools other than package management software. For example, a local administrator may download unpackaged source code, compile it, and install it. This may cause the state of the local system to fall out of synchronization with the state of the package manager’s database. The local administrator will be required to take additional measures, such as manually managing some dependencies or integrating the changes into the package manager.
There are tools available to ensure that locally compiled packages are integrated with the package management. For distributions based on .deb and .rpm files as well as Slackware Linux, there is CheckInstall, and for recipe-based systems such as Gentoo Linux and hybrid systems such as Arch Linux, it is possible to write a recipe first, which then ensures that the package fits into the local package database.[citation needed]
Maintenance of configuration
Particularly troublesome with software upgrades are upgrades of configuration files. Since package managers, at least on Unix systems, originated as extensions of file archiving utilities, they can usually only either overwrite or retain configuration files, rather than applying rules to them. There are exceptions to this that usually apply to kernel configuration (which, if broken, will render the computer unusable after a restart). Problems can be caused if the format of configuration files changes; for instance, if the old configuration file does not explicitly disable new options that should be disabled. Some package managers, such as Debian‘s dpkg, allow configuration during installation. In other situations, it is desirable to install packages with the default configuration and then overwrite this configuration, for instance, in headless installations to a large number of computers. This kind of pre-configured installation is also supported by dpkg.
Repositories
To give users more control over the kinds of software that they are allowing to be installed on their system (and sometimes due to legal or convenience reasons on the distributors’ side), software is often downloaded from a number of software repositories.[5]
Upgrade suppression
When a user interacts with the package management software to bring about an upgrade, it is customary to present the user with the list of actions to be executed (usually the list of packages to be upgraded, and possibly giving the old and new version numbers), and allow the user to either accept the upgrade in bulk, or select individual packages for upgrades. Many package managers can be configured to never upgrade certain packages, or to upgrade them only when critical vulnerabilities or instabilities are found in the previous version, as defined by the packager of the software. This process is sometimes called version pinning.
For instance:
yum supports this with the syntax exclude=openoffice*[6]
pacman with IgnorePkg= openoffice[7] (to suppress upgrading openoffice in both cases)
dpkg and dselect support this partially through the hold flag in package selections
APT extends the hold flag through the complex “pinning” mechanism[8] (Users can also blacklist a package[9])
portage supports this through the package.mask configuration file
Cascading package removal
Some of the more advanced package management features offer “cascading package removal”,[7] in which all packages that depend on the target package and all packages that only the target package depends on, are also removed.
Comparison of commands
Although the commands are specific for every particular package manager, they are to a large extent translatable, as most package managers offer similar functions.
emerge -avtuDN --with-bdeps=y @world or emerge --update --pretend @world
delete orphans+config
zypper rm -u
pacman -Rsn $(pacman -Qdtq)
apt autoremove
dnf erase PKG
emerge --depclean
show orphans
zypper pa --orphaned --unneeded
pacman -Qdt
package-cleanup --quiet --leaves --exclude-bin
emerge -caD or emerge --depclean --pretend
update all
zypper up
pacman -Syu
apt upgrade
dnf update
emerge --update --deep --with-bdeps=y @world
The Arch Linux Pacman/Rosetta wiki offers an extensive overview.[11]
Prevalence
Package managers like dpkg have existed as early as 1994.[12]
Linux distributions oriented to binary packages rely heavily on package management systems as their primary means of managing and maintaining software. Mobile operating systems such as Android (Linux-based), iOS (Unix-like), and Windows Phone rely almost exclusively on their respective vendors’ app stores and thus use their own dedicated package management systems.
A package manager is often called an “install manager”, which can lead to a confusion between package managers and installers. The differences include:This box:
It is entirely at the discretion of the installer. It could be a file within the app’s folder, or among the operating system’s files and folders. At best, they may register themselves with an uninstallers list without exposing installation information.
There could be as many formats as the number of apps
Package format compatibility
Can be consumed as long as the package manager supports it. Either newer versions of the package manager keep supporting it or the user does not upgrade the package manager.
The installer is always compatible with its archive format, if it uses any. However, installers, like all computer programs, may be affected by software rot.
Comparison with build automation utility
Most software configuration management systems treat building software and deploying software as separate, independent steps. A build automation utility typically takes human-readable source code files already on a computer, and automates the process of converting them into a binary executable package on the same computer. Later a package manager typically running on some other computer downloads those pre-built binary executable packages over the internet and installs them.
However, both kinds of tools have many commonalities:
For example, the dependency graphtopological sorting used in a package manager to handle dependencies between binary components is also used in a build manager to handle the dependency between source components.
For example, many makefiles support not only building executables, but also installing them with make install.
For example, every package manager for a source-based distribution – Portage, Sorcery, Homebrew, etc. – supports converting human-readable source code to binary executables and installing it.
A few tools, such as Maak and A-A-P, are designed to handle both building and deployment, and can be used as either a build automation utility or as a package manager or both.[13]
Common package managers and formats
Universal package manager
Also known as binary repository manager, it is a software tool designed to optimize the download and storage of binary files, artifacts and packages used and produced in the software development process.[14] These package managers aim to standardize the way enterprises treat all package types. They give users the ability to apply security and compliance metrics across all artifact types. Universal package managers have been referred to as being at the center of a DevOps toolchain.[15]
Each package manager relies on the format and metadata of the packages it can manage. That is, package managers need groups of files to be bundled for the specific package manager along with appropriate metadata, such as dependencies. Often, a core set of utilities manages the basic installation from these packages and multiple package managers use these utilities to provide additional functionality.
For example, yum relies on rpm as a backend. Yum extends the functionality of the backend by adding features such as simple configuration for maintaining a network of systems. As another example, the Synaptic Package Manager provides a graphical user interface by using the Advanced Packaging Tool (apt) library, which, in turn, relies on dpkg for core functionality.
By the nature of free and open source software, packages under similar and compatible licenses are available for use on a number of operating systems. These packages can be combined and distributed using configurable and internally complex packaging systems to handle many permutations of software and manage version-specific dependencies and conflicts. Some packaging systems of free and open source software are also themselves released as free and open source software. One typical difference between package management in proprietary operating systems, such as Mac OS X and Windows, and those in free and open source software, such as Linux, is that free and open source software systems permit third-party packages to also be installed and upgraded through the same mechanism, whereas the package managers of Mac OS X and Windows will only upgrade software provided by Apple and Microsoft, respectively (with the exception of some third party drivers in Windows). The ability to continuously upgrade third party software is typically added by adding the URL of the corresponding repository to the package management’s configuration file.
Beside the system-level application managers, there are some add-on package managers for operating systems with limited capabilities and for programming languages in which developers need the latest libraries.
In contrast to system-level package managers, application-level package managers focus on a small part of the software system. They typically reside within a directory tree that is not maintained by the system-level package manager, such as c:\cygwin or /usr/local/fink. However, this might not be the case for the package managers that deal with programming libraries, leading to a possible conflict as both package managers may claim to “own” a file and might break upgrades.
Impact
Ian Murdock had commented that package management is “the single biggest advancement Linux has brought to the industry”, that it blurs the boundaries between operating system and applications, and that it makes it “easier to push new innovations […] into the marketplace and […] evolve the OS”.[16]
