Buddha Dharma and DevOps compiled by Cloud Monk Losang Jinpa, Ph.D, MCSE/MCT, Retired in Pacific Mountain Northwest
Tag:DevOps
DevOps – Deliver innovation faster with simple, reliable tools for Continuous Integration / Continuous Delivery (CI/CD) for Agile teams to share code, track work, and ship software.
Securing DevOps explores how the techniques of DevOps and security should be applied together to make cloud services safer. This introductory book reviews the latest practices used in securing web applications and their infrastructure and teaches you techniques to integrate security directly into your product. You’ll also learn the core concepts of DevOps, such as continuous integration, continuous delivery, and infrastructure as a service.
Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.
About the Technology
An application running in the cloud can benefit from incredible efficiencies, but they come with unique security threats too. A DevOps team’s highest priority is understanding those risks and hardening the system against them.
About the Book
Securing DevOps teaches you the essential techniques to secure your cloud services. Using compelling case studies, it shows you how to build security into automated testing, continuous delivery, and other core DevOps processes. This experience-rich book is filled with mission-critical strategies to protect web applications against attacks, deter fraud attempts, and make your services safer when operating at scale. You’ll also learn to identify, assess, and secure the unique vulnerabilities posed by cloud deployments and automation tools commonly used in modern infrastructures.
What’s inside
An approach to continuous security
Implementing test-driven security in DevOps
Security techniques for cloud services
Watching for fraud and responding to incidents
Security testing and risk assessment
About the Reader
Readers should be comfortable with Linux and standard DevOps practices like CI, CD, and unit testing.
About the Author
Julien Vehent is a security architect and DevOps advocate. He leads the Firefox Operations Security team at Mozilla, and is responsible for the security of Firefox’s high-traffic cloud services and public websites.
With the rise of DevOps, low-cost cloud computing, and container technologies, the way Java developers approach development today has changed dramatically. This practical guide helps you take advantage of microservices, serverless, and cloud native technologies using the latest DevOps techniques to simplify your build process and create hyperproductive teams.
Stephen Chin, Melissa McKay, Ixchel Ruiz, and Baruch Sadogursky help you evaluate an array of options. The list includes source control with Git, build declaration with Maven and Gradle, CI/CD with CircleCI, package management with Artifactory, containerization with Docker and Kubernetes, and much more. Whether you’re building applications with Jakarta EE, Spring Boot, Dropwizard, MicroProfile, Micronaut, or Quarkus, this comprehensive guide has you covered.
Explore software lifecycle best practices
Use DevSecOps methodologies to facilitate software development and delivery
Understand the business value of DevSecOps best practices
Manage and secure software dependencies
Develop and deploy applications using containers and cloud native technologies
Manage and administrate source control repositories and development processes
Use automation to set up and administer build pipelines
Identify common deployment patterns and antipatterns
Maintain and monitor software after deployment
About the Author
Stephen Chin is Head of Developer Relations at JFrog and author of The Definitive Guide to Modern Client Development, Raspberry Pi with Java, and Pro JavaFX Platform. He has keynoted numerous Java conferences around the world including Devoxx, JNation, JavaOne, Joker, and Open Source India. Stephen is an avid motorcyclist who has done evangelism tours in Europe, Japan, and Brazil, interviewing hackers in their natural habitat. When he is not traveling, he enjoys teaching kids how to do embedded and robot programming together with his teenage daughter. You can follow his hacking adventures at: http://steveonjava.com/.
Melissa McKay is currently a Developer Advocate with the JFrog Developer Relations team. She has been active in the software industry 20 years and her background and experience spans a slew of technologies and tools used in the development and operation of enterprise products and services. Melissa is a mom, software developer, Java geek, huge promoter of Java UNconferences, and is always on the lookout for ways to grow, learn, and improve development processes. She is active in the developer community, has spoken at CodeOne, Java Dev Day Mexico and assists with organizing the JCrete and JAlba Unconferences as well as Devoxx4Kids events.
Ixchel Ruiz has developed software applications and tools since 2000. Her research interests include Java, dynamic languages, client-side technologies, and testing. She is a Java Champion, Groundbreaker Ambassador, Hackergarten enthusiast, open source advocate, JUG leader, public speaker, and mentor.
Baruch Sadogursky (a.k.a JBaruch) is the Chief Sticker Officer @JFrog (also, Head of DevOps Advocacy) at JFrog. His passion is speaking about technology. Well, speaking in general, but doing it about technology makes him look smart, and 19 years of hi-tech experience sure helps. When he’s not on stage (or on a plane to get there), he learns about technology, people and how they work, or more precisely, don’t work together.
He is a co-author of the Liquid Software book, a CNCF ambassador and a passionate conference speaker on DevOps, DevSecOps, digital transformation, containers and cloud-native, artifact management and other topics, and is a regular at the industry’s most prestigious events including DockerCon, Devoxx, DevOps Days, OSCON, Qcon, JavaOne and many others. You can see some of his talks at jfrog.com/shownotes
Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers on virtual machines or bare metal hosts[10]
“A DevOps toolchain is a set or combination of tools that aid in the delivery, development, and management of software applications throughout the systems development life cycle, as coordinated by an organization that uses DevOps practices.
“In software, a toolchain is the set of programming tools that is used to perform a complex software development task or to create a software product, which is typically another computer program or a set of related programs. In general, the tools forming a toolchain are executed consecutively so the output or resulting environment state of each tool becomes the input or starting environment for the next one, but the term is also used when referring to a set of related tools that are not necessarily executed consecutively.[3][4][5]
As DevOps is a set of practices that emphasizes the collaboration and communication of both software developers and other information technology (IT) professionals, while automating the process of software delivery and infrastructure changes, its implementation can include the definition of the series of tools used at various stages of the lifecycle; because DevOps is a cultural shift and collaboration between development and operations, there is no one product that can be considered a single DevOps tool. Instead a collection of tools, potentially from a variety of vendors, are used in one or more stages of the lifecycle.[6][7]” (WP)
Plan is composed of two things: “define” and “plan”.[8] This activity refers to the business value and application requirements. Specifically “Plan” activities include:
Tools and vendors in this category often overlap with other categories. Because DevOps is about breaking down silos, this is reflective in the activities and product solutions.[clarification needed]
Verify
Verify is directly associated with ensuring the quality of the software release; activities designed to ensure code quality is maintained and the highest quality is deployed to production.[8] The main activities in this are:
Packaging refers to the activities involved once the release is ready for deployment, often also referred to as staging or Preproduction / “preprod”.[8] This often includes tasks and activities such as:
Approval/preapprovals
Package configuration
Triggered releases
Release staging and holding
Release
Release related activities include schedule, orchestration, provisioning and deploying software into production and targeted environment.[9] The specific Release activities include:
Configure activities fall under the operation side of DevOps. Once software is deployed, there may be additional IT infrastructure provisioning and configuration activities required.[8] Specific activities including:
Infrastructure storage, database and network provisioning and configuring
Monitoring is an important link in a DevOps toolchain. It allows IT organization to identify specific issues of specific releases and to understand the impact on end-users.[8] A summary of Monitor related activities are:
Information from monitoring activities often impacts Plan activities required for changes and for new release cycles.
Version Control
Version Control is an important link in a DevOps toolchain and a component of software configuration management. Version Control is the management of changes to documents, computer programs, large web sites, and other collections of information.[8] A summary of Version Control related activities are:
Non-linear development
Distributed development
Compatibility with existent systems and protocols
Toolkit-based design
Information from Version Control often supports Release activities required for changes and for new release cycles.
^ Garner Market Trends: DevOps – Not a Market, but Tool-Centric Philosophy That supports a Continuous Delivery Value Chain (Report). Gartner. 18 February 2015.
^ abcdefg Avoid Failure by Developing a Toolchain that Enables DevOps (Report). Gartner. 16 March 2016.
^ Best Practices in Change, Configuration and Release Management (Report). Gartner. 14 July 2010.
^ Roger S. Pressman (2009). Software Engineering: A Practitioner’s Approach (7th International ed.). New York: McGraw-Hill.
Maven addresses two aspects of building software: how software is built, and its dependencies. Unlike earlier tools like Apache Ant, it uses conventions for the build procedure, and only exceptions need to be written down. An XML file describes the software project being built, its dependencies on other external modules and components, the build order, directories, and required plug-ins. It comes with pre-defined targets for performing certain well-defined tasks such as compilation of code and its packaging. Maven dynamically downloads Java libraries and Maven plug-ins from one or more repositories such as the Maven 2 Central Repository, and stores them in a local cache.[2] This local cache of downloaded artifacts can also be updated with artifacts created by local projects. Public repositories can also be updated.
Maven is built using a plugin-based architecture that allows it to make use of any application controllable through standard input. A C/C++ native plugin is maintained for Maven 2.[3]
Alternative technologies like Gradle and sbt as build tools do not rely on XML, but keep the key concepts Maven introduced. With Apache Ivy, a dedicated dependency manager was developed as well that also supports Maven repositories.[4]
The number of artifacts on Maven’s central repository has grown rapidly
Maven, created by Jason van Zyl, began as a sub-project of Apache Turbine in 2002. In 2003, it was voted on and accepted as a top level Apache Software Foundation project. In July 2004, Maven’s release was the critical first milestone, v1.0. Maven 2 was declared v2.0 in October 2005 after about six months in beta cycles. Maven 3.0 was released in October 2010 being mostly backwards compatible with Maven 2.
Maven 3.0 information began trickling out in 2008. After eight alpha releases, the first beta version of Maven 3.0 was released in April 2010. Maven 3.0 has reworked the core Project Builder infrastructure resulting in the POM’s file-based representation being decoupled from its in-memory object representation. This has expanded the possibility for Maven 3.0 add-ons to leverage non-XML based project definition files. Languages suggested include Ruby (already in private prototype by Jason van Zyl), YAML, and Groovy.
Special attention was given to ensuring backward compatibility of Maven 3 to Maven 2. For most projects, upgrading to Maven 3 will not require any adjustments of their project structure. The first beta of Maven 3 saw the introduction of a parallel build feature which leverages a configurable number of cores on a multi-core machine and is especially suited for large multi-module projects.
A directory structure for a Java project auto-generated by Maven
Maven projects are configured using a Project Object Model, which is stored in a pom.xml-file. An example file looks like:
<project><!-- model version is always 4.0.0 for Maven 2.x POMs --><modelVersion>4.0.0</modelVersion><!-- project coordinates, i.e. a group of values which uniquely identify this project --><groupId>com.mycompany.app</groupId><artifactId>my-app</artifactId><version>1.0</version><!-- library dependencies --><dependencies><dependency><!-- coordinates of the required library --><groupId>junit</groupId><artifactId>junit</artifactId><version>3.8.1</version><!-- this dependency is only used for running and compiling tests --><scope>test</scope></dependency></dependencies></project>
This POM only defines a unique identifier for the project (coordinates) and its dependency on the JUnit framework. However, that is already enough for building the project and running the unit tests associated with the project. Maven accomplishes this by embracing the idea of Convention over Configuration, that is, Maven provides default values for the project’s configuration.
The directory structure of a normal idiomatic Maven project has the following directory entries:
Directory name
Purpose
project home
Contains the pom.xml and all subdirectories.
src/main/java
Contains the deliverable Java sourcecode for the project.
src/main/resources
Contains the deliverable resources for the project, such as property files.
src/test/java
Contains the testing Java sourcecode (JUnit or TestNG test cases, for example) for the project.
src/test/resources
Contains resources necessary for testing.
The command mvn package will compile all the Java files, run any tests, and package the deliverable code and resources into target/my-app-1.0.jar (assuming the artifactId is my-app and the version is 1.0.)
Using Maven, the user provides only configuration for the project, while the configurable plug-ins do the actual work of compiling the project, cleaning target directories, running unit tests, generating API documentation and so on. In general, users should not have to write plugins themselves. Contrast this with Ant and make, in which one writes imperative procedures for doing the aforementioned tasks.
A Project Object Model (POM) provides all the configuration for a single project. General configuration covers the project’s name, its owner and its dependencies on other projects. One can also configure individual phases of the build process, which are implemented as plugins. For example, one can configure the compiler-plugin to use Java version 1.5 for compilation, or specify packaging the project even if some unit tests fail.
Larger projects should be divided into several modules, or sub-projects, each with its own POM. One can then write a root POM through which one can compile all the modules with a single command. POMs can also inherit configuration from other POMs. All POMs inherit from the Super POM[7] by default. The Super POM provides default configuration, such as default source directories, default plugins, and so on.
Most of Maven’s functionality is in plug-ins. A plugin provides a set of goals that can be executed using the command mvn [plugin-name]:[goal-name]. For example, a Java project can be compiled with the compiler-plugin’s compile-goal[8] by running mvn compiler:compile.
There are Maven plugins for building, testing, source control management, running a web server, generating Eclipse project files, and much more.[9] Plugins are introduced and configured in a <plugins>-section of a pom.xml file. Some basic plugins are included in every project by default, and they have sensible default settings.
However, it would be cumbersome if the archetypal build sequence of building, testing and packaging a software project required running each respective goal manually:
mvn compiler:compile
mvn surefire:test
mvn jar:jar
Maven’s lifecycle concept handles this issue.
Plugins are the primary way to extend Maven. Developing a Maven plugin can be done by extending the org.apache.maven.plugin.AbstractMojo class. Example code and explanation for a Maven plugin to create a cloud-based virtual machine running an application server is given in the article Automate development and management of cloud virtual machines.[10]
The build lifecycle is a list of named phases that can be used to give order to goal execution. One of Maven’s standard lifecycles is the default lifecycle, which includes the following phases, in this order:[11]
validate
generate-sources
process-sources
generate-resources
process-resources
compile
process-test-sources
process-test-resources
test-compile
test
package
install
deploy
Goals provided by plugins can be associated with different phases of the lifecycle. For example, by default, the goal “compiler:compile” is associated with the “compile” phase, while the goal “surefire:test” is associated with the “test” phase. When the mvn test command is executed, Maven runs all goals associated with each of the phases up to and including the “test” phase. In such a case, Maven runs the “resources:resources” goal associated with the “process-resources” phase, then “compiler:compile”, and so on until it finally runs the “surefire:test” goal.
Maven also has standard phases for cleaning the project and for generating a project site. If cleaning were part of the default lifecycle, the project would be cleaned every time it was built. This is clearly undesirable, so cleaning has been given its own lifecycle.
Standard lifecycles enable users new to a project the ability to accurately build, test and install every Maven project by issuing the single command mvn install. By default, Maven packages the POM file in generated JAR and WAR files. Tools like diet4j[12] can use this information to recursively resolve and run Maven modules at run-time without requiring an “uber”-jar that contains all project code.
A central feature in Maven is dependency management. Maven’s dependency-handling mechanism is organized around a coordinate system identifying individual artifacts such as software libraries or modules. The POM example above references the JUnit coordinates as a direct dependency of the project. A project that needs, say, the Hibernate library simply has to declare Hibernate’s project coordinates in its POM. Maven will automatically download the dependency and the dependencies that Hibernate itself needs (called transitive dependencies) and store them in the user’s local repository. Maven 2 Central Repository[2] is used by default to search for libraries, but one can configure the repositories to be used (e.g., company-private repositories) within the POM.
The fundamental difference between Maven and Ant is that Maven’s design regards all projects as having a certain structure and a set of supported task work-flows (e.g., getting resources from source control, compiling the project, unit testing, etc.). While most software projects in effect support these operations and actually do have a well-defined structure, Maven requires that this structure and the operation implementation details be defined in the POM file. Thus, Maven relies on a convention on how to define projects and on the list of work-flows that are generally supported in all projects.[13]
There are search engines such as The Central Repository Search Engine[14] which can be used to find out coordinates for different open-source libraries and frameworks.
Projects developed on a single machine can depend on each other through the local repository. The local repository is a simple folder structure that acts both as a cache for downloaded dependencies and as a centralized storage place for locally built artifacts. The Maven command mvn install builds a project and places its binaries in the local repository. Then other projects can utilize this project by specifying its coordinates in their POMs.
Add-ons to several popular integrated development environments targeting the Java programming language exist to provide integration of Maven with the IDE’s build mechanism and source editing tools, allowing Maven to compile projects from within the IDE, and also to set the classpath for code completion, highlighting compiler errors, etc. Examples of popular IDEs supporting development with Maven include:
These add-ons also provide the ability to edit the POM or use the POM to determine a project’s complete set of dependencies directly within the IDE.
Some built-in features of IDEs are forfeited when the IDE no longer performs compilation. For example, Eclipse’s JDT has the ability to recompile a single Java source file after it has been edited. Many IDEs work with a flat set of projects instead of the hierarchy of folders preferred by Maven. This complicates the use of SCM systems in IDEs when using Maven.[15][16][17]
Kubuntu (/kʊˈbʊntuː/kuu-BUUN-too)[4] is an official flavour of the UbuntuLinux distributionoperating system that uses the KDE Plasma Desktop instead of the GNOME desktop environment. As part of the Ubuntu project, Kubuntu uses the same underlying systems. Every package in Kubuntu shares the same repositories as Ubuntu,[5] and it is released regularly on the same schedule as Ubuntu.[6]
Kubuntu was sponsored by Canonical Ltd. until 2012 and then directly by Blue Systems. Now, employees of Blue Systems contribute upstream, to KDE and Debian, and Kubuntu development is led by community contributors. During the changeover, Kubuntu retained the use of Ubuntu project servers and existing developers.[7]
Kubuntu was born on 10 December 2004 at the Ubuntu Mataro Conference in Mataró, Spain.
The Linux Mint project was created by Clément Lefèbvre and is actively maintained by the Linux Mint Team and community.[8]
Linux Mint began in 2006 with a beta release, 1.0, code-named ‘Ada’,[9] based on Kubuntu. Linux Mint 2.0 ‘Barbara’ was the first version to use Ubuntu as its codebase. It had few users until the release of Linux Mint 3.0, ‘Cassandra’.[10][11]
Ubuntu is released every six months, with long-term support (LTS) releases every two years.[7][20][21] As of 22 October 2020, the most recent long-term support release is 20.04 (“Focal Fossa”), which is supported until 2025 under public support and until 2030 as a paid option. The latest standard release is 20.10 (“Groovy Gorilla”), which is supported for nine months.
Ubuntu is developed by Canonical,[22] and a community of other developers, under a meritocratic governance model.[7][23] Canonical provides security updates and support for each Ubuntu release, starting from the release date and until the release reaches its designated end-of-life (EOL) date.[7][24][25] Canonical generates revenue through the sale of premium services related to Ubuntu.[26][27]
Ubuntu is named after the Nguni philosophy of ubuntu, which Canonical indicates means “humanity to others” with a connotation of “I am what I am because of who we all are”.[7]
Tails, or The Amnesic Incognito Live System, is a security-focusedDebian-based Linux distribution aimed at preserving privacy and anonymity.[4] All its incoming and outgoing connections are forced to go through Tor,[5] and any non-anonymous connections are blocked. The system is designed to be booted as a live DVD or live USB, and will leave no digital footprint on the machine unless explicitly told to do so. The Tor Project provided financial support for its development in the beginnings of the project.[6] Tails comes with UEFI Secure Boot.
History:
Tails was first released on 23 June 2009. It is the next iteration of development on Incognito, a discontinued Gentoo-based Linux distribution.[7] The Tor Project provided financial support for its development in the beginnings of the project.[6] Tails also received funding from the Open Technology Fund, Mozilla, and the Freedom of the Press Foundation.[8]
Note: Due to the fact that Tails includes uBlock Origin (compared to the normal Tor Browser Bundle), it could be subject to an attack to determine if the user is using Tails (since the userbase for Tails is less than the Tor Browser Bundle) by checking if the website is blocking advertising.[14] Although this can be avoided by disabling uBlock Origin.
“Raspberry Pi OS[3] (formerly Raspbian) is a Debian-based Linux distributionoperating system for Raspberry Pi. Since 2015 it has been officially provided by the Raspberry Pi Foundation as the primary operating system for the Raspberry Pi family of compact single-board computers.[4] The first version of Raspbian was created by Mike Thompson and Peter Green as an independent project.[5] The initial build was completed in June 2012.[6]
Previous Raspberry Pi OS versions were 32bit only and based on Debian, taking the name Raspbian. Since the more recent 64bit versions no longer use Debian, the name was changed to Raspberry Pi OS for both the 64bit and 32bit versions. As of 1 February 2021, the 64-bit version is in beta and is not suitable for general use.[7][8]
Raspberry Pi OS is highly optimized for the Raspberry Pi line of compact single-board computers with ARM CPUs. It runs on every Raspberry Pi except the Pico microcontroller. Raspberry Pi OS uses a modified LXDE as its desktop environment with the Openbox stacking window manager, along with a unique theme. The distribution is shipped with a copy of the algebra program Wolfram Mathematica[4] and a version of Minecraft called Minecraft: Pi Edition, as well as a lightweight version of the Chromium web browser.”
Although KNOPPIX is primarily designed to be used as a Live CD, it can also be installed on a hard disk like a typical operating system. Computers that support booting from USB devices can load KNOPPIX from a live USB flash drive or memory card.
There are two main editions: the traditional compact-disc (700 megabytes) edition and the DVD (4.7 gigabytes) “Maxi” edition. However, it appears that the CD edition has not been updated since June of 2013.[3] Each main edition has two language-specific editions: English and German.
Knoppix can be used to copy files easily from hard drives with inaccessible operating systems. To quickly and more safely use Linux software, the Live CD can be used instead of installing another OS.”
It was developed by Mati Aharoni and Devon Kearns of Offensive Security through the rewrite of BackTrack, their previous information security testing Linux distribution based on Knoppix. Originally, it was designed with a focus on kernel auditing, from which it got its name KernelAuditingLinux. The name is sometimes incorrectly assumed to come from Kali the Hindu goddess.[8][9] The third core developer, Raphaël Hertzog, joined them as a Debian expert.[10][11]
Kali Linux is based on the Debian Testing branch. Most packages Kali uses are imported from the Debian repositories.[12]
Kali Linux’s popularity grew when it was featured in multiple episodes of the TV series Mr. Robot. Tools highlighted in the show and provided by Kali Linux include Bluesniff, Bluetooth Scanner (btscanner), John the Ripper, Metasploit Framework, Nmap, Shellshock, and Wget.[13][14][15]“
