I happened upon a really great Data Fabric model done by Alan McSweeney and thought it would be useful to have this Data Fabric Framework in modelled Archimate 3.0.

Original Data Fabric model

Alan kindly sent me the original to look at in more detail and be able to cut and paste quicker. He gave me permission to convert it and even collaborate on making it it better in the process.

Archimate Data Fabric (high level) model

This is a high level view of the various components necessary for a holistic Data Fabric framework. We have a more detailed model for each of these component packages in each of the sections below.

Archimate Data Fabric (more detailed) model

This model is the A0 sized page that contains all the information needed. Each of the packages in this model, are modelled separately in the sections below with the same names as the fabric component name.

Fabric Component 01 – External Interacting Parties model

These are the range of external parties that supply data to and access data from the enterprise.

Fabric Component 02 – External Party Interaction Zones, Applications, Channels and Facilities model

These are the set of applications and data interface and exchange points provided specifically to External Interacting Parties to allow them supply data to and access data from the enterprise.  These can be hosted internally or externally or a mix of both.

Fabric Component 03 – External Party Interaction Zones Data Stores model

These are applications and sets of data created by the enterprise to be externally facing where external parties can access information and interact with the enterprise.

Fabric Component 04 – External Third Party Applications model

These are third-party applications (such as social media platforms) that contain information about the enterprise or that are used by the enterprise to present information to or interact with
External Interacting Parties or where the enterprise is referred to, affecting the perception or brand of the enterprise.

Fabric Component 05 – External Data Sensors model

Sources of remote data measurements.

Fabric Component 06 – External Data Devices model

These are devices connected with services offered by the enterprise (such as ATMs and Kiosks).

Fabric Component 07 – Data Intake Gateway model

This is the set of facilities for handling data supplied to the enterprise including validation and transformation including a possible integration or service bus. This can be hosted internally or externally or a mix of both.

Fabric Component 08 – Line of Business Applications model

This represents the set of line of business applications deployed on enterprise owned and managed infrastructure used by business functions to operate their business processes.

Fabric Component 09 – Line of Business Applications hosted outside the organisation model

This represents the set of line of business applications deployed on external infrastructure used by business functions to operate their business processes This includes cloud facilities such as external data storage and XaaS facilities and an integration service to connect external data to internal data

Fabric Component 10 – Desktop Applications model

These are applications used by individual users to view and author documents.

Fabric Component 11 – Data Storage Platforms model

Operational Data Stores – These are the various operational data stores used by the Line of Business Applications. Unstructured Data Stores – This provides structured access to documents and information including externally hosted applications providing these facilities.

Fabric Component 12 – External Application Operational Data Stores model

These are the various operational data stores used by the Line of Business Applications used by Line of Business Applications Hosted Outside the Organisation.

Fabric Component 13 – Document Centric Applications model

• Doc Management Systems – These are systems used to manage transactional and ad-hoc structured and unstructured documents in a formal and controlled manner, including the metadata assigned to documents.
• Doc Sharing & Collaboration – These are tools used within the enterprise to share and collaborate on the authoring of documents.
• Document and Information Portal – This provides structured access to documents and information including externally hosted applications providing these facilities.

Fabric Component 14 – Cloud Extension to Data Mastering, Reporting and Analysis Facilities model

These are facilities to create and manage master data and data extracted from operational data to create a data warehouse and data extracts for reporting and analysis. This includes an extract, transformation and load facility These can be hosted internally or externally or a mix of both.

Fabric Component 15 – Data Reporting and Analysis Facilities model

This represents the set of facilities to extract operational data from business applications, create, store and manage reference and master data, create and store enduring data and analyse the data including reporting, visualisation, mining and modelling. These can be hosted internally or externally or a mix of both.

Fabric Component 16 – Data Mastering model

These are facilities to create and manage master data and data extracted from operational data to create a data warehouse and data extracts for reporting and analysis. This includes an extract, transformation and load facility These can be hosted internally or externally or a mix of both.

Conclusion

You are free to download the printable versions in .PDF form. We may have more discussion and refine these models. Of course we also welcome comments and recommendations for improving these models as well.

Downloads

Data Fabric Architecture (High Level) PDF. (A3 or A2 size)

Data Fabric Architecture (More Detailed Level) PDF. (A1 or A0 size)

The post Data Fabric Framework – Archimate 3.0 model appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Orchestration in Archimate – Part 9.

Ceilometer = Telemetry-Montoring-Collecting

Ceilometer project is for monitoring and collecting data regarding to resource usage in the cloud infrastructure.

Mission: Ceilometer is to provide an infrastructure to collect any information needed regarding OpenStack projects. It was designed so that rating engines could use this single source to transform events into billable items which we label as “metering”. As the project started to come to life, collecting an increasing number of meters across multiple projects, the OpenStack community started to realize that a secondary goal could be added to Ceilometer: become a standard way to meter, regardless of the purpose of the collection. This data can then be pushed to any set of targets using provided publishers mentioned in Publishers section.

If you divide a billing process into a 3 step process, as is commonly done in the telco industry, the steps are:

1. Metering
2. Rating
3. Billing

Ceilometer’s initial goal was, and still is, strictly limited to step one ‘Metering’. This is a choice made from the beginning not to go into rating or billing, as the variety of possibilities seemed too large for the project to ever deliver a solution that would fit everyone’s needs, from private to public clouds.  Ceilometer is a component of the Telemetry project. Its data can be used to provide customer billing, resource tracking, and alarming capabilities across all OpenStack core components.

Ceilometer = Telemetry-Montoring-Collecting Structural Logical Architecture

Previously, ceilometer used mysql as for backend storage. Currently, it is passed off to another Application Collaboration called gnocchi that does the backend job using mongodb. Ceilometer also has a Command Line Interface (CLI).

Each of Ceilometer’s services are designed to scale horizontally. Additional workers and nodes can be added depending on the expected load. Ceilometer offers two core services:

• polling agent – daemon designed to poll OpenStack services and build Meters. Polling agents, which is the less preferred method, will poll some API or other tool to collect information at a regular interval. The polling approach is less preferred due to the load it can impose on the API services.
• notification agent – daemon designed to listen to notifications on message queue, convert them to Events and Samples, and apply pipeline actions.  The Notification agent which takes messages generated on the notification bus and transforms them into Ceilometer samples or events. This is the preferred method of data collection. If you are working on some OpenStack related project and are using the Oslo library, you are kindly invited to come and talk to one of the project members to learn how you could quickly add instrumentation for your project. The heart of the system is the notification daemon (agent-notification) which monitors the message queue for data sent by other OpenStack components such as Nova, Glance, Cinder, Neutron, Swift, Keystone, and Heat, as well as Ceilometer internal communication.

Ceilometer = Telemetry-Montoring-Collecting Behavioural Services Architecture

These are the REST & CLI Services offered by Ceilometer for Monitoring used to be found here: https://docs.openstack.org/developer/ceilometer/webapi/v2.html. They are now deprecated as the storage elements have moved over to Aodh [Alarms], Gnocchi [Metrics] and Panko [Events].

Archimate Model Download

This file is shared under the GNU General Public License (GPL). You are free to download it and use it as you like. You can either make use of your own Archmate Tool and download it in .archmate format, or use your own tool that support the Open Groups open Exchange File format. These files are both compressed into a .zip file.

References

• https://Openstack.org
• UDemy.com – OpenStack essentials course
• https://www.slideshare.net/mirantis/openstack-architecture-43160012
• https://www.slideshare.net/prk1980/cloud-orchestration-major-tools-comparision
• https://www.slideshare.net/openstack/intro-grizzlyarchv1-19109550
• https://www.slideshare.net/Cloudenablers/openstack-heat-how-autoscaling-works-52244215
• https://www.slideshare.net/openstackindia/openstack-heat-51867693
• http://www.sparkmycloud.com/blog/telemetry-service-in-openstack/

The post OpenStack Cloud Metering in Archimate – Part 10 appeared first on Agile Enterprise Architecture.

his blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Identity in Archimate – Part 8.

Heat = Orchestration

Mission: Heat as the OpenStack Orchestration program is to create a human- and machine-accessible service for managing the entire lifecycle of infrastructure and applications within OpenStack clouds. Why Heat? It makes Clouds Rise!

Heat is the main project in the OpenStack Orchestration program. It implements an orchestration engine to launch multiple composite cloud applications based on templates in the form of text files that can be treated like code. A native Heat template format is evolving, but Heat also endeavours to provide compatibility with the AWS CloudFormation template format, so that many existing CloudFormation templates can be launched on OpenStack. Heat provides both an OpenStack-native ReST API and a CloudFormation-compatible Query API.

Heat = Orchestration Structural Logical Architecture

• A Heat template describes the infrastructure for a cloud application in a text file that is readable and writable by humans, and can be checked into version control, diffed, etc. Infrastructure resources that can be described include: servers, floating ips, volumes, security groups, users, etc.
• Heat also provides an autoscaling service that integrates with Telemetry, so you can include a scaling group as a resource in a template.
• Templates can also specify the relationships between resources (e.g. this volume is connected to this server). This enables Heat to call out to the OpenStack APIs to create all of your infrastructure in the correct order to completely launch your application.
• Heat manages the whole lifecycle of the application – when you need to change your infrastructure, simply modify the template and use it to update your existing stack. Heat knows how to make the necessary changes. It will delete all of the resources when you are finished with the application, too.
• Heat primarily manages infrastructure, but the templates integrate well with software configuration management tools such as Puppet and Chef. The Heat team is working on providing even better integration between infrastructure and software.

This Architecture appears to be in a state of constant refactoring and enhancement, so follow the Architecture here: https://wiki.openstack.org/wiki/Heat/Vision

Heat = Orchestration Behavioural Services Architecture

Orchestration services to create and manage the life-cycle of cloud applications are described like this:

• How do we model the cloud application deployment (YAML/JSON)? Heat provides Template (Openstack:HOT & AWS Cloud Formation:CFN)
• How do we configure the cloud application? Heat provides Software-Configuration
• How do we customize the cloud application? Heat provides Parameters & Environments
• How do we create and manage the cloud application? Heat provides Stack
• How do we model the cloud application element such as instances, volumes, etc ? Heat provides Resource (as resource plug-in)
• How do we manage the cloud application scalability? Heat provides Auto-scaling
• How do we track the progress of life-cycle operations ? Heat provides Events

Next

The next blog in the series is OpenStack Cloud Metering in Archimate – Part 10

The post OpenStack Cloud Orchestration in Archimate – Part 9 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on Openstack Cloud Object Storage in Archimate – Part 7

Keystone = Identity

Keystone is an OpenStack service that provides API client authentication, service discovery, and distributed multi-tenant authorization by implementing OpenStack’s Identity API. It supports LDAP, OAuth, OpenID Connect, SAML and SQL. It offeres Role Based Access Control (RBAC).

Keystone = Identity Structural Logical Architecture

Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication.
Keystone handles API requests as well as providing configurable keystone (service & admin APIs) catalog, policy, token and identity services.
Standard backends include LDAP or SQL, as well as Key Value OpenStack token backend, catalog backend, policy backend and identity backend stores (KVS).
Identity Service Most people will use this as a point of customization for their current authentication services.

Keystone = Identity Behavioural Services Architecture

These are the REST & CLI Services offered by Keystone for Identity. For each Service below there are one or many sub-services or commands that can be found here:  https://developer.openstack.org/api-ref/identity/v3/ 

1. User gets Token from Keystone.
2. Token includes the list of user Projects and Roles in them.
3. User calls the Service specifying the Token.
4. Service interprets the Roles:
   1. Service consults its policy.json file.
   2. Policy.json specifies the list of available rules.
   3. Policy.json specifies which rules are enforced for operations and resources.

Next

The next blog in the series is OpenStack Cloud Orchestration in Archimate – Part 9.

The post OpenStack Cloud Identity in Archimate – Part 8 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Images in Archimate – Part 6

Swift = Object Storage

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound.

Swift is a highly available, distributed, eventually consistent object/blob store. Organizations can use Swift to store lots of data efficiently, safely, and cheaply. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound.

Swift = Object Storage Structural Logical Architecture

The Object Storage system organizes data in a hierarchy, as follows:

• Account. Represents the top-level of the hierarchy.  Your service provider creates your account and you own all resources in that account. The account defines a namespace for containers. A container might have the same name in two different accounts. In the OpenStack environment, account is synonymous with a project or tenant.
• Container. Defines a namespace for objects. An object with the same name in two different containers represents two different objects. You can create any number of containers within an account. In addition to containing objects, you can also use the container to control access to objects by using an access control list (ACL). You cannot store an ACL with individual objects. In addition, you configure and control many other features, such as object versioning, at the container level. You can bulk-delete up to 10,000 containers in a single request. You can set a storage policy on a container with predefined names and definitions from your cloud provider.
• Object. Stores data content, such as documents, images, and so on. You can also store custom metadata with an object.

Swift = Object Storage Behavioural Services Architecture

These are the REST & CLI Services offered by Swift for Object storage. For each Service below there are one or many sub-services or commands that can be found here: https://developer.openstack.org/api-ref/object-storage/index.html 

Next

The next blog in the series is OpenStack Cloud Identity in Archimate – Part 8

The post OpenStack Cloud Object Storage in Archimate – Part 7 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Block Storage in Archimate – Part 5.

Glance = Images

Mission: The Image service (glance) provides a service where users can upload and discover data assets that are meant to be used with other services. This currently includes images and metadata definitions.

What is a virtual machine image? A virtual machine image is a single file which contains a virtual disk that has a bootable operating system installed on it. Virtual machine images come in different formats: Raw, AKI, AMI, ARI, ISO, OVF, QCOW2, UEC Tarball, VDI, VHD, VHDX, VMDK.

Glance image services include discovering, registering, retrieving and format converting virtual machine (VM) images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.

Glance = Images Structural Logical Architecture

Glance image services include discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image. VM images made available through Glance can be stored in a variety of locations from simple filesystems to object-storage systems like the OpenStack Swift project.

Glance = Images Behavioural Services Architecture

These are the REST & CLI Services offered by Glance for Images. For each Service below there are one or many sub-services or commands that can be found here: https://developer.openstack.org/api-ref/image/v2/index.html

Next

The next blog in the series is OpenStack Cloud Object Storage in Archimate – Part 7

The post OpenStack Cloud Images in Archimate – Part 6 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Compute in Archimate – Part 4.

Cinder = Block Storage

Cinder is a Block Storage service for OpenStack.

Mission: To implement services and libraries to provide on demand, self-service access to Block Storage resources. Provide Software Defined Block Storage via abstraction and automation on top of various traditional back-end block storage devices.

The block storage system manages the creation, attaching and detaching of the block devices to servers. Block storage volumes are fully integrated into OpenStack Compute and the Dashboard allowing for cloud users to manage their own storage needs. In addition to local Linux server storage, it can use storage platforms including Ceph, CloudByte, Coraid, EMC (VMAX and VNX), GlusterFS, IBM Storage (Storwize family, SAN Volume Controller, XIV Storage System, and GPFS), Linux LIO, NetApp, Nexenta, Scality, SolidFire and HP (StoreVirtual and StoreServ 3Par families).

Cinder = Block Storage Structural Logical Architecture

Cinder virtualizes the management of block storage devices in software. Cinder provides end users with a self-service REST or CLI API to request and consume those resources without requiring any knowledge of where their storage is actually deployed or on what type of device. This is done through the use of either a reference implementation (LVM) or plugin drivers for other storage.

Cinder uses a sql-based central database that is shared by all Cinder services in the system. The amount and depth of the data fits into a sql database quite well. For small deployments this seems like an optimal solution. For larger deployments, and especially if security is a concern, cinder will be moving towards multiple data stores with some kind of aggregation system.

The Auth Manager component responsible for users/projects/and roles. Can back-end to DB or LDAP. This is not a separate binary, but rather a python class that is used by most components in the system.

The scheduler: decides which host gets each volume.

A volume: manages dynamically attachable block devices. There can be many of these of different types each of which has a specific driver.

Cinder = Block Storage Behavioural Services Architecture

These are the REST & CLI Services offered by Cinder for Block Storage. For each Service below there are one or many sub-services or commands that can be found here: https://developer.openstack.org/api-ref/block-storage/v3

Next

The next blog in the series is OpenStack Cloud Images in Archimate – Part 6.

The post OpenStack Cloud Block Storage in Archimate – Part 5 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Networking in Archimate – Part 3.

Nova = Compute

Mission: To implement services and associated libraries to provide massively scalable, on demand, self-service access to compute resources, including bare metal, virtual machines, and containers.

Nova = Compute Structural Logical Architecture

Nova Application Collaboration is comprised of multiple server processes, each performing different functions. The user-facing interface is a REST API, while internally Nova components communicate via an RPC message passing mechanism called the Conductor Component.

The API servers process REST requests, which typically involve database reads/writes, optionally sending RPC messages to other Nova services, and generating responses to the REST calls. RPC messaging is done via the oslo.messaging library, an abstraction on top of message queues. Most of the major nova components can be run on multiple servers, and have a manager that is listening for RPC messages. The one major exception is nova-compute, where a single process runs on the hypervisor it is managing (except when using the VMware or Ironic drivers).

Nova also uses a central database that is (logically) shared between all components.

Nova = Compute Behavioural Services Architecture

These are the REST & CLI Services offered by Nova for Compute. For each Service below there are one or many sub-services or commands that can be found here: https://developer.openstack.org/api-ref/compute (We also chose not to show all the many Deprecated services)

Next

The next blog in the series is OpenStack Cloud Block Storage in Archimate – Part 5.

The post OpenStack Cloud Compute in Archimate – Part 4 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from the previous Blog on OpenStack Cloud Dashboard in Archimate – Part 2.

Neutron = Networking

Mission: To implement services and associated libraries to provide on-demand, scalable, and technology-agnostic network abstraction.

Openstack Neutron offers Network-as-a-Service via REST API’s. It implements Virtual VLAN Networks with Level 2 isolation. It also implements Virtual ports where devices can connect to the virtual network. Neutron implements a pluggable back-end to abstract the API away from different technology choices (e.g. Open vSwitch, Cisco UCS/Nexus). Offers virtual switches, bridges, L3 routing, floating IPs, NAT, DHCP, VPN and Load Balancing.

Implements Software-Defined networking which is the physical separation of network control plane from the forwarding plane using the OpenFlow protocol to control several devices. Uses either Command Line Interface (CLI) or Horizon as a GUI to configure the networking.

Neutron = Networking Structural Logical Architecture

The structural architecture of Neutron is shown in the diagram below. Please note however that even though this all looks like it resides on one single node, components can be split out across multiple Compute Nodes, Controller nodes and Network Nodes.

These are the REST & CLI Services offered by Neutron. For each Service below there are one or many sub-services or commands that can be found here: https://developer.openstack.org/api-ref/networking/v2/index.html

Next

The next blog in the series is OpenStack Cloud Compute in Archimate – Part 4.

The post OpenStack Cloud Networking in Archimate – Part 3 appeared first on Agile Enterprise Architecture.

This blog on Openstack modelled using Archimate follows on from Openstack Cloud in Archimate – Part 1.

Horizon = Dashboard Functionality

Mission: Horizon is the canonical implementation of OpenStack’s dashboard, which is extensible and provides a web based user interface to OpenStack services.

Horizon = Dashboard Structural Logical Architecture

• Is “stateless” — doesn’t require a database.
• Delegates error handling to the back-end.
• Doesn’t support all the API functions exposed by the other Openstack Applications.
• Can use memcached or database to store sessions.
• Gets updated via API polling.
• Uses Django Project, which is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. See more detail on Django here: https://www.slideshare.net/balakumarp/django-framework
• It can be extended.

As mentioned in Part 1., the Openstack Cloud can be managed via a command line interface (CLI). This Dashboard makes is a lot easier to facilitate simple queries and changes, but does not support all the ever evolving services served by all the relevant Applications. To get all the functionality, one would need to connect to the CLI manually or REST services using the API programmatically.

Horizon = Dashboard Behavioural Services Architecture

Horizon does not offer or provide any of its own services, it simply consumes all the services from the other applications. Since in latest blogs we will show all Services offered by each application, nothing will be modelled here.

Next

This next blog in the series is OpenStack Cloud Networking in Archimate – Part 3.

The post OpenStack Cloud Dashboard in Archimate – Part 2 appeared first on Agile Enterprise Architecture.