Oracle Database Backups to the Amazon Cloud

The traditional way of performing backups includes using Oracle RMAN in combination with media management layer software ( typically Netbackup, Tivoli or similar ), which writes backup data to remote robotic tape unit. Tapes are then stored offsite to a secure location. It is well known fact that tape media poses certain challenges in reliability and physical manipulation areas.

Cloud-based backups’ main attraction is that they are inherently disk based, always accessible, offsite and there are no capex expenditures. All tape related costs are thus eliminated. On the other hand new costs will be incurred for Cloud backups storage and service. Data is transferred over the public network i.e. Internet.

Cloud based backups can be used for quick database refresh or duplication of source databases to any target environment. Practically you have an unlimited amount of storage that can be instantly attached to any database server as temporary storage for backups and restores. For example, you might want to create new QA environment using development database as data source. This can be achieved by backing up development database  to Amazon S3, then restoring to QA.

Technology to perform tightly integrated Oracle backups to Amazon Cloud ( S3) is available. Please refer to http://www.oracle.com/technetwork/database/features/availability/twp-oracledbcloudbackup-130129.pdf for technical details. RMAN is integrated with Amazon via Oracle Secure Backup (OSB) Cloud Module which automatically directs backups to Amazon S3 storage. Backups can be encrypted and run in parallel over multiple channels to comply with security and performance requirements.

Simple change in your RMAN configuration parameters will redirect backups to the Amazon Cloud. RMAN parameters will have to be carefully configured to take maximum advantage of compression and parallel execution in order to minimize impact of network speed to data transfer rates.

For databases larger than couple of hundred of Gigabytes it would be impossible to rely on standard Internet, out-of-the-box data transfer rates. Amazon AWS Direct Connect service lets you establish direct connection from your on-premise network to Amazon VPC using one or more 1Gbps and 10Gbps connections. There is no charge for IN data transfers, which makes it ideally suited for backup purposes.

Open source fast file transfer protocol called Tsunami UDP provides faster data transfers than what is possible with ftp.

Additional products like Aspera that boost network data transfer rates are being introduced that make it possible to move terabytes of data on a daily basis. Please refer to http://aws.amazon.com/solutions/solution-providers/aspera/. There is additional cost associated with Aspera usage.

Restore, i.e., DR tactics will not have to be significantly modified to take advantage of cloud based backups.

OSB Cloud module is currently available for Linux 32 and 64, SPARC 64, and Windows 32 bit environments.

Oracle Disaster Recovery Site Hosted by Amazon Cloud

DR sites are typically built as an exact replica of the primary site. Application and database software is installed on DR site and sits there mostly unused, waiting for a disaster to happen. DR site is very expensive proposition than only large companies are able to afford. Amazon AWS is an interesting alternative to having your own DR site.

Oracle databases on DR side are in Data Guard configuration with a primary site and actively apply archive log files shipped from there. Pay per use, scalable Amazon Cloud model makes it an attractive alternative to creating and maintaining your own DR site. During normal usage you will use only as many resources  CPU, memory) as is required to keep Oracle Data Guard active. Once disaster strikes you can switch over to DR site, add CPU and memory to your database and make it able to withstand regular load until primary site becomes functional again. As soon as primary site is repaired then new switchover can be initiated to fall back to the original configuration i.e. Amazon AWS again becomes your DR site.

Technology to build Oracle Data Guard databases in Amazon AWS is available. You can install Oracle software, create and configure Data Guard physical standby databases and enable archive log shipping between primary and DR sites.

Oracle Data Guard physical standby database requires same version of operating system software on both primary and DR sites. Amazon EC2 currently supports various flavors of Linux as well as Windows operating system. It means that having Oracle Data Guard DR site hosted by Amazon AWS will be possible for customers running Linux and Windows operating systems.

For databases larger than couple of hundred of Gigabytes it would be impossible to rely on standard Internet, out-of-the-box data transfer rates. Amazon AWS Direct Connect service lets you establish direct connection from your on-premise network to Amazon VPC using one or more 1Gbps and 10Gbps connections. There is no charge for IN data transfers, which makes it ideally suited for DR site purposes.

Open source fast file transfer protocol called Tsunami UDP provides faster data transfers than what is possible with ftp.

Products like Aspera that boost network data transfer rates are being introduced that make it possible to move terabytes of data on a daily basis. Please refer to http://aws.amazon.com/solutions/solution-providers/aspera/. There is additional cost associated with Aspera usage.

OpenNebula: Open Source Cloud Setup

What Is OpenNebula ?

It is cloud management solution – industry standard open source cloud computing tool to manage the complexity and heterogeneity of distributed data center infrastructures.

OpenNebula is a fully open-source management toolkit for on-premise Infrastructure as a Service (IaaS) cloud computing. OpenNebula can be primarily used as a virtualization tool to manage your virtual infrastructure in the data-center or cluster, which is usually referred as Private Cloud. OpenNebula supports Hybrid Cloud to combine local infrastructure with public cloud-based infrastructure, enabling highly scalable hosting environments. OpenNebula also supports Public Clouds by providing Cloud interfaces to expose its functionality for virtual machine, storage and network management.

OpenNebula can be primarily used as a virtualization tool to manage your virtual infrastructure in the data-center or cluster.

OpenNebula is an open and flexible tool that fits into existing data center environments to build any type of Cloud deployment. OpenNebula can be primarily used as a virtualization tool to manage your virtual infrastructure in the data-center or cluster.

Functionality and Capabilities

How are networks managed by OpenNebula?

OpenNebula contains a built-in Virtual Network Manager that enables the mapping of virtual networks onto physical ones. In this way, physical networks can be breakdown into smaller networks that, in turn, can be used to isolate virtual machines in different virtual networks and to communicate those machines in the same isolated virtual networks.

Which VM managers can OpenNebula use?

Physical cluster nodes can have KVM, Xen, or VMware hypervisors installed.

What interfaces does OpenNebula provide?

Native API of OpenNebula is offered via XML-RPC and its Java and Ruby OCA bindings. Managing of virtual machines, physical cluster nodes and virtual networks can be accomplished using this interface. OpenNebula implements the EC2 Query, OGF OCCI and vCloud APIs to demonstrate its support for the development of new Cloud interfaces.

What does OpenNebula offer?

OpenNebula provides a powerful, scalable and secure multi-tenant cloud platform for fast delivery and elasticity of virtual resources.

  • The Image Repository system allows to set up and share images, which can be operative systems or data, to be used in Virtual Machines easily.
  • The Template Repository system allows to register Virtual Machine definitions in the system, to be instantiated later as Virtual Machine instances.
  • Virtual Networking is provided to interconnect Virtual Machines, they can be defined as fixed or ranged networks.
  • Once a Template is instantiated to a Virtual Machine, there are a number of operations that can be performed to control their lifecycle, such as migration (live and cold), stop, resume, cancel, etc. These operations are available both from theCLI and the Sunstone GUI.

What are the Main Components?

Interfaces & APIs: OpenNebula provides many different interfaces that can be used to interact with the functionality offered to manage physical and virtual resources. There are two main ways to interface OpenNebula: command line interface and the Sunstone GUI. There are also several cloud interfaces that can be used to create public clouds: OCCIandEC2 Query.


Users and Groups: OpenNebula supports user accounts and groups, as well as variousauthentication and authorization mechanisms. This feature can be used to create isolated compartments within the same cloud, implementing multi-tenancy. Moreover, a powerfulAccess Control Listmechanism is in place to allow different role management, allowing a fine grain permission granting.

Networking: An easily adaptable and customizable network subsystem is present in OpenNebula in order to better integrate with the specific network requirements of existing datacenters. Support for VLANs and Open vSwitch are also featured.

Hosts and Virtualization: Various hypervisors are supported in the virtualization manager, with the ability to control the lifecycle of Virtual Machines, as well as monitor them. This monitorization also applies to the physical hosts.

Advanced Setups

OpenNebula supports the following types of cloud deployments:

  • Multiple zones a VDCs: The OpenNebula Zones component (oZones) allows for the centralized management of multiple instances of OpenNebula, called Zones, managing in turn potentially different administrative domains. This zones can be further compartmentalized by grouping physical hosts in Virtual Data Centers (VDCs).
  • Hybrid: OpenNebula gives support to build a Hybrid Cloud is an extension of a Private Cloud to combine local resources with resources from remote Cloud providers. A whole Public Cloud provider can be encapsulated as a local resource to be able to use extra computational capacity to satisfy peak demands.

Canadian Government Shared Services Infrastructure Zones deployed in Amazon Web Services Public Cloud

According to IT Shared Services Security Domain & Zones Architecture document:
a server will typically need to “reside in” or “connect to” other specialized zones.

Readers will note that some servers are in Shared Services Canada Data Center – for example Monitoring, Management Restricted Zones, while other servers are in Amazon Web Services Public Cloud. Storage RZ can entirely reside in AWS S3 storage for backups and files of all purposes, while AWS EBS volumes can serve as block storage i.e. for file systems.

According to the same Government document servers also “reside” within the ITSS Management RZ from which they can be managed by SSC. Through this connection they can be monitored for availability and performance (i.e. by monitoring services located within the Monitoring RZ), and receive utility services (e.g. patch, update, network time, name to address resolution) from servers located within the Management PAZ.

Amazon Web Services can naturally blend with Canadian Shared Services as one of ready made, immediately available building blocks. Integration effort is negligible, since AWS components completely adhere to Cloud principles of on-demand, elastic, scalable service provisioning.

How Amazon Web Services Complies to Canadian Government Policy Requirements

Canadian Government documents ( IT Shared Services Security Domain & Zones Architecture ) specify that:
Given that each department maintains responsibility for a significant portion of the total solution, including the security controls to be implemented, it is unlikely that the risk of grouping multiple client departments into a single SSC domain, and the sharing of zones that this implies, would be acceptable.
For this reason a domain per department is recommended for IaaS type services.

AWS is addressing this requirement by ability to create new security domain for each new customer Canadian Government agency. AWS Virtual Private Cloud is completely capable of providing all necessary features needed for implementation of security domain.

Government standards envision various levels of interaction between departments and service providers as per picture below:

AWS is able to serve Government of Canada needs at many levels of involvement.

  • As Co-location provider via AWS dedicated instances program.
  • As IaaS provider via EC2, ESB, S3 offerings
  • As PaaS provider via RDS, Elastic MapReduce and many other offerings

Government standards require that following practices are maintained:

Applications are created and tested in the development environment, after which they are “promoted” (i.e. moved) to staging where they are tested in a production like environment. Finally, once the application has successfully passed client acceptance within staging, it is “promoted” to the production-hosting environment.

Each of these three environments is depicted as within a separate SSC security domain.

AWS offers capabilities to create separate domains for each stage so that proper change management procedure is followed.

How Government of Canada Shared Services can utilize Amazon Public Cloud

Canadian Government Shared Services standards envision inclusion and usage of public Cloud providers like Amazon Web Services in Government infrastructure. Government documents ( IT Shared Services Security Domain & Zones Architecture document ) specify that standard and guidelines contained in this document should be transposable to the use of similar shared services offered through a public cloud provider under contract to the GC.

We will have a look at how specific Canadian Government requirements can be easily, comprehensively and immediately addressed by superb Amazon Web Services infrastructure. We will start by looking at Canadian Government requirements from more generic guidelines to very specific, detailed specification and describe  corresponding AWS features.

Above mentioned document specifies that:

A security domain is an environment or context that includes a set of resources and a set of entities that have the right to access the resources as defined by a common security policy and administered by a single authority. 

A network security zone is a networking environment with a well-defined boundary, a Network Security Zone Authority, and a standard level of susceptibility to network threats. The concept of network security zones is generally applied during the implementation of a security domain as a way to satisfy some of the domain policy requirements.

Shared Services involvement with departmental IT is at various levels of engagement – from providing email service  to complete set of data center services ( network, storage, application, database ). In Cloud parlance intent is for Shared Services to provide IaaS, PaaS and SaaS services to internal clients.

Dedicated IT on  above picture would be government departmental IT organization. Hosting provider is Shared Services ( or Public Cloud provider like Amazon Web Services, under contract to Government of Canada ).

Network security zone as defined by Government of Canada is easily implemented by ready-made, available on demand, scalable Amazon Virtual Private Centre. Customer’s  Network on picture below would be Government Agency Departmental IT which can extend into AWS using Amazon VPC feature.

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a private, isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define. With Amazon VPC, you can define a virtual network topology that closely resembles a traditional network that you might operate in your own datacenter. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways.

Government standards envision that:

The client may also require (and have control over the configuration of) perimeters between the tiers of the application architecture (e.g. between their application RZ -restricted zone and their database RZ). 

Amazon Web Services Multi-Tier Security Architecture is configurable to limit access between tiers. You can easily customize the network configuration for your Amazon VPC. For example, you can create a public-facing subnet for your webservers that has access to the Internet, and place your backend systems such as databases or application servers in a private-facing subnet with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to Amazon EC2 instances in each subnet.

Additionally, you can create a Hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate datacenter.

IT Shared Services

Security Domain & Zones

Architecture

Deploying Oracle Databases to Amazon AWS (EC2, RDS)

Amazon recently added Oracle database hosting capabilities to its RDS service offering. You can rent an Oracle database in a pay-as-you-go fashion now. We are going to explore if corporations should be utilizing Amazon AWS Oracle Database related services (EC2, RDS ), how it should be used, where possible savings and potential trouble points are. With services like Amazon AWS it doesn’t matter where your hardware and software physically is – it could be in a room next to you or in some other country. It is much easier and cheaper to procure and get new servers up and running.

[Read more...]

Move Your Oracle Databases to Amazon EC2 Cloud

Image representing Amazon Web Services as depi...

Image via CrunchBase

Amazon Web Services EC2 Cloud is full scale public data center offering services that are in many aspects far ahead of ancient practices present in regular IT environments.

[Read more...]

Enterprise Class Oracle Databases in the Public Cloud?

Panorama in Calfiornia

Image via Wikipedia

Most of enterprise class shops today run their Oracle databases on either HP-UX, AIX or Sun OS operating systems. Is it possible to move these databases to the public cloud, and, if so, who are providers who can help with such a move?

[Read more...]

Amazon Federal Government Cloud

Image representing Amazon Web Services as depi...

Image via CrunchBase

Federal CIO Vivek Kundra instituted a “Cloud First” policy. Policy states that agencies “must migrate three applications to the cloud within the next 18 months”Amazon Web Services created Amazon Federal Government service to help agencies comply with this and similar initiatives.

What is AWS GovCloud?

AWS GovCloud is an ”Amazon Web Services Region designed to allow US government agencies and contractors to move more sensitive workloads into the cloud by addressing their specific regulatory and compliance requirements.

AWS GovCloud is physically and logically accessible by US persons only, government agencies can now manage more heavily regulated data in AWS while remainingcompliant with strict federal requirements.

AWS has received Federal Information Security Management Act (FISMA) Moderate Authorization and Accreditation from the U.S. General Services Administration.

With the addition of FISMA Moderate, the AWS security and compliance framework now covers FISMA Low and Moderate, PCI DSS Level 1, FIPS 140-2, ISO 27001, and SAS-70 type II. AWS also provides an environment that enables businesses to comply with HIPAA regulations.

Many AWS services are now available in AWS GovCloud:
Amazon Elastic Compute Cloud (Amazon EC2)  delivers scalable, pay-as-you-go compute capacity in the cloud; this is where your virtual machines and instances will be hosted.

Amazon Simple Storage Service (Amazon S3) – provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time, from anywhere on the Web; you can think of S3 as unlimited cheap disk based storage, an ideal replacement for tape.

Amazon Elastic Block Store (EBS) – provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance; EBS volumes will be hosting your file systems and data.

Amazon Virtual Private Cloud (Amazon VPC) – provision a private, isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define.

AWS Identity and Access Management (IAM) – enables you to securely control access to AWS services and resources for your users.

Amazon CloudWatch – provides monitoring for AWS cloud

Amazon is offering variety of instance types ( including dedicated instances ) and charging models to help you pick and choose i.e. tailor services that will best fit your particular needs.

What to migrate first ?

It is relatively easy and painless to migrate your applications to AWS. It typically takes around two weeks for medium complexity application to have it up and running in AWS.

You will need to come up with the list of applications best suited for cloud migration. Our experiences show that you can safely start with less mission critical, relatively self-contained applications and then progress towards more complex deployments. Systems we recommend to be migrated first are:

- software build/deployment system

- time & attendance

- monitoring systems

- asset tracking & verification

- vendor payment system

- recruiting systems

- ldap sync system

- internal blogging system

Other uses

Virtual Private Cloud is an ideal destination for tape replacement disk based backups. Some vendors ( Oracle, for example ), already created products which enable simple RMAN redirections to AWS S3 storage for backup purposes.

Building your DR site in VPC cloud is another type of AWS use that is a natural fit for the public cloud. AWS is inherently remote destination and certified to host a wide variety of software.

Databases on DR side are in Data Guard configuration with a primary site and actively apply archive log files shipped from there. Pay per use, scalable Amazon Cloud model makes it an attractive alternative to creating and maintaining your own DR site. During normal usage you will use only as many resources  CPU, memory) as is required to keep Oracle Data Guard active. Once disaster strikes you can switch over to DR site, add CPU and memory to your database and make it able to withstand regular load until primary site becomes functional again. As soon as primary site is repaired then new switchover can be initiated to fall back to the original configuration i.e. Amazon AWS again becomes your DR site.

AWS is also offering Multi A-Z zones – a Relational Database Service feature . A simple click will suffice to create an Oracle or MySQL physical standby database – an exercise which easily takes a day or two if done manually, even by an experienced DBA. Your primary database will thus be instantly protected by a robust standby database. When database is created or modified to run as a Multi-AZ deployment, Amazon RDS will automatically provision and manage physical standby database in a different Availability Zone. Availability Zone is independent infrastructure in a physically separate location.

Read Replica is an AWS feature that addresses one of notoriously difficult problems in RDBMS world – horizontal scaling. Single click will create read-only replica of your relational database that can be used for query purposes, thus offloading main database server for OLTP activities.

Oracle RMAN  (Recovery Manager) catalog database contains backup metadata for all enterprise wide Oracle databases. Each database backed up through RMAN needs connection to a central backup repository database. RMAN catalog  is purged and maintained by removing obsolete backup records, crosschecking with existing media etc. If central repository grows too big then backup performance will suffer since it will take too long for RMAN to locate metadata.

Amazon Web Services gives us the ability to quickly and easily create either brand new RMAN catalog database or move an existing catalog to it.

Conclusion

There is a wide variety of applications that can make immediate use of great AWS GovCloud. Amazon practically defined cloud computing. Big shift towards Cloud environment has started. It is now clear that this change is similar in magnitude to the shift from mainframe to client-server computing two decades ago.

Amazon Web Services is the pioneer and market leader in Cloud computing space. Other vendors are playing catch up and do not come close to the breadth and scale of AWS offerings. Services and features Amazon provides are quite extensive and cover many of the enterprise-class computing needs. APIs and command line interfaces are available for each service, which makes scripting and automation achievable. Documentation is publicly available and there is large ecosystem of organizations and individuals proficient in use of AWS.  Amazon Web Services unsurpassed global presence and size makes it an easy choice for government Cloud IaaS provider.

Follow

Get every new post delivered to your Inbox.

Join 75 other followers