In today’s article, we will look again bit into deployment of Oracle DB over NFS protocol on NetApp storage. This article is based on practical experience with deployment of SAP using Oracle (10g) as a back-end as well as on experience with Oracle DB deployments (10g & 11g). Most parts of this article are from a documentation I have created for my last customer, who has been implementing DR solution between two sites with third site for Backup and Recovery. This article is a part of Oracle over NFS on NetApp series, during entire series, we will introduce you to our approach towards projects and as example we chose to give you deep insight into Oracle deployment which we are using as a show-case. In todays episode, we will create aggregates, volumes, QTrees and vFiler.
Aggregates
As we have designed in Part 2 of this series, we have two aggregates made out of 300GB SAS drives. Both aggregates have raid-group size of 19 spindles.
toaster01a> aggr create aggr_oradata -r 19 -T SAS -t raid_dp -d 2b.02.18 2b.02.19 2b.02.20 toaster01a> aggr add aggr_oradata 54 toaster01a> aggr options aggr_oradata nosnap on toaster01a> snap reserve -A aggr_oradata 0 toaster01a> aggr create aggr_oramisc -r 19 -T SAS -t raid_dp -d 0a.00.1 0a.00.2 0a.00.3 toaster01a> aggr add aggr_oramisc 16 toaster01a> aggr options aggr_oramisc nosnap on toaster01a> snap reserve -A aggr_oramisc 0
toaster02a> aggr create aggr_oradata -r 19 -T SAS -t raid_dp -d 2b.02.18 2b.02.19 2b.02.20 toaster02a> aggr add aggr_oradata 54 toaster02a> aggr options aggr_oradata nosnap on toaster02a> snap reserve -A aggr_oradata 0 toaster02a> aggr create aggr_oramisc -r 19 -T SAS -t raid_dp -d 0a.00.1 0a.00.2 0a.00.3 toaster02a> aggr add aggr_oramisc 16 toaster02a> aggr options aggr_oramisc nosnap on toaster02a> snap reserve -A aggr_oramisc 0
Volumes
As we have designed in Part 2 of this series, we have five mirrored volumes between primary and DR site, and one independent volumes in each site. In addition to these volumes, we also need to create vFiler root volumes, 200MB in size at each site. These will be also independent volumes without mirroring.
toaster01a> vol create vf_wr01_oraarch_m_wr01 aggr_oramisc 700g toaster01a> qtree security /vol/vf_wr01_oraarch_m_wr01 unix toaster01a> qtree create /vol/vf_wr01_oraarch_m_wr01/arch toaster01a> vol options vf_wr01_oraarch_m_wr01 nosnap on toaster01a> vol options vf_wr01_oraarch_m_wr01 nosnapdir on toaster01a> vol options vf_wr01_oraarch_m_wr01 no_atime_update on toaster01a> vol options vf_wr01_oraarch_m_wr01 minra on toaster01a> vol create vf_wr01_oralogtemp_m_wr01 aggr_oramisc 64g toaster01a> qtree security /vol/vf_wr01_oralogtemp_m_wr01 unix toaster01a> qtree create /vol/vf_wr01_oralogtemp_m_wr01/temp toaster01a> vol options vf_wr01_oralogtemp_m_wr01 nosnap on toaster01a> vol options vf_wr01_oralogtemp_m_wr01 nosnapdir on toaster01a> vol options vf_wr01_oralogtemp_m_wr01 no_atime_update on toaster01a> vol options vf_wr01_oralogtemp_m_wr01 minra on toaster01a> vol create vf_wr01_oraredo_m_wr01 aggr_oramisc 4g toaster01a> qtree security /vol/vf_wr01_oraredo_m_wr01 unix toaster01a> qtree create /vol/vf_wr01_oraredo_m_wr01/log toaster01a> vol options vf_wr01_oraredo_m_wr01 nosnap on toaster01a> vol options vf_wr01_oraredo_m_wr01 nosnapdir on toaster01a> vol options vf_wr01_oraredo_m_wr01 no_atime_update on toaster01a> vol options vf_wr01_oraredo_m_wr01 minra on toaster01a> vol create vf_wr01_oractrl_m_wr01 aggr_oramisc 4g toaster01a> qtree security /vol/vf_wr01_oractrl_m_wr01 unix toaster01a> qtree create /vol/vf_wr01_oractrl_m_wr01/ctrl toaster01a> vol options vf_wr01_oractrl_m_wr01 nosnap on toaster01a> vol options vf_wr01_oractrl_m_wr01 nosnapdir on toaster01a> vol options vf_wr01_oractrl_m_wr01 no_atime_update on toaster01a> vol options vf_wr01_oractrl_m_wr01 minra on toaster01a> vol create vf_wr01_oracle_wr01 aggr_oramisc 32g toaster01a> qtree security /vol/vf_wr01_oracle_wr01 aggr_oramisc unix toaster01a> qtree create /vol/vf_wr01_oracle_wr01 aggr_oramisc/bin toaster01a> vol options vf_wr01_oracle_wr01 nosnap on toaster01a> vol options vf_wr01_oracle_wr01 nosnapdir on toaster01a> vol options vf_wr01_oracle_wr01 no_atime_update on toaster01a> vol options vf_wr01_oracle_wr01 minra on toaster01a> vol create vf_wr01_oradata_m_wr01 aggr_oradata 7689g toaster01a> qtree security /vol/vf_wr01_oradata_m_wr01 unix toaster01a> qtree create /vol/vf_wr01_oradata_m_wr01/data toaster01a> vol options vf_wr01_oradata_m_wr01 nosnap on toaster01a> vol options vf_wr01_oradata_m_wr01 nosnapdir on toaster01a> vol options vf_wr01_oradata_m_wr01 no_atime_update on
toaster02a> vol create vf_wr01dr_oraarch_m_wr01 aggr_oramisc 700g toaster02a> qtree security /vol/vf_wr01dr_oraarch_m_wr01 unix toaster02a> qtree create /vol/vf_wr01dr_oraarch_m_wr01/arch toaster02a> vol options vf_wr01dr_oraarch_m_wr01 nosnap on toaster02a> vol options vf_wr01dr_oraarch_m_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oraarch_m_wr01 no_atime_update on toaster02a> vol options vf_wr01dr_oraarch_m_wr01 minra on toaster02a> vol create vf_wr01dr_oralogtemp_m_wr01 aggr_oramisc 64g toaster02a> qtree security /vol/vf_wr01dr_oralogtemp_m_wr01 unix toaster02a> qtree create /vol/vf_wr01dr_oralogtemp_m_wr01/temp toaster02a> vol options vf_wr01dr_oralogtemp_m_wr01 nosnap on toaster02a> vol options vf_wr01dr_oralogtemp_m_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oralogtemp_m_wr01 no_atime_update on toaster02a> vol options vf_wr01dr_oralogtemp_m_wr01 minra on toaster02a> vol create vf_wr01dr_oraredo_m_wr01 aggr_oramisc 4g toaster02a> qtree security /vol/vf_wr01dr_oraredo_m_wr01 unix toaster02a> qtree create /vol/vf_wr01dr_oraredo_m_wr01/log toaster02a> vol options vf_wr01dr_oraredo_m_wr01 nosnap on toaster02a> vol options vf_wr01dr_oraredo_m_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oraredo_m_wr01 no_atime_update on toaster02a> vol options vf_wr01dr_oraredo_m_wr01 minra on toaster02a> vol create vf_wr01dr_oractrl_m_wr01 aggr_oramisc 4g toaster02a> qtree security /vol/vf_wr01dr_oractrl_m_wr01 unix toaster02a> qtree create /vol/vf_wr01dr_oractrl_m_wr01/ctrl toaster02a> vol options vf_wr01dr_oractrl_m_wr01 nosnap on toaster02a> vol options vf_wr01dr_oractrl_m_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oractrl_m_wr01 no_atime_update on toaster02a> vol options vf_wr01dr_oractrl_m_wr01 minra on toaster02a> vol create vf_wr01dr_oracle_wr01 aggr_oramisc 32g toaster02a> qtree security /vol/vf_wr01dr_oracle_wr01 aggr_oramisc unix toaster02a> qtree create /vol/vf_wr01dr_oracle_wr01 aggr_oramisc/bin toaster02a> vol options vf_wr01dr_oracle_wr01 nosnap on toaster02a> vol options vf_wr01dr_oracle_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oracle_wr01 no_atime_update on toaster02a> vol options vf_wr01dr_oracle_wr01 minra on toaster02a> vol create vf_wr01dr_oradata_m_wr01 aggr_oradata 7689g toaster02a> qtree security /vol/vf_wr01dr_oradata_m_wr01 unix toaster02a> qtree create /vol/vf_wr01dr_oradata_m_wr01/data toaster02a> vol options vf_wr01dr_oradata_m_wr01 nosnap on toaster02a> vol options vf_wr01dr_oradata_m_wr01 nosnapdir on toaster02a> vol options vf_wr01dr_oradata_m_wr01 no_atime_update on
Virtual interfaces and other networking configuration
As illustrated in picture below, from Part 2 of this series, we have two connections from each filer towards two Cisco Catalyst 6500 switches.
- Logical design for Primary and DR site
In fact, we have in each site, two controllers, that create a HA pair – cluster. Each controller have two 10Gb/s cards and these are connected to our storage switches. We will create a active/active virtual interface out of these interfaces. We need to create these interfaces on both nodes of a cluster. For storage network we will use 192.168.x.0/24 network subnet where x is a vlan ID. First 10 addresses (0-10) are dedicated for different network devices, gateways,.. IP address .11 will be our vFiler IP address and IP addresses 101-254 will be addresses for servers. We will also use Jumbo-frames (MTU 9000) on all interfaces to increase throughput and reduce latency and processing overhead.
toaster01a> vif create multi vif0 -b rr e1a e2a toaster01a> vlan create vif0 101 toaster01a> ipspace create wr01_ipspace vif0-101 toaster01a> ifconfig vif0-101 mtusize 9000 partner vif0-101 toaster01a> vlan add vif0 106 toaster01a> ifconfig vif0-106 192.168.106.11 netmask 255.255.255.0 mtusize 9000 partner vif0-106 toaster01b> vif create multi vif0 -b rr e1a e2a toaster01b> vlan create vif0 101 toaster01b> ipspace create wr01_ipspace vif0-101 toaster01b> ifconfig vif0-101 mtusize 9000 partner vif0-101 toaster01b> vlan add vif0 106 toaster01a> ifconfig vif0-106 192.168.106.12 netmask 255.255.255.0 mtusize 9000 partner vif0-106
toaster02a> vif create multi vif0 -b rr e1a e2a toaster02a> vlan create vif0 102 toaster02a> ipspace create wr01dr_ipspace vif0-102 toaster02a> ifconfig vif0-102 netmask 255.255.255.0 mtusize 9000 partner vif0-102 toaster02a> vlan add vif0 106 toaster02a> ifconfig vif0-106 192.168.106.13 netmask 255.255.255.0 mtusize 9000 partner vif0-106 toaster02b> vif create multi vif0 -b rr e1a e2a toaster02b> vlan create vif0 102 toaster02b> ipspace create wr01dr_ipspace vif0-102 toaster02b> ifconfig vif0-102 mtusize 9000 partner vif0-102 toaster02b> vlan add vif0 106 toaster02b> ifconfig vif0-106 192.168.106.14 netmask 255.255.255.0 mtusize 9000 partner vif0-106
To make this configuration persistent, we need to modify /etc/rc files on both cluster nodes and add following lines:
toaster01a
vif create multi vif0 -b rr e1a e2a # create a virtual active/active virtual interface vlan create vif0 101 # create NFS vlan ifconfig vif0-101 mtusize 9000 partner vif0-101 vlan add vif0 106 # create SnapMirror vlan ifconfig vif0-106 192.168.106.11 netmask 255.255.255.0 mtusize 9000 partner vif0-106
toaster01b
vif create multi vif0 -b rr e1a e2a # create a virtual active/active virtual interface vlan create vif0 101 # create NFS vlan ifconfig vif0-101 mtusize 9000 partner vif0-101 vlan add vif0 106 # create SnapMirror vlan ifconfig vif0-106 192.168.106.12 netmask 255.255.255.0 mtusize 9000 partner vif0-106
toaster02a
vif create multi vif0 -b rr e1a e2a # create a virtual active/active virtual interface vlan create vif0 102 # create NFS vlan ifconfig vif0-102 mtusize 9000 partner vif0-102 vlan add vif0 106 # create SnapMirror vlan ifconfig vif0-106 192.168.106.13 netmask 255.255.255.0 mtusize 9000 partner vif0-106
toaster02b
vif create multi vif0 -b rr e1a e2a # create a virtual active/active virtual interface vlan create vif0 102 # create NFS vlan ifconfig vif0-102 mtusize 9000 partner vif0-102 vlan add vif0 106 # create SnapMirror vlan ifconfig vif0-106 192.168.106.14 netmask 255.255.255.0 mtusize 9000 partner vif0-106
Flow-control consideration
Ethernet offers a flow control mechanism, PAUSE frames has been defined in IEEE standard 802.3x in 1997. In Ethernet Storage Best Practices NetApp recommends to either disable Ethernet flow-control (ifconfig <interface name> flowcontrol none) or to configure end devices to only send PAUSE frames and switches to only receive PAUSE frames. In our environment, we use second recommendation, Ethernet flow-control is enabled by default, both receive and send.
Since flow-control works on a link-to-link basis, we can not control it on a vlan tagged virtual interfaces, but we need to configure it on entire virtual interface.
toaster1a> ifconfig vif0 flowcontrol send toaster1b> ifconfig vif0 flowcontrol send toaster2a> ifconfig vif0 flowcontrol send toaster2b> ifconfig vif0 flowcontrol send
Also, to make a configuration persistent, we need to modify /etc/rc files. On a switch side we disable flow-control send and enable flow-control receive.
interface Port-channel1 description toaster1a vif0 switchport mode trunk flowcontrol receive on no cdp enable mtu 9198 spanning-tree guard loop spanning-tree portfast end interface Port-channel2 description toaster1b vif0 switchport mode trunk flowcontrol receive on no cdp enable mtu 9198 spanning-tree guard loop spanning-tree portfast end interface Port-channel1 description toaster2a vif0 switchport mode trunk flowcontrol receive on no cdp enable mtu 9198 spanning-tree guard loop spanning-tree portfast end interface Port-channel2 description toaster2b vif0 switchport mode trunk flowcontrol receive on no cdp enable mtu 9198 spanning-tree guard loop spanning-tree portfast end
vFilers
vFilers, are a logical containers for network IP addresses, network interfaces, volumes and QTrees. This functionality is enabled by multistore license and enables you to logically separate multiple virtual NAS storage containers on one physical device. Multistore technology is a key enabler for Secure Multi-Tenancy and works with IP based protocols – NFS, CIFS, HTTP, FTP and iSCSI.
We will have two vFilers, one in primary datacenter, second in disaster recovery datacenter.
toaster1a> vol create vf_wr01_root aggr_oramisc 100m toaster1a> vfiler create vf_wr01 -s wr01_ipspace -i 192.168.101.11 /vol/vf_wr01_root toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oraarch_m_wr01 toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oralogtemp_m_wr01 toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oraredo_m_wr01 toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oractrl_m_wr01 toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oracle_wr01 toaster1a> vfiler add vf_wr01 /vol/vf_wr01_oradata_m_wr01
toaster2a> vol create vf_wr01dr_root aggr_oramisc 100m toaster2a> vfiler create vf_wr01dr -s wr01dr_ipspace -i 192.168.102.11 /vol/vf_wr01dr_root toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oraarch_m_wr01 toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oralogtemp_m_wr01 toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oraredo_m_wr01 toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oractrl_m_wr01 toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oracle_wr01 toaster2a> vfiler add vf_wr01dr /vol/vf_wr01dr_oradata_m_wr01
Now we have a vFilers provisioned, later in this series, we will also change some options for a vFilers.
Next time
In next episode, we will export needed mount points and we will also look into what kernel and other parameters we need to tune on our Linux boxes for Oracle specific parameters as well as for NetApp/NFS specialties.
Discussion
No comments yet.