Exadata Redundancy

1) “External” redundancy diskgroups do not provide ASM mirroring.
2) “External” redundancy diskgroups rely on the redundancy provided outside Oracle (e.g. RAID 1, etc) at Storage level.
3) Therefore, on “External” redundancy diskgroups, if one disk member fails or is affected, then the diskgroup is dismounted or cannot be mounted back again, this behavior is expected, because an external redundancy diskgroup cannot tolerate the failure of any disk in the diskgroup.
4) Any kind of disk failure causes ASM to dismount the diskgroup or unable to mount it back.
5) For this reason, if the redundancy provided outside Oracle(ASM) is not offering the complete/correct coverage against corruptions or physical disk failures (expected redundancy), it is recommendable to implement/use "Normal" or "High" redundancy diskgroups instead, because ASM ("Normal" or "High" redundancy) provides redundancy with the use of failure groups, which are defined during diskgroup creation. The diskgroup type determines how ASM mirrors files. When you create a diskgroup, you indicate whether the diskgroup is a "Normal" redundancy diskgroup (2-way mirroring for most files by default) or a "High" redundancy diskgroup (3-way mirroring) or an "External" redundancy diskgroup (no mirroring by Oracle ASM).
6) Failgroups definition is specific to each storage setup, but you should follow these guidelines on "Normal" redundancy diskgroups:

a) All the failgroups (each failgroup), in the same diskgroup, must have the same number of disks to maintain the diskgroup redundancy (mirroring).

b) All the disks, in the same diskgroup, must have the same uniform size to guarantee the required redundancy as well.

c) All the disks, in the same diskgroup, must have the same I/O physical characteristics to provide a uniform I/O throughput among all the disks in the same diskgroup.

Note: Those rules must be followed when normal or high redundancy diskgroups are created and configured, otherwise it is a bad diskgroup design and ASM redundancy will not be guaranteed, thus ASM mirroring will be in risk and I/O performance will be compromised as well

Mirroring protects data by storing copies of data across different disks, providing access to user data in the event of a disk failure. When you create an ASM disk group, you specify a level of redundancy—this redundancy level dictates how many copies of data are maintained:

Normal redundancy = two-way mirror
High redundancy = three-way mirror
External redundancy = no ASM mirroring; ASM uses mirroring functionality in the storage array/subsystem, if available, to provide protection
With Exadata, Oracle protects and mirrors storage exclusively with ASM normal or high redundancy; there is no external redundancy alternative on Exadata.

The redundancy level controls how many disk failures are tolerated without ASM un-mounting the disk group or losing data.

When ASM allocates an extent for a mirrored file, it allocates a primary copy and one or two mirror copies. Oracle places the mirror copy on a disk that’s part of a different failure group. Failure groups are where mirror copies are stored. If you lost disks in either the primary location or failure group(s) (but not both or all), Oracle would continue to operate normally on the surviving copy.

ASM doesn’t mirror physical disks or LUNs like traditional RAID—it mirrors database extents. This is a very important design aspect of ASM—with normal or high redundancy, extents are mirrored on sectors from disks on one or more failure groups. These failure groups consist of disks different from the primary extents.

On Oracle Exadata, Oracle always places mirrored extents on grid disks located in a different storage server. This provides the flexibility that if you lost an entire storage server, you would still have access to either the primary or mirrored extents.