No Data Found Error with Oracle Logical Standby

New wiki entry posted for an issue that I encountered with replication stalling between the primary and logical standby databases that I am configuring using Oracle Database 10g Data Guard. There were errors stating that "no data found" when trying to complete the SQL Apply statements to the logical standby and I was able to reinstantiate the object having the issue (a table in this case) from the primary server to get the replication processing again. It took a bit of resource, but the procedure was clearly stated in the Oracle documentation, that I have been very pleased with thus far.

Oracle Database 10gR2 Data Guard Logical Standby Configuration

I've completed my "easier to follow" walkthrough of a logical standby database using Oracle 10g Data Guard on Solaris 10. I installed this configuration for a new Remedy ARS configuration that we are migrating to that will offer replication to a failover database instance that can be open for reporting at all times and used as a redundant server in the event that the primary fails.

Although it is a bit more work to create a logical standby (you have to configure it as a physical standby first and then convert to a logical standby), the benefit of having the standby database always open for read-only queries far outweighs the work involved. A physical standby database can be put in a read-only mode to be able to query, but this pauses all redo log application from the primary until it is taken out of read-only mode. Anyway, here is the link to the wiki article that contains this configuration and the steps to manually switchover the roles of the two servers: http://www.amcpu.org/wiki/index.php?title=Oracle_Database_10_Data_Guard_Logical_Standby

Using awk to transpose flat files

Here's a pretty nifty awk script that will take a flat file database that has entries segmented in stanzas rather than single lines. By stanzas I mean a group of lines, in which each entry is delimited by a blank line. Here's how it works:

Here is your flat file database:

Name:Tony Cesaro
Age:25
E-mail:acesaro@gmail.com

Name:Jim Jones
Age:22
E-mail:jim@gmail.com

Here is the file that you are trying to create:

Tony Cesaro,25,acesaro@gmail.com
Jim Jones,22,jim@gmail.com

Here is the script, utilizing the power of awk, showing how to achieve this:

#!/usr/bin/ksh

FFDB=/home/acesaro/people.txt
awk -F"\n" 'BEGIN { RS=""} {print $1" "$2" "$3" "$4}' $FFDB | awk '{print $2","$4","$6"}'

Now let's dissect this quickly:

Obvious, setting the variable:
FFDB=/home/acesaro/people.txt

Transpose the stanzas to single lines; the $1, $2, etc. are set to the values of each line in each stanza:
awk -F"\n" 'BEGIN { RS=""} {print $1" "$2" "$3" "$4}' $FFDB

And finally, use a typical awk line to only print out the values on each line that we want, using -f: to specify ":" as the delimiter and separating each variable with a comma to create a CSV:

awk -F: '{print $2","$4","$6"}'

This is a handy way to translate flat files between formats that previously seemed pretty difficult...to me at least. :)

I utilized this at work to take the flat file DBs that NerveCenter uses to store the nodes that it polls and their associated properties and convert them into a tab delimited lookup file for Netcool.

Netgroups with LDAP for NFS authorization

Since the success of our LDAP implementation for centralized user/password naming services, we have started to utilize LDAP for many more naming services such as hosts, netgroups, Sun Desktop Manager, and user authorization for various applications that support it. Netgroups have proven to be a simple, handy solution to authorizing access to systems and NFS shares, something that we really couldn't live without after transitioning from NIS+ to LDAP.

Here is an example implementation of utilizing a netgroup "nfs-clients" for authorization to NFS shares. Very simple and effective in only allowing those servers that you trust to mount up NFS shares from central fileservers. I'm quite sure there is some spoofing methods (including DNS cache poisoning) that could drastically affect the effectiveness of this method, but it's better and leaving an NFS share open to the [internal] world to mount up.

As most of my postings, the context of this setup is Sun Java Directory Server 6.x with Solaris-based LDAP clients performing queries against the directory. Small modifications can make this work on other *nix/LDAP platforms.

• If you don't already have it, create a new OU called ou=netgroup,dc=your,dc=domain.
• Create a new object under this new OU, with objectclasses of: top and nisNetgroup
• Configure the attributes of this nisNetgroup object with similar attributes as below, remembering that the nisNetgroupTriple attribute is multi-valued and will hold an entry for each host (must be listed as FQDNs!!!) that will be allowed to mount NFS shares that specify this netgroup in /etc/dfs/dfstab
  

  nisNetgroupTriple: (nfsclient1.domain.net,,)
  nisNetgroupTriple: (nfsclient2.domain.net,,)
  objectClass: nisNetgroup
  objectClass: top
  cn: nfs-clients

• 
  
• Modify the DUA profile that the LDAP clients are configured with to include a ServiceSearchDescriptor such as the following, to ensure that nsswitch looking in the right branch of the directory tree for the netgroup information:
  

  netgroup:ou=netgroup,dc=your,dc=domain?sub

• Now, modify your dfstab entry to specify options for the share(s) to reference netgroups to allow ro/rw access and mapping of the root user:
  

  share -F nfs -o ro=nfs-clients,root=nfs-clients /export/home

• Of course, run the shareall command to reshare this NFS export with the updated options, and it would probably be a good idea to restart nscd (name service cache daemon) as well.
• On your NFS/LDAP clients, make sure that the value for the netgroup lookup is set to LDAP:
  

  # grep netgroup /etc/nsswitch.conf
  netgroup: ldap
  

• Mount it up on one of the servers specified in the netgroup, nfs-clients, and attempt to mount it on a server not listed in that netgroup to validate that authorization control is working properly.
  

I'll make another post that is VERY similar to this one explaining how to set up host lookups in LDAP also. This is handy if you want to store some host/ip address mapping records in LDAP versus DNS (believe me, there are situations that call for this). I would suggest sticking with DNS for this purpose as much as possible, as it is optimized for host lookups, but like I said there are situations where this is helpful.

Common tuning tips and pitfalls to avoid

One major aspect of setting up systems that is very much neglected in the practice of system administration is testing and tuning (and testing again and tuning more...). It is a shame that I don't get the chance to test as much as I would like to before deploying a system to the production environment, but I have learned a few things through reactive actions that I would like to share to hopefully prevent myself and others from running into such issues after a system is deployed.
If you're in a similar situation as me, without the abundance of human resources that are allocated to specific roles (i.e. architecture, capacity planning, engineering, lab/testing, etc.) and have to handle all of these necessary capacities yourself, take the following into consideration when going through your "checklist" to ensure you've "dotted your i's and crossed your t's."

• Make sure that your network interfaces are set to full duplex and the link speed that you intended to be set at. This sounds so simple, but is one of those configurations that seems to come back to bite me every once in a while. In Solaris (10+) this can be quickly checked by running: dladm showdev or kstat bge:0 (example)
  
• Check the max file descriptors per process (and total) setting for the user that your services run under. This is an OS level tuning parameter that defines how many files/ports/pipes/etc. a given process can open and usually defaults at 256, but should probably be increased. At some point, this has an impact on the system by increasing memory utilization, so caution should be taken when setting this high. A safe value to stay under is 65535. I've seen this the root cause of far too many service impacting outages and should be prevented as much as possible.
• Validate that your database has enough tablespace to grow as needed and make sure it is set to auto-extend if desired. Also, make sure that archive log files are handled effectively by removing them after backing them up (with RMAN in the Oracle world) and do not utilize more local disk space than you feel they should.
• Choose the right processor architecture for the software that will run on it, and load test as much (and as brutal) as possible to ensure stability at higher than expected production loads. As an example, there are software packages that are highly multi-threaded, but choke on a extremely multi-threaded processors (such as the Sun Niagra [T1/T2] series) due to lack of highly-depended on resources such as FPUs.
• more to come in future posts...
  

Using idsconfig with Sun DSEE 6.0

Despite the awesomely advanced management interface to Sun DSEE 6.0, I have found that the handy /usr/lib/ldap/idsconfig command that actually initializes the directory with the correct OUs, searchServiceDescriptors, and profiles that the Solaris ldapclient command uses to initialize the OS to use LDAP as a naming service. Also, the idsconfig command is no longer even mentioned in the DSEE 6.0 installation guide nor the administration guide (the logical place to describe this very helpful tool).

So, through a bit more of digging I happened to come across the command and a description of how to modify it to work with DSEE 6.0 in the Sun Java System Identity Synchronization for Windows 6.0 Deployment Planning Guide in Appendix A. (Pluggable Authentication Modules), of all places! Anyway, here an excerpt from that guide that will get you on your way with the idsconfig command:

For PAM to work with Directory Server 6.0, you must edit the /usr/lib/ldap/idsconfig script and change 5 to 6 in the following code:

if [ "${IDS_MAJVER}" != "5" ]; then

While executing the idsconfig command-line tool, you need to know which values to assign to the various configuration parameters. If you do not know, use the default values when prompted (other than the configuration parameters 1, 2, and 4).

Finally, I was able to initialize the directory as I was able to with previous versions of the Sun Directory Server. Especially helpful are the profiles that the idsconfig command creates in the directory under cn=profilename,ou=profiles,dc=yourdomain,dc=net which allows the LDAP clients to simply download the profile with all of the configuration options preset and simply copy them locally to become initialized. This saves a lot of time when converting a bunch of systems from file or nis for naming services in the nsswitch.conf file.

For a complete how-to of initializing the Sun Directory Server, please reference my wiki page: http://www.amcpu.org/wiki/index.php?title=SJSDS_Installation_Guide.

Also, for a walkthrough of configuring Solaris LDAP clients, check out this wiki entry: http://www.amcpu.org/wiki/index.php?title=SJSDS_Configure_LDAP_Clients.

Sun Directory Server EE 6.0 Rules!

I've been tinkering with Sun's latest LDAP directory server, and I must say that I am very impressed. The time to get up and running with an LDAP server that has self-signed SSL certificates, replication with neighbor servers, and an awesome web management interface is minimal compared to previous versions. The server comes packages with the Java Enterprise System Identity Management Suite and can be configured during the installation or afterwards. I like to have full control over the configuration, so I select to "Configure Later" which allows me to run the DS Control Center setup script later on.

I know that there are a lot more exciting blog postings on the Sun Blogs, but I wanted to show my appreciation for such a great improvement in Sun's LDAP offering. You'll see many more postings along this topic as I go through the steps of rolling out the DSEE 6.0 along with Sun Access Manager 7.1 (authentication services / SSO) and Sun Identity Manager 7.0 (user provisioning and access control).