Error in Agent trying to install Grid Control

An error showed when I tried to install Oracle Grid Control 1o.2.0 on a Windows 2003 SE x32. The error was pretty much the same as the error shown with the regular Enterprise Manager Console on this platform, it has to do with the time zone issue I have already reported some time ago in this blog.

Sometimes because of the timeout involved during the installation process, it may happen that after a second attempt the Agent can be installed, afterwards then take a look at the log files, there may be some major issue that requires special attention.

I took a look at the \log directory and looked for the emdctl.trc file, there I found the following lines:

2010-02-19 15:30:52 Thread-1744 ERROR main: nmectla_agentctl: Error connecting to https://tango.oracle.com:3872/emd/main/. Returning status code 1
2010-02-19 15:30:53 Thread-2872 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset =-360,and testTZoffset for GMT:0 do not match
2010-02-19 15:30:54 Thread-2872 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset =-360,and testTZoffset for GMT:0 do not match
2010-02-19 15:49:34 Thread-2552 ERROR main: nmectla_agentctl: Error connecting to https://tango.oracle.com:3872/emd/main/. Returning status code 1
2010-02-19 15:49:44 Thread-2660 WARN http: snmehl_connect: connect failed to (tango.oracle.com:3872): No connection could be made because the target machine actively refused it. (error = 10061)


By the way, tango.oracle.com is a fictitious server and does not have anything to do with Oracle corp.

At the log file a particular error came to may attention, the TZ Error. This one has to do with the time zone changes that took place some time back. I will apply the latest patchset on top of it, so I am not too much concerned about fixing it at the time, but since I want to have a 'clean' install I worked around this by commenting the line found at the \config\emd.properties file (a routine backup is highly advisable) and commented the last line. I changed the time zone in the windows machine and by the time the assistant is re-run it takes the right Time Zone.

emd.properties

###HRM: agentTZRegion=GMT
agentTZRegion=America/Chicago

That's it, my installation took place and I can proceed with the next tasks.

My recent experience with VMWare

VMware ate my four CPU Cores

Today I noticed the high amount of CPU consumed by the vmware-authd.exe process. It raised the CPU consumption to 100% and leaves no CPU resources to any other process in the system.

The environment is a Windows 7 professional 64 bits with 4GB Ram and 4 cores (all of them at the top 100%).

The most recent post I found so far by googling is from Oct 19th 2009, and the author of the post states that a bug was filed (483679).

vmware-authd.exe is the executable for the VMware Authorization and Authentication Service for starting and accessing virtual machines. This process is required if you are not logged in with administrative privileges (which by the way is my case).

I shuted down this service. In the VMWare 2.0 edition the VMWare Auth. service is dependent from the VMWare Host Agent which provides remote command and administrative control over this VMWare Server host. After shutting down both processes the CPU monitor was back to normal.

This process is required in the VMWare Server 2.0 version, otherwise it will no be possible to launch the console. By shutting down and restarting the services the problem seems to be fixed. Most probably it has to do with a bug.

On the other hand, I don't have anything against the new Tomcat based console, but I miss the regular windows based console.


Host Unreachable

When trying to connect to my virtual machines through the network, they replied with a 'host unreachable' error, among other unpleasant related network errors.

The current environment I have:

• DHCP on host real network adapter
• Loopback adapter on real host
• A fixed IP address through a bridged virtual network adapter
  

In this scenario the loopback adapter was required since I am installing an Oracle 11g Rel. 1 / 2 on this platform. The virtual machines were not visible or either were partially and intermittently visible.

I put the virtual network adapter to have a dynamic address provided by the DHCP server and it began to work. However I still need a fixed IP Address at the virtual machine since I am installing Oracle RDBMS Oracle 11g Rel 1 on top of it. So I installed a Loopback adapter inside the Virtual Machine.

I went through several Google references and most of them talked about de-installing and re-installing the VM protocol from the real network adapter, but it only lead me in this particular case to a waste of time and a server reboot. So far, everything is properly working.

When I pinged the host server I noticed a lot of time waiting for a reply, and when I tried to access a shared path from it my local machine replied with a timeout. Two issues were involved here, first the user at the host server is a domain user, not a local user, and the second one, it has to solve the hostname first, which took too long, so I added the address from the host to the hosts file at the virtual machine.

Network Communication to the VM was Deeeeeeadly Slooooooooooow

It is not enough eh!, well I tried to perfom file transfer from my host to the virtual machine, it happened that the performance was around 20Kb/s. so figure out what it was to transfer the Oracle XE installer executable (200,000 Kb), around 2:45 hrs to transfer the whole file, I don't want to image how long it would last to transfer the more than 1G file to install Ora11gR1 to the VM. Google you are the Geek's Nirvana, after googling a while I found a reference that stated some network parameters had to be configured, it was the advanced properties for the network adapter at the physical host.


At the advanced properties of the network adapter ther is one named Large Send Offload v2 (IPv4), this must turned off (disabled), it boosted the network performance to the Virtual Machine. A definition of what this parameter does can be found at this Microsoft Tech Note.

"DisableTaskOffload
Disables offloading of processor tasks to the network adapter. Offloading is designed to optimize performance of Windows 2000.

Network Driver Interface Specification (NDIS) 5.0 lets TCP take full advantage of intelligence in network adapters by letting the adapter do some of the tasks that the processor normally performs. Offloading these tasks to the network adapter leaves the processor free for tasks that only it can perform."

Addendum
A document that may help with some additional performance hints can be found here "VMWare ESX Server Performance Tuning Best Practices".

I go back to the VMWare, this VMWare session was very instructional for me.

Performance Problems with my Workstation

My computer was deadly slow, several processes took over the computer resources, such as CPU, which almost always was permanently at 100%, IO resources, several processes were reading several Gb's of data, (what for?), it was a nightmare. The bigger the computers grow the slower they can become. I have a Laptop I use as a server with several VM Machines running when required which serve as test environments. I use Windows XP, yeap, I personally don't like Vista and if possible I avoided new releases, except for Windows 2003 and Windows 2008.

One of the resource consuming processes was the jqs.exe, a process that runs in the background with low priority but which can lead your computer to IO starvation. JQS is an acronym after Java Quick Starter, it is used to speed up the launch of java applets, but if you are not launching Java applets too much frequently it is not worth keeping it running. Now, if you want to disable the Java Quick Starter process you can do that in the Windows Control Panel. You find a Java entry there which will open the Java Control Panel. A click on Advanced and the selection of Miscellaneous will display the activated Java Quick Starter entry. Uncheck the box to disable the process. You have gotten rid of the jqs.exe process. Hasta la vista baby! ...

CCSCHST.EXE is another heavy weight process I want to get rid of, this process creates another IO and CPU bottleneck. I have Norton Antivirus, this was not a good choice for an anti-vrus, it deadly slowed down the computer performance. It is being said that it has to do with a conflict between the Microsoft update process and Norton.

I disabled the automatic windows update feature from the control panel. A red alert is displayed, this let me remember that I have to manually connect to the Microsoft site to perform the updates. Here it is an article that describes what is going on with this issue Norton Vs. Microsoft . I disabled the NAV security features so that Norton does not try to start the live update on its own. I choosed to manually launch the updates.
In the case of Norton AntiVirus I had to leave several features to the minimum, or even disable them from the NAV control panel.

Another nice application that consumes a lot of resources, particularly during system startup is the Weather Channel desktop, it is a very useful and nice application, particularly considering these unpredictable winter weather, but neither my machine nor my work are willing to go through the additional overhead required to startup this application at windows start time. So I disabled it from starting at windows start time. I'll go through the Weather channel web page to keep updated about the weather.

I also have the Virtual Machine server from VMWare, it requires several resources during startup, but this is one of the application I use the most, so there is not too many options but to let it be.

And finally firefox, I love FireFox, but I do not know what happened to the latest releases, particularly 3.X. There have been several upgrades, it added new features, and it is pretty nice, but it requires more resources. Either way after getting rid of all of the ballast my machine had to carry on during startup and production time, there are enough free system resources for fireFox, so this is definitely out of my personal deinstallation list.

The Oracle Open World 2009 is just a few weeks ahead

The last OOW2008 was for me the first time I visited this event. This is the time to meet partners, colleagues and friends. There are tons of information, a lot of activity during the day evening and night. It all starts early, between 8:00 and 9:00 a.m., depending on the scheduled activities and the interest in key conferences or other early activities. If you have already confirmed your assistance make sure you have accessed the scheduler builder tool. By now most of the hands-on workshops have no more seats available, this is the first activity that is in high demand. There are lots of conferences and most of them are overlapped, so it may be difficult to be at two or more places at the same time unless you develop the ubiquity ability.
The schedule builder is a very important tool during the event. This allows you to better plan ahead how your time will be distributed, when you are going to take breaks, when you are meeting friends and partners, visit the exhibition hall, assist to the demo grounds or perform other relevant activities.
The last time I was there I didn't carry my iPod with me and my cell phone didn't have wireless capabilities. Never do that. It is very useful to have your schedule on your handheld device and it is important to have internet, it becomes a very useful mean of communication. There are computers where you can quickly access your email and internet, but since they are public, it becomes difficult to remain more than ten minutes when several folks are waiting in line. I suggest you to take your portable wireless device with you.
Last year the electronic presentations were not available but for those who buy them. There is a CDRom you can buy. So if you are interested you can get them from Oracle.
This year I will be signing my book, the scheduled time slot will be on Tuesday 13th at 16:30 hrs at the library. I hope to see you there.
This is just the beginning, there will be a lot of things more to comment about this event.

The Oracle ACE

What is the Oracle ACE Award?

"Oracle ACEs and Oracle ACE Directors are known for their strong credentials as Oracle community enthusiasts and advocates, with candidates nominated by anyone in the Oracle Technology and Applications communities. The baseline requirements are the same for both designations; however, Oracle ACE Directors work more closely and formally with Oracle in terms of their community activity."



Ref. Oracle ACE Program

Andrew Clarke mentioned in his blog how things have changed since he was first nominated to receive this Award back in 2003: "The bar has certainly been raised since I was nominated. If I look at some of the people who are now ACEs I think I must have walked into the wrong club." He also mentions it is something that happens sometime along the way, and certainly it is, it happens when you least expect it. You cannot nominate yourself, "someone else spotted your efforts" and it is just a way to give you a Big Thank You for providing meaningful and valuable contributions to the Oracle Community.

I have been in the Oracle community for 20 years so far, but it was not because of those 20 years I received this Award, but specifically because of the last five years, I have shared knowledge with the community in several ways, by participating in the Oracle Forums, sharing knowledge by writing an Oracle related blog, and by writing an Oracle book.

In this Universe, whatever you do, right or wrong follows the third Newton's Law "For every action there is an equal and opposite reaction." and this ACE Award is the practical manifestation of this law.

I just want to say a Big Thank You to the Oracle Community.

What does it feel to be twenty?

One day is a very long time if you have to wait for something to happen. Twenty years are as fast as a wink of an eye if you keep on making things happen.

The 80's, the Glowing Decade

Back in the 80's talking about mobile phones was not popular at all. The cassio fx100 was my programmable scientific calculator used to solve integrals by means of numerical methods. In the music, Madonna, Billy Joel, Elton John, Cindy Lauper, Aerosmith, Brian Adams, Dire Straits, Duran Duran, Eurythmics, Peter Gabriel, Huey Lewis & the news, Robert Palmer, Bruce Springsteen, Survivor, Tears for Fears, Van Halen, ZZ Top, only to name a few, were the singers we used to hear with the walkMan (some sort of iPod of the 80's) wich was our preferred music gadget to play audio tapes.
At that time the fluorescent pink, yellow, green and blue were the fashion colors found in every guard robe. People dressed in those colors easily glowed in the night with the Discotheque lights. These noisy colors still survive today only as text markers.
At that time Java was an Island found in the Pacific Ocean and Cobol was still the programming language for the business applications. Meanwhile Pascal was an emerging structured programming language which mostly remained in the research labs. Most of the people related to informatics at the time felt in love with Pascal but ended married with C Language until the word Java changed its widely known meaning to the programming language for the internet.

20 Years of professional Career devoted to Oracle

Oracle has been evolving and it is as challenging, aggressive and innovative as the first time I met it. It was Oracle version 5.0 the first Oracle version I used to issue my first SQL statements. 16 Mb RAM, two hard disks of 512 Mb and 320Mb inside an HP9000 runnig HPUX 9.0 were the leading edge technology standards capable of carrying the data payload required to move the Metropolitan University data. Today this amount of computer power is hilarious, but if compared with the average PC of the time (640K RAM + Extended memory + 40Mb HD + 256 Colors monitor) this was a powerful Server.
My professional career has evolved side by side with Oracle. I have run at the same pace Oracle runs, stuck to it as its shadow. During this time I have seen a lot of people come and go to the Oracle community, and very few of them still join me today. Even though very few things have changed. The two main Kernel directives were not very different from today's kernel directives,
I. High Availability
II. High Performance
Oracle is conceived as a highly fault tolerate rdbms, whose memory and physical structures provide a good balance between recoverability and performance. I have met all the major and minor releases since then up to the latest available 11g release today and I can say that even though Oracle's main directives are the same, each day it becomes more robust and more complex.
It's been a long journey that has run at a very fast pace. I have met a lot of people, I have met a lot of places. And at the end here I am still working with Oracle. I enjoy this professional activity as much as the first time I met Oracle, it's been some sort of falling in love at the first sight that has ended in a marriage so fresh and new as the first time.
This is the train of a never ending trip where a lot of people climb up and others descend from it, but the most important part of it is the trip itself, not the destination.

ORA-00600: [LibraryCacheNotEmptyOnClose]

Recently a database in 9iR2 (9.2.0.8.0) was migrated to 10gR2 (10.2.0.4.0), this database was already having a hard time with memory issues and frequent ORA-04030 errors. At the time the workaround for this issue was to increase the value of the PGA_AGGREGATE_TARGET instance parameter. Since 10g consumes more resources than 9i this error re-appeared and the local operators found easy to rebounce the oracle instance when this problem was more frequently shown. However, after the migration to 10gR2 this procedures throwed the above mentioned ORA-00600 error.

Googling around I found that this errors is due to a bug (Bug 4483084 OERI[LibraryCacheNotEmptyOnClose] on shutdown), and it is harmless, so this can be ignored. There are people who have never seen this error but they started facing it after migration to 64 bit platforms.

According to Shervin Sheidaei, he stated that purging the memory region prior to the shutdown process using a shutdown event trigger the problem can be worked around.

Code taken from his blog:
CREATE or replace TRIGGER flush_shared_pool
BEFORE SHUTDOWN ON DATABASE
BEGIN
execute immediate 'ALTER SYSTEM FLUSH SHARED_POOL';
EXCEPTION
WHEN OTHERS THEN
RAISE_APPLICATION_ERROR (num => -20000, msg => 'Error flushing pool');
END;

Packt Publishing Press Release

Oracle 10g/11g Data and Database Management Utilities is a new book from Packt that helps Oracle professionals improve performance and manageability using the advanced features of Oracle Utilities. Written by Hector R. Madrid, a certified DBA and Java curriculum instructor for the Oracle University, this book helps users to master the utilities to optimize the efficiency, management, and performance of their daily database tasks.

Database interaction is a part of the daily routine for all database professionals, and by using Oracle Utility tools such as Oracle Data Pump and Oracle SQL*Loader, users can benefit from improved maintenance windows related to data management tasks, optimized backups, faster data transfers, and more reliable security, thereby allowing them to do more with the same amount of time and resources.

Using the advanced features of direct export or import utilities, readers will learn to improve performance and manageability of different databases. Readers will make use of the Oracle Scheduler to specify maintenance windows, assign priorities, configure job classes and many more features. With the help of the Oracle Universal Installer tool, the installation tasks becomes more efficient, and allows users to execute effective default and comprehensive database creations.

Readers will also learn about the Oracle Wallet Manager and how it is used to increase the security in an Oracle environment, protect the backups, and manage its certificates. Using OPatch, users will be able to manage software updates related to Critical Patch Updates (CPU) or individual patches and perform installations in batch environments. They will also learn to use DBCA to create and configure a database and manage it in an ASM environment.

Oracle professionals interested in working with several powerful tools to help improve performance and manageability of utilities among different databases will find this book useful. This book is out now and is available from Packt. Fore more information, please visit http://www.packtpub.com/oracle-10g-11g-data-and-database-management-utilities/book

Does your database look complicated? Are you finding it difficult to interact with it? Database interaction is a part of the daily routine for all database professionals. Using Oracle Utilities the user can benefit from improved maintenance windows, optimized backups, faster data transfers, and more reliable security and in general can do more with the same time and resources.

"Oracle 10g/11g Data and Database Management Utilities" is written using a practical approach that guides you through different practical scenarios. It provides a brief introduction to the topics; this way you can quickly get to know the main features, start being productive with the tool, and grow with it at a fast pace. If you wish to read through the book extract, please feel free to click here: Book Extract

My recently published Book

"Oracle 10g/11g Data and Database Management Utilities" is the tittle of my recently released book that was published by Packt Publishing. This book represents for me my debut as an Oracle Author. This book was written after all the cases I have faced for already 20 years of professional carreer as an Oracle Professional.

There are times when a process doesn't seem to end, and when it is about to reach the final stages something happens and it breaks up. If we had a tool to monitor and at lest stop the process before this fatal ending it would be great. The tool actually exists, but the problem with several DBA's or DBO's is that they don't know this tool exist and even if some of them know this exists, very few of them know how to master it so they can use it in a practical environment.

This is not the only case I expose in my book, there are other scenarios that relate to tools such as DataPump. Today there are a lot, of people out there that still talk about the traditional exp/imp. Data Pump is a very old tool, considering this was first released in 2003 along with the 10g Rel. 1 RDBMS, and since the technology gets obsolete in five years (avg.) this can be considered that Data Pump has already had three evolution stages (10gR1, 10gR2, 11gR1 and very soon 11gR2). What about exp/imp, it is more than an obsoleted tool that has already given what it has to give and is maintained only for upward compatibility.

Another tool has to do with sql*loader, you may say it is not new and you are right, but how about using it to load CLOB, BLOB files? Can this be used to create External Tables and have access to log files so they can be analyzed at the database level by means of SQL commands? The answer is yes, and SQL*Loader can be used in a very creative way. You can get a network performance meter by means of a sqlloader exeternal table, you can have another view of the alert log and you can query it directly from the database.

There are several topics I address in the book, but overall I do a practical approach, I introduce practical examples that can help the DBA in the day by day tasks so the DBA can perform its dutys in a more efficient way.

Yet another CRS-0223: Resource 'ResourceName' has placement error

Recently I faced this error. The scenario took place after a full RAC reboot process. The difference with error I had already documented here was that it was useless to restart each RAC service. Either way it was sending the placement error for all RAC related services. In this case I took a look at the Metalink Note: 726925.1 "srvctl start instance fails with PRKP-1001; srvctl trace shows error connecting to CRSD".

First I checked the alert.log file related to the cluster operation. It reported nothing unusual, everything seemed to be normal.
Next I took a look at the output of the command:

crsctl check crs

This command reported everything was working just as normal. Definitely it had to do with the syncronization at startup time. It is pretty weired since a normal node reboot should not lead to such inconsistency. I must point out that the environment used was 10gR2 (10.2.0.1.0) on RHEL4 (Red Hat Enterprise Linux AS release 4 (Nahant Update 3) 2.6.16 xenU (32-bit)), this was a scenario faced while I was teaching the RAC 10g course for Oracle. Since this environment is not patched at the start of the course, I would not be surprised to find out that this is due to an already filed bug.

The procedure was to kill (as root) all crsd.bin process on all participating nodes, then a simple crs_stop -all / crs_start -all was just enough to put everything back to normal.