Tuesday, July 29, 2008

ORA-06544 when performing network expdp between 11g and 10g

When I was trying to perform a network export datapump between 11g and 10g I found an ORA-06544 error.

[oracle@alpha admin]$ impdp SCOTT/TIGER tables=emp_10g,dept_10g network_link=db10g

Import: Release 11.1.0.6.0 - Production on Tuesday, 29 July, 2008 22:37:01

Copyright (c) 2003, 2007, Oracle. All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-39006: internal error
ORA-39113: Unable to determine database version
ORA-04052: error occurred when looking up remote object SYS.DBMS_UTILITY@DB10G
ORA-00604: error occurred at recursive SQL level 3
ORA-06544: PL/SQL: internal error, arguments: [55916], [], [], [], [], [], [], []
ORA-06553: PLS-801: internal error [55916]
ORA-02063: preceding 2 lines from DB10G

ORA-39097: Data Pump job encountered unexpected error -4052


This error is an interoperability error due to the bug 4511371, which is fixed by applying the 10.1.0.5.0 patchset for 10gR1 and 10.2.0.2.0 patchset for 10gR2.

Ref. Metalink note 4511371.8

Thursday, July 24, 2008

ORA-39001 When executing Export DataPump on RAC

While I was performing a data pump test export from one of the demo accounts a very weired error showed up. I have looked for it at the web, metalink, forums, and it seems to be the first case.

Scenario
A simple expdp proces trying to export whatever data shows up this:

-bash-3.00$ expdp sh/sh dumpfile=shSchema directory='default_dp_dest' schemas=sh
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORA-39001: invalid argument value
ORA-39000: bad dump file specification
ORA-31641: unable to create dump file "/home/oracle/default_dp_dest/shSchema.dmp"
ORA-27041: unable to open file
Linux Error: 22: Invalid argument
Additional information: 2


I have already made some expdp testing, and it has worked, the only difference during this testing is that it was made on a RAC environment.

When taking a look at the background dump dest trace files I found this:

-bash-3.00$ more rdbkdi1_dm00_31645.trc
/u01/app/admin/RDBKDI/bdump/rdbkdi1_dm00_31645.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name: Linux
Node name: eg6886
Release:
2.6.16-xenU
Version: #2 SMP Tue Oct 24 16:33:50 CDT 2006
Machine: i686
Instance name: RDBKDI1
Redo thread mounted by this instance: 1
Oracle process number: 39
Unix process pid: 31645, image: oracle@eg6886 (DM00)
*** ACTION NAME:(EXPHR01) 2008-07-24 19:38:22.781
*** MODULE NAME:(Data Pump Master) 2008-07-24 19:38:22.781
*** SERVICE NAME:(SYS$USERS) 2008-07-24 19:38:22.781
*** SESSION ID:(125.291) 2008-07-24 19:38:22.781
kswsdlaqsub: unexpected exception err=604, err2=24010
kswsdlaqsub: unexpected exception err=604, err2=24010


If ever someone finds this error solution or reference to it I'll be thankful to hear about it.

Tuesday, July 22, 2008

OMS version not checked yet ...

When a grid control agent was installed during a RAC setup an error showed up at the end of the agent configuration assistant.

The agent was started successfully, but it was not possible to force the agent to have the XML files uploaded to the grid server, it was when the dreaded error "EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet.." showed up.

The problem basically was due to a password typing error during the configuration phase.


The Troubleshooting Procedure Outline was:

Setup the Agent Home environment
export AGENT_HOME=/u01/app/oracle/pruduct/10.2.0/agent10g
export PATH=$AGENT_HOME/bin:$PATH


Startup the Agent
emctl start agent

Clear the Agent status
emctl clearstate agent

Reset Credentials
emctl secure agent

Retry and verify synchronization operations
emctl upload agent
emctl status agent


###
### EM Agent Troubleshooting log
###

$ export AGENT_HOME=/u01/app/oracle/product/10.2.0/agent10g
$ export PATH=$AGENT_HOME/bin:$PATH

$ # Start agent
$ emctl start agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Starting agent ......... started.
$
$
$ # Trying to force XML files uploading
mct$ emctl upload agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..


$ # Clear EM Agent status
$ emctl clearstate agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
EMD clearstate completed successfully


$ # Configuring OMS Credentials
$ emctl secure agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Enter Agent Registration password : *******
Agent successfully stopped... Done.
Securing agent... Started.
Requesting an HTTPS Upload URL from the OMS... Done.
Requesting an Oracle Wallet and Agent Key from the OMS... Done.
Check if HTTPS Upload URL is accessible from the agent... Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode... Done.
EMD_URL set in /u01/app/oracle/product/10.2.0/agent10g/eg6881.us.oracle.com/sysm
an/config/emd.properties
Securing agent... Successful.
Agent successfully restarted... Done.


$ # Retrying operations
$ emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 10.2.0.1.0
OMS Version : 10.2.0.1.0
Protocol Version : 10.2.0.0.0
Agent Home : /u01/app/oracle/product/10.2.0/agent10g/eg6881.us.oracle.com
Agent binaries : /u01/app/oracle/product/10.2.0/agent10g
Agent Process ID : 5307
Parent Process ID : 5260
Agent URL : https://eg6881.us.oracle.com:3872/emd/main
Repository URL : https://eg6876.us.oracle.com:1159/em/upload
Started at : 2008-07-22 15:40:37
Started by user : oracle
Last Reload : 2008-07-22 15:40:37
Last successful upload : 2008-07-22 15:40:58
Total Megabytes of XML files uploaded so far : 0.79
Number of XML files pending upload : 18
Size of XML files pending upload(MB) : 1.86
Available disk space on upload filesystem : 27.54%
Last successful heartbeat to OMS : 2008-07-22 15:40:43
---------------------------------------------------------------
Agent is Running and Ready


$ emctl upload agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully


$ emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 10.2.0.1.0
OMS Version : 10.2.0.1.0
Protocol Version : 10.2.0.0.0
Agent Home : /u01/app/oracle/product/10.2.0/agent10g/eg6881.us.oracle.com
Agent binaries : /u01/app/oracle/product/10.2.0/agent10g
Agent Process ID : 5307
Parent Process ID : 5260
Agent URL : https://eg6881.us.oracle.com:3872/emd/main
Repository URL : https://eg6876.us.oracle.com:1159/em/upload
Started at : 2008-07-22 15:40:37
Started by user : oracle
Last Reload : 2008-07-22 15:40:37
Last successful upload : 2008-07-22 15:41:13
Total Megabytes of XML files uploaded so far : 2.65
Number of XML files pending upload : 1
Size of XML files pending upload(MB) : 0.03
Available disk space on upload filesystem : 27.54%
Last successful heartbeat to OMS : 2008-07-22 15:40:43
---------------------------------------------------------------
Agent is Running and Ready
$

Friday, May 23, 2008

Blog not properly displayed in Firefox

Recently, without a warning, my blog site stopped being properly displayed, when drilling down on the circumstances that prevented it from properly displaying at least, so far, I have found that when it is displayed with Firefox browsers the left navigation bar 'disappears'. If displayed with Internet Explorer it is properly displayed.

I had not realized about this fact, but when Eduardo Legatti asked me what happened with the left navigation bar it was when I entered as a regular non authenticated user, I saw something happened with my Blog Archive and my Topics menus, those were gone. Eduardo told me he uses also Firefox.

In my personal opinion my default choice is Firefox, just like 24% of the blog users, not Internet Explorer, unless due to VPN restrictions or bank access requirement. I have tried to change my layout, and unsuccessfully it behaves the same. So the workaround in the mean time is to click on the item menu or use Internet Explorer :(

Some people have reported differences between firefox and internet explorer [Resolving Browser Display Discrepancies Between IE and Firefox].

... After a couple of hours after publishing this post the left navigation bar is back again, it looks like there was some issue wiht the cache. I have removed the cache from firefox, but I am not sure this was the actual solution.

Thursday, May 22, 2008

Missing libraries when installing EM Grid Control

When I was trying to install and setup an Enterprise Manager Grid Control 10g R2 (10.2.0.1.0) on an Oracle Enterprise Linux 4 (OEL4) platform, I surprisingly found I didn't meet the package list install requirement, it stated there were three libraries missing, I did install and reinstall them from the OEL4 disks, unsuccessfully, since the error still persist. I should have STWF (Searched The Web First) for further cases where something curious like this had happened to somebody( Linux OEM Grid Control 10.2.0.1.0 installation issue), and yes, it happened to this individual who was very kind in documenting this at the Oracle forums.

The error is found with these three missing RPM's:
  • compat-libstdc++-296-2.96-132.7.2 (failed)
  • libstdc++devel-3.4.3-22.1 (failed)
  • openmotif-21-2.1.30-11 (failed)
It can be seen that openmotif-21 is misspelled, it should read openmotif21, without the hyphen.
I did further search at Metalink [Note:343364.1, Prerequisites and Install Information for EM 10g Grid Control Components on Red Hat EL 4.0 Update 2 (and higher updates) Platforms] and I found that when installing EM Grid Control with a fresh database on 10g, it always shows this library error. Experience always comes when it is no longer required. The installer has a flaw, so it will show an error even if the three missing libraries are installed on the system, it is because a bad list of RPM's in the installer. This error can be user checked and proceed with installation, it will succeed. Just, this error is ever found, I suggest to proceed with caution, and make sure the list of RPM's meet the minimum requirements, if this manual check is successful, the missing RPM's error can be skipped.

rpm -q glibc glibc-2.3.4-2.13
rpm -q make make-3.80-5
rpm -q binutils binutils-2.15.92.0.2-15
rpm -q gcc gcc-3.4.4-2
rpm -q libaio libaio-0.3.103-3
rpm -q glibc-common glibc-common-2.3.4-2.13
rpm -q setarch setarch-1.6-1
rpm -q pdksh pdksh-5.2.14-30.3
rpm -q openmotif21 openmotif21-2.1.30-11.RHEL4.4
rpm -q sysstat sysstat-5.0.5-1
rpm -q gnome-libs gnome-libs-1.4.1.2.90-44.1
rpm -q libstdc++ libstdc++-3.4.4-2
rpm -q libstdc++-devel libstdc++-devel-3.4.4-2
rpm -q compat-libstdc++-296 compat-libstdc++-296-2.96-132.7.2
rpm -q compat-db compat-db-4.1.25-9
rpm -q control-center control-center-2.8.0-12.rhel4.2
rpm -q xscreensaver xscreensaver-4.18-5.rhel4.9


Enjoy!

Thursday, May 15, 2008

CRS-0223: Resource 'ResourceName' has placement error

While I was attempting to configure a RAC environment, I noticed some services were started and some other weren't, when I tried to start the services, the 'placement' error showed up, this was due because when trying to start the crs with the crsctl start crs command on both nodes, there was not enough time between the start process in node 1 and node2, so the VIP address has switched its control from node 1 to node2, and when I tried to bring back the service to node 1 it by explicitly starting it on node 1 the error showed up.

The Oracle error manual states:

Oracle Error : CRS-0223: Resource '%s' has placement error.
Cause: There was no host available on which to failover/start the resource based on the Placement Policy for the resource.
Action: Check the target host for the resource and restart the resource using the crs_start command.

$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....A1.inst application ONLINE OFFLINE
ora....A2.inst application ONLINE ONLINE eg3704
ora.RDBHRA.db application ONLINE ONLINE eg3704
ora....SM1.asm application ONLINE OFFLINE
ora....03.lsnr application ONLINE OFFLINE
ora.eg3703.gsd application ONLINE ONLINE eg3703
ora.eg3703.ons application ONLINE ONLINE eg3703
ora.eg3703.vip application ONLINE ONLINE eg3704
ora....SM2.asm application ONLINE ONLINE eg3704
ora....04.lsnr application ONLINE ONLINE eg3704
ora.eg3704.gsd application ONLINE ONLINE eg3704
ora.eg3704.ons application ONLINE ONLINE eg3704
ora.eg3704.vip application ONLINE ONLINE eg3704


[oracle@eg3703 bin]$ ./crs_start ora.eg3703.LISTENER_EG3703.lsnr
eg3703 : CRS-1018:
Resource ora.eg3703.vip (application) is already running on eg3704
eg3704 : CRS-1019: Resource ora.eg3703.LISTENER_EG3703.lsnr (application) cannot run on eg3704

CRS-0223: Resource 'ora.eg3703.LISTENER_EG3703.lsnr' has placement error.
[oracle@eg3703 bin]$ ./crs_start ora.eg3703.ASM1.asm
eg3703 : CRS-1018: Resource ora.eg3703.vip (application) is already running on eg3704
eg3704 : CRS-1019: Resource ora.eg3703.ASM1.asm (application) cannot run on eg3704

CRS-0223: Resource 'ora.eg3703.ASM1.asm' has placement error.


At first hand, the procedure was to shutdown services and start them in order, so when that each service could take its place.


Name Type Target State Host
------------------------------------------------------------
ora....A1.inst application ONLINE ONLINE eg3703
ora....A2.inst application ONLINE ONLINE eg3704
ora.RDBHRA.db application ONLINE ONLINE eg3703
ora....SM1.asm application ONLINE ONLINE eg3703
ora....03.lsnr application ONLINE ONLINE eg3703
ora.eg3703.gsd application ONLINE ONLINE eg3703
ora.eg3703.ons application ONLINE ONLINE eg3703
ora.eg3703.vip application ONLINE ONLINE eg3703
ora....SM2.asm application ONLINE ONLINE eg3704
ora....04.lsnr application ONLINE ONLINE eg3704
ora.eg3704.gsd application ONLINE ONLINE eg3704
ora.eg3704.ons application ONLINE ONLINE eg3704
ora.eg3704.vip application ONLINE ONLINE eg3704

Adding a voting disk online fails

This happened on a RHEL4/Ora10gR2 Ver. 10.2.0.1.0.
In a cluster environment it is important to keep RAC availability at maximum and avoid unnecessary either planned or unplanned down times. That is why the online commands and functionality are for.

When attempting to add a Voting disk to my RAC configuration an error, "Cluster is not in a ready state for online disk addition" after issuing this command:

$ sudo /u01/crs1020/bin/crsctl add css votedisk /dev/raw/raw8

It was not possible to perform this task, even though the manual states it is possible, after a bit of research in metalink I found this metalink note:"ADDING VOTING DISK ONLINE CRASH THE CRS" This is due to the bug 4898020 and the platforms affected are, so far, Linux at the 10.2.0.1.0 release, One more reason to keep updated with the latest patchset available.The only workaround for this is to shutdown the crs and retry the operation with the force flag

So in my case the complete procedure to perform this task that could have been directly performed online was:

1. Stop all application processes

$ /u01/crs1020/bin/srvctl stop nodeapps -n eg7910
$ /u01/crs1020/bin/srvctl stop nodeapps -n eg7909

2. Verify services were down

$ /u01/crs1020/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....H1.inst application OFFLINE OFFLINE
ora....H2.inst application OFFLINE OFFLINE
ora.RDBHRH.db application OFFLINE OFFLINE
ora....SM1.asm application OFFLINE OFFLINE
ora....09.lsnr application OFFLINE OFFLINE
ora.eg7909.gsd application OFFLINE OFFLINE
ora.eg7909.ons application OFFLINE OFFLINE
ora.eg7909.vip application OFFLINE OFFLINE
ora....SM2.asm application OFFLINE OFFLINE
ora....10.lsnr application OFFLINE OFFLINE
ora.eg7910.gsd application OFFLINE OFFLINE
ora.eg7910.ons application OFFLINE OFFLINE
ora.eg7910.vip application OFFLINE OFFLINE


3. Stop CRS on both nodes

$ sudo /u01/crs1020/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

$ ssh eg7910 sudo /u01/crs1020/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.


4. Add the votedisks with the force flag.

$ sudo /u01/crs1020/bin/crsctl add css votedisk /dev/raw/raw8 -force
Now formatting voting disk: /dev/raw/raw8
successful addition of votedisk /dev/raw/raw8.


$ sudo /u01/crs1020/bin/crsctl add css votedisk /dev/raw/raw9 -force
Now formatting voting disk: /dev/raw/raw9
successful addition of votedisk /dev/raw/raw9.


5. Verify the voting disks were added

$ /u01/crs1020/bin/crsctl query css votedisk
0. 0 /dev/raw/raw7
1. 0 /dev/raw/raw8
2. 0 /dev/raw/raw9
located 3 votedisk(s).


6. Restart CRS and applications.

$ sudo /u01/crs1020/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
$ ssh eg7910 sudo /u01/crs1020/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly


This should have happened on line, and this could mean a momentarily loss of RAC availability.

Thursday, May 08, 2008

How to manually remove a service

When performing an Oracle deinstall, sometimes services are left behind, such as the TNSListener Service. In order for you to get rid of it, there are two options, first one, remove it from the regedit, this choice must be carefully executed, you don't want to remove other service.
1. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
2. Locate the service name, right click it and select Delete

It is always advisable to have a regedit backup prior to performing this task.

The second choice is the sc command
sc delete

This command is also used to manually create services
sc create

The complete syntax options for the sc command line utility is:
DESCRIPTION:
SC is a command line program used for communicating with the
NT Service Controller and services.
USAGE:
sc

Wednesday, May 07, 2008

Why is Varchar2 oversizing bad?

Apparently the same space consumption would be the same if a Varchar2(4) is used versus a Varchar2(4000), if you require one byte, then one byte will be allocated, and performance should be the same, but when taking a look at the performance statistics there are some surprises that arise.
When fetching information from Oracle, buffers and fetching structures are being prepared beforehand to be able to manage the maximum column lenght, applications such as toad may run out of memory trying to allocate resources to retrieve the information from an unnecessarily oversized varchar2 column.

Let's analyze this case:
Table 1: Varchar2LenghtDemo

SQL> desc Varchar2LenghtDemo
Name Null? Type
----------------------------------------- -------- ----------------------
ID NUMBER(10)
VCCOL VARCHAR2(4000)
VCCOL2 VARCHAR2(4000)
VCCOL3 VARCHAR2(4000)
VCCOL4 VARCHAR2(4000)
VCCOL5 VARCHAR2(4000)
VCCOL6 VARCHAR2(4000)
VCCOL7 VARCHAR2(4000)
VCCOL8 VARCHAR2(4000)
VCCOL9 VARCHAR2(4000)
VCCOL10 VARCHAR2(4000)
VCCOL11 VARCHAR2(4000)
VCCOL12 VARCHAR2(4000)
VCCOL13 VARCHAR2(4000)
VCCOL14 VARCHAR2(4000)
VCCOL15 VARCHAR2(4000)
VCCOL16 VARCHAR2(4000)
VCCOL17 VARCHAR2(4000)
VCCOL18 VARCHAR2(4000)
VCCOL19 VARCHAR2(4000)
VCCOL20 VARCHAR2(4000)

23:39:55 SQL> select count(*) from Varchar2LenghtDemo;

COUNT(*)
----------
8388608

Elapsed: 00:00:10.18

Statistics
----------------------------------------------------------
5 recursive calls
0 db block gets
55719 consistent gets
55669 physical reads
0 redo size
414 bytes sent via SQL*Net to client
385 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed


If we compare the statistics from one table and the other:
SQL> desc Varchar2LenghtDemoShort
Name Null? Type
----------------------------------------- -------- --------------------

ID NUMBER(10)
VCCOL VARCHAR2(4)
VCCOL2 VARCHAR2(4)
VCCOL3 VARCHAR2(4)
VCCOL4 VARCHAR2(4)
VCCOL5 VARCHAR2(4)
VCCOL6 VARCHAR2(4)
VCCOL7 VARCHAR2(4)
VCCOL8 VARCHAR2(4)
VCCOL9 VARCHAR2(4)
VCCOL10 VARCHAR2(4)
VCCOL11 VARCHAR2(4)
VCCOL12 VARCHAR2(4)
VCCOL13 VARCHAR2(4)
VCCOL14 VARCHAR2(4)
VCCOL15 VARCHAR2(4)
VCCOL16 VARCHAR2(4)
VCCOL17 VARCHAR2(4)
VCCOL18 VARCHAR2(4)
VCCOL19 VARCHAR2(4)
VCCOL20 VARCHAR2(4)

23:43:02 SQL> r
1* select count(*) from Varchar2LenghtDemoShort

COUNT(*)
----------
8388608

Elapsed: 00:00:05.50

Statistics
----------------------------------------------------------
0 recursive calls
0 db block gets
55756 consistent gets
22077 physical reads
0 redo size
414 bytes sent via SQL*Net to client
385 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

Almost all statistics are the same, except for these two:
Operation: Select count(*) from table

Table Rows Time Elapsed Phy Rds
Varchar(4) 8,388,608 00:00:05.50 22,077
Varchar(4000) 8,388,608 00:00:10.18 55,669

Since the number of physical reads is meaningful different a question arises:
How many rows per block? Is the row density the same?
Table          Avg Rows per Block
Varchar(4) 150.99
Varchar(4000) 150.99

Quite surprising, Row density is the same, but physical blocks processed are doubled.


Conclusion

Even though Varchar2 columns allow a variable number of characters and it will just use the number of bytes required, it is absolutely better to properly size the actual requirements, otherwise a meaningful performance issue unnecessarily will be created.

Thursday, May 01, 2008

Dump an Oracle Block to the trace

This is the script I use to dump an Oracle block.

set pagesize 0
set def =
set def &
column header_block new_value m_block
column header_file new_value m_file

select
header_file,
header_block + &m_offset header_block
from
dba_segments
where
segment_name = upper('&m_segment')
and nvl(partition_name,'xxx') = nvl(upper('&m_partition'),'xxx')
and owner = upper('&m_owner')
;

alter system dump datafile &m_file block min &m_block block max &m_block;