Tuesday, February 05, 2008

Issues Installing Patchset 10.2.0.3.0 on Windows XP

When attempting to install the Patchset 10.2.0.3.0 upgrade OUI complainted about a msvcr71.dll file in use. All Oracle services were shutdown, as directed by the Installation Guide. So this file shouldn't be in use by any Oracle executable. So where the culprit is? I found it by means of the Process Explorer, this is an utility that can be downloaded from the microsoft site, by navigating through the url: http://www.sysinternals.com/, the Microsoft Technet site can be reache, a couple of clicks and you're done, you can download it. A useful tool is the ListDLL too.

I found a svchost.exe process grabbing the msvcr71.dll file, I just killed it and the Installation process just went on to the end. I should point out that killing an OS process is a very serious matter, don't kill it unless you are absolutely sure what you are killing. DLL's shouldn't be locked by third party processes, so Oracle can just smoothly perform the upgrade. By means of the graphical tool previously mentioned I managed to know which process was locking the DLL.

Side Note, msvcr71.dll: is a module containing standard C library functions such asprintf, memcpy, and cos. It is a part of the Microsoft C RuntimeLibrary.

Thursday, January 31, 2008

Orastack

The orastack utility is a tool provided by oracle to set the stack size reserved or commited on a per thread basis in the Oracle Server on Windows Operating Systems. I had to use it because on a 10gR2 Winx32 database frequent ORA-04030 errors displayed along with TNS-12518.

The database must be shutdown prior to issue the command, otherwise this error will show up:

C:\Oracle\product\10.2.0\db_1\BIN>orastack oracle.exe 524288
Couldn't open file with CreateFile()
GetLastError() == 32

The default value is 1024K, it can be reduced to 512K, but it is not recommended to reduce the value below 300K, otherwise the ORA-01331 may start to show up.

Syntax:
  • orastack {no arguments} will display a useful README 1st.
  • orastack executableName will display the current settings
  • orastack executableName newValue will reset the current value to the specified newValue

ORA-04030 After 10gR2 Upgrade on Windows 2003

I have faced this exact situation, a large awful Error stack with errors such as:
ORA-04030, TNS-12518.

The particular environment where I faced these issues was:
Oracle 10gR2 (10.2.0.1.0) for Windows x32
Windows 2003 Enterprise Edition 32 bits

It was Right after migrating a user processes intensive application from 9iR2 (9.2.0.8.0) to the above referred 10gR2 release.
I had documented this issue with the metalik Note: 10gR2 Dedicated Connections Intermittently Fail with TNS-12518 Doc ID: Note:371983.1

One of the recommendations was to reduce memory consumption, and among one of the solutions was the use of orastack.exe to achieve this reduction. Right now I am documenting the issues and drawbacks of performing this procedure.

Aonother recommendation is to reduce the SGA consumption, this was achieved by reducing the values of the SGA_TARGET and SGA_MAX_SIZE instance parameters.


Update 11/Feb/2008

The orastack utility was helpful in reducing both, the Oracle and tnslsnr executables stack size. Additionally, a reduction in users' contention greatly helped in reducing the frequency of ORA-04030 errors, it doesn't mean this error was completely wiped out, I am still waiting for the 10.2.0.4.0 patchset to assess if I can completely get rid of this issue.

Wednesday, January 30, 2008

Forums Code of Conduct

Having actively participated at the Oracle forums, I have collected some useful recommendations for making life at forums easier for everybody.

1. Have a meaningful subject line.
The title should properly summarize the problem you are facing, this will attract people who may already have had a similar issue more easily.

2. Give details about versions and technologies (DB, OS, working environment). Depending on the version combinations a suitable solution can or can't be found, if not properly specified the answer scope may be unnecessarily wide and blind guesses will start to be posted.

3. Give detailed error messages. Posting monosyllabic or partially specified error messages or being non specific won't bring you specific answers for sure.

4. If possible provide steps to reproduce the problem. A problem could be reproduced by the posters and configuration issues could be easily spotted. Lab environments could be easily set up.

5. Search before you post. Don't try to reinvent the wheel when someone else has already done it for you. A suitable answer can be found, a 10 minute goggling effort could find you an answer.

6. Get a name. Anonymous users are like ghosts without a name. user615770, for example, says nothing about a person. Everybody can take a few minutes to give themselves a real id, this will make posters more responsible for what they are postings, since they a real person with a real name and they are not hiding behind the mask of a generic and anonymous User Id.

7. If you solve it on your own, share the solution and mark problems with [Solved] in your thread title. There are people who not only feedback if the solutions provided helped in solving their issue, they never track their threads (specially seen when people mark their threads as Urgent). Thinking about other people who are looking for a similar solution, they never know if the provided steps lead to a successful or an unsuccessful result, or even if another workaround was found.

8. Never judge people because of their english level. Most of the people at the international forums aren't English native speakers (including myself), and they do their best to make themselves clear and understandable, let's help them instead of laughing at them, particularly if posting on a technical forum, which should not be used as an English language level assessment tool.

9. Never use discriminatory comments, nor references to ethnic groups or religions, nor make any comment that could be offensive to anybody, in a technical forum, strictly technical comments are never offensive to anybody.

10. Kill Uppercase and exclamation marks. This is considered as yelling and it could be offensive for some people, including me. 'Please' is always an advisable and suggestible keyword.

11. Think of your threads and contributions as if they were written on stone. More people than you could imagine is reading or will read the thread, and each time this thread is read people will read it as is.

12. RTFM is not an answer. I have seen several times self-qualified senior level whatever providing this kind of answers. RTFM is equal as saying "don't bother me, stop @%##~ me", and this is not an acceptable technical answer, specially for those who pretend to be self qualified as senior.

13. Address the community with respect. Self explanatory.

Thanks to Shay Shmeltzer's Blog out of which, this code of conduct proposal is based.

Issues found when upgrading 9.2.0 to 10.2.0

Note: I strongly encourage people to read the Upgrade guide, read metalink Notes, get involved with 10g new features and setup a testing environment prior to perform the actual upgrade on a production environment.

ORA-00093: _shared_pool_reserved_min_alloc must be between 4000 and 0


In my particular case, the reason why I faced this issue was because I enabled ASMM and I didn't declare a value for SGA_TARGET. All parameters managed by ASMM were set to zero, but I forgot to explicitly set SGA_TARGET and SGA_MAX_SIZE. After setting suitable values for these parameters I was able to start my database. _shared_pool_reserved_min_alloc is a hidden parameter and this error appears due to ASMM missconfiguration.

This was on a manually upgraded database. I followed the below mentioned documentation, which was of great help.

Upgrade Guide
Oracle 10g New Features

Metalink Notes:
  • Complete checklist for manual upgrades of Oracle databases: 421191.1
Heap size nK exceeds notification threshold (2048K) warning
An error similar to this one is frequently seen at the alert.log file. Several queries which used to work without any issue, suddenly start to be reported at the alert.log file without apparent reason.
Memory Notification: Library Cache Object loaded into SGA
Heap size 11369K exceeds notification threshold (2048K)
Details in trace file d:\oracle\admin\aonmxdbf\udump\aonmxdbf_ora_3624.trc
KGL object name :EXPLAIN PLAN SET STATEMENT_ID = 'TRACE156742' INTO DBO.IW_PLAN_TABLE FOR Select

This is actually not an error, and the database is not about to crash, it only means a Heap size is greater than the value registered as a threshold. This is an issue on 10.2.0.1.0, this issue has been fixed on later patchsets. If you want to get rid of this warning, then an internal parameter has to be modified to a value greated than 2M. This should be made from a sql plus prompt connected as sysdba:

SQL> alter system set "_kgl_large_heap_warning_threshold"=8388608 scope=spfile ;


ORA-04030: out of process memory
An error similar to the one shown below is written at the alert.log file, a process has run out of memory. The error manual is funny, according to it you don't have to do any thing.

ORA-04030: out of process memory when trying to allocate string bytes (string,string)
Cause: Operating system process private memory has been exhausted
Action: none

Errors in file d:\oracle\admin\aonmxdbf\udump\aonmxdbf_ora_3176.trc:
ORA-04030: out of process memory when trying to allocate 66172 bytes (pga heap,kco buffer)
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [unable_to_trans_pc] [PC:0x60504019] [ADDR:0x4]
[UNABLE_TO_WRITE] []

This error showed up when an end user tried to access an application with a view ranked as one of the top queries in the database. I have increased the PGA_AGGREGATE_TARGET instance parameter and I have enabled ASMM.

I should further clarify, this happened on a Windows 2003 environment, and I found out that the boot didn't have the /3G switch required to access more than 2G RAM memory. So I have set it too since the memory requirements were very close to the 2G limit.

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /3GB



The ORA-04030 was accompanied by TNS-12518 (TNS:listener could not hand off client connection)

It happened intermitently and when workload increased. As detailed on my other post this was due to excesive resources allocated by all the new features for this release. It is documented at the metalink note 371983.1.


Instance parameters compatible and log_archive_format
Once the database was upgraded, according to the install guide, the compatible parameter remained at 9.2.0.0.0, once the upgrade process was stabilized the parameter changed to 10.2.0.1.0, this made instance parameter such as log_archive_format to have a mandatory different format: it must contain the archive format mask %s %r %t.



Sunday, November 04, 2007

Zayle Letzel Trojan

"Todo por ti Letzel"

What's the Zayle Trojan
The Zayle or Letzel is a trojan virus that attacks the USB devices, such as regular pen drives, USB storages, digital cameras, ipods, mp3 players, etc. This can be recognized when you see the hidden system file autorun.inf along an executable named crsvc.exe (Zarteck), this prevents the pen drive from being safely removed, and it may prevent the device from being formatted.

The autorun.inf file can be opened with notepad and a couple of text images can be seen (as shown below).

Zayle could be a rearrange of the name Elyza, and the message posted by the hacker reads "TODO POR TI LETZEL", which means "All for you Letzel". This could be dedicated either to the hacker's girlfriend or this could be dedicated to the architect Jan Letzel, who created the Hiroshima Peace Memorial, just mere speculation.


HowTo get rid of Zayle
  1. Disable the system restore feature. Go to the Control Panel -> System -> System Restore And disable the Restore system by clicking on the Deactivate System Restore on all units check box. Accept it and close this dialog window.
  2. From the Task Manager find a process named crsvc.exe and kill it.
  3. Disable the 'hide protected Operating system files' option from the folder options. Go to Tools -> folder options -> See -> Hide Protected operating system files and deselect this option, so the hidden system files appear listed on the explorer.
  4. From the explorer, do not open the pendrive with double click, this will re-enable the trojan. Explore it instead by right clicking on the drive letter icon.
  5. Delete the autorun.inf and the crsv.exe files.
  6. Open the regedit, Start menu -> run (or click [Win]+r then type regedit on the dialog box and accept. On the regedit explorer look for HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run, and there you will find an entry named syslog that calls the crsvc.exe file, remove this entry and restart your computer.
  7. The infection is gone.

Zayle's Signature
Autorun signature left by this trojan at the autorun.inf file when it infects the pendrive (very creative and romantic):

[autorun]
open=crsvc.exe
shell\1=ZAYLE
shell\1\Command=crsvc.exe
shellexecute=crsvc.exe

TODO POR TI LETZEL (All for you Letzel)
[A heart text image]


[A rose text image]

Saturday, October 27, 2007

Enterprise Manager Troubleshooting

Most frequently seen problems with EM console have to do with non properly configured networking environments and skipping the dhcp warning at the installation phase. Proper networking configuration prerequisites have to be met before proceeding with installation.

1. DHCP issues.
Assigning a dynamic IP address makes the EM console to fail. Since the repository is configured with the environment it collects at install time, if a dynamic address is assigned, next time the computer unplugs (most frequently seen on laptop computers) EM stops working.

2. Use of IE with Enhanced Security Option enabled.
This is another issue frequently seen, when IE Enhanced Security option is enabled EM stops working, even though the console can be started.

3. Enterprise Manager can be accessed but Performance and maintenance tab request again for login credentials
This is another issue due to improperly configuring the networking and host name resolution environment. Oracle reads the hosts file to establish both the IP Address and the fully qualified host name. If it fails to properly read the hosts file, or if this doesn't properly identify the host, then the localhost.localdomain will be taken to configure the EM Console. When Attempting to solve the tns entry to access the performance tab or the maintenance tab Oracle won't be able to identify the host declared at the tns entry and it will fail to access, even though the provided credentials and the tns entry seem to be well configured.

4. Changing hostname or fixed IP Address
When networking changes happen then an EM reconfiguration is required.

For further references on HowTo troubleshoot and reconfigure Enterprise Manager I suggest to refer to this link --> Enterprise Manager DB Control Console.

Thursday, October 25, 2007

Who's Who at Oracle Forums

The more time I spend on the Oracle forums the more people I get to know, even though I have never physically seen most of them, only by the kind of regular answers I have been able to 'see' Who's who.

Sybrand Bakker AKA sybrandb
It's funny to see people on the forums who are self qualified as 'Senior Oracle DBA', and when trying to google them on the web the only kind of references obtained so far are at an extreme poor, simple and moronic (thanks Howard for this vocabulary addendum), no white papers found, no meaningful forum participation, no semminars, no references, no history, no blog, no personal web page, no nothing but simple racist, aggressive and rude answers at the Oracle related forums.

I have always said that reading the manual is the first source of Oracle information, but for heavens sake!, RTFM is not a 'Senior DBA Level Answer' I would like to hear, and guess what, just issue a search at the Oracle forum for the RTFM string, and you'll see who is one of the posters who most frequently provides this as an answer.

I though this gentleman was participating at the Oracle forums only, but when googling for 'Sybrand Bakker' I have found several other interesting references on the web with the same signature and the same 'Seior Oracle DBA' Level Answers. Just a xenofobic who shows a lot of aggressivity, lack of manners and lack of knowledge. I couldn't even say if this fellow poster has the OCA and I have not been able to find someone who makes a good reference for Mr. Bakker.

I still cannot understand why someone who definitely hates to write for free and whose contributions are less than binary rubbish, wastes his time writing for nothing but to build up a very ugly reputation.

Sunday, October 07, 2007

My roots


Madrid surname was first found in Old Castile in the heart of Spain, where the name originated in Visigothic times. It means one who came from Madrid, the capital of Spain. When Jews converted to Christianity in Spain in the 15th century, whether voluntarily or by force, they often took a last name based on their town or city or origin.

Some of the first settlers of this family name or some of its variants were: Among the early explorers of the New World were Juan De Madrid, who came to America in 1510; Gonzalo De Madrid, who came to the Dominican Repulic in 1560.

I was born in Mexico City, my roots are both, my family came from Hidalgo, where immigrants from England and Spain settled attracted by the gold and silver mines at Real del Monte, where in 1830 the Cornish first installed the steam mining machinery. One of the family branches later became farmers and settled in Tulancingo, where my mom was born. On my father's side, his family came from Huauchinango to Tulancingo, and there was where my parents met, however they didn't get married there. They lost contact and it was several years later when they met again in Mexico City, where finally I was born.

Monday, October 01, 2007

Is it possible to decrypt the Oracle password?

Oracle's hashed password algorithm, even though is supposed to be secret, it has been shown to be weak due to several vulnerabilities.
Weak Salt SelectionNon-random salt values.Lack of alphabetic case preservation.

On 1993 there was a post on the comp.databases.oracle newsgroup which describes the algorithm in detail, identifying an unknown fixed key as an input parameter. The key was later published on a book named "Special Ops". This has provided enough information to reproduce the algorithm:

1. Concatenate the username and the password to produce a plain text string;

2. Convert the plain text string to uppercase characters;

3. Convert the plain text string to multi-byte storage format; ASCII characters have the high byte set to 0x00;

4. Encrypt the plain text string (padded with 0s if necessary to the next even block length) using the DES algorithm in cipher block chaining (CBC) mode with a fixed key value of0x0123456789ABCDEF;

5. Encrypt the plain text string again with DES-CBC, but using the last block of the output of the previous step (ignoring parity bits) as the encryption key. The last block of the output is converted into a printable string to produce the password hash value.

Let's suppose a user's password is about 12-16 characters, then the algorithm would perform between 6 and 8 DES encryptions to compute the hash value.

The hash algorithm has another weaknesses, it can be calculated out from rainbow tables, which are precomputed hashed passwords which can be used to compare the actual hash password value later.

Ref. An Assessment of the Oracle Password Hashing Algorithm
Carlos CidInformation Security GroupRoyal Holloway, University of London
carlos.cid@rhul.ac.uk
Joshua WrightSANS Institutejwright@sans.org
18. Oct. 2005