Intro
I’m not saying every day is like this, and I’m compressing several days into one narrative, but you’ll quickly get the idea and see the difficulties we face. As I like to joke this is why we make the medium bucks.
The single remaining guy responsible for the in-house application environment has finally convinced the powers that be to upgrade IBM WebSphere from a five-year-old version to version 8.5. We traditionally use a web server front-end which I have traditionally supported. So I get tapped to figure out what to do for new web servers.
I get three enormous zip files from him and nothing else.
I happen upon a documentation file containing a link to an IBM web site and not much else. I go there. The installation mentions using IBM Installation Manager. Never heard of it. I ask the guy for that.
Get it and unpack. Try to find documentation on how to install the Installation Manager and none seems to exist. Isn’t that ironic?
I wing it and try to run a file with the promising name of install:
$ sudo ./install
sudo ./install
00:02.01 ERROR [main] org.eclipse.equinox.log.internal.ExtendedLogReaderServiceFactory safeLogged
Application error
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
at org.eclipse.swt.SWT.error(SWT.java:4387)
at org.eclipse.swt.widgets.Display.createDisplay(Display.java:913)
at org.eclipse.swt.widgets.Display.create(Display.java:899)
at org.eclipse.swt.graphics.Device.<init>(Device.java:156)
...
Install:
An error has occurred. See the log file
/tmp/IBMinstall/configuration/1420812667336.log. |
The logfile referred to contains this “helpful” information:
!SESSION 2015-01-09 09:11:05.439 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.6.0_24
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=solaris, ARCH=sparc, WS=gtk, NL=en
Framework arguments: -toolId install -accessRights admin input @osgi.install.area/install.xml
Command-line arguments: -os solaris -ws gtk -arch sparc -toolId install -accessRights admin input @osgi.install.area/insta
ll.xml
!ENTRY org.eclipse.osgi 4 0 2015-01-09 09:11:12.346
!MESSAGE Application error
!STACK 1
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
at org.eclipse.swt.SWT.error(SWT.java:4387)
at org.eclipse.swt.widgets.Display.createDisplay(Display.java:913)
at org.eclipse.swt.widgets.Display.create(Display.java:899)
at org.eclipse.swt.graphics.Device.<init>(Device.java:156)
at org.eclipse.swt.widgets.Display.<init>(Display.java:497)
at org.eclipse.swt.widgets.Display.<init>(Display.java:488)
at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:669)
at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:161)
at com.ibm.cic.agent.internal.ui.AgentUIApplication.initDisplay(AgentUIApplication.java:140)
at com.ibm.cic.agent.internal.ui.AgentUIApplication.launch(AgentUIApplication.java:162)
at com.ibm.cic.agent.internal.ui.AgentUIApplication.start(AgentUIApplication.java:64)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
at org.eclipse.equinox.launcher.Main.main(Main.java:1414) |
The references to Display hint that my display is goofed up. Which it is. I have no X display.
So I have to export the DISPLAY to another utility server where I can run vncserver.
Oops. That server was rebooted and so there is no vncserver currently running. I launch that:
$ vncserver :2
Now I can connect to it from my desktop using the VNC client, fire up an xterm and allow others to export their displays to it:
$ xhost +
Now I go back to Solaris and set my DISPLAY environment variable:
$ export DISPLAY=vncserver_name:2
And re-install. This time it comes up. The screen dialogs are very sluggish but very simple. I get it going just before 9:30 AM. The status bar creeps over to the right veerrrry slowly. at 10 AM it is finally done – for a package of size 297 MB! But I can do other work in the meantime. Hey, they can’t do backups any longer on a firewalled subnet. may be a problem with resolving the backup server’s name in this domain. Can I look into it? Yes, the domain name is missing when I query the authoritative nameservers. The guy next to me, I happen to know, is the administrator of this special domain. I ask him to look into it.
Meanwhile I unzip disk 1 of the WAS 8.5 download and hunt for the documentation. I find it in readme_plugins/en/readme_en.html. It doesn’t have much, just a few links to IBM web sites. After a few wrong leads I decide there is no direct link. I want to install the plugin file. So I have to interact with the online documentation a bit to get what I want. The documentation is thorough to the point of being bloated and effectively masks whatever it is you actually need out of it. I think I am getting close now after about 15 clicks and skimming loads of crap. The bread crumb trail looks like this so far:
WebSphere Application Server Network Deployment 8.5.5
Network Deployment (Distributed Operating Systmes), Version 8.5
Setting up intermediary services (who knew?)
Implementing a web server plugin
Installing and configuring web server plugins
Installing and uninstalling the Web Server Plug-ins on distributed operating systems
I’m still not sure I’ve struck meat yet. I just feel I am getting close now! No, actually there is another level:
Installing the Web Server plugins using the GUI
From this document, which actually contains some useful information, I get the imp[ression that I may need a repository set up, whatever that is.
I find and launch the IBM Installation Manager regardless to see what it does. I found its path as /opt/IBM/InstallationManager/eclipse/IBMIM. Click on the Install option and sure enough it complains I have no repository setup. It offers a link to do that.
After some futzing it seems to lead me to click on a repository config file in /opt/IBM/InstallationManager/eclipse/repository.config. But that may be a fools errand because when I re-launch it says the repository is not connected. Huh?
So then I try to specify a URL as repository, but to connect that I need an IBM username/password which i don’t have. I ask my colleague for one.
Meanwhile I re-examine the unzipped 1 of 3 zip file for WAS 8.5 and I see a repsitory.config file there! So after some fumbling with the slow and awkward Installation Manager GUI I manage to indicate that as my repository config file and delete the original one I had configured. This looks promising. Now I see an option to select IBM WebServer plugins. Looking good.
Interruption. You know that SHA2 certificate you got last year? We don’t think it’s really gong to work and can you get an SHA1 one instead? I am doubtful at this late stage but I promise to ask my contacts and fire off some emails.
The installation needs disk2 so I have unzip that one; then disk3. Now I’m out of space and move things around before unzipping that one. I am soon able to hit the Install button and seven minutes later the 389 MB package is installed.
I see it hasn’t asked me which web server I use and where it is and all that. So clearly I need some more steps. Rummaging around I come across /opt/IBM/WebSphere8.5/Plugins/bin/ConfigureApachePlugin.sh, which sounds pretty promising.
I run that and see there are a bunch of switches I have to provide values for. No problem. I get those and it runs. I examine what it has done to my config file and it looks partially promising and partially puzzling. It relies on an environment variable which I don’t think it has defined.
I stop the server and it already complains about that very thing:
httpd: Syntax error on line 344 of /usr/local/apache203/conf/httpd.conf: Syntax error on line 183 of /usr/local/apache203/conf/vhosts/secure-siteinfo.conf: Cannot load /usr/local/apache203/${WAS_PLUGIN_DRIVER} into server: ld.so.1: httpd: fatal: /usr/local/apache203/${WAS_PLUGIN_DRIVER}: open failed: No such file or directory |
I define that variable. And try to stop it again. The next error kind of scares me:
httpd: Syntax error on line 344 of /usr/local/apache203/conf/httpd.conf: Syntax error on line 183 of /usr/local/apache203/conf/vhosts/secure-siteinfo.conf: Cannot load /opt/IBM/WebSphere8.5/Plugins/bin/64bits/mod_was_ap22_http.so into server: ld.so.1: httpd: fatal: /opt/IBM/WebSphere8.5/Plugins/bin/64bits/mod_was_ap22_http.so: wrong ELF class: ELFCLASS64 |
To me that hints I may have the wrong architecture installed. I run some control tests:
$ file /opt/IBM/WebSphere8.5/Plugins/bin/64bits/mod_was_ap22_http.so
/opt/IBM/WebSphere8.5/Plugins/bin/64bits/mod_was_ap22_http.so: ELF 64-bit MSB dynamic lib SPARCV9 Version 1, dynamically linked, not stripped |
and now compared to my apache binary:
$ file /usr/local/apache2/bin/httpd
/usr/local/apache2/bin/httpd: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped |
I check with the system administrator if he had ever provided me a 64-bit apahce package for Solaris. After some checking we realize that Solaris 10 does provide an apache package but it is 32-bit.
I have an idea. I can simply change the path to the shared object file in my environment definition:
export WAS_PLUGIN_DRIVER=/opt/IBM/WebSphere8.5/Plugins/bin/32bits/mod_was_ap22_http.so |
I had originally specified 64bits. Maybe this will be compatible. My first thought is that I installed the wrong package and would have to ask for a different download.
Yess! It now stops. And it starts. And I can access its homepage.
Now go into its config and change its home page to the same as used by the Sun Java System web server.
Find a page that actually calls out to WebSphere by examining the log files and grepping for js (just a hunch). I find something. Try to reproduce it with curl on the real web server and I get a not found. Hmm. Work harder to match up the host header to the vhosts mentioned in the plugin config file. Specifying the right host it gives me a redirect and sets some cookies. I know the web server isn’t programmed to do that so I must have reached the back-end WebSphere app server and now I have something to test with. Test against the port running apache with this WAS config file and it produces the same result! A redirect and some cookies. Great. The hardest part is over. Now a control. We’ll remove the plugin config line in the apache config and re-try it. Yup. 404 not found. We really are communicating to the app server.
No way I am going to go through that pain for each and every server where this is needed. I’ll just tar up the needed files and untar them on any server where this is needed.
But I wonder if I should use the provided apache instead.
Interruption. We’ve received a corrupt pdf file in email two months ago. The vendor is mad at us because we are the only ones with this problem. Could our systems have corrupted an attachment? This is kind of an interesting question and deserves some rumination. The quick reaction is no we don’t do that. But years of experience tell me that exceptions abound. I open the attachment. Yup, corrupted. I save the file in an effort to examine the bytes. Then I see it has 0 length, That’s peculiar. I’ve never seen that around here. Then I think to check our mail server log files two months back for their record. I quickly find it and see that its size was reported as 34000 bytes. That strikes me as kind of large for a message with no attachment, but kind of small for a pdf attachment. I share my results with the requester.
Answer: they can still issue an SHA1 CERT. But probably only one which has a year’s duration. I tell the customer for this certificate that all is not rosy as they will probably use an obscure CA which is not accepted by all his customers, so there is no way out without experiencing some pain here.
Unix admin tells me they’re now getting alerts about running out of disk space on the filesystem and system where I put my WebSphere installation downloads. I move another one of those puppies (1 GB in size) to /tmp.