Intro
I’m just documenting what the support tech had me do to fix this scary issue.
The details
This was a SiteScope v 11.24 instance running on a RHEL 6.6 VM.
2015-12-08 05:12:56,768 [SiteScope Main Thread] (SiteScopeSupport.java:721) ERROR - SiteScope unexpected shutdown
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy106.postInit(Unknown Source)
at com.mercury.sitescope.platform.configmanager.ConfigManager$PersistencyAdaptor.initPersistObjectsAfterLoad(ConfigManager.java:1967)
at com.mercury.sitescope.platform.configmanager.ConfigManager$PersistencyAdaptor.initialize(ConfigManager.java:1247)
at com.mercury.sitescope.platform.configmanager.ConfigManager$PersistencyAdaptor.access$300(ConfigManager.java:1112)
at com.mercury.sitescope.platform.configmanager.ConfigManager.initialize(ConfigManager.java:145)
at com.mercury.sitescope.platform.configmanager.ConfigManagerSession.initialize(ConfigManagerSession.java:153)
at com.mercury.sitescope.bootstrap.SiteScopeSupport.initializeSiteScope(SiteScopeSupport.java:592)
at com.mercury.sitescope.bootstrap.SiteScopeSupport.configureSiteScope(SiteScopeSupport.java:629)
at com.mercury.sitescope.bootstrap.SiteScopeSupport.siteScopeMain(SiteScopeSupport.java:678)
at com.mercury.sitescope.web.servlet.InitSiteScope$SiteScopeMainThread.run(InitSiteScope.java:233)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.mercury.sitescope.platform.configmanager.ManagedObjectConfigRef$ManagedObjectProxyHandler.invoke(ManagedObjectConfigRef.java:290)
... 10 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String
at com.mercury.sitescope.entities.monitors.MonitorGroup.readDynamic(MonitorGroup.java:445)
at com.mercury.sitescope.entities.monitors.MonitorGroup.postInit(MonitorGroup.java:2001)
... 14 more
2015-12-08 05:12:56,776 [SiteScope Main Thread] (SiteScopeShutdown.java:51) INFO - Shutting down SiteScope reason Exception: java.lang.reflect.UndeclaredThrowableException null...
2015-12-08 05:12:56,784 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1061) INFO - Stopping dynamic counters flow...
2015-12-08 05:12:56,832 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1112) INFO - Waiting 40 secs to allow monitors to complete.
2015-12-08 05:13:36,854 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1280) INFO - Average Monitors Running: 0
2015-12-08 05:13:36,855 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1281) INFO - Peak Monitors Per Minute: 0 at 7:00 pm 12/31/69
2015-12-08 05:13:36,855 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1283) INFO - Peak Monitors Running: 0.0 at 7:00 pm 12/31/69
2015-12-08 05:13:36,855 [ShutdownThread at Tue Dec 08 05:12:56 EST 2015] (SiteScopeGroup.java:1284) INFO - Peak Monitors Waiting: 0 at 7:00 pm 12/31/69 |
And this just kept happening and happening.
The solution
The support tech from HPE had me go in the groups directory and delete all files except those ending in .dyn and .config. Those directories are in the /opt/HP/SiteScope directory on my installation.
In the persistency directory we deleted all files ending in .tmp. But we made saved copies of the entire original groups and persistency directories elsewhere just in case.
The results
HP siteScope started just fine after that! A healthy siteScope startup includes lines like these:
2015-12-08 08:59:12,149 [SiteScope Main] (SiteScopeGroup.java:995) INFO - Open your web browser to: 2015-12-08 08:59:12,149 [SiteScope Main] (SiteScopeGroup.java:996) INFO - http://10.192.136.89:8080 2015-12-08 08:59:12,180 [SiteScope Main] (SiteScopeGroup.java:324) INFO - Starting common scheduler... 2015-12-08 08:59:12,273 [SiteScope Main] (SiteScopeGroup.java:344) INFO - Starting maintenance scheduler... 2015-12-08 08:59:12,381 [SiteScope Main] (SiteScopeGroup.java:609) INFO - Starting topaz manager 2015-12-08 08:59:12,384 [SiteScope Main] (SiteScopeGroup.java:611) INFO - Topaz manager started. 2015-12-08 08:59:12,386 [SiteScope Main] (SiteScopeGroup.java:457) INFO - Starting monitor scheduler... 2015-12-08 08:59:12,386 [SiteScope Main] (SiteScopeGroup.java:462) INFO - Starting report scheduler... 2015-12-08 08:59:12,398 [SiteScope Main] (SiteScopeGroup.java:477) INFO - Starting analytics scheduler... 2015-12-08 08:59:12,398 [SiteScope Main] (SiteScopeGroup.java:488) INFO - 2015-12-08 08:59:12,398 [SiteScope Main] (SiteScopeGroup.java:489) INFO - SiteScope is active... 2015-12-08 08:59:12,493 [SiteScope Main] (SiteScopeGroup.java:513) INFO - SiteScope Starting all monitors 2015-12-08 08:59:12,830 [SiteScope Main] (SiteScopeGroup.java:517) INFO - SiteScope Start monitors completed 2015-12-08 08:59:13,037 [SiteScope Main] (SiteScopeGroup.java:532) INFO - SiteScope Startup Completed 2015-12-08 08:59:13,215 [SiteScope Main] (SiteScopeSupport.java:713) INFO - SiteScope 11.24.241 build 165 process started at Tue Dec 08 08:59:13 EST 2015 2015-12-08 08:59:13,215 [SiteScope Main] (SiteScopeSupport.java:714) INFO - SiteScope Start took 26 sec |
Especially that last line.
The hypothesis
As the server had crashed the hypothesis is that one of the files must have gotten corrupted.
Conclusion
HP SiteScope which could not start was fixed by removing some older files. These files are not needed, either – your configuration will not be lost when you delete them.