Setup
I decided to run an X Server on my Windows 10 laptop. I only need it for Cognos gateway configuration, but when you need it, you need it. Of course an X Server listens on port 6000, so hosts outside of your PC have to be able to initiate a TCP connection to your PC with destination port 6000. So that port has to be open. The software I use for the X Server is Mobatek XTerm.
Here is the Powershell command to disable the block of TCP port 6000.
The Powershell window needs to be run as administrator. The change is permanent: it suffices to run it once.
Conclusion
And, because inquiring minds want to know, did it work? Yes, it worked and I could send my cogconfig X window to my Mobatek X Server. I had to look for a new Window. It was slow.
In a lot of ways, the last few years have seen 3D printing “going mainstream,” so to speak. People are no longer astonished by the sheer capabilities of the technology, and we’re getting more accustomed to seeing stories about one interesting product or another having been 3D printed. People are also gaining more access to 3D printing; in some cases the costs of the devices have fallen, though we have posted about building one’s own 3D printer as well, and this is another option some savvier consumers are embracing.
The process of getting to this point, at which 3D printing is at least somewhat familiar to a lot of people, has been complex, fascinating, and enjoyable. But now that 3D printing is mainstream, what’s next for the technology? It’s hard to say with certainty, but we can look to a few current trends and developments and predict some of the ways that the technology could become more popular still.
Further Understanding Of Capabilities
Even if people are no longer astonished by what 3D printing can do in general, further understanding of the full range of the technology’s capabilities is likely to come with time. In a sense, the initial learning curve concerned the basic concept – that a 3D object could be printed into empty space by a machine according to a digital design. Next, people will inevitably come to learn about all of the various twists that can exist within that broad concept. For instance, people will get more used to the idea that 3D printers can create metal objects, and not only work with plastics; people will come to be familiar with larger 3D printers on a scale that can create car parts, or potentially even small homes and urban planning features. Ideas like these don’t necessarily fall into the most basic understanding of 3D printing, but they’ll inevitably be more widely understood in short time.
Combination With Other Advanced Manufacturing
This is perhaps more of a concern for large companies and significant manufacturing operations. Nevertheless, it’s a good bet that the next few years will see more people come to understand 3D printing as one of a few advanced manufacturing methods that can contribute to the creation of modern products. The technology’s most noteworthy counterpart may be injection molding, which is not as new, but which has grown vastly more sophisticated and capable. This is basically a process by which a given material is heated and molded to fit a shape, such that it hardens in that shape and becomes a final product. Fictiv describes injection molding processes as being ideal for design validation or for higher volume production, which largely conveys the idea that they are similarly useful to 3D printing. The truth though is that people may come to realize that the combination of the two technologies can benefit entire industries. For instance, there are some cases in which prototypes and early models are created via 3D printing (a better process for trial-and-error), and then used as the basis for large-scale molding efforts to design product lines.
3D-Printed Food
The very notion of 3D-printed food can sound absurd on its face. It reads very much like something from a science fiction novel or a fantasy film in which food can simply be created, and there are no more resource shortages. Amazingly enough though, while it’s not all quite so glossy and miraculous as the sci-fi version, 3D-printed food has become a reality. Hackaday explains how 3D-printed food can work in a piece written just last year, and the concept is actually surprisingly straightforward. Basically, pastes made of organic materials and broken down ingredients (such as, in one example, peas and seaweed) can be printed into mock versions of regular food items (in the same example, a steak). The same article acknowledges that taste isn’t necessarily ideal just yet, but the very idea of creating food products from a mishmash of edible materials is exciting. It’s a starting point for what could be a massive step toward addressing food shortages, and it’s likely to become one of the main topics driving further interest in 3D printing.
Command-Based Projects
This is something we don’t hear all that much about yet, but which could make for a significant step particularly with regard to consumer 3D printing. Singularity Hub took a look at 3D printing innovations we might see in the next five years or so and introduced this idea in an interesting way, stating that “hey” will become the “most frequently used command in design engineering.” Specifically the suggestion is that we’ll be able to say, “Hey 3D printer, design me a new pair of shoes” in much the same way we ask Alexa or Siri to make shopping lists or tell us the weather. If this proves to be true, it’s likely we’ll see a significant spike in consumer 3D printer purchases. For that matter, it could significantly change much of the retail industry.
Ultimately one of the cool things about 3D printing is that we’re all still figuring it out. It likely has countless applications we have’t yet considered or imagined. The suggestions here are grounded in current trends and speculation though, and should play a significant role in keeping this technology fresh and interesting.
Intro
Today I came across a simple but useful tool which runs on Windows systems that will help determine if a remote host is listening on a particular port. I wanted to share that information.
The details PortQry is attractive because of its simplicity, plus, it is supported and distributed by Microsoft themselves. The help section reads like this:
PortQry version 2.0
Displays the state of TCP and UDP ports
Command line mode: portqry -n name_to_query [-options]
Interactive mode: portqry -i [-n name_to_query] [-options]
Local Mode: portqry -local | -wpid pid| -wport port [-options]
Command line mode:
portqry -n name_to_query [-p protocol] [-e || -r || -o endpoint(s)] [-q]
[-l logfile] [-sp source_port] [-sl] [-cn SNMP community name]
Command line mode options explained:
-n [name_to_query] IP address or name of system to query
-p [protocol] TCP or UDP or BOTH (default is TCP)
-e [endpoint] single port to query (valid range: 1-65535)
-r [end point range] range of ports to query (start:end)
-o [end point order] range of ports to query in an order (x,y,z)
-l [logfile] name of text log file to create
-y overwrites existing text log file without prompting
-sp [source port] initial source port to use for query
-sl 'slow link delay' waits longer for UDP replies from remote systems
-nr by-passes default IP address-to-name resolution
ignored unless an IP address is specified after -n
-cn specifies SNMP community name for query
ignored unless querying an SNMP port
must be delimited with !
-q 'quiet' operation runs with no output
returns 0 if port is listening
returns 1 if port is not listening
returns 2 if port is listening or filtered
Notes: PortQry runs on Windows 2000 and later systems
Defaults: TCP, port 80, no log file, slow link delay off
Hit Ctrl-c to terminate prematurely
examples:
portqry -n myserver.com -e 25
portqry -n 10.0.0.1 -e 53 -p UDP -i
portqry -n host1.dev.reskit.com -r 21:445
portqry -n 10.0.0.1 -o 25,445,1024 -p both -sp 53
portqry -n host2 -cn !my community name! -e 161 -p udp
...
PortQry version 2.0
Displays the state of TCP and UDP ports
Command line mode: portqry -n name_to_query [-options]
Interactive mode: portqry -i [-n name_to_query] [-options]
Local Mode: portqry -local | -wpid pid| -wport port [-options]
Command line mode:
portqry -n name_to_query [-p protocol] [-e || -r || -o endpoint(s)] [-q]
[-l logfile] [-sp source_port] [-sl] [-cn SNMP community name]
Command line mode options explained:
-n [name_to_query] IP address or name of system to query
-p [protocol] TCP or UDP or BOTH (default is TCP)
-e [endpoint] single port to query (valid range: 1-65535)
-r [end point range] range of ports to query (start:end)
-o [end point order] range of ports to query in an order (x,y,z)
-l [logfile] name of text log file to create
-y overwrites existing text log file without prompting
-sp [source port] initial source port to use for query
-sl 'slow link delay' waits longer for UDP replies from remote systems
-nr by-passes default IP address-to-name resolution
ignored unless an IP address is specified after -n
-cn specifies SNMP community name for query
ignored unless querying an SNMP port
must be delimited with !
-q 'quiet' operation runs with no output
returns 0 if port is listening
returns 1 if port is not listening
returns 2 if port is listening or filtered
Notes: PortQry runs on Windows 2000 and later systems
Defaults: TCP, port 80, no log file, slow link delay off
Hit Ctrl-c to terminate prematurely
examples:
portqry -n myserver.com -e 25
portqry -n 10.0.0.1 -e 53 -p UDP -i
portqry -n host1.dev.reskit.com -r 21:445
portqry -n 10.0.0.1 -o 25,445,1024 -p both -sp 53
portqry -n host2 -cn !my community name! -e 161 -p udp
...
The PortQry “install” consisted of unzipping a ZIP file, so, no install at all, and no special permissions needed, which is a plus in my book.
nmap
Of course there is always nmap. I never really got into it so much, but clearly you can go nuts with it. One advantage is that it is available on linux and MacOS as well. But in my opinion it is a heavy-handed install.
Intro
I panned Zabbix in this post: DIY monitoring. But I have compelling reasons to revisit it. I have to say it has matured, but there remain some very frustrating things about it, especially when compared with SiteScope (now owned by Microfocus) which is so much more intuitive.
But I am impressed by the breadth of the user base and the documentation. But learning how to do any specific thing is still an exercise in futility.
I am going to try to structure this post as a problems encountered, and how they were resolved.
Current production version as of this writing? Answer: 7.0
Zabbix Manual does not work in Firefox
That’s right. I can’t even read the manual in my version of Firefox. Its sections do not expand. Solution: use Chrome
Which database?
You may see references to MYSQL in Zabbix docs. MYSQL is basically dead. what should you do?
Zabbix quick install on Redhat
Answer
Install mariadb which has replaced MYSQL and supports the same commands such as the mysql from the screenshot. On my Redhat instance I have installed these mariadb-related repositories:
Terminology confusion
what is a host, a host group, a template, an item, a web scenario, a trigger, a media type?
Answer
Don’t ask me. When I make progress I’ll post it here.
Web scenario specific issues
Can different web scenarios use different proxies?
Answer: Yes, no problem. In really old versions this was not possible. See web scenario screenshot below.
Can the proxy be a variable so that the same web scenario can be used for different proxies?
Answer: Yes. Let’s say you attach a web scenario to a host. In that host’s configuration you can define a “macro” which sets the variable value. e.g., the value of HTTP_PROXY in my example. I think you can do the same from a template, but I’m getting ahead of myself.
Similarly, can you do basic proxy auth and hide the credentials in a MACRO? Answer: I think so. I did it once at any rate. See above screenshot.
Why does my google.com web scenario work whereas my amazon.com scenario not when they’re exactly the same except for the URL? Answer: some ideas, but the logging information is bad. Amazon does not take to bots hitting it for health check reasons. It may work better to change the agent type to Linux|Chrome, which is what I am trying now. Here’s my original answer: Even with command-line curl I get an error through this proxy. That can’t be good:
$ curl ‐vikL www.amazon.com
My amazon.com web scenario is not working (status of 1), yet in dashboard does not return any obvious warning or error or red color. Why? Answer: no idea. Maybe you have to define a trigger?
Say you’re on the Monitoring|latest data screen. Does the data get auto-updated? Answer: yes, it seems to refresh every 30 seconds.
In Zabbix Latest Data can you control the history displayed via url parameters? By default only one hour of history is displayed. Answer: There is an undocumented feature I have discovered which permits this. Let’s say your normal URL for your direct link to the latest data of item 1234 is https://drjohns.com/history.php?action=showgraph&itemids[]=1234. The modified version of that to display the last day of data is: https://drjohns.com/history.php?action=showgraph&from=now-1d&to=now&itemids[]=1234
In latest data viewing the graph for one item which has a trigger, sometimes the trigger line is displayed as a dashed line and sometimes not at all. Answer: From what I can tell the threshold line is only displayed if the threshold was entered as a number in the trigger condition. Strange. Unfortunate if true.
Why is the official FAQ so useless? Answer: no idea how a piece of software otherwise so feature-rich could have such a useless FAQ.
Zabbix costs nothing. Is it still actively supported? Answer: it seems very actively supported for some reason. Not sure what the revenue model is, however…
Can I force one or more web scenarios to be run immediately? I do this all the time in SiteScope. Answer: I guess not. There is no obvious way.
Suppose you have defined an item. what is the item key? Answer: You define it. Best to make it unique and use contiguous characters. I’m seeing it’s very important…
What is the equivalent to SiteScope’s script monitors? Answer: Either ssh check or external check.
How would you set up a simple PING monitor, i.e., to see if your host is up? Answer: Create an item as a “simple check”, e.g., with the name ping this host, and the key icmpping[{HOST.IP},3]. That can go into a template, by the way. If it succeeded it will return a 1.
I’ve made an error in my script for an external check. Why does Latest data show nothing at all? Answer: no idea. If the error is bad enough Zabbix will disable the item on you, so it’s not really running any longer. But even when it doesn’t do that, a lot of times I simply see no output whatsoever. Very frustrating.
Help! The Latest Data graph’s Y axis only shows 0’s and 5’s. Answer: Another wonderful Zabbix feature, this happens because your Units are too long. Even “per minute” as Units can get you into trouble if it is trying to draw a Y axis with values 22.0 22.5 23.0, etc: you’ll only see the .0’s and the .5’s. Change units to a maximum of seven characters such as “per min.”
Why is the output from an ssh check truncated, where does the rest go? Answer: no idea.
How do you increase the information contained in the zabbix server log? Let’s say your zabbix server is running normally. Then run this command: zabbix_server ‐R log_level_increase
You can run it multiple times to keep increasing the verbosity (log level), I think.
Attempting to use ssh items with key authentication fails with :”Public key authentication failed: Callback returned error” Initially I thought Zabbix was broken with regards to ssh public key authentication. I can get it to work with password. I can use my public/private key to authenticate by hand from command-line as root. Turns out running command such as sudo -u zabbix ssh … showed that my zabbix account did not have permissions to write to its home directory (which did not even exist). I guess this is a case of RTFM, because they do go over all those steps in the manual. I fixed up permissions and now it works for me, yeah.
Where should the scripts for external checks go? In my install it is /usr/lib/zabbix/externalscripts.
Why is the behaviour of triggers inconsistent. sometimes the same trigger has expected behaviour, sometimes not. Answer: No idea. Very frustrating. See more on that topic below.
How do you force a web scenario check when you are using templates? Answer: No idea.
Why do (resolved) Problems disappear no matter how you search for them if they are older than, say, 30 minutes? Answer: No idea. Just another stupid feature I guess.
Why does it say No media defined for user even though user has been set up with email as his media? Answer: no idea.
Why do too many errors disable an ssh check so that you get Status Disabled and have no graceful way to recover? Answer: no idea. It makes sense that Zabbix should not subject itself to too many consecutive errors. But once you’ve fixed the underlying problem the only recovery I can figure is to delete the item and recreate it. or delete the host and re-create it. Not cool.
I heard dependent items are the way to go to parse complex data coming out of a rich text item. How do you do that? Answer: Yes they are. I have gotten them to work and really give me the fine-grained control I’ve always wanted. I hope to show a real-life example soon. To get started creating a dependent item you can right-click on the dots of an item, or create a new item and choose type Dependent Item.
I am looking at Latest data and one item is grayed out and has no data. Why? Answer: almost no idea. This happens to me in a dependent item formed by a regular expression where the regular expression does not match the content. I am trying to make my RegEx more flexible to match both good and error conditions.
Why do my dependent items, when running a Check Now, say Cannot send request: wrong data type, yet they are producing data just fine when viewed through Latest data? Answer: this happens if you ran a Check Now on your template rather than when viewing an individual host. Make sure you select a host before you run Check Now. Actually, even still it does not work, so final answer: no idea.
Why do some regular expressions check out just fine on regex101.com yet produce a match for value of type “string”: pattern does not match error in Zabbix? Answer: Some idea. Fancy regular expressions do not seem to work for some reason.
Every time I add an item it takes the absolute maximum amount of time before I see data, whether or not I run check Now until I turn blue in the face. Why? Answer: no idea. Very frustrating.
If the Zabbix server is in one time zone and I am in another, can I have my view of timestamps customized to my time zone? Otherwise I see all times in the timezone of the Zabbix server. Answer: You are out of luck. The suggestion is to run two GUIs, one in your time zone. But there is a but. Support for this has been announced for v 5.20. Stay tuned…
My DNS queries using net.dns don’t do anything. Why? Answer: no idea. Maybe your host is not running an actual Zabbiox agent? That’ll do it. Forget that net.dns check if you can’t install an agent. Zabbix has no agentless DNS monitor for some strange reason.
A DNS query which returns many address records fails (such as querying an AD domain), though occasionally succeeds. Why?Answer: So your key looks something like this, right? net.dns.record[10.1.2.3,my-AD-domain.net,A,10,2,tcp]. And when you do the query through dig it works fine, right? E.g., dig +tcp my-AD-domain.net @10.1.2.3. And you’ve set the Zabbix response to type text? It seems to be just another Zabbix bug. You may have to use a script instead. Zabbix support has been able to reproduce this bug and they are working on it as we speak.
What does Check/Execute Now really do? Answer: this sometimes works and causes the itme to be run outside of its scheduled window, but it may takes a few seconds. However, if you have enough permissions, what you can do when you’re looking at an item for a specific Host is to run Test. Then Get Value. I sometimes get Permission Denied, however.
I want to show multiple things on a dashboard widget graph like an item plus its baseline (Ed: see references for calculating a baseline). What’s the best way? Answer: You can use the add new data set feature for instance to add your baseline. In your additional data set you put your baselines. Then I like to make the width 2, transparency 0 and fill 0. This will turn it into a thin bold line with a complementary color while not messing too much with the original colors of your items. The interface is squirrely, but, hey, it’s Zabbix, what did you expect?
I have a lot of hosts I want to add to a template. Does that Mass Update feature actually work? Answer: yes. Use it. It will save you time.
Help! I accidentally deleted an entire template. I meant to just delete one of its macros. Is there a revert? Answer: it doesn’t look like it. Hope you remember what you did…
It seems if I choose units in an item which have too many characters, e.g., client connections, the graph (in Latest Data) cuts it off and doesn’t even display the scale? Answer: seems so. It’s a bug. This won’t happen when using vector graphs in Dashboard. The graphs in Latest Data are PNG and limited to short Units, e.g., mbps. Changing to vector graphs has been in the roadmap but then disappeared.
Can I create a baseline? Nope. It’s on the roadmap. However, see this clever idea for building one on your own without too much effort.
I’ve put a few things on the same Dashboard graph. Why don’t they align? There are these big gaps. Zabbix runs the items when it feels like, and the result is gaps in data which Zabbix makes no attempt to conceal at the beginning and end of a graph. You can use Scheduling Intervals on your items to gain some control over this. See this article for details.
Besides cloning the whole thing, how can I change the name of a Dashboard? Answer: If you just click to edit a Dashboard the name appears fixed. However, click on the gear icon and that gives you the option to edit the dashboard name. It’s kind of an undocumented feature.
My SNMP MIB has bytes in/out for an interface when what I really want is bandwidth, i.e., Megabits per second. A little preprocessing on a 64-bit bytes value and you are there (32 bit values may roll over too frequently). See this article for details.
In functions like avg (sec|#num,<time_shift>), why is the time_shift argument so restricted? It can’t be a macro, contain a formula like 1w-30m, or anything semi-sophisticated. It just accepts a dumb literal like 5h? Answer: It’s just another shortcoming in Zabbix. How much did you pay for it? 🙂
I have an SNMP template with items for a hostgroup of dispersed servers. Some work fine. The one in Asia returns a few values, but not all. I am using Bulk Request.Answer (to your implied question!) You must have bad performance to that one. Use a Zabbix proxy with a longer timeout for SNMP requests. I was in that situation and that worked for me.
SNMPv3 situation. I have two identical virtual servers monitored by the same Zabbix proxy. Only one works. Command-line testing of snmpwalk looks fine. What could it be?’Answer: We are fighting this now. In our case the SNMP v3 engineIDs are identical on the two virtual servers because they were from the same image, whereas, if you read the specs, they are supposed to be unique, like a MAC address. Who knew? And, yes, once we made the engineIDs unique, they were fine in Zabbix.
Riddle: when is 80% not 80%? Answer: when pulling in used storage on a filesystem via SNMP and comparing it to storage size! I had carefully gotten a filesystem 83% full based on the output of df -m. But my trigger, set to go off at 80%, never went off. How could it be? The 83% includes some kind of reserved user space on the filesystem which is not included when you do the calculation directly. So I was at 78% or so in actuality. I changed the trigger to 75%.
My trigger for a DNS item, which relies on a simple diff(), goes off from time-to-time yet the response is the same. Why? Answer: We have seen this behavior for a CNAME DNS item. The response changed the case of the returned FQDN from time-to-time, and that is enough to set off the Zabbix diff()-based trigger! We pre-processed the output with a RegEx to just get the bits we wanted to examine to fix this.
Related question. My diff() trigger for a DNS item does NOT go off when the server actually goes down. What’s up with that? Answer: Although you might expect a suddenly unavailable server constitutes a “difference,” in Zabbix’s contorted view of reality it does not. I recommend an additional trigger using the function nodata().
Does the new feature of login using SAML actually work? Answer: Yes, we are using it in Zabbix v 5.0.
My OIDs for my filesystems keeps shifting around. What to do? Answer: Use low-level discovery. It’s yet another layer of abstraction and confusion, but it’s probably worth it. I intend to write up my approach in my practical Zabbix examples blog post.
After an Zabbix agent item goes bad (no data), Zabbix refuses to test it for a full 30 minutes after it went bad, despite an update interval of 5 minutes. Why? Answer: In one of the worst architectural decisions of all time, Zabbix created the concept of unsupported items. It works something like this: the very moment when you need to be told Hey there’s something wrong here is when Zabbix goes quiet. Your item became unsupported, which is like being in the penalty box for 30 minutes, during which time nothing works like you naively expected it to. Even the fact that your item becameunsupported is almost impossible to find out from a trigger. An example of software which treats this situation correctly is Microfocus SiteScope. In Zabbix in version 5.0 there’s a global timeout for all unsupported items. Ours is set to 30 minutes, you see. In some cases that may make sense and prevent Zabbix from consuming too many resources trying to measure things which don’t work. I find it annoying. For DNS, specifically, best to use a key of type net.dns and not net.dns.record. That returns a simple 0 or 1 and does not become unsupported if the dns server can’t be reached. V 5.2 will provide some more options around this issue. For a HTTP agent and I suppose many other items, it’s best to create triggers which use the nodata() function, which can somewhat compensate for this glaring weakness in Zabbix. If you run Zabbix v 5.2, you should use the new preprocessing rule “Check for not supported value” and then set new value e.g. “Error”. Then the Item won’t become unsupported and can also be used for triggers.
We’ve got SNMP items set up for a host. What’s the best way to alert for a total outage? Answer: I just learned this. This is closely related to the previous question. To avoid that whole unsupported item thing, you make a Zabbix internal item. the key is literally this: zabbix[host,snmp,available] and type is numeric unsigned. This wil continue to poll even if the other host items became unsupported. This is another poorly documented Zabbix feature.
While trying to set up a host for SNMP monitoring I get the error Cannot update Host. Cannot find host interface on host_name for item key item_name. Answer: You probably used an interface type of Agent instead of SNMP. Under Interfaces for the host, add one for type SNMP and remove the Agent one. Or, maybe the reverse: your item type is of type Zabbix agent but your host’s interface is of type SNMP – that combo also produces this error.
In Zabbix my SNMP item shows error No such instance currently exists at this OID, yet my snmpwalk for same shows it works. Why? Answer: In my case I switched to snmpget for my independent testing and reproduced that error, and found that I needed a literal .”0″ at the end of the OID (specifically for swap used on an F5 device). Once I included the .”0″ (with the double-quotes) in the OID in Zabbix it began to work. In another case I could do the snmpget from the same zabbix proxy where I was getting this error message. The custom MIB was right there in /usr/share/snmp/mibs on the Zabbix proxy. Zabbix hadn’t been started in awhile. I restarted it and the problem went away.
I wish to use a DNS value instead of an IP in net.tcp.service[service,IP,port] because I use geoDNS or round-robin DNS. Can I? Answer: It seems to work, yes.
Can I send alerts to MS teams? Answer: This is obviously a fake question. But the answer is Yes. You set up a Connector in a MS Teams channel. It’s pretty staight forward and it’s pretty cool. I’ll try to publish more in my Zabbix tips post if I have time.
Get a lot of false positives? Answer: Yes! On F5 equipment this one is vexing me:
Resolved: BIG-IP is unreachable via SNMP for 15 minutes
And for others (pool member unavailable for a few minutes) I tried to require two consecutive failures before sending an alert. Basically still working on it.
I have a bunch of HTTP items on this one Zabbix proxy. They all sort of go bad at the same time (false positives) and Zabbix says this agent is unreachable for five minutes around the same time. Answer: Seen that. Short term it may be advisable to create a dependent trigger: https://www.zabbix.com/documentation/5.0/manual/config/triggers/dependencies Mid-term I am going to ask support about this problem.
Why is the name field truncated in Monitoring | Latest Data, with no possibility to increase it? Answer: If you have Show Details selected you see very few characters. Deselect that.
What, Zabbix version 5.2 RPMs are not available for RHEL 7? Answer: that is correct, unfortunately, as of this writing. You can run as high as v 5.0.7. We are trying to pressure them to provide this compatibility. Lots of people still run Redhat v 7.
Can you send reminder alerts periodically for a problem which persists? Answer: Yes you can. For instance, every four hours. Read all about it in the manual, under Action | Escalations, and look at their examples. However, the documentation is at odds with the product’s behaviour if you have multiple alerts with different durations defined. I am studying it…
Is Zabbix affected by the same hack that infected SolarWinds? Answer: No idea. Let’s see. Developed in Eastern Europe. Basically, no one’s saying. Let’s hope not.
Is Zabbix stupid enough to send multiple alerts for the same problem? Answer: In a word, yes. If you are unlucky enough to have defined overlapping alert conditions in your various alerts, Zabbix will make no effort to consolidate them.
What does it mean when I look at a host and I see inaccessible template? Answer: Most likely explanation is that you don’t have permission to see that template.
Can the y-axis be drawn in a logarithmic scale in a dashboard graph? I have low values (time for a DNS query) which sometimes soar to high ones. Answer: No. This feature has been requested now for almost 10 years and still is lacking. I will try to make a feature request.
Why does our Zabbix agent time out so often? The message is Zabbix agent on hostname is unreachable for five minutes. The problem is sporadic but it really interferes with the items like our simple net.dns checks. Answer: If you use a lot of net.dns agent items you can actually cause this behavior if you are running agent2. The default agent item is passive. We had better luck using an Active Agent item. We had severe but random timeouts and they all went away.
Our Webhook to MS Teams was working fine. Then we set up a new one to a new channel which wouldn’t work at all. A brief error message says invalid Webhook or something. What’s the fix? Answer: It is a known bug which is fixed in v 5.0.8. Of course a lot else could be wrong. In fairness Microsoft changes the format for webhooks from time-to-time so that could be the problem. This Microsoft page is a great resource to do your own testing of the Webhook: Sending messages to Connectors and Webhooks – Teams | Microsoft Docs
The formatting of alert emails is screwy, especially with line breaks in the wrong places. Can I force it to send HTML email to gain more control? Answer: Sort of. You can define a media type where you use HTML email instead of plain text email. I personally don’t have access to do that. But it is not possible to selectively use HTML email within the Custom email form of the alert setup screen. With the more straightforward custom emails, the trick is to put in extra line breaks. A single solitary linebreak is sometimes ignored, especially if the sequence is MACRO-FUNCTION linebreak more text. But if you use two consecutive linebreaks it will inject two linebreaks.
I swear Zabbix is ignoring my macros in trigger functions used in templates which refer to time values in minutes, and just filling in 0 instead. Is that even possible? Answer: I’m still investigating this one. I will withold my customary sardonic comments about Zabbix until I know who or what is to blame. [Later] I’m thinking this one is on me, not Zabbix.
Do Zabbix items, particularly HTTP items, have the concept of a hidden field to hide confidential data such as passwords from others with the same level of access? Answer: Apparently not. But if you believe in the terrible idea of security by obscurity, you can obscure values by stuffing them into a macro.
My Zabbix admin won’t let me get creative. No external items, no ssh items, etc. I can run some interesting scripts on my linux server. How to stuff the results into Zabbix? Answer: Install zabbix_sender utility on your linux. Then set up an item of type Zabbix trapper. The link to the RPM for zabbix_sender is in the references.
These days nothing is either black or white. So when a trigger fires, it’s likely it will return to good status, and then bad, and then good, etc. The alerts are killing us and casual users tend to discount all of them. What to do? Answer: This is common-sense, but, a very good strategy in these cases is to define a recovery expression for that trigger that looks at the average value for the last 3600 seconds and requires it to be in the good range before the trigger that all is good gets sent out as an alert.
I’m using the dynamic host feature in a dashboard. Unfortunately, one of my hosts has a really short name that matches so many other hosts that it never appears in the drop-down list. What to do? Answer: Click the “select” button to the right of the search field. Then you can choose the host group and from there the host. Or rename the host to somethng more unique.
I wish to add some explanatory text in the dashboard I’ve created. Is it possible? Answer: This is laughably kludgy, but you can do this with a map widget. What you can do is to create a map, add a text box to it, and put your desired text into the text box. But it is hard to get the sizing correct as things shrink when putting the widget on the dashboard.
My top hosts widget is now displaying 0’s. Answer: This happened after we upgraded from v 6.0 to 6.0.8. In characteristically Zabbix illogical fashion, if you now sort by BottomN instead of TopN you should see the expected results (highest on top). Not all our widgets displayed this bug!
I have an item which only runs once a week. Monitoring > Latest Data doesn’t show any values. Is that a bug or feature? Answer: There is a setting somewhere where you can change this behavior. Set it to last two weeks and all will be well.
While using the pyzabbix Zabbix api I had trouble switching from username/password to use an authentication token. Answer: Perhaps yuo installed both py-zabbix as well as pyzabbix? I’m confused by this. as there is some overlap. To use the token auth method – preferred by experts – uninstall both these packages and re-install only pyzabbix. I will give an example in my other Zabbix blog post, Practical Zabbix examples.
The trigger.create api call says a dependent triggerid must be passed? Is that really mandatory? It makes no sense. Answer: No. I experimented with it and found you can just leave the dependencies out altogether. The documentation is wrong.
I need to create about 100 custom alerts. Is there seriously no way to do this via the api? Answer: apparently not.
What’s the correct way to send a compound filter expression via the api? Answer: Watch out! If you are trying to filter on suppressed problems, do not put a reference to suppressed in your filter. Instead it goes outside the filter like so: zapi.problem.get(…,suppressed=False,filter={‘name’:…})
Monitoring > Problems > History view is slow. Then it grays out periodically. Answer: Zabbix is spending all its time figuring out which host groups you have access to. To speed things up, explicitly enter only your accessible host groups in the filter.
geoMAP in Zabbix 6.0 is cool until you blow up a continent and see all the local geographical names written in their native language. So Asian placenames are inscrutable to Enlgih speakers. Is there any fix? Answer: You are probably using the provider, OpenStreetMap in this case, which is using localized names. You can switch providers (global setting).
I’m using a RegEx in the regsub function on an LLD macro. What flavor of RegEx are supported and what characters need to be escaped? Answer: Supposedly Perl-compatible (PCRE) RegExes are supported. For anything remotely complex, enclose your RegEx in double-quotes. Then, for good measure put a backslash (\) in front of any double-quote (“) you require as a match character, and a backslash in front of any slash (/) match character, plus the usual rules.
Why am I seeing the same host graph twice? Answer: This is a bug I have personally discovered in Zabbix 6.0. It occurs when you have a template with just a single item and a single graph. They will be working on it as of August 2022.
In latest data I see: Value of type “string” is not suitable for value type “numeric unsigned.” Why? Answer: I got this in Zabbix 6.4 when I used zabbix_sender with argument -o 36 which I thought would feed in the integer 36. But no, it got interpreted as a string. I tried to introduce a preprcoessing step but I could not get it to work. In the end I created a dependent item with a RegEx to convert it. I made the original item type character. I could not beat this in a simple way.
I can’t get my new agent to be seen by its Zabbix proxy. Error is failed to accept an incoming connection: from [agent]: reading first byte from connection failed: [104] Connection reset. Answer: You may be running a Palo Alto firewall perhaps? They will permit the tcp handshake and then drop the connection with a “reset both sides.” which produces this error. Thus super simplified connection tests you run by hand with nc/nmp may appear to work.
I just created my first passive agent item. The agents log shows this error: failed to process an incoming connection from x.y.z.w: cannot accept unencrypted connection. Answer: I solved this issue by editing the agent’s config to have these lines TLSAccept=unencrypted and #TLSAccept=psk
I can only measure what I seek to know as an item with type Zabbix agent, e.g., net.dns.record for testing a dns server from a Zabbix agent. Yet I want that item to be associated with the target and not so much with the agent running the item so I can create normal triggers. Answer: On your host dns server, create a calculated item and use a function such as last(/host/key) to pull in the results from the agent! Wish I had known that about four years ago!
Does changing the name of a host change its hostid? Answer: No. We have a multi-stage discovery process which relies on this fact.
Does the hosts IP filter accept a subnet mask? Answer: No, it is very primitive. It does accept a partial IP, strangely enough, so 10.9.9 matches 10.9.9.0/24.
Why is the simple item type so darn simple? In udp-land the only thing you can check is ntp. Answer: It is unfortunate, isn’t it? I need a simple DNS service check.
In preprocessing of an item, do the things in Validation produce a boolean value? Answer: That would make sense, wouldn’t it? So of course in keeping with the mantra that nothing in Zabbix should be intuitive, that is not the case. See my Prtactical Zabbix Examples for a workaround.
I have an item which is never runs, yet testing it or executing it produces the exepected results. Answer: You will do a head-slap. This happened to us recently when someone set the Update Interval to 300. It just never ran. You need to put the time quantifier in the update Interval such as 300s or 5m!
I get this error, even while testing an SSH agent item type with known good username/password: Password authentication failed: Access denied for ‘password’. Authentication that can continue: publickey,password,keyboard-interactive. Answer: When this happened to me I tested the ssh from the Zabbix proxy and found it had an old key for the host in /root/.ssh/known_hosts. I manually deleted the offending entry and all was good.
A word about SSH checks and triggers
Through the school of hard knocks I have learned that my ssh check is clipping the output from the executed command. So you know that partial data you see when you look at latest data, and thought it was truncating it for display purposes? Nuh, ah. That’s all you’re getting to go up against in your trigger, which sucks. It’s something like 260 characters. I got lucky in a sense to discover this early by running an ssh check against dns resolution of amazon.com. The response I got varied almost every 60 seconds depending on whether or not the response came out of the dns cache. So this was an excellent testbed to learn about the flakiness of triggers as well as waste an entire day.
Another thing about triggers with a regex. As far as I can tell the logic is reversed. So you think you’re defining the OK condition when you seek to match the output and have it given the value of 1. But instead try to match the desired output for the OK condition, but assign it a value of 0. I guess. Only that approach seems to work for me. And getting the regex to treat multiple lines as a unit was also a little tricky. I think by default it favored testing only against the last line.
So let’s say my output as scraped from Monitoring|Latest Data alternated between either
proxy1>test dns amazon.com
Performing DNS lookup for: amazon.com
DNS Response data:
Official Host Name: amazon.com
Resolved Addresses:
205.251.242.103
176.32.98.166
176.32.103.205
Cache TTL: 1, cache HIT
DNS Resolver Response: Success
proxy1>test dns amazon.com
Performing DNS lookup for: amazon.com
DNS Response data:
Official Host Name: amazon.com
Resolved Addresses:
205.251.242.103
176.32.98.166
176.32.103.205
Cache TTL: 1, cache HIT
DNS Resolver Response: Success
or
proxy1>test dns amazon.com
Performing DNS lookup for: amazon.com
Sending A query for amazon.com to 192.168.135.145.
Sending A query for amazon.com to 8.8.8.8.
DNS Response data:
Official Host Name: amazon.com
Resolved Addresses:
20
proxy1>test dns amazon.com
Performing DNS lookup for: amazon.com
Sending A query for amazon.com to 192.168.135.145.
Sending A query for amazon.com to 8.8.8.8.
DNS Response data:
Official Host Name: amazon.com
Resolved Addresses:
20
, then here is my iregexp expression which seems to do the correct thing (treat both of these outcomes as successes):
{proxy1:ssh.run[resolve DNS,1.2.3.4,22,utf-8].iregexp("(?s)((205\.251\.|176\.32\.)|Sending A query.+\s20)")}=0
{proxy1:ssh.run[resolve DNS,1.2.3.4,22,utf-8].iregexp("(?s)((205\.251\.|176\.32\.)|Sending A query.+\s20)")}=0
Note that the (?s) at the beginning helps, I think, to treat the newline character as just another character which matches “.”. I may have an extra set of parentheses around the outermost alternating expression, but I can only experiment so much…
I ran various tests such as to change just one of the numbers to make sure it triggered.
I now think I will get better, i.e., more complete, results if I make the item of type text rather than character, at least that switch definitely helped with another truncated output I was getting from another ssh check. So, yes, now I am capturing all the output. So, note to self, use type text unless you have really brief output from your ssh check.
So with all that gained knowledge, my simplified expression now reads like this:
{proxy:ssh.run[resolve a dns name,1.2.3.4,22,utf-8].iregexp("(205\.251\.|176\.32\.)")}=0
{proxy:ssh.run[resolve a dns name,1.2.3.4,22,utf-8].iregexp("(205\.251\.|176\.32\.)")}=0
Here’s a CPU trigger. From a show status it focuses on the line:
CPU utilization: 29%
and so if I want to trigger a problem for 95% or higher CPU, this expression works for me:
So I will accept the results as OK as long as at most one of five packets was dropped.
A lesson learned from SNMP monitoring of F5 devices
My F5 BigIP devices began producing problems as soon as we set up the SNMP monitoring. Something like this:
Node /Common/drj-10_1_2_3 is not available in some capacity: blue (4)
Node /Common/drj-10_1_2_3 is not available in some capacity: blue (4)
It never seemed to matter until now that my nodes appear blue. But perhaps SNMP is enforcing a best practice and expecting nodes to not be blue, meaning to be monitored. And it turns out you can set up a default monitor for your nodes (I use gateway_icmp). It’s found in Nodes | Default Monitor. I’m not sure why this is not better documented by F5. After this, many legacy nodes turn red so I am cleaning them up… But my conclusion is that I have learned something about my own systems from the act of implementing this monitoring, and that’s a good thing.
To be continued…
References and related
A good commercial solution for infrastructure monitoring: Microfocus SiteScope.
Just to put it out there: If you like Zabbix you may also like Specto. Specto is an open-source tool for monitoring web sites (“synthetic” monitoring). I know one major organization which uses it so it can’t be too bad. https://specto.sourceforge.net/
Since this document is such a mess I’m starting to document some of my interesting items and Practical Zabbix examples in this newer and cleaner post. It includes the baseline calculation formula.
Intro
This may be really obvious to some, but I did not realize until recently that you can adjust the touch screen sensitivity on a Samsung smartphone. Who knew?
Why it matters
I have a family member with a glass screen protector. I watched her struggle tapping and swiping multiple times to get it to respond. No one mentioned a way to fix it. Then i helped apply a new glass screen protector after the old one broke and i did an RTFM. The directions mentioned you can increase the touch sensitivity in settings. It’s no longer where it said. but in Display settings.
Intro My jump server is getting old and they’re threatening to cut it off. A jump server is a server from which you launch CLI terminal sessions into your linux servers. Since my laptop has firewall access to all the same servers I wondered if I could build up a productive environment right within Windows 10 on my own laptop. For me this would be running GNU screen as a terminal multiplexer since I hop between terminal screens all day.
More details Windows 10 is coming around to more fully integrating with Linux! it’s about time. WSL, windows subsystem for Linux, is all about that. And things like bash shell, ubuntu and OpenUSE Linux are available from the windows store. But that was not an option for me. My organizaiton has shut all that down.
So I thought back to my days as a Cygwin user those many years ago… Could I get GNU screen running within Cygwin environment on Windows 10? Well, yes, I can with just a few tweaks.
I think the initial Cygwin install required admin privileges, but once installed to run it does not.
Within Cygwin screenis an optional package and you can run their setup program to search and install it.
Here is my .screenrc file
defscrollback 4000
#change init sequence to not switch width
termcapinfo xterm Z0=\E[?3h:Z1=\E[?3l:is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l
# Make the output buffer large for (fast) xterms.
termcapinfo xterm* OL=10000
# tell screen that xterm can switch to dark background and has function
# keys.
termcapinfo xterm 'VR=\E[?5h:VN=\E[?5l'
termcapinfo xterm 'k1=\E[11~:k2=\E[12~:k3=\E[13~:k4=\E[14~'
termcapinfo xterm 'kh=\E[1~:kI=\E[2~:kD=\E[3~:kH=\E[4~:kP=\E[H:kN=\E[6~'
# special xterm hardstatus: use the window title.
termcapinfo xterm 'hs:ts=\E]2;:fs=\007:ds=\E]2;screen\007'
#terminfo xterm 'vb=\E[?5h
defscrollback 4000
#change init sequence to not switch width
termcapinfo xterm Z0=\E[?3h:Z1=\E[?3l:is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l
# Make the output buffer large for (fast) xterms.
termcapinfo xterm* OL=10000
# tell screen that xterm can switch to dark background and has function
# keys.
termcapinfo xterm 'VR=\E[?5h:VN=\E[?5l'
termcapinfo xterm 'k1=\E[11~:k2=\E[12~:k3=\E[13~:k4=\E[14~'
termcapinfo xterm 'kh=\E[1~:kI=\E[2~:kD=\E[3~:kH=\E[4~:kP=\E[H:kN=\E[6~'
# special xterm hardstatus: use the window title.
termcapinfo xterm 'hs:ts=\E]2;:fs=\007:ds=\E]2;screen\007'
#terminfo xterm 'vb=\E[?5h
lt;200/>\E[?5l' termcapinfo xterm 'vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l' # emulate part of the 'K' charset termcapinfo xterm 'XC=K%,%\E(B,[\304,\\\\\326,]\334,{\344,|\366,}\374,~\337' # xterm-52 tweaks: # - uses background color for delete operations termcapinfo xterm ut #from https://stackoverflow.com/questions/359109/using-the-scrollwheel-in-gnu-screen termcapinfo xterm* ti@:te@ escape ^\\ # changes espace sequence password
lt;200/>\E[?5l' termcapinfo xterm 'vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l' # emulate part of the 'K' charset termcapinfo xterm 'XC=K%,%\E(B,[\304,\\\\\326,]\334,{\344,|\366,}\374,~\337' # xterm-52 tweaks: # - uses background color for delete operations termcapinfo xterm ut #from https://stackoverflow.com/questions/359109/using-the-scrollwheel-in-gnu-screen termcapinfo xterm* ti@:te@ escape ^\\ # changes espace sequence password
Note that in my .screenrc I use <Ctrl-\> as my escape sequence, so, e.g., to pop to the previous screen it is <Ctrl-\> <Ctrl-\>. I’m not sure that’s standard but my fingers will remember that to my dying day. They probably still remember some of those EDT/TPU VAX editor commands to this day!
Compare and contrast Here are my day 0 observations.
ssh, curl, nslookup and tracert are coming from the underlying Windows system (do a which curl to see that) so that means you get the dumb version your system has.
So there is no dig, and no nc or netcat.
touch, cat, mkdir and vi behave pretty normally. man pages are installed, which can be a help.
If you use proxy, a funny thing can happen and your environment variables can get mixed. You may have inherited an HTTP_PROXY environment variable form the system, but the alias you copied from a linux jump server probably defines an http_proxy environment variable (lower case). And both can co-exist! As to which one curl would then use, who knows? Better just stick to working with the upper-case one and NOT define another in lower case.
For awhile it looked like scrolling was not working at all when screen was running. Then i found that tip I reference at the bottom of my .screenrc file which makes scrolling work via the mouse’s scroll wheel, which isn’t too bad.
Old friends like ls, grep, echo and while (built-in bash command) are available however. dig can be installed from the bind-utils package.
A lot of other packages are optionally available, including a whole X-Windows environment, which I used to run in the past but hope to avoid this time around.
No crontabs however (to have cron daemon requires installing admin privileges) which kind of hurts.
Simple output redirection seems to work, as does job control, e.g.,
ping -t 8.8.8.8 > /dev/null 2>&1 &
ping -t 8.8.8.8 > /dev/null 2>&1 &
Not sure why you’d want to run the above command, but this nice example shows that the /dev/null device exists, and the ping command is inherited from your Windows environment hence the -t option to run it indefinitely, and that it will create a background process which you can view and control with jobs / kill.
Now I typically move my laptop off the work environment each night, so all my ssh logins will be lost, unlike the jump server situation. But our jump server isn’t that stable anyway so no big loss I’d say…
I am sooo used to highlighting text in Teraterm, which is my current environment, and that being sufficient to put that text into the clipboard, that I keep doing that in this environment. But it doesn’t work. I have to use the CMD window convention of highlighting the text and then hitting ENTER to get it into the clipboard. oops. That was because I had been launching Cygwin from a CMD window. Now I am launching from a proper Cygwin shortcut and simple text highlighting works, BUT, right-clicking to paste it in brings up a menu rather than just doing it! So there’s that difference now… Instead of right-click I can quickly paste the text in doing a SHIFT-Insert.
ssh will get you
By default you end up using the Windows-10 supplied ssh, and that works pretty well. But when you’re ready to advance and need to put some thing into a .ssh/config file, forget about it. In principle it’s possible in Windows 10, but it’s too complex. Just install the ssh package. That in turn permits you the facility familiar to you where you can create a ~/.ssh/config file.
How to set your userid by default for your ssh logins
First make sure you install the Cygwin ssh package and are using that one. A which ssh should come back with /usr/bin/ssh.
My config file looks like this:
Host * User drjohn
That sets my default userid to be drjohn on any random server I ssh to.
New ssh error pops up
Unable to negotiate with 50.17.188.196 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
This only happened when I switched from my Windows ssh to the Cygwin one. This is, of course, when connecting to a system (ironically, a firewall) with an old image. I think the only solution to be able to access these old systems is to switch back to the Windows 10 ssh – after all we never got rid of it and it used to work. Since all my customary ssh’s are aliased, this works well enough. I just made an alias like this
alias oldFW='screen -t oldFW /cygdrive/c/windows/system32/openssh/ssh.exe [email protected]'
since on my system the Windows 10 openssh is installed there in the system32 folder.
How do you get multiple login sessions (shells) within your screen to the localhost?
Well, you can’t just do a su – and you probably don’t have an ssh daemon running locally, so this is more of a non-trivial question than it first appears.
I define a bunch of aliases. My alias for getting an additional shell on the Windows 10 machine is this:
alias local=’screen -t localhost bash –login -i’
A word on package management I don’t know why I was afraid of installing packages when I first tried Cygwin over a decade ago. Now for me that’s the key – to understand and practice installing packages because it’s actually really easy when you’re used to it.
The key is to simply keep your initial install setup hanging around, setup-x86_64.exe. In my case it’s in my downloads directory. Example usage: I wondered if I could install a decent version of ping rather than continually suffer with the dumb DOS version. So, fire up the above-mentioned executable. Go through a few screens (where it remembers the answers from the initial install), then search for the package (Yes, it’s there!), and select to install the most recent version from the drop-down. A few more clicks and it’s done and available in your path. it’s that easy… Not sure about uninstalling because you almost never need to do that. It seems maybe a thousand packages are available? so no, there’s no yum or zypper or rpm or apt-get, but who really needs those anyway?
As a concrete example, I am learning about SNMP. So I got something running on a Bluecoat proxy, and I wanted to see what I could see. The guide recommended using snmpwalk, which of course I did not have. So I learned which package it is in with a DDG search, then ran the Cygwin setup, found that package, installed it, and voila, there was snmpwalk in my path. And it worked, by the way. Easy peasy. Creating your own scripts If you have the funny situation, like me, where you had enough privileges to install Cygwin, perhaps by temporarily assigning your account the Admin role, but when you use it day-to-day, you do not have admin privileges, you will find yourself unable to create files in some of the system directories like /usr/local/bin – permission denied! But in your home directory you will be able to edit files.
So what I did is to create a bin directory under my home directory, where I plan to add my home-grown scripts such as mimeencode, and make sure my PATH includes this directory with a statement like
export PATH=$PATH:${HOME}/bin
export PATH=$PATH:${HOME}/bin
which I put in my .alias file, which in turn I source from .bashrc.
2021 update: The fate of the screen package
I read somewhere the screen utility which I love is beyond repair and will have to be replaced by something else. Too bad. I’ve used it for about 10 years now.
X Windows
In a previous iteration of Cygwin I had installed the X Server components though I left it out this time around. For an X Server running on my PC, which I do need from time-to-time, I use MobaXterm. Seems to work OK for my purposes, which are very minimal. But I prefer to use Cygwin over MobaXterm for the command line stuff I do.
Conclusion GNU screen for Windows is indeed possible, but you gotta run it on top of Cygwin. It’s of interest that after all these years Cygwin is still viable on Windows 10. Cygwin can be run in a pretty lightweight fashion if you avoid the X-Windows stuff. There are some quirks but it is surprisingly linux-like at the end of the day. I believe it is really suitable as a replacement for a linux jump server. screen, for the uninitiated, is a temrinal multiplexer, which means it makes it very fast for you to switch between multiple terminal windows.
Some things are a bit different.
I think I will use this both at work and at home… Nope! My home PC runs too darn slow to ever use the Cygwin environment. My work laptop has SSD which probably helps keep performance good.
It is possible to set up an ssh default user.
It is possible to create multiple local shells within one screen within one Cygwin terminal.
So it is really possible to have your Linux command line. I use it every day…
2022 update
WSL2 is the way to go now. The setup can be little tricky, however, but it is worth it. You get a full hypervisor environment, not an emulator as you have with Cygwin. I write it up here. References and related
If you have a linux jump server that runs screen, or just want to ssh to a linux server, teraterm can be a good choice (as opposed to putty or built-in ssh). These days it can be found here: https://osdn.net/projects/ttssh2/releases/
To have an X Server running locally, MobaXterm seems a good choice. It looks like it’s free: https://mobaxterm.mobatek.net/
Intro
I am just beginning to use Microsoft’s Azure cloud environment. Although I am inclined to be a fan of AWS, I haven’t looked at AWS networking for awhile, and the last time I did something I felt totally lost in trying to understand their terminology.
But in spite of my natural inclination to support everything Amazon, I gotta admit that Azure was a good, usable environment for what I needed to do; swap the public IPs on two VMs.
The details
I was not getting any help whatsoever from with my organization. But I did at least get sufficient access to the Resource Group where my Redhat 7.4 VM was running. That was a godsend.
In Azure network interfaces are resources. They have IPs like 10.0.1.4, 10.0.1.7, etc.
Public IP addresses are resources. They have IPs apprporiate for your region. They are typically associated with a network interface.
A network interface in turn is associated to a VM, typically.
For some reason which no one could explain to me, I could no longer patch my RHEL 7.4 server. That began about September 2019. Meantime, I was using an application which relied on a built-in package. Now Redhat always ships with old versions of everything, so running this old version plus lack of patches really put pressure on me to upgrade to a new OS. Can you do an in-place upgrade? As far as I can tell, no. I went with SLES 15 SP1 on a new VM within the same resource group and data center since I have some familiarity with Suse Linux. That OS had a newer version of that open source package, plus it could be patched.
But the IP of the old server was embedded in several places and switching it was not an option. What to do? What to do? Can you even swap IPs on two VMs within the same Resource Group? Who knows?
Well, it turns out you can. The documentation on the topic is pretty good and cleared up some things for me. Particularly the first two links in the references at the bottom.
I took this approach.
Changed the IP from dynamic to Static (go to configure section when looking at this resource). This should have been done from the get-go, but wasn’t. Who knew?
Dissociate the IP from the network interface.
Changed the second IP from dynamic to static.
Dissociate this IP from its network interface.
Associate IP to network interface of the SLES 15 server.
Associate second IP to network interface of the RHEL server.
And that’s it…. It worked like a charm.
Then I cleaned up some old public IP addresses which weren’t being used. You have to remember there is a shortage of IPs. So a lot of the quirk you encounter are due to their utilizing Ips as sparingly as possible. makes sense to me. For instance you can have a “public IP” resource which has no value! it may not get a value until its absolutely needed by virtue of being associated with a network interface on an active server. Stuff like that…
Conclusion
Yes, you can indeed swap public IPs on two servers if they belong to the same Resource Group and I guess the same data center. I know because I did it. As a bonus I found that the Azure documentation is pretty clear and sufficiently detailed.
Intro
This is yet another tale of things in the IT world often do not turn out the way it seems at first blush. Or possibly a tale of just when you think you’ve seen it all after decades in the industry, something new (to you) occurs.
What’s going on
The firewall team was all busy so when this strange problem occurred Friday they called in the second string: me. I consider some of the team to be less-than-customer focused so I try to compensate for them and for my lack of knowledge about the firewall by applying a more customer-first attitude. In other words, a sympathetic listening ear. These days it can be hard just to find someone to complain to about your It problem, and I am keenly aware of that.
There was some strange communication which wasn’t working, mediated by a firewall I had never accessed and was not sure i even had access to. So of course I was asked to join a big conference call where an ongoing debugging session was taking place.
I refused.
I hate being blindsided, and i hate not having answers, making me sound even less competent than I already am.
But what I did do is being my research to see what the system is, if perhaps I had access, etc.
Yes. I found that through a management system I have access to I had access to view the policies on that particular firewall and view the logs as well.
So once I had that up, I agreed to join the call.
They had one server communicating to three different systems. Only one of the three systems was being reached. Yes the other two were on the same subnet. Two of our firewalls were between the system and the three servers.
And, yes, i could see some drops. The interesting TCP error stated: TCP packet out of state, first packet isn’t SYN.
No problem. routing must be screwed up such that we have asymmetric routing. It happens all the time. Right? well these systems are really appliances with only some basic networking information configurable, not real debugging facility, and really no ability to add a host route.
I could not establish a shell session onto the firewall – not sure what the password naming scheme was that they used.
Then a real firewall guy comes on the call. But his connectivity is messed up, so I keep with the debug session, if nothing else than to support him since four eyes is more effective than just two. He shares the routing tables of our two in-line firewalls. It’s hard to understand as these are all new subnets for me, some are ones that don’t look right. But just focusing on possible host routes for any of these three servers, I don’t see anything amiss.
Firewall policy
And, in firewall policy I see the entire subnet has this traffic permitted. There is no rule specific to one or the other of these systems.
So what do we have up until now?
A purist firewall administrator attitude would be as follows:
The firewall treats all these systems the same, therefore this cannot be a firewall problem. Talk to your networking or system people. Have a nice day.
Well, in fact there was some serious question about the network switch as well. So we had a network guy on the call. So they dug up the MAC addresses of these systems, from which they found the switch ports. Then they checked the port configuration. Ah, some complex 802.1x authentication was configured. As I understand this means the device would not even be allowed onto the subnet until it passed some kind of Radius authentication. So they removed this 802.1x stuff and just made sure that port was assigned to the right vlan.
Still, the problem persisted.
I think the other firewall guy was also new to this equipment. Eventually, though, he tries to do a packet trace of the one that’s working versus the one that isn’t.
You know, I never saw the results of those traces, but I’m pretty sure, reading between the lines, that they surprised him, meaning, they did not fit the hypothesis of the asymmetric routing.
In these situations there is the main communication in the mian session, then side communications going on, like between me and the firewall guy. But it is all chaotic. Acoustics are mediocre, accents are hard to understand. So the net transfer of information is pretty low. Statements, even important ones, often have to be repeated multiple times (rebroadcasts) to assure everyone “gets it.”
Typical questions were asked. When did this last work? what had changed? There were a couple changes. Some kind of networking thing (I forget what), and then the firewalls changed management systems after that. The firewall change seemed closer in time to the last known success.
You acquire more and more information as you dig into problems. It’s hard to judge which is relevant at the time and which lines if inquiry are a complete waste of time. A good incident manager or project manager can sense which are the more productive lines of investigation and nurture those discussions while suppressing the noise.
Actually it was the networking guy who found the Checkpoint link below. I looked at it. the firewall guy was muttering something about badly behaved, older applications that might exhibit this behaviour.
So we agreed to take the suggested steps, which would basically allow these out of state packets. Drat. The firewall returned an error.
But I continue to refresh the firewall logs. The communication was occurring about every minute. Lo and behold, I see the older drops, and then accepts for the last few minutes! I think it worked. I tell them to check.
They check their end. Sure enough. Communication beginning to work…
The customer tries to make assertion that this was a firewall problem all along. Not so fast. Firewall guy says, well, the firewall is doing exactly what it’s supposed to be doing. who’s right?
We’re all good for now, but we state this is a kludge for today and a follow-up meeting needs to occur.
So what happened?
I think the single most important thing is that the firewall guy switched his problem hypothesis from Must be asymmetric routing, to Maybe it’s a badly behaved application. Meaning what? What if you have an application that establishes a TCP connection, and then to beat idle timeouts, sends a KEEP ALIVE packet every minute? Well, now, suppose your firewall is rebooted in the middle of that because it has changed management stations and needs to reload policy? What might the situation look like to it?
It you were unlucky, it just might see these KEEP ALIVE TCP packets without having the connection in its connection table, in other words, exactly the situation we are observing!
What should have happened?
It would have been great if the communication were forced to be re-established form time-to-time, even once a day. This problem had been going on for days.
But, given this very stupid behaviour on the part of this application, if the app people had been aware they should have forced their application to re-establish the TCP connection after the firewall reboot. Probably, for the one that did work, it had been forced to re-establish.
A firewall person has to be sufficiently aware to realize this could be happening, and advise the app owner on what to do to prevent it.
Conclusion
So whose problem is it?
To the app people it looks like a firewall issue, cut-and-dried. To a firewall guy it looks like an application issue, cut-and-dried. I see both sides. It is some of both. An app owner has to understand enough about firewalls to see that this type of thing can occur. Assigning blame to one side or the other, as most people are wont to do, is not productive. Only a team effort could have revealed this issue. And recall that the “fix” is actually a kludge that lowers security.
Intro
A company known to me was a contented user of WebEx until they noticed a strange behaviour: their calls were losing quality or even dropped exactly after one hour. No one, most especially the vendor of WebEx, had the slightest idea of the root cause. Read on to see how this fascinating case is playing out.
Scene: company offices, Sao Paolo, Brazil
Triago, a very competent IT professional located in Sao Paolo was the first to report the problem. More-or-less it went like this:
– when he uses the call-my-computer feature in WebEx the call quality is fine, until he has been on the meeting for one hour. At the one hour mark the voice quality of others (from his perspective) dropped dramatically. Sometimes the call was completely lost. Then, about five minutes later, the quality was OK again.
– the problem only occurred when in the office or using VPN, i.e., when using the company network
– others in South America are having the same issue
– he can use the same company laptop, on the Internet, and will not have the problem
After the usual finger-pointing amongst various vendors a debugging plan was created.
It’s going really well. There have been about 12 test calls, stretched out over the last five months. You have to admire the chutzpah of US software vendors who sell to major customers and then still manage to treat them like crap come time for support.
The pattern, more-or-less, goes like this. test call with several vendors plus Triago and I in the US. Wait around for an hour, produce the problem. Wait for software vendor to “analyze”. Wait for two weeks. Some small insight may be gleaned by them. Conclusion: another test is needed, we didn’t have all the traces we need. rinse and repeat.
Scene: A soulless office park somewhere in northern New Jersey
To be continued, literally…
Scene: an enterprise-class server room somewhere in Research triangle Park, North Carolina
If one uses the company’s guest WiFi, one uses the company’s firewall, but not the company’s proxy server. This test succeeds. But, unfortunately, one also uses UDP rather than TCP for the communication because that is the default. See the references for communication requirements.
So one thought is to knock out the ability to use UDP by blocking UDP port 9000, thereby forcing use of TCP. Testing that today….
Intro
Because someone else on the team had a good experience with it, and so i can exchange notes and get some tips form them, I went ahead and ordered the Anet A8 3D printer + an extra roll of PLA. That is, despite the vocal negative reviews which exist.
Although normally I don’t consider myself very mechanical, I guess I’m pretty good at following instructions. So I’m watching their detailed Youtube video and mimicking their actions, step-by-step. And I’m having a blast! It’s like having an erector set all over again.
And I’m in constant awe that so much could be had for so little. There must be, what, a couple hundred parts, including five stepper motors, several limit switches, a circuit board, lcd, power supply, steel rods, belts, arylic(?) parts, and 0.5 Kg of PLA – all for $140?? I feel like its raw value is several times that. Hey, I just bought two m5 screws and nuts from Home Depot and paid almost $3.
Not exactly like the video
Things were going great, until the kit actually deviated form the parts shown in the video! Especially where the kit provided a two 3D-printed parts shaped significantly differently from what they show. Assembly slowed down after that.
The rods did not fit through the 3D parts as they should have – I had to bore them out a bit with one of the yet-unused threaded rods that came with the kit! Which did work by the way + a lot of muscle and hand strength.
Partially assembled – extruder not yet inserted.
Many gray hairs later I finished the assembly. If you’re familiar with furniture assembly of a semi-complex piece like a desk with top shelves, this is about three times more complicated. You keep hoping that OK, now it’ll get easier and I’ll cruise through this. But it never does! Every stage features unique steps presenting their own unique challenges.
In my case my cooling fan on the extruder, which looks suspiciously like a 3D part, hangs below the extruder nozzle, so I don’t think I can use it.
Then once it was assembled my first test print, a simple box, went awry. Around the third layer the whole thing started moving around on the print bed! Once again I was hoping that the hard part was the assembly and then I could cruise to printing to my heart’s content. Wrong. Now comes a whole new set of challenges unique to 3D printing, and you have to master that as well. This is nothing at all like going to Staples to get an ink jet printer where the most challenging thing you’ll have to do is change an ink cartridge. Nothing at all like that. This is more like constructing a model railroad yourself.
Things get smoother
So I took everyone’s advice and installed Cura 3D as my slicer. Then I just decided to go for it and print out my first upgraded part: a center nozzle fan. It worked really, really well!
My very first print – a center nozzle fan
You can almost see form the picture that the quality was really good. I had to sandpaper the chute a little for it to fit – apparently ABS sandpapers easily – and voila, it snapped into place.
Next I printed a filament guide. Also no problems there.
Then my filament broke.
Some terms
slicer – software which takes an STL file an translates it into a series of layer-by-layer movements. Cura 3D is the slicer I use.
gcode – the file format that the anet a8 printer understands. You take an STL file, put it into your slicer, and have it produce a gcode file for you that you print.
PLA – the type of plastic most often used for 3D printing at home
