An OpEd by Marcus Adams. Opinions stated here do not necessarily reflect the stance of Appalachian Insider. More information about the author can be found at the bottom.

Some parts of our infrastructure are incredibly vulnerable and the sad thing is, they don’t have to be.

Attacks on American infrastructure have been increasing, and will probably continue to do so into the future. This past Wednesday FBI Directory Christopher Wray stated, “China’s hackers are positioning on American infrastructure in preparation to wreak havoc and cause real-world harm to American citizens and communities, if or when China decides the time has come to strike”.

I used Microsoft’s AI image generator to generate most of this image, but overlaid a screenshot of a real world HMI onto his display.

Infrastructure is an attractive target for attackers because it represents something where a single target can have a widespread effect. You take down a water company and all of a sudden thousands or even millions of customers are without safe drinking water, including government facilities reliant on that service. It is therefore imperitive that our nation’s critical infrastructure be protected. The sad reality however is that much of it is not. All you have to do is swing by Shodan or look up one of the VNC bots posting to social media and you’ll see some stuff that should absolutely not be on the internet.

So, I’m going to discuss one problem that I see popping up quite a lot and suggest a solution. Don’t get me wrong, there are many disparate problems with the various systems and processes that make up our national infrastructure, but I don’t know enough details about how all those systems to work to sit down and write a book redfining national security. Instead I’m going to focus on what I have actually seen and what I actually know.

There’s a bot on Mastodon I follow called “VNC Resolver”. This isn’t the only one I’ve ever seen, and it doesn’t do anything malicious, but what it does is scan and post links to unsecure VNC servers that it finds on the internet. The information it posts is all legally and publicly available if you’re looking. VNC is a protocol that lets you view or control the screen of remote devices. When used correctly it’s a perfectly valid tool that is not malicious in and of itself. Look at this page for more than just a few seconds though and you’ll see that many of the screenshots it posts appear to be HMI interfaces for various machinery; generators, grain dispensers, etc. These devices should not be internet accessible. That’s not to say they shouldn’t be accessible via the internet from remote locations, but that can be accomplished without exposing them to unauthorized access. You can absolutely use the internet as a backbone for remote access, without being “on” the internet.

A screenshot of the “VNC Resolver” Mastodon account posting information about an exposed system.

First, let’s make some deducations based on what we can see immediately:

The system has no password on the VNC connection
The system is either connected directly to the internet, or the port for VNC has been manually forwarded thru the router to this device.
The person is most likely using a Verizon mobile hotspot, based on the “myvzw(.)com” domain name. Therefore, this device is most likely on a remote site where the owner’s home or business internet can’t reach.

It doesn’t do any good to complain about things if we aren’t prepared to offer a solution that meets this customer’s needs. What goal is trying to be accomplished here? Obviously, this is a system that the owner wants or needs to be able to monitor and manage remotely, and by “remotely”, I mean from outside their own home network. The solution this product’s vendor has chosen has been to just stick a cellular hotspot to it, turn on VNC and leave the firewall wide open. Let’s improve on that.

These are the major steps I would take to fix this specific situation, and others like it:

Put a password on VNC
Put a router between the cellular hotspot and the device
Set up a VPN connection for securely accessing that device thru the router

None of these steps require major changes to the existing devices or software products being used. You could obviously go above and beyond, but these steps can be accomplished for not much more than the cost of a decent router to be used at the remote site.

The first step should be to put a password on the VNC connection. Most (all?) VNC clients support saving the password for a particular connection, so password protecting the VNC connection doesn’t have to bother the farmer who just wants to manage their machinery. A simple password would stop the majority of drive-by attacks.

A screenshot of the “Remmina” VNC client software. I have highlighted the “User password” block to show that the password can be saved.

The next thing I would do is put a router between the cellular hotspot and the device itself so the device you’re monitoring is not directly connected to the internet. If the cellular hotspot “is” a router and it has had port forwarding turned on for the device, port forwarding should be disabled so that the port for VNC is not directly visible as being “open” to the outside internet.

In that router, we’re going to look for two settings. One of them is “dynamic DNS”. Most routers have this setting, but basically there are a variety of services/providers who will give you a domain name that you can then have linked to your current public IP address in real time, and your router can be configured to do this automatically. What this accomplishes is it allows this device to be found remotely, instead of with an IP address that may change, with a fixed domain name like “farmerjohngraindispenser1.ddns.net” that doesn’t change. I would register a domain name with one of these providers (I personally use noip.com), either free or paid, and then configure this router to use that service and domain name. The specifics on how you do this in each router are different, so you’ll have to reference the documentation for your router specifically.

A screenshot, taken from the TP-Link website, showing how the DDNS setup page might look on one of their routers.

The next thing I would do is set up a VPN connection. Many routers come with built-in VPN server functionality. You’ve probably heard tons of marketing jargon about how VPNs keep you secure, but what they are actually intended to do is to make you “logically” appear to be a member of the remote network you’re connecting to. It creates a secure, encrypted tunnel between you and that remote network. Let’s imagine you have various network services, like a shared drive, maybe a network printer and other things you would like to access from anywhere in the world. Forwarding all those ports would be a security nightmare. Not only would you be making your attack surface much larger, but you would have to tightly control each service individually to make sure it’s as secure as possible without breaking its functionality. Instead, the safer option would be to set up a VPN server and only forward one port, the one to connect to the VPN.

A screenshot of the “VPN Dashboard” on my GL-iNet router where you can set up a VPN server

Setting up a VPN server on the router would allow somebody with an account/profile on that service to place themselves logically on that same network, regardless of what network they are physically operating on. This means that you would be able to access things like VNC servers on that remote network without having to open the VNC port to the whole world. Any attackers looking at this network would only see a single open port; the one for the VPN server, which is useless to them without a pre-configured account on that service. For the end-user, once everything was set up properly, the only additional work that would be required on their part “might” be to click a button to connect the VPN.

A screenshot from my laptop showing that to connect to my “Home” VPN, all I have to do is click a single button, which I have circled in red

Following these steps would accomplish the following:

Password protect the VNC connection
Hide the existence of the VNC service from the internet altogether
Create a secure, encrypted tunnel between the remote site and the monitoring station that the user intends to actually interact with at their home/office
Faciliate the same end-user experience of being able to use the systems they already have to remotely manage their devices

If this system is sold as a package or bundle from a vendor, there’s no reason whatsoever that it couldn’t be built this way and still be simple to deploy and use out-of-the box. Security is hard sometimes because it often comes at the cost of use-ability. The more secure something is, “generally” the harder it is to use. It’s a constant struggle to find that balance between security and use-ability. But a lot of the time that complexity can be abstracted away and presented to users in a way where they don’t have to worry about the nitty gritty details about “how” something works, only that it does. This is one of those situations.

Looking thru some of these posts has me deeply concerned about what kinds of companies are building and selling these products to folks without telling them that they are vulnerable. And this isn’t just an American problem; it’s a worldwide one. Many of the devices listed on the Mastodon account I referenced earlier are not in America. They’re all over the world.

An image of some kind of device located in Istanbul; pulled from the “VNC Resolver” Mastodon page

Any company that would sell a product and set it up in this way, especially to critical infrastructure, needs to be put out of business. With systems like these connected to our farms, our water treatment plants, etc., hackers don’t have to pull a bunch of Hollywood hacker nonsense, all they have to do is open VNC Viewer, punch in an IP address and they have easy access to systems that could be used to annoy, or even cause harm to untold numbers of people. And it would be more understandable if the solutions to these problems were hard, and some of them might be. But with this specific issue, the exposed VNC servers, the vendors of these products aren’t even trying. They’re throwing together the cheapest solution they can come up with that works “right now” to make a quick buck, without putting any thought into the long term security of the things they create.

An image of a “hacker”, generated by Microsoft’s AI image generator

A few weeks ago a hacker group that goes by the “People’s Cyber Army of Russia” took over a water treatment plant and were able to manipulate machinery and cause one of the water tanks to overflow. A couple of weeks before that I was checking out an image of an exposed VNC server and discussing with folks on the internet what we thought it might be, and we eventually came to the conclusion that it appeared to be some kind of water treatment plant. I don’t recall where this one in particular was located, but it’s a weird little anecdote about how I was literally looking at an exposed water treatment plant a few weeks before one got hacked by the Russians. Other Russian groups, like “Sandworm”, have been conducting operations against water treatment facilities across the US.

Part of me wants to start following up on these exposed devices; contacting their owners and letting them know that they are vulnerable to attack. Part of me however realizes that not everybody may appreciate that message, and may even decide to come after me, assuming I’m some sort of scammer or hacker myself. The governor of Missouri accused somebody of being a “hacker” and tried to hold them legally liable for looking at the source code for a web page and telling them about a problem they found. Apparently pressing the F12 key in your web browser makes you a hacker. This was a perfect example of how many people in positions of leadership are technologically incompetent, so one misunderstanding about your intentions, or the nature of your activities and they have the power to put you in jail over something they don’t understand.

So, I think for now at least, instead of contacting folks directly, I just want to put information out there. To let people know that this is a problem, that the solution isn’t difficult, and any vendor who tells you it is doesn’t deserve your money. Is my solution perfect? Absolutely not. Everything can be hacked. There could be a vulnerability in the firmware of the router you choose that allows a remote attacker to gain control over it, there could be a vulnerability in the version of the VPN server your device uses, etc. There is no such thing as perfect cybersecurity. However, if you reduce your footprint and what people are able to easily find out about you, and make it hard enough to get into your systems, attackers will probably just move on to lower hanging fruit.

An image of a tree with fruit, and some small computer looking objects on it while a man is picking some of the fruit. Generated with Microsoft’s AI image generator.

So stop exposing VNC directly to the internet. This is stupid, grade school level cybersecurity mistakes that professional adults who are selling a product to other adults shouldn’t be making. It reeks of either gross incompetence or malice and honestly, it makes you wonder whether there shouldn’t be some kind of federal investigation into stuff like this. To be clear, I don’t blame the farmers or the government workers, I blame the crooks who build these systems and sell them knowing full well how vulnerable they are and what they’re going to be used for. In summary, if you currently have a system at a remote site that uses VNC exposed to the internet to allow you to manage it remotely, here’s the things I suggest you do to secure it:

Add a password to VNC
Put a router between the cellular hotspot and the device you’re monitoring. If there already is one, stop forwarding the port for VNC
Establish a free (or paid) domain name with a service provider like dyndns or noip, and configure the router to keep it updated
Set up a VPN connection between that router and the device at your home/office you use to monitor or interface with that remote system

Let’s help keep not just America and Americans, but the world safe. It’s one thing when a hacker scams you out of a few dollars by making you think you have a virus on your computer. It’s another thing entirely when nation states are giving a paycheck to those hackers to target our food, water and health care systems. Some of this stuff will be hard to fix, but we’ve got to start or something terrible, that we have the power to prevent, is going to happen.

Shameless Personal Plug

Writing blogs like this takes time and effort. If you have found value in my work or if it has benefited you in any way, I would appreciate it if you would consider donating to help compensate me for my time. There are several ways you can do this, so here they are in no particular order.

PayPal: https://paypal.me/gerowen

LiberaPay: https://liberapay.com/gerowen

Bitcoin (BTC):

bc1q86c5j7wvf6cw78tf8x3szxy5gnxg4gj8mw4sy2