Favorite Free Applications to Troubleshoot Visual Manufacturing and Windows – Part 10 Wireshark

January 2, 2021

I have been working with Infor Visual Manufacturing ERP system for roughly 21 years now, and have been assisting users at other companies with Visual Manufacturing issues for the last 20 years. In 2019 with Yahoo announcing the eventual shut down of their Yahoo Groups platform, I worked with two other long-term members of the Visual Manufacturing VMIUGLIST and VMIUG-TEC groups on Yahoo to set up those groups on the Google Groups platform. Since the groups were set up on Google Groups, I have periodically posted technical articles of various topics to the VMIUG-TEC group to help group members solve various types problems and to take better advantage of the Visual Manufacturing package. The most recent article series describes various free operating system commands and utilities that I have used to troubleshoot various types of issues with Visual Manufacturing, Windows servers, and networks. The article that I published in the group a few days ago is titled: “How To: Favorite Free Applications to Troubleshoot Visual and Windows – Part 10 Wireshark”. Eleven years ago on this blog I posted a series of of articles titled Network Monitoring Experimentations, where I showed how to use various functionality in Wireshark. As there is a strong tie-in with that article series and the latest article that I posted to the VMIUG-TEC group, I decided to re-post the VMIUG-TEC article here (with no changes).

————————————————————————————

This article is the tenth article in the series of articles that describe free applications and utilities for troubleshooting Visual Manufacturing problems and general Windows reliability.  There are many such applications and utilities, but this article series will cover those that I have personally found helpful over the years.  Today’s article describes Wireshark (https://www.wireshark.org/download.html ), which is a very popular network activity capture and analysis tool.  Prior to 2006 Wireshark was commonly known as Ethereal, and was renamed to Wireshark when the original developer started working for a different company in 2006.  If I recall correctly, I first started using Ethereal in either late 2001 or early 2002 while testing the effectiveness of an iptables-based Linux firewall that I built from the ground up using a Linux firewalls book and a copy of Red Hat Linux.  Since that time, Ethereal and Wireshark have quite possibly assisted me in the resolution of more computer / network problems than any other tool that I have described so far in this article series.  A few such examples (including Wireshark screen captures when possible):

• Greatly increase the throughput of file transfers.  A scheduled file transfer from a Windows Server 2016 server to a Synology NAS over a gigabit network connection executed at an average rate of 34 MB/s as long as the server was otherwise idle.  If another scheduled file transfer, for example copying Oracle backup files to the Windows server, was also in process, both file transfers would transfer at a maximum rate of four MB/s.  If BackupExec running on another server was actively backing up files on the Windows Server 2016 server, the speed of the scheduled file transfer would drop to just four KB/s!  With the help of Wireshark problems with the network switch, network cables, and Windows server configuration with four network ports connected to the same switch (network team created with the Microsoft Network Adapter Multiplexor Driver) could be eliminated as problems, even though some unexpected behavior caused by “Large Send Offload” being enabled was found in the Wireshark’s logs.  Once the problem with the scheduled file transfer was fixed, the scheduled file transfer executed at a rate faster than 113 MB/s (close to the maximum real-world throughput for gigabit network connections) even while BackupExec was copying files from the server at a rate of about 103 MB/s.  Two actual problems were found in this case.  1) When processes (ROBOCOPY in this case, which was mentioned in part two of this article series) are run in a Windows scheduled task, those processes are given a low I/O priority (as shown in Windows Resource Monitor) and a low memory priority.  To fix that problem, I exported the Windows scheduled task to an XML file, changed the <Priority>7</Priority> value to <Priority>4</Priority>, saved the XML file, deleted the original scheduled task, and then imported the previously exported XML file.  That change prevented Windows from automatically throttling down the disk activity of the process running in the scheduled task when another process also required disk activity.  2) The ROBOCOPY command that was running in the scheduled task also required a bit of adjustment to eliminate file caching, resumable copying, and intermediate progress updates.  So, the ROBOCOPY command changed from: ROBOCOPY /MIR /R:3 /Z /XO C:\Shares\Backup Y:\Backup To: ROBOCOPY /MIR /R:3 /J /NP /XO C:\Shares\Backup Y:\Backup
  
• Determine how it was possible to ping the IP address of a “transparent” firewall located between the corporate network and programmable logic controllers (PLCs), yet it was not possible to access the configuration web interface of the “transparent” firewall.  Wireshark analysis of ARP broadcast traffic found that when devices were trying to resolve the MAC address of the IP address which was assigned to the firewall, another device that was thought to be discarded a few years ago also answered the ARP requests in addition to the firewall.  Wireshark identified the MAC address of both devices using the IP address, making it possible to hunt down the rogue device.
  
• Pinpoint the cause of several multi-hour network outages to an old, barely used 3COM network switch that repeatedly fell off the network and then reappeared on the network.  That switch had the spanning tree protocol enabled, as did all other managed switches, and used default settings for the spanning tree protocol (which assign the same Priority value of 32768 to all devices, causing the device with the alphabetically lowest MAC address on the network to become the Root device to which all other switches attempt to map the lowest cost communication path).  Replacing that old switch with a newer non-3COM managed switch fixed that problem without requiring any adjustment to other switches (even though another switch near the servers should have been configured as the root switch).
• Locate apparently offline / broken security cameras operating unexpectedly in the wrong network VLAN due to an inexperienced employee unplugging network cables and then reconnecting those network cables into the wrong network switch ports.
  
• Collect evidence of a PLC becoming confused due to receiving excessive ARP packets, leading to the PLC unintentionally disabling client computer access to the network virus scanner server by poisoning the MAC routing tables in network switches due to the PLC specifying an invalid corrupt Target MAC address with the virus scanner server’s IP address specified as the Target IP address in unrequested ARP response packets.
• Discover that optimizing network performance per Microsoft’s Server 2003 documentation leads to a Nagle – Delayed ACK problem when the Windows server attempts to copy files from a Linux server that is using SAMBA (to allow Windows computers to access files stored on the Linux server).
  
• Decrease the Visual Manufacturing 6.5.2 startup time from more than 10 seconds to approximately one second.  The Wireshark log indicated that a change was needed to the database specifications in the SQL.INI file (eliminate TNS: from the remotedbname= lines), and also an undocumented missing file named NetHasp.ini had to be created in the Visual EXE folder.
• Pinpoint the source of a flood of ARP packets on the network that were causing multiple PLCs (protected behind “transparent” firewalls) to malfunction and stop responding on the network.  Wireshark showed that a single computer was repeatedly sending out ARP requests trying to sequentially resolve the MAC address of all IP addresses in multiple class C subnets (with a subnet mask of 255.255.0.0, the computer would repeatedly check the first four class C subnets).  Wireshark identified the MAC address and IP address of the sender of those packets, and then either the TRACERT command or a DNS lookup was performed to determine the computer that was sending the ARP packets (if TRACERT and DNS failed to locate the name of the device, the MAC routing tables in the various managed switches would have been checked to locate the switch port into which only the sender’s MAC address is listed to pinpoint the device).  Once the computer was located, it was connected wirelessly to a cheap Linksys router (specifically bought for this purpose), along with two other computers that were running Wireshark, and then Process Monitor (described in part five of this article series) was simultaneously run on the computer to determine what malware or other software on the Lenovo laptop was flooding the network with ARP packets.  Process Monitor nicely documents when applications send information across the network, but unfortunately it does not capture ARP traffic, which operates on Open Systems Interconnection (OSI) layer two ( https://en.wikipedia.org/wiki/OSI_model ), so Process Monitor could not quickly show which process(es) on the computer was sending the large number of ARP packets.  Wireshark running on the other computers showed that when connected to the cheap router the Lenovo laptop would repeatedly send out ARP packets sequentially by IP address in the class C subnet, and if any device responded to an ARP request, that device would then be pinged on a frequent basis without the computer sending out another ARP packet for the responding IP address (at least for a period of time when the entry aged from the computer’s resolved ARP table).  While Process Monitor could not identify the process sending out ARP packets, it did identify the programs that were running / accessing ARP.EXE (which was discussed in part two of this article series – the ARP command does not send the ARP packets, but when executed will list MAC addresses that are associated with IP addresses); a process named Lenovo.Modern.ImController.PluginHost.CompanionApp.exe (the only entry with a “Process Create” operation entry) and NortonSecurity.exe as shown in the second screen capture below (the Process Monitor log file became corrupt somehow by putting the computer to sleep with that log file still loaded in the program, but fortunately I had exported the log file to a CSV file and loaded that file into Excel to search for processes accessing ARP, so a screen capture of the information from Excel is shown below).  In Process Monitor I located one of the entries for Lenovo.Modern.ImController.PluginHost.CompanionApp.exe, and double-clicked that entry – the displayed information is shown in the third screen capture below.  From the Command Line that process appeared to be related to Lenovo Wifi Security.  A bit of research indicated that the Lenovo Wifi Security software may come preinstalled on Lenovo laptops, and may not have a legitimate purpose for remaining installed on the computer.  Temporarily disabling that program on the computer eliminated the flood of ARP packets sent across the network.
  

Quick Overview of Wireshark Functionality

From quickly viewing the above screen captures, you may have noticed that not all screen captures show the same columns of information in the top section of the window for each captured packet.  While it may not be obvious, Wireshark allows quickly adding additional columns for each packet row simply by right-clicking an entry in the decoded packet area near the bottom of the Wireshark window, and selecting Apply as Column from the popup menu.  Once added as a column, left-clicking that column’s header permits sorting the packet rows in ascending or descending order by the values contained in the column.  The right-click menu also contains the Apply as Filter item, which may be used to restrict the packets displayed to just those packets that have a similar value for the right-clicked statistic value.

Wireshark is available pre-compiled for Windows and Mac, and the source code may be downloaded and compiled for various other operating systems; see the download link at the start of this article for access to the setup programs and/or source code.  When installing the Windows version of Wireshark, the default values suggested by the setup program are probably good enough for most purposes, although you may want to select Wireshark Desktop Icon under the Create Shortcuts heading.

After installing Wireshark, upon starting Wireshark the top-half of the Wireshark window will show previously opened or saved Wireshark captures, while the bottom-half of the window shows network activity for the various network interfaces installed in your computer.  Starting a Wireshark capture may be as simple as double-clicking one of the listed network interfaces that shows some activity in the graph to the right of the network interface name.  For the below screen capture I single-clicked the wireless network interface to indicate the connection that I would likely select to use for the packet capture.

Selecting Options… from the Capture menu also allows starting a packet capture using a specific network interface installed in your computer, and also offers the ability to enable or disable promiscuous mode network packet captures.  Normally, network interfaces will ignore all (non-broadcast) network traffic that is NOT specifically directed to the network interface’s MAC or IP address.  Enabling promiscuous mode captures allows Wireshark to also show and log those packets that are NOT specifically directed to the network interface’s MAC or IP address; note that an upstream network switch, when functioning normally, will typically filter out most network traffic that is not specifically for the network interface’s MAC and/or IP address, unless the computer’s network port in the switch is configured as a “monitor port”, in which case a copy of all traffic passing through the switch would be also sent to the device connected to that “monitor port”.  The actual availability of promiscuous mode captures on wireless networks may be dependent on the wireless card drivers installed, so it may not be possible to capture all transmitted wireless traffic using Nagios on a computer connected wirelessly through a wireless access point.

To end a Wireshark capture, select Stop from the Capture menu, or click the red square toolbar button near the left of the toolbar (the shark-fin shaped button at the far-left of the toolbar may be used to start a capture in addition to the other methods previously mentioned).  Once capturing stops, Wireshark applies the configured color rules to the packets in the top section of the window to help draw attention to entries that may be of interest.  The color-coded vertical bar at the right of the window shows color codes of packets that were captured immediately before or after the displayed packets, allowing you to quickly jump to other potentially interesting sections of the packet log by clicking an area on that vertical bar.  The bottom half of the window shows the decoded information for the selected network packet, along with the raw bytes and ASCII character representation of the selected network packet.

By right-clicking a packet, and then selecting an item from the Follow submenu it is possible to easily apply a filter to only show the packets that are directly related to the selected packet, as well as an attempted ASCII character representation of the bytes in the packets.

The screen capture below shows the result of selecting TCP Stream from the menu for the [TCP Out-Of-Order] packet – this technique might be used to view the SQL statements sent to a database server, and the results sent back from the server.  Note that tcp.stream eq 110 appears near the top-left of the window – that is the automatically created and applied filter.  To again display all captured packets, click the X button at the far right of the green filter text area:

When troubleshooting problems with the help of a Wireshark packet capture, recognizing potential problem patterns in the packet capture may be difficult.  Recognizing patterns in the packet captures is often a significant step toward finding and fixing the source of a particular problem, so learning to recognize patterns is important.  As shown in the screen captures above, Wireshark makes effective use of color coding to draw attention to certain captured packets.  Wireshark’s Expert Information option on the Analyze menu processes the current packet capture, looking for network packets of potential interest, and then groups those packets by category.  IMPORTANT: Always make certain that the packet capture is saved before using the Expert Information option, as Wireshark will sometimes silently crash when summarizing the network packets.  [Note that if Wireshark does crash before you have saved a packet capture, the packet capture might have been written to a file with a name such as “wireshark_Wi-Fi 2CLAIW0.pcapng” (where “Wi-Fi 2” is the name of the network interface that was selected for the capture, and “CLAIW0” is a randomized set of characters) in your configured temp files folder, so you may be able to recover an unsaved Wireshark capture simply by renaming and then opening that file.]  Click the right pointing arrow next to a group name to see a summary of the packets that fall into the group, and click one of those packets to scroll the main Wireshark window to that packet.

Filters

As mentioned earlier, it is possible to right-click an entry in the decoded packet information to apply a filter which then causes Wireshark to only show other captured packets that have the same decoded value.  You are also able to manually create a filter specification by typing into the textbox that is located just below the toolbar.  As you type, Wireshark will change the background color of that textbox to either red or green depending on whether or not the filter is executable, and suggestions will be offered as you type to help you complete the filter so that it may be executed.  A couple of example filters are shown below:

Find packets set to or received from the IP address 23.38.130.175 (one of the IP addresses for support.infor.com):

ip.src == 23.38.130.175 || ip.dst == 23.38.130.175

Find ARP broadcast packets that are sent by the device having the MAC address 70:b3:d5:80:61:a4 that do not have a destination MAC address of 0a:01:05:1c:d1:64:

((frame.protocols == "eth:ethertype:arp") && (eth.src == 70:b3:d5:80:61:a4)) && !(eth.dst == 0a:01:05:1c:d1:64)

Show only OSI layer 2 network packets (technically, ARP is layer 2, but this filter eliminates ARP packets):

not ip and not arp

For additional Wireshark filter examples, see the “Wireshark Tutorial: Display Filter Expressions”  article created by Brad Duncan, or review the Wireshark documentation.

Potentially Important Additional Columns

The first time Wireshark starts only a small number of columns of information will appear for each network packet.  Depending on the type of problems that you are investigating, you may need to add additional columns of information to quickly understand the reason why certain packets appear in the packet capture.  For the screen captures that follow, I selected a TCP type network packet to view the decoded information in that packet.  As mentioned previously in this article, by right-clicking information in the decoded packet, you are able to quickly add additional columns of information for each packet in the upper window area.

In the Frame section of the decoded packet, you may want to add a column for “Time delta from previous frame”.  If you are checking for performance / communication speed problems and you are able to sufficiently eliminate unrelated information from the packet capture using filters, adding this column may help you quickly zero in on sections of the communication with the longest unexpected delays as this column will show the amount of elapsed time between the packet and the displayed packet that was logged immediately before the packet.  If you are not able to filter out unrelated communication from the Wireshark output, you may want to show the “Time since previous frame in this TCP stream” (mentioned below) instead.

If you are troubleshooting packet retransmits found in the packet capture or suspected network loops, you may want to add a column for “Identification” and “Time to Live” in the Internet Protocol Version section.  Each packet sent should have a unique “Identification” value, even if the packet is intentionally retransmitted by one of the devices.  If retransmissions appear in the Wireshark capture with the same “Identification” value, that potentially indicates that a harmful network loop exists that is causing packets to be incorrectly repeated on the network.  The “Time to Live” value typically starts at either 255, 128, or 64, and is decreased by one for every router that the packet passes through.  So, if duplicate packets appear on the network with the same “Identification” value, but different “Time to Live” values, that could indicate a routing problem that is incorrectly reintroducing packets on the network after being sent through a router.

Adding columns for “Sequence Number”, “Next Sequence Number”, “Acknowledgement Number”, “Calculated window size”, and “Time since previous frame in this TCP stream” may be helpful when investigating either unexpected network slowness or retransmissions.  If a packet appears with the same “Sequence Number”, but with a different “Identification” value, that probably indicates that the sender did not receive an acknowledgement (ACK) for that packet within a certain time limit.  If a packet appears with the same “Sequence Number” and “Identification” value, that probably indicates that there is a network loop causing duplicate packets to appear on the network.  When a device sends an ACK, it actually specifies the “Next Sequence Number” that the other device may send, rather than referring to the “Sequence Number” of the received packet (for some reason I frequently forget this acknowledgement numbering approach, which is why I am mentioning it here).  The “Time since previous frame in this TCP stream” statistic indicates the amount of elapsed time that has passed since the previous packet in a specific TCP conversation.  This statistic is very helpful when it is not possible to fully filter potentially unrelated network traffic from the packet capture.  If an unexpectedly large value appears in the column when added, then the next investigative step is to check the packets that appear in the packet capture between the current packet and the previous packet in the TCP conversation to see if, for example, a DNS lookup failed, which resulted in an eventual WINS broadcast that failed.  If nothing obvious appears in the packet capture, then the problem may be application, computer, or user “think time” causing the delay.  If the “Time since previous frame in this TCP stream” statistic does not appear in the decoded packet, first verify that you have selected a TCP type packet, right-click in the decoded packet information, and select “Protocol Preferences” – “Calculate conversation timestamps” from the popup menu.

Analyzing Performance Problems with Statistics

A quick glance at the Statistics menu in Wireshark suggests that there is substantial functionality in Wireshark to help analyze various types of potential network problems.  In this article I will only scratch the surface of the items found in the Statistics menu.  Assume that I have the Visual EXE files installed locally on a computer, that computer is connected to the company network using a VPN connection over the internet, and the person using the computer would like to start Visual Manufacturing and then the Purchase Requisition Entry Visual module.  Running Visual modules over a VPN connection using this approach is not recommended as Visual modules tend to be very chatty over the network, and with relatively high latency internet connections, Visual modules may take several seconds to respond to even the simplest command.  Out of curiosity, I wondered if Wireshark could help indicate what is causing the slowness, so I created a packet capture of Visual Manufacturing starting (from a shortcut that provided the username and password), and then opening the Purchase Requisition Entry module when connected over a VPN connection.  My first thought when ending that capture was that a whole lot of network communication was captured in the 63 second log capture.

After ending the capture, I found the network packet where the client computer first connected to the database server, and clicked that packet to select it.  From the Statistics menu I selected “TCP Stream Graphs” and then “Time Sequence (Stevens)” from the submenu.  The below screen capture shows the graph that appeared, which shows the number of network packets transmitted over time.  Fast rises indicate a lot of network packets transmitted in a short period of time, while slow rises in the network packets indicate that some sort of delay was preventing continued rapid communication with the database server.

Next, from the Statistics menu I selected “Flow Graph”.  The flow graph shows a summary of the network communication that appeared in the network capture in an easily understood format by IP address, which might be helpful to determine what was happening on the network when the client’s communication with the database server slowed, as shown in the previous screen capture.  In the following screen capture I replaced the displayed IP addresses with the function of the device (ex: Visual Client, File Server, and DB Server).  As pictured, the first 1.08 seconds was spent accessing various Visual INI files stored on a server with three potentially problematic packets, there was a 0.2 second delay with no network activity, and then communication with the database server started.  By scrolling down and to the right, additional communication, potentially unrelated to the Visual startup test, is revealed.

A lot of helpful information, including sample network captures showing various types of network activity, is accessible from Wireshark’s Help menu.  Other resources that I have found useful include the following videos and reference material:

• Excellent introduction to how network communication works: SF18EU – 01: Back to the Basics (Hansang Bae)
• Excellent introduction to Wireshark information: TCP Fundamentals Part 1 – Wireshark Talks at Sharkfest
• Part two of the above presentation: TCP Fundamentals Part 2 – Wireshark Talks at Sharkfest
• (Partially repeats sections of above two videos) Is It the Client, Network, or Server? – Packet Analysis with Wireshark – Sharkfest Talks
• Decent overview of problem solving steps for network troubleshooting: SF19US – 30 Using Wireshark to solve real problems for real people (Kary Rogers)
• Sharkfest 2019 presentations: https://sharkfestus.wireshark.org/sf19 (search Youtube for Sharkfest SF20V for the most recent videos)
• My seven-part Wireshark series written 11 years ago “Network Monitoring Experimentations”
• Investigating Network Loops
• Duplicate Packets and TCP Retransmissions
• Excellent suggestions for specifying additional display columns and recognizing patterns in Wireshark captures: SF18US – 34: Patterns in TCP Retransmissions (Scott Reid)

Do not expect to learn everything about Wireshark in a few hours or days.  Instead, learn the basics of using Wireshark, and learn about the more advanced functionality as needed when troubleshooting various network or performance problems.  Of course, if you prefer to binge watch instructional videos, there are a number of very good 70+ minute long videos on YouTube from previous Sharkfest conferences to enjoy.

————————————————————————————

VMIUGLIST and VMIUG-TEC Google Groups

Access to the VMIUGLIST and VMIUG-TEC Google Groups is restricted. Anyone who is a user of Infor Visual Manufacturing, works for Infor, or works for one of the official Infor sales channel partners will be authorized to access the groups.  To keep spam under control, we ask that when requesting to join the groups, the following information is provided:

• The Visual version you currently use.
• The company you are working for.
• Your company email address.
• Your position and why you wish to join.

The new groups may be found using the following Google search page (incidentally, once you are a member of the groups, including the phrase VMIUG in a search will also allow searching for content within the groups): https://groups.google.com/forum/#!search/vmiug