Sunday 11 October 2015

Check Point - Acceleration

SecureXL: Security Acceleration

SecureXL is a technology interface that accelerates multiple, intensive security operations, including operations carried out by Check Point's Stateful Inspection Firewall.
SecureXL accelerates Firewall and VPN performance by remembering certain attributes of packets and packet flows that have already been validated by the Firewall/VPN application. Then, validation of related packets and connections is delegated to the SecureXL API; this validation is done at the hardware interrupt level.

Packets attempting to establish a new TCP connection (or a comparable UDP) are handled in the slow path. Once the first packet is seen by the Firewall and suitable connections information is off-loaded to an appliance OS, further packets are handled at the OS's interrupt-level code.

SecureXL improved non-encrypted firewall traffic throughput and encrypted VPN traffic throughput. Packet acceleration is also referred to as throughput acceleration as it matches on the familiar 5-tuple of source address, destination address, source port, destination port and protocol. However, only packets during the specific TCP/UDP connection can be accelerated.

Session rate acceleration: SecureXL also reduces the overhead in establishing certain kinds of new connections, improving new connection rate(connections per second), connection setup/teardown rate (sessions per second) and throughput in certain high-connection rate traffic environment.

From a 5-tuple validation, the source port of a flow may be masked off, effectively providing a global match for source port. These new connection setup packets matching 4 out of 5 tuples avoid a round trip to the firewall application and limit the computing overhead. Security is not impacted because the OS continues to track the state of the new connection using Stateful inspection.

Application Layer Protocol (example HTTP)

Web pages consist of multiple components. Using HTTP 1.0, each component is downloaded from S2C using a separate TCP connection. This action involves substantial overhead in connection setup and tear-down and protective firewall connection tracking.

Once a connection involving a flow to port 80 is approved by the Firewall application for the web client (resulting from the first HTTP request), a template is created and stored. All subsequent connection setups carrying those additional requests can share that same template approval because it's ok that the source port differs. Establishing those subsequent connections does not involve a round trip to the Firewall, resulting in faster processing through the server Firewall.

HTTP 1.0 creates a separate connection for each HTTP component. The newer HTTP 1.1 improves the protocol's performance by permitting not only parallel, but also persistent and pipelined server connections. The server may keep the connection alive after sending the end of a component which avoids the need to create a new connection to send the next component.

FTP and many types of VOIP have handlers which precludes acceleration. There are several factors that preclude a packet from being accelerated (SDF, QoS, connections that have a Handler, multicast packets, etc.)
There are factors that can preclude templating if all other parameters are met for acceleration: time objects, dynamic objects, domain objects, source port ranges, IPS features not supported in Acceleration, NAT, Encrypted connections.

Once templating is disabled in the Rule Base, all connections matching rules lower in the Rule Base cannot be templated. Use fwaccel stat to determine at which rule templating is disabled and move the most used rules above that rule for session acceleration.

CoreXL: Multicore Acceleration 


Wednesday 30 September 2015

Check Point - Clustering

Virtual Routing Redundancy Protocol 

VRRP cluster can be used for High Availability or Load Sharing. The check point implementation of VRRP includes additional functionality called Monitored Circuit VRRP which prevents black holes. 
You cannot deploy a standalone deployment (Security Gateway and SMS on the same computer) in a Gaia VRRP cluster.

A VRRP router might participate in more than one VRID. The VRID mappings and priorities are different for each VRID. 

Monitored Circuit VRRP eliminates black holes caused by asymmetric routes that can be created if only one interface on the master fails as opposed to the entire platform. Monitored Circuit VRRP monitors all of the VRRP-configured interfaces on the platform. If an interface fails, the master release its priority over all of the VRRP-configured interfaces. To release the priority, Gaia subtracts the Priority Delta from the priority to calculate the Effective Priority. Make sure to calculate the Priority delta value so that Gaia releases priority over all interfaces on a virtual router to let failover occur when one interface fails.

If the platforms run firewall software, you must configure the firewall policies to accept VRRP packets. The Multicast address assigned for VRRP is 224.0.0.18. If the policy does not accept packets to 224.0.0.18, Firewall platforms in the same VRRP group take on Master state.

With Monitored Circuit VRRP, some ethernet switches might not recognize the VRRP MAC address after a master to backup change. This is because many switches cache the MAC address related to the Ethernet device attached to a port. When the change to a backup router occurs, the MAC address for virtual router shifts to a different port and switches that cache the MAC address might not change to the correct port during VRRP change. To prevent this, replace the switch with a hub; disable MAC address caching or set the address ageing value sufficiently low; also enable portfast.

Cluster XL

ClusterXL provides both load sharing and high availability solutions.ClusterXL must be installed in a distributed configuration in which the SMS and the cluster members are on different machines. ClusterXL is part of the standard security gateway installation.

A Critical Device is a device that is critical to the operation of the cluster member. It is also known as a Problem Notification (PNote). It can be hardware or a process. The fwd and cphad processes as well as the Security Policy itself are predefined as critical devices. Use cphaprob command to add.

Cluster Control Process (CCP) is used specifically for clustered environments to allow gateways to report their own states and learn about the states of other members in the cluster. It is essential means by witch State Synchronization works to provide failover in the event an active member goes down.
There is no need to add a rule to the Rule Base that accepts CCP. When clustering is configured on the gateways, an implied rule is created making this provision.

ClusterXL uses unique physical IP and MAC addresses for the cluster members and virtual IP addresses to represent the cluster itself. Virtual IP addresses do not belong to an actual machine interface.

Cluster Synchronization  

In order to make sure each Gateway cluster member is aware of the connections going through the other members, a mechanism called State Synchronization exists witch allows status information about connections on the Security Gateways to be shared between the members. Every IP based service including TCP and UDP, recognized by the security gateway is synchronized. State synchronization is used both by ClusterXL and by third-party OPSEC-certified clustering products. 2 modes
   Full Synchronization - Transfers all Firewall Kernel table information from one cluster member to another. It's handled by the fwd daemon, using an encrypted TCP connection. Full synchronization is used for initial transfers of state information for thousands of connections. If a cluster member is brought up after failing down, it will perform full sync. Once all members are synchronized, only updates are transferred via delta sync.

   Delta Synchronization - Transfers changes in the Kernel tables between cluster members. Delta sync is handled by the Firewall Kernel using UDP Multicast or Broadcast on port 8116.

A user authenticated connection through a cluster member will be lost if the cluster member fails. However, a Client Authenticated or Session Authenticated connection will not be lost.
When failover, accounting information that was accumulated on the failed member but not yet reported to the SMS is lost.

Checkpoint recommends securing the synchronization interfaces by using a dedicated syn network or connecting the physical network interfaces of the cluster members directly.

ClusterXL: Load Sharing

Machines in a ClusterXL load sharing configuration must be synchronized. Machines in a ClusterXL HA configuration do not have to be synchronized but connections wll be lost upon failover if they are not. Multicast and unicast are 2 available modes in a load sharing environment.

Multicast Load Sharing 

Every member of the cluster receives all of the packets sent to the cluster IP address. ClusterXL decision algorithm on all cluster members decides which cluster member should perform enforcement processing on the packet. Only that machine processes the packet and sends the packet to its destination The other machines drop the packet.

Unicast Load Sharing

In this mode, one machine called the Pivot machine receives all traffic from a router with a Unicast configuration and redistributes the packets to the other machines in the cluster. The Pivot machine is chosen automatically by ClusterXL.

The Pivot is the only machine that communicates with the router and the router uses only the Pivot's Unicast MAC address to communicate to the cluster.

Sticky connections - A connection is sticky when all of its packets are handled, in either direction, by a single cluster member. In HA mode, all connections are routed through the same cluster member. In load sharing mode, this is not the case but certain connections can be made sticky by enabling the Sticky Decision Function (SDF).

Perform a Manual Failover of the Firewall Cluster

The best practice method for initiating a manual failover: run below command on an active cluster member that creates a problem notification entry with no refresh time in a problem state.
     cphaprob -d STOP -s problem -t 0 register
Running the command cphaprob list on this machine will show an entry named STOP. To remove the problematic STOP entry from the cluster member,
     cphaprob -d STOP unregister

Also can be done from expert mode
     clusterXL_admin down
     clusterXL_admin up

A manual failover can also be induced from the Gateways status screen in Smart View Monitor via Stop Cluster member.

ClusterXL CCP on the cluster members uses Multicasts by default as it's more efficient. If the connecting switch is incapable of forwarding Multicast, change the CCP mode to Broadcast
     cphaconf set_ccp broadcast
     cphaconf set_ccp multicast   //to change back to multicast

--------------------------------------------------------------------------------------------------------------------

Management High Availability

The SMS consists of several databases with information on different aspects of the system such as objects, users and policy information. In the absence of SMS, essential operations performed by the gateways, such as fetching of the Security Policy and the retrieval of the CRL, cannot take place.

In Management HA, the Active SMS always has one or more backup Standby SMS. These standby SMS must all be of the same operating system and version. In a Management HA deployment, the first installed SMS is specified as the Primary SMS.

The Secondary SMS is created with empty databases that are filled with information received from the Active SMS. Secondary SMS is ready once
  • It is represented on the Primary SMS by a network object
  • SIC has been initialized between it and the Primary SMS
  • Manual synchronization has been completed with the Primary SMS for the first time
All management operations are done by the Active SMS. The transition from Standby to Active must be initiated manually. The Standby SMS are synchronized to the Active SMS so they are kept up-to-date with all changes in the databases and Security Policy Security Gateways can fetch the Security Policy and retrieve a CRL from both SMS.

In order for Management HA to function properly, there must be a backup of Database (such as Objects and Users), Certificate information such as Certificate Authority data and CRL, and the installed Security Policy.

Manual or Automatic synchronization. Synchronization status can be viewed in the Management High Availability Servers window or in SmartView Monitor depending on whether you are connected to the Active or Standby SMS. The possible statuses are : never been synchronized, synchronized, lagging (the peer SMS has not been synchronized since the Active SMS has changes applied to it), advanced (the peer SMS is more up-to-date), collision (the active SMS and its peer have different installed policies and/or databases).

Saturday 26 September 2015

Wireshark notes - 4 - Tips

Try to keep my Wireshark trace files to 100 MB size maximum.

Define a useful naming scheme for your trace files as soon as possible. Consider including capture location, capture purpose and any notes about the trace file in your trace file names.
sw1-msmith-slowsalesforce.pcapng
sw1-msmith-backgroundidle.pcapng
local-gspicer-slowbrowse.pcapng
local-gspicer-uploadstuck.pcapng
fs2-disconnects.pcapng
rtr2side1-slowpath.pcapng
rtr2side2-slowpath.pcapng


Tips for Analyzing TCP-Based Applications

-Look at the TCP handshake to get a snapshot of round trip time.
   If capturing at the client, measure the time between the SYN and the SYN/ACK.
   If capturing at the server, measure the time between the SYN/ACK and ACK.
-Open SYN and SYN/ACK packets and examine TCP peer capabilities (TCP Options).
   Decent MSS size?
   SACK supported by both?
   Window Scaling supported by both?
   Decent scaling factor?
-Launch the IO Graph and look for drops in throughput.
   Add the Bad TCP coloring rule filter to the IO Graph to correlate drops in throughput with TCP issues (the Golden Graph).
-Open the Expert Infos to view detected problems.
   Focus on Errors, Warnings and Notes.
   Expand sections and click on packets to jump to that location in the trace file and explore further.
-View and sort the TCP Delta column (tcp.time_delta).
   Sort the column from high to low and examine delays.
   Do not get distracted by "normal delays" (refer to Do not Focus on "Normal" or Acceptable Delays).
 -View and sort the Calculated window size field to look for issues.
   Do not worry about FIN or RST packets with Window 0 values.
   Look for low window size values and delays in close proximity.

Tips for Locating the Cause of Intermittent Problems

Consider using a Ring Buffer during the capture process. To capture intermittent problems, set up a capture machine close to one of the machines that experiences the problem. Start capturing traffic to a file set and define the number of files to be saved by the Ring Buffer. Do not set an auto stop condition—stop the capture as soon as possible after the problem occurs.


When you stop capturing the last file is displayed. Work backwards through this file and then the
other files in the file set to locate the problem. Select File | File Set | List Files to view and navigate
between files in the file set.

Tips for Detecting WLAN Problems

You need to capture the 802.11 Management, Control and Data frames, the 802.11 header, and have a pseudoheader applied. Management and Control frames are necessary to identify problems with associating and authenticating to a WLAN. Data frames provide us with the actual throughput rates on a WLAN.

Tips for Sanitizing Trace Files

Security rule: Never share trace files that may contain confidential information. Use TraceWrangler that was created specifically to sanitize .pcapng files.

Tips for When you get stuck

Search www.ietf.org, www.wiresharkbook.com/resources.html, and also consider asking for help at ask.wireshark.org. 


Saturday 19 September 2015

Wireshark notes - 3 - Application Errors and Advanced IO Graph


dns.flags.rcode > 0
http.response.code >= 400 or
http.response.code > 399

HTTP response code
1xx: Informational—Request received, continuing process
2xx: Success—The action was successfully received, understood, and accepted
3xx: Redirection—Further action must be taken in order to complete the request
4xx: Client Error—The request contains bad syntax or cannot be fulfilled
5xx: Server Error—The server failed to fulfill an apparently valid request


SMB response code(NT status) of 0 indicates the request was successful.
smb.nt_status > 0 || smb2.nt_status > 0


SIP is a request/response-based application. SIP can run over UDP or TCP. When SIP is configured to run over TCP, we hope to see an ACK to our SIP request in a reasonable amount of time and then a successful response. SIP response codes are
1xx: Provisional — request received, continuing to process the request.
2xx: Success — the action was successfully received, understood, and accepted.
3xx: Redirection — further action needs to be taken in order to complete the request.
4xx: Client Error — the request contains bad syntax or cannot be fulfilled at this server.
5xx: Server Error — the server failed to fulfill an apparently valid request.
6xx: Global Failure — the request cannot be fulfilled at any server.

sip.Status-Code >= 400 or
sip.Status-Code > 399

----------------------------------------------------------------------------------------------------

A picture is worth a thousand packets

Use an IO Graph to compare the throughput of separate conversations
Use an IO Graph to compare application throughput based on port numbers in use Consider using Advanced IO Graphs when you need the Calc functions (such as MIN, AVG, MAX)

When the application runs over TCP and you have the option of using an application name filter (such as http), it is recommend you use a port-based filter (such as tcp.port==80) instead in order to include the TCP overhead (such as TCP handshake packets, ACKs, FINs, and RSTs) in your graph.


The Advanced IO Graph offers Calc functions for summing the contents of a field, counting the occurrences of a field and more.
  -Use Calc: SUM(*) to add the contents of a numerical field, such as tcp.len, which does not exist in a packet, but is Wireshark's field to count just data bytes in packets.
  -Use Calc: COUNT FRAMES(*) to count the occurrence of specific type of frame or Expert Infos item such as tcp.analysis.retransmission.
  -Use Calc: COUNT FIELDS(*) to count the occurrence of a field, such as the IP ID (ip.id) field which occurs twice in some ICMP packets.
  -Use Calc: MIN(*), AVG(*) and MAX(*) to graph the minimum, average and maximum value of a numerical field, such as the tcp.window_size field.
  -Use Calc: LOAD(*) to graph response time fields, such as smb.time.

There is no field in a packet called tcp.len, but Wireshark uses this value to define the number of data bytes in each TCP segment. Tcp.len value does not count header values.

---------------------------------------------------

Detect Consistently low throughput due to low packet sizes

Low packet sizes may be caused by an application that intentionally wants to transfer smaller amounts of data. Low packet sizes can also be an indication of a low Maximum Segment Size (MSS) setting. For MSS setting, check tcp handshake.


Identify Queuing Delays along a Path

Interconnecting devices can inject delays by queuing (holding the packets temporarily before forwarding them) along a path. Consider using a traffic generator to detect queuing along a path. A tool such as iPerf/jPerf can be used to transmit traffic at a steady rate.



Correlate drops in Throughput with TCP Problems (the Golden Graph)

This graph can determine if throughput issues are related to network problems such as lost packets or
zero window sizes. This is a great graph to build whenever anyone complains about slow performance of a TCP-based application.


Graph Time Delays

This is a great way to identify slow responses for an application that does not have a delta time function.

Graph High TCP Delta Time (TCP-Based Application)

Some TCP-based applications (such as HTTP and SMB) have a delta time tracking function in Wireshark. If the application does not have the delta time tracking function built into the dissector, you can still graph high delta times using tcp.time_delta.


Graph Other Network Problems

You can graph window size issues based on the TCP analysis flag (tcp.analysis.zero_window) or the actual Calculated window size file value.
You can graph packet loss and recovery processes using the TCP analysis flags for each part of the process. 
Although TCP time-sequence graph can be very busy, it can depict not only packet loss but it can also depict selective ACKs.




The above graph clearly depicts the points in the trace where Wireshark noticed packet loss. In addition, the graph depicts the packet loss recovery process by graphing Duplicate ACKs and Retransmissions.



Thursday 10 September 2015

Wireshark notes - 2

Expert Info Messages

Previous Segment not Captured

tcp.analysis.lost_segment

Packet loss recovery method #1 - Fast Recovery
If the receiver supports Fast Recovery and notices the jump in sequence number value, it will immediately begin sending Duplicate Acknowledgments requesting sequence number 7,920. Upon receipt of 4 identical ACK (can be more than 4), sender retransmit the packet.


Packet loss recovery method #2 - Sender Retransmission Timeout (RTO)
If the sender notices that a data packet has not been acknowledged within its Retransmission Timeout (RTO) timer value, it will retransmit the packet.



To determine how many packets were lost, add 3 colums - sequence number, next sequence number and acknowledgement number.



Location of the capture



Since we know the sequence number of the packet that is missing, we can use that information to
determine if we see the original and the Retransmission or just the Retransmission. That will tell us if
we are upstream or downstream from packet loss. Filter tcp.seq==9164761.

Duplicate ACKs

tcp.analysis.duplicate_ack Duplicate ACKs are an indication that a host supports Fast Recovery and noticed that a packet arrived with a sequence number beyond the calculated next sequence number. Duplicate ACKs are usually a sign of packet loss, but Duplicate ACKs can also be an indication of out-of-order packets.

If the packet with the missing sequence number arrives within 3 ms, Wireshark marks that packet as
Out-of-Order (tcp.analysis.out_of_order). If the packet with the missing sequence number arrives later than 3 ms. later, Wireshark will indicate that the packet is a either a Retransmission or a Fast Retransmission.

These Duplicate ACKs complain about a missing sequence number. If SACK is in use we should see
only the missing packets being retransmitted. The SACK Left Edge and SACK Right Edge fields in
the TCP Options area acknowledge other data packets received while the Acknowledgment Number
field still indicates the desired missing sequence number.
If SACK is not in use we may see many unnecessary retransmissions as the sender retransmits every
data packet starting at the missing sequence number.


Out-of-Order Packets

tcp.analysis.out_of_order  Out-of-order packets may not affect performance if there is very little time(1-3 ms) between their expected arrival and their actual arrival. If out-of-order packets arrive after quite a delay, or there are many out-of-order packets, there may be a noticeable degradation in performance. TCP cannot pass received data up to the application until all the bytes are in the correct order.

Determining if a packet is Out of order, Retransmission or Fast Retransmission

Fast Retransmission

tcp.analysis.fast_retransmission  Fast Retransmissions are triggered by receipt of three identical ACKs (the original ACK and two Duplicate ACKs).

Retransmission 

tcp.analysis.retransmission  Standard Retransmissions are not triggered by Duplicate ACKs. Standard Retransmissions are triggered by a Retransmission Time Out (RTO) at the sender. The RTO timer is used to ensure data delivery continues even if the TCP peer stops communicating (with ACKs). When the RTO timer expires without receiving an ACK for the data packet, the sender retransmits the unacknowledged data packet.




You do not want to spend time troubleshooting Retransmissions or Fast Retransmissions when these
packets are actually Out-of-Order packets that did not arrive within 3 ms of the higher Sequence
Number field value.

Remember, Duplicate ACKs lead to Fast Retransmissions. An expired RTO at the sender leads to
Retransmissions. Each of these is an indication of packet loss which typically occurs at interconnecting devices. Capturing at different points on the network can help you find the point of
packet loss.
Applications cannot pick up data from the buffer until all sequential bytes have been received. Outof-
Order problems typically aren't felt by network users unless there is a large gap in time between
the expected arrival time and actual arrival time.

ACKed Unseen Segment

tcp.analysis.ack_lost_segment  This Expert Infos warning indicates that Wireshark sees an ACK, but it did not see the data packet that is being acknowledged.



Zero Window

tcp.analysis.zero_window  Each side of a TCP conversation advertises its receive buffer space in the Window Size Value field (tcp.window_size_value). When a receiving application cannot pull data out of the receive buffer fast enough, this advertised Window Size value can drop to zero



The Window Size Value field indicates the actual Window Size being advertised. When Window
Scaling is in use, Wireshark multiplies the Scaling Factor by the advertised Window Size Value field
to provide the scaled Window Size (Calculated window size field). The TCP FIN or RST packets would not be colored by the Bad TCP.



Window Full

tcp.analysis.window_full  Window Full is an indication that the target will be out of receive buffer space when the data packet arrives. Use 'Bytes in Flight" to watch a stuck application.

Zero Window Probe and Zero Window Probe ACK

tcp.analysis.zero_window_probe, tcp.analysis.zero_window_probe_ack  Zero Window Probe packets may be sent by a host to a TCP peer that is advertising a Zero Window condition in the hope of eliciting a Window Update response. Keep in mind that a host may send a TCP Keep Alive (decrementing the sequence number by 1) instead of a Zero Window Probe packet.


Tuesday 8 September 2015

Wireshark notes - 1

some checksum algorithms are able to recover the error simply by calculating what the error is and just repari it. Some can't and so, retransmit if the protocol supports.

Flow Graph --> Statistics - Flow Graph
 -TCP handshake in full view - review the handshake and patterns associated with communications
 -Helps to find errors in communications


TCP Stream Graph --> Statistics - TCP Steam Graph


Service Response Time --> Statistics -
 For protocls SMB, LDAP, others.


Analyzing Packet Lengths
 -shows fragmentation problems
 -tiny packets problems

IO Graph (covered in foundation videos)
===================================

If the application is TCP-based, you should use a display filter based on the port number in order to view the TCP overhead (such as the TCP handshake, ACKs and connection tear down) as well as the application traffic. For example, the filter tcp.port==21 would display the FTP command channel traffic, including the TCP handshake, ACKs, and the TCP connection teardown packets.


Sample useful display filters

eth.addr == d4:85:64:a7:bf:a3
ip.addr==10.1.1.1
http.request.method  --to view all HTTP client request packets
dns.flags.rcode > 0  -- to identify DNS error responses
tcp.window_size < 1000 -- to identify advertised buffer space issue. Look for window update messages

Use !/not with ==/eq when you filter on a field name that matches two fields such as ip.addr, tcp.port or udp.port.
Use (!=) when you filter on a field name that only matches one field such as dns.flags.rocde or tcp.dstport

tcp.flags.syn == 1

Change the TCP Dissector Reassembly Setting to Properly Measure HTTP Response Times


Use Statistics --> Conversations to find top talkers.
Use Statistics --> IO Graph to quickly spot a throughput problem.

=======================================================
Wireshark profiles are saved in Personal configuration folder.

Normal or acceptable delays should be ignored in the trace file. They are
 -Delays before DNS queries,
 -delays before TCP FIN or Reset packets,
 -delays before a client sends a request to server,
 -delays before keep-alive or zero window probes(zero window probe is sent during a zero window situation to determine if more buffer space is avail at the target),
 -delays before TLS encrypted alert followed by a TCP FIN or RST
 -delays before a periodic set of packets in a connection that is otherwise idle (applicaiton's own keep alive packet)
 -

Knowing what "normal" delay times are will help. These delays below do matter
 -delays before a server responds with a SYN/ACK
 -delays before a client completes the 3-way TCP handshake
 -delays before a server sends a response
 -delays before the next packet in a data stream (buffer space)
 -delays before an ACK from a TCP peer (delays before transmitted data is ACKed)
 -delays before a window update (tcp.window.size; no expert info warning for this "low window size" problem.

Various time measurements and application response time measurements
Delta time (frame.time_delta)
Delta displayed time (frame.time_delta_displayed and Delta time displayed)
TCP delta time (tcp.time_delta)
DNS response time (dns.time)
HTTP response time (http.time)
SMB response time (smb.time)


Using IO Graph to display latency


Calculating conversation timestamps of TCP

Wireshark numbers each separate TCP conversation with a TCP Stream index (tcp.stream) value starting with 0. After you have enabled the Calculate conversation timestamp preference setting, Time since previous frame in this TCP stream (tcp.time_delta) will be visible at the end of the TCP header. Unlike the basic delta time value, this time value tracks the time from the end of one packet in a TCP conversation (aka "stream") to the end of the next packet in that same TCP conversation.

The TCP Delta column is a key column to add when troubleshooting TCP-based applications. It's one of the first steps I use when locating the cause of poor performance of a TCP-based applications on a network.

In large trace file, to find the most active TCP conversation, use Conversations menu. From there, click on TCP/UDP and sort by Bytes. Right click on the converation with highest bytes and apply as filter.


Obtain Round Trip Time (RTT) using TCP Handshake
If capturing at the clinet, look at the tcp.time_delta value between client's TCP SYN packet and server's TCP SYN/ACK response. If at server, look at value between server's TCP SYN/ACK and clinet TCP ACK response. If capturing inside the infrastructure, add up the delta time between TCP SYN and ACK packets of the handshake.

To filter the first 2 packets of TCP handshake - tcp.flags.syn==1
To filter SYN/ACK - (tcp.flags.syn==1 && tcp.flags.ack==1)
To filter ACK, - (tcp.seq==1 && tcp.ack==1)
To filter SYN/ACK and ACK - (tcp.flags.syn==1 && tcp.flags.ack==1) || (tcp.seq==1 && tcp.ack==1) && tcp.len==0 && tcp.flags.fin==0


filter: tcp.time_delta > 1 && tcp.flags.fin==0 && tcp.flags.reset==0 && !http.request.method=="GET"
Using IO Graph for TCP Delay


Identify High HTTP Response Time

The Http response time field is called http.time which exists only in Http response packets.




Saturday 22 August 2015

Checkpoint - some commands

To change default shell for admin account to bash and allow WinSCP connections,
   chsh -s /bin/bash admin

To return the setting back to default shell,
   chsh -s /etc/cli.sh admin

--------------------------------------------------

fw stat  -- run on gateway
fw unloadlocal   -- run on gateway to uninstall the firewall policy
fw fetch localhost -- run on gateway

fwm load Standard A-GW-01 --on management server

cpinfo
cpinfo -o <output file name>.txt
fw ctl pstat --monitor health and sync status on GW

fw monitor
fw monitor -e "accept src=10.1.1.201 or dst=10.1.1.201;" -o monitor2.out

Checkpoint - fw ctl pstat

Only run on Security gateway.

monitor the heath of your CP box., especially Syc Status.

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 466729198, retransmitted : 241305, retrans reqs : 6089, acks : 809
Sync packets received:
total : 77283541, were queued : 6715, dropped by net : 6079
retrans reqs : 37462, received 175 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Delta Sync memory usage: currently using XX KB mem
Callback statistics: handled 138 cb, average delay : 2, max delay : 34
Number of Pending packets currently held: 1
Packets released due to timeout: 18


Explanation:

Version: new
This line must appear if synchronization is configured (versions above 4.1)

Status: Able to Send/Receive sync packets
If sync is unable to either send or receive packets, there is a problem

Sync packets sent:
total : 466729198, retransmitted : 241305, retrans reqs : 6089, acks : 809
TOTAL number of sync packets is non-zero and increasing
RETRANS REQS may increase under load

Sync packets received:
total : 77283541, were queued : 6715, dropped by net : 6079
QUEUED value never decreases - A non-zero value does not indicate a problem
DROPPED BY NET number may indicate network congestion

The “dropped by net” counter is incremented when the cluster member receives a sync packet with a sequence number which is higher than the expected seq num. This means packets with lower seq where lost somewhere along the way, and we need to find out where.

retrans reqs : 37462, received 175 acks
RETRANS REQS growing very fast may indicate that the load is becoming too high

retrans reqs for illegal seq : 0
May indicate a sync problem

dropped updates as a result of sync overload: 0
In a heavily loaded system, the cluster member may drop synchronization updates sent from another cluster member

Delta Sync memory usage: currently using XX KB mem
This statistic only appears for a non-zero value.
It requires memory only while full sync is occurring at other times, Delta sync requires no memory

Callback statistics: handled 138 cb, average delay : 2, max delay : 34
This statistic only appears for a non-zero value.
AVERAGE DELAY should be 1-5 packets, otherwise indicates an overload of sync traffic

Number of Pending packets currently held: 1
This statistic only appears for a non-zero value.

Packets released due to timeout: 18
This statistic only appears for a non-zero value.
If the it is large (more than 100 pending packets), and the "Number of Pending packets currently held" is small, you should take action to reduce the number of pending packets.
To tackle this problem, try google "Reducing the Number of Pending Packets".

Checkpoint - tcpdump and fw monitor

What’s the difference between tcpdump and fw monitor ?

Tcpdump displays traffic coming or leaving to/from a firewall interface while fw monitor would also tell you how the packet is going through the firewall including routing and NAT decisions.

Let’s get deeper

FW MONITOR

It captures traffic at 4 important points in the firewall namely i, I, o & O. You would see them in the capture in the same sequence.
i – Preinbound, just where the packet is received on the interface. If you see only this then the packet is dropped by address spoofing or the access rule.
I – Postinbound, where the packet has gone across the incoming interface. If you don’t see the next line in capture after this at ‘o’, you could infer that it’s a routing issue.
For both i & I the interface is the incoming, where the packet enters the firewall.
o – Preoutbound, the place where the packet is received at the exit interface within the firewall. If this is the point beyond which the capture is not seen, then you may need to check NAT.
O- Postoutbound, If you see this then make you are sure that the packet has left the firewall and the ACL, route and NAT all are working as expected.

fw monitor -e "accept src=10.1.1.201 or dst=10.1.1.201;" -o monitor2.out


TCPDUMP.

It captures at position i & O of firewall monitor, and you can be sure the traffic has left the firewall. This is similar to the way captures work on a Cisco PIX/ASA.

So, which one you use ?

Consider you run tcpdump and see the incoming traffic but don’t see the traffic leaving the exit interface. You can guess it’s a routing or a NAT issue. But to make a sure shot without wasting time by looking in routes or the NAT rules you could run fw monitor and know what the issue is.

Tuesday 2 June 2015

Redhat Linux, CentOS - Part 2


Processes

A process is created in memory when a program or command is initiated. It is assigned with a unique identification number known as process identification (PID). The background system processes are called daemons and critical to system functionality.
PPID - process ID of the parent process. TTY column shows the terminal on which the process was started. Console represents system console and ? indicates that the process is a daemon.

$ps -ef == process status -e (every) -f (full)



$top also can be used.
$pidof crond  == to list PID of a given process name
$ps -U root == list all processes owned by root. -G can be used for group


There are 5 process states and each process is in one state at any given time - running, sleeping, waiting, stopped and zombie.
sleeping - process is currently waiting for input from a user or another process
waiting - process has received the input it has been waiting for and it is now ready to run as soon as its turn arrives
stopped - process is currently halted and will not run even when its turn comes unless it is sent a signal.
zombie - process is dead; takes up no resource.

Listing Open Files

A file is opened when the process or program stored in it is executed and closed when it is no longer required. To see which files are open, which processes are using them, and who the owners are, use lsof (list open files) command.

Sunday 31 May 2015

Redhat Linux, CentOS - Basic

Basic Commands

who
who am i
w
logname
id
groups

last
last reboot
utmpdump /var/log/wtmp  == recent login and reboot

lastb -- list unsuccessful logins
utmpdump /var/log/btmp

uname -a --> uname --all == display OS name
date --set "Tue Jul 24 07:48:00 SGT 2014"

cal  == calendar
which == shows absolute path of the command that will execute
whereis == displays the binary name and full pathname of command along with locaiton of man

Compression/Zip Commands

tar cvf /tmp/home.tar /home     == create home.tar from /home
tar rvf /tmp/home.tar /etc/xinetd.d  ==append /etc/xinetd.d to home.tar
tar tvf /tmp/home.tar  == list content of home.tar
tar cvf /tmp/files.tar /etc/host.conf /etc/ntp.conf /etc/yum.conf
tar xvf /tmp/files.tar ==to extract

tar cvzf /tmp/home.tar.gz /home ==create a tarball and compress it with gzip
tar cvjf /tmp/home.tar.bz /home ==create a tarball and compress it with bzip

zip /tmp/files.zip /etc/host.conf /etc/ntp.conf /etc/yum.conf == compressed & add a zip file
unzip /tmp/files.zip

gzip /root/install.log /root/install.log.syslog == compress the files & add .gz extension
gunzip /root/install.log.gz
gzip -d install.log.syslog.gz

bzip2 /root/install.log /root/install.log.syslog == compress the files & add .bz2 extension
bunzip2 /root/install.log.bz2 /root/install.log.syslog.bz2
bzip2 -d /root/install.log.bz2

Linux File Systems Tree

/bin (binary) contains crucial user executable commmands.

/lib (library) contains shared library files required by kernel and other programs.

/sbin (system binary) contains most commands required at system boot up, crucial system administration commands that are not intended for normal users (require root priviledge)
/etc (etcetera) holds most system configuration files (eg. sysconfig, default). Dynamic data files.

lost+found directory hold files that become orphan(a file that has lost its name) after system crash; hold dynamic information.

/root - default home directory for root user

/net - all available NFS file systems on the network get mounted under their corresponding hostnames beneath /net.

/media - is used to automatically mount removable media.

/boot (boot file system) -Disk-based contains linux kernel, bootloader, boot config files and other files required to boot RHEL/Cent OS.

/var (variable file system) -Disk-based contains data that frequently changes while system is operational (e.g /var/log)
  /var/log contains most system log files./var/spool/mail - user mailboxes. /var/opt contains log, status for softwares installed in /opt. /var/spool holds print jobs, cron jobs, email messages, other queued items being sent out. /var/tmp contains large temp file or temp files that need to exist for extended periods of time than what is allowed in  /tmp, are stored here. These files are not automatically deleted after system reboot.

/usr (unix system resource file system) -Disk-based contains general files related to system.
  /usr/bin - additional user executable commands. /usr/sbin - additional system administration commands. /usr/local - system administrator repository for storing commands andtools that administrators download from the web, develop in-house or obain elsewhere. usr/share - location for man pages, documentation, etc.

/tmp (temporary file system) -Disk-based is a repository for temp files. Many programs create temp files as they run or as they are being installed.

/opt (optional file system) -Disk-based hold additional software installed on system. A sub-directory is created for each.

/home (home file system) -Disk-based hold user home directories.

/dev (device file system) -Virtual contains device files for hardware and virtual devices. Some key sub-directories are disks, pts and vg00 - hard disks, pseudo terminals and root volume group.

/proc (process file system) -Virtual maintains info about current state of the running kernel including details on CPU, memory, paritioning, running processes. It contains dynamic files and it is automatically maintained by the system.

/sys (system file system) -Virtual stores and maintains info about currenlty configure hardware. It is automatically maintained by the system.

/selinux (selinux file system) -Virtual stores all current settings for SELinux if installed.

Absolute and Relative Path

Absolute path points to a file or direcotry in relation to the root (/); always starts with (/). Relative path points to a file or dir in relation to your current location; starts with ./folder or .. or sub-directory name.

File Types

Normal files, directories
Executable files - a file that has x in the 4th, 7th or 10th field of ll output.

Symbolic link files - a link; begins with the letter l and there is an arrow pointing to the linked file or directory.

Named Pipe Files - allows 2 unrelated process running on the same system or on 2 different systems to communicate with each other and exchange data. It's unidirectiona; starts with 'p' in ll output.

Socket Files - is a named pipe that works in both directions; 2 way named pipe; used in client server programs; starts with 's' in ll ouput.

Files and Directory Operatoins

Creating files
touch (creates an empty file and update its timestamp) - $touch file1
cat (creates short text files) - $cat > newfile
vi, vim, nano

Displaying files
cat, more, less, head, tail, nano. view, vi, vim display the file in vi editor.
tail -f  == to view the content of the log file in real time

Copying files and directories
$cp file1 newfile1  == file1 is copied and name newfile1 in the same directory
$cp file1 scripts.dir1 == file1 is copied to scripts.dir1 directory
By default,when copying a file, the destination is overwritten and a warning message is not generted. To avoid this, use -i option which prompts for confirmation before overwriting.

$cp -r folder1 folder2  == using Recursive, copy folder1 along with its contents to folder2
-i option can be used together as well.

Moving and renaming files and directories
$mv -i file1 dir1 == move file1 into dir1
$mv newfile newfile1 == rename as newfile1
$mv dir1! dir2 == move dir1 into dir2 (dir2 must exist)
$mv dir1 dir2 == rename as dir2 (dir2 must exist)

Removing files and directories
$rm newfile == remove newfile
$rm dir1 == remove empty directory dir1
$rm -r dir1 == remove non-empty directory dir1



Files and Directory Control Attribute

$lsattr file1
$chattr +i file1 == make file1 cannot be changed, renamed, deleted

Pattern Matching

$grep thurein /etc/passwd
$grep thurein /etc/passwd /etc/group
$grep thurein /etc/passwd /etc/group /etc/hosts == display only file names which contain the pattern thurein
$grep -v root /etc/group == serach 'root' in /etc/group and exclude the lines that contains 'root'
$grep ^root /etc/passwd == search all line begins with 'root'
$grep bash$/etc/passwd == search all line end with 'bash'
$grep -i root /etc/passwd  == search root; ignores the letter case

Finding files

$find . -name newfile == search for newfile in my home directory
$find ~ -size -1M == find files smaller than 1MB
$find /usr -size +10M == find files lager than 10MB in /usr
$find /home -user thurein -not -gorup thurein == find files in /home with ownership set to thurein and group membership set to any group but thurein
$find /etc/rc.d -mtime +120 == find files in the /etc/rc.d directory that were modified more than 120 days ago
$find /etc/rc.d -atime -90 == find files in the /etc/rc.d directory that have not been accessed in the last 90 days
$find /etc/rc.d -mtime 10 == find files in the /etc/rc.d directory that were modified exactly 10 days ago

$locate passwd
$locate -n 3 passwd == display the first 3 file names from the results

Sorting

$sort file10
$sort -r file10
$sort -k 2 -n file10 == sort this file numberically on the second column
$ll / | sort
$ll / | sort -k 9 == sort according to filename
$ll -a /etc/skel | sort -k 6 -k 7 == sort on the 6th and then 7th comlumn

Linking files and directories

Each file in the system has a unique number assigned to it.The number is called inode(index node).

Soft link - it's like a shortcut pointing to an actual file. A soft link can cross file system boundaries and can be used to link directories.

[thurein@localhost temp]$ ln -s newfile newfilelink
[thurein@localhost temp]$ ll -i
total 4
655497 -rw-rw-r--. 1 thurein thurein 64 May 30 23:56 newfile
655498 lrwxrwxrwx. 1 thurein thurein  7 May 31 03:05 newfilelink -> newfile

Hard link - A hard link associates two or more files with a single inode number.These files have identical permissions, ownership, time stamp and file contents. A hard link cannot cross file system boundaries and cannot be used to link directories.

[thurein@localhost temp]$ ln newfile newfile10
[thurein@localhost temp]$ ll -i
total 8
655497 -rw-rw-r--. 2 thurein thurein 64 May 30 23:56 newfile
655497 -rw-rw-r--. 2 thurein thurein 64 May 30 23:56 newfile10

File Permission

3 Permission classes - User(u), Group(g) and Others(o)
4 Permission types - Read(r), Write(w), Execute(x) - executes a file or cd into the directory, Access Denied(-)
The output of ll command list the file/dir type and permission settings. The first character indicates the type of file or d for directory.  The next nine characters - three groups of three characters - show read(r),write(w),execute(x),or  none(-) permission for the three user classes: user, group, and others, respectively.

Changing Access Permission
chmod can modify the permission by using symbolic or octal notation.



File Permission

3 Permission classes - User(u), Group(g) and Others(o)
4 Permission types - Read(r), Write(w), Execute(x) - executes a file or cd into the directory, Access Denied(-)
The output of ll command list the file/dir type and permission settings. The first character indicates the type of file or d for directory.  The next nine characters - three groups of three characters - show read(r),write(w),execute(x),or  none(-) permission for the three user classes: user, group, and others, respectively.

Changing Access Permission

chmod can modify the permission by using symbolic or octal notation.

$chomd u+x file1
$chomd go+w file1
$chomd o-w file1

$chmod 544 file2
$chmod 744 file2

Default file and directory permission is handled by umask value. Subtract umask value from 666(file) and 777(dir) to have default permission.
$umask
$umask -S

Every file and directory has an owner associated with it. chown and chgrp commands are used to alter ownership and group membership.
$chown user2 file1
$chgrp user2g file1
$chown user2:user2g file1
$chown -R user2:user2g dir1 == recursive for all files and sub-directories

Special permission - setuid, setgid, sticky bit may be set up on executable files and directories.
The sticky bit is typically set on public writable directories (or other directories with rw permissions for everyone) to protect files and sub-directories owned by regular users from being deleted or moved by other regular users.

[thurein@localhost ~]$ ll -d /tmp
drwxrwxrwt. 14 root root 4096 May 31 19:40 /tmp

To set the sticky bit on /var, do either
$chmod 1755 /var
$chmod o+1 /var
$chmod 755 /var == unset sticky bit
$chmod o-1 /var == unset sticky bit
$find / -type d -perm -1000 == find all directories with sticky bit set

Manipulating Input Output

Redirecting Standard input and output
$mailx user1 < /etc/group
$sort 11.out > sort.out
$sort 11.out >> sort.out  == to append the output to sort.out

Command line history is stored in user's home directory/.bash_history. Also can use
$history

~ can be used with echo, ls, cd that refers to a location in the directory structure.
~  == $HOME directory of the user
~+ == current directory
~- == previous directory

(|) pipe sends output of one command as input to another command.
$ll /etc | more
$ll /etc | grep root | grep -i apr | nl

The tee filter can be used to send an output to more than one destination.
$ll /etc | nl | tee /tmp/ll.out
$date | tee -a /tmp/ll.out  == append the output to a file



Friday 29 May 2015

AllocPSA - Installation notes

CentOS 6.6

Steps
install apache/httpd
  -change permission -- sudo chown -R apache:apache /var/www/
  -sudo chmod 755 /var/www
  -restart server and can browse home page..not sure if this 3 steps is necessary
install mysql - yum install mysql-server, and start
install php - yum install php php-mysql


Follow instruction from http://sourceforge.net/p/allocpsa/discussion/562149/thread/12ba9198/

403 access denied error is sovled by changing SElinux directory label
  ls -Z
  chcon -Rv --type=httpd_sys_content_t html
Enable php mbstring extension - if not 500 internal error - php fatal error utf8
  yum install php-mbstring
Enable php GD extension
  yum install gd gd-devel php-gd

Created a folder /var/local/alloc and give apache permission to write

Follow instruction by accessing http://serverip/allocxxx..
Connect to mySQL -> mysql -u root -p


Other Notes
find / -name web*

install webmin (use wget)
open port 10000 in iptables(/etc/sysconfig/iptables) or firewall-cm
  -iptables fw traffic can be viewed at /var/log/messages but check config at /etc/rsyslog.conf

httpd log file is at /var/log/httpd/error-log