How to Record URL and User Information in ISA 2004 Firewall Logs and Reports

One of the most common questions I see on the ISAServer.org Web boards and mailing list is how to get user and URL information in the ISA firewall's logs and reports. The ISA firewall creates reports using ISA log summaries. The log summaries are derived from the ISA firewall's Web Proxy filter and Firewall service logs. If you want to see user information and URLs (instead of IP addresses) in the reports, you've got to get that information into the logs first.

You need to address the following issues to get the user information into the logs:

• Configure the Web Proxy log to record user and URL information
  
• Configure all machines on the internal network as Web Proxy and Firewall clients
  
• Remove anonymous access rules that would apply before authenticated access rules
  
• Install LogHostname to get URLs for SecureNAT clients
  

NOTE:

This article is not a comprehensive review of the ISA firewall's logging and reporting features. Check out the last chapter in our book Configuring ISA Server 2004 for more information on the ISA firewall's logging and reporting feature set.

Configuring the Logs to Record User Information

The ISA firewall's Firewall service and Web Proxy filter log files record user names by default (for the Web proxy filter, only if the rule requires authentication). However, it is possible that someone disabled the logging of user names. You can confirm that that the Client user name field is logged by checking the Firewall Logging Properties dialog box:

1. In this ISA firewall console, expand the server name and click the Monitoring node.
   
2. On the Monitoring node, click the Logging tab in the details pane.
   
3. Click the Tasks tab in the Task Pane. Click the Configure Firewall Logging link.
   
4. In the Firewall Logging Properties dialog box, click the Fields tab.
   
5. On the Fields tab, scroll down the list of fields in the Include the selected fields in the log dialog box. Confirm that there is a checkmark in the Client Username checkbox. There are many of useful fields that you can log, so take some time to check out the other options on the Fields tab of the Firewall Logging Properties dialog box.
   
6. Figure 1
   
7. Click Apply and then click OK.
   

Note that there is no option to log the URL in the Firewall Logging Properties dialog box. The reason for this is that the Firewall client doesn't send the URL for Web sites accessed via the Firewall client. However, we'll correct this issue later with the Web proxy client configuration.

Now we need to confirm that the Web proxy filter logs include the username and URL fields so that this information appears in the ISA firewall's logs and reports:

1. In the ISA firewall console, expand the server name and then click the Monitoring node.
   
2. On the Monitoring node, click the Logging tab in the details pane.
   
3. Click the Tasks tab in the Task Pane. Click the Configure Web Proxy Logging link.
   
4. In the Web Proxy Logging Properties dialog box, click the Fields tab.
   
5. On the Fields tab, confirm that there are checkmarks in the Client Username and URL checkboxes.
   
6. Figure 2
   
7. Click Apply and then click OK.
   

Notice that only the Web Proxy filter log maintains URL information. This is important because it confirms that clients must be configured as Web proxy clients (or use the LogHostname filter)

Configure Machines as both Web Proxy and Firewall Clients

User information associated with a connection request is only available when requests sent by Web Proxy and/or Firewall clients. The SecureNAT client is cannot send user information with its connection requests. The SecureNAT client can't send user information with its connection requests because the SecureNAT client connections are not proxied to either the ISA firewall's Web proxy filter or directly to the ISA firewall's Firewall service.

Proxy communications are client/server communications and the client device requires a client-side software component. There is no client-side software to send user information to the firewall with SecureNAT clients. Web Proxy and Firewall client requests are proxied to the ISA firewall with the aid of the Firewall client software and Web browser configuration.

Firewall client users must be logged into the same domain as the ISA firewall, or logged into a domain the ISA firewall trusts. The Firewall client always sends user credentials in the background, and the Firewall client always sends user credentials even when Access Rules may not require authentication.

You can tell when the Firewall client has sent user credentials to the ISA firewall for a connection not requiring authentication because a questions mark (?) will appear next to the user name in the Session tab of the Monitoring node. The Firewall service never prompts the user for credentials. If you see a log on dialog box, it is not the Firewall service asking you for credentials.

ATTENTION:

This is a critical design and planning issue. In order to fully leverage the authentication, authorization, access control, and protocol support provided by the Firewall client software, the ISA firewall must be a member of the user domain, or be a member of a domain that trusts the user domain. You lose a significant amount of ISA firewall security and flexibility when you don't make the ISA firewall a domain member.

In contrast to the Firewall client authentication scheme, the user does not need to be logged on to the domain when using the Web Proxy client and the ISA firewall doesn't need to be a domain member. However, if the ISA firewall isn't a domain member, or if the ISA firewall is a domain member but the user is not logged into the domain, then transparent authentication via integrated authentication won't be available. If basic authentication is enabled, then the user will be challenged with a log on dialog box.

In order to configure clients as Web Proxy and Firewall clients, you need to:

• Configure the Firewall client listener
  
• Configure the Web Proxy listener and authentication support
  
• Install the Firewall client software on the client machines
  
• Configure the browsers as Web proxy client
  

Perform the following steps to configure the Firewall client listener:

1. In the ISA firewall console, expand the server name and then expand the Configuration node. Click the Networks node.
   
2. On the Networks node, click the Networks tab in the details pane.
   
3. On the Networks tab, click the ISA firewall Network for which you want to configure the Firewall client listener. In this example, we'll configure the listener for the default Internal Network.
   
4. In the Internal Properties dialog box, click the Firewall Client tab.
   
5. On the Firewall Client tab, enter the FQDN clients on the default Internal Network will use to connect to the IP address of the ISA firewall's internal on the default Internal Network. Always use FQDNs, don't use single-label NetBIOS names and don't use IP addresses. Make sure the correct entries for this FQDN are included in your DNS. The Web browser configuration on the Firewall client computer frame enables you to configure the browser using the Firewall client configuration. When you install the Firewall client, you can automatically configure the machine as a Web proxy client. You have the option to configure the browser as a Web proxy client by select one or more of the following options: Automatically detect settings, Use automatic configuration script or Use a Web proxy server. We go into full details on the implications of these Web proxy settings in Configuration ISA Server 2004, so I won't go over them again here. In general, you're best off configuring WPAD entries and using the Automatically detect settings option. For a great review of the ISA firewall's autodiscovery feature, check out Stefaan Pouseele's article Understanding the Web Proxy and Firewall Client Automatic ConfigurationUnderstanding_the_Web_Proxy_and_Firewall_Client_Automatic_Configuration.php
   
6. Figure 3
   
7. Click the Auto Discovery tab. Put a checkmark in the Publish automatic discovery information checkbox. Leave the default value of 80 in the Use this port for automatic discovery requests text box.
   
8. Leave the Internal Properties dialog box open for the next procedure.
   

The next step is to configure the Web proxy listener on the same network:

1. Click the Web Proxy tab. Confirm that there is a checkmark in the Enable Web Proxy clients checkbox.
   
2. Figure 4
   
3. Confirm that there is a checkmark in the Enable HTTP checkbox and that the default port is 8080 (you can change this, but there is rarely a reason to do so).
   
4. Click the Authentication button. In the Authentication dialog box you can set the authentication methods you want to support. Integrated authentication enables transparent authentication for domain users. If you have non-domain users, or Mac users, then you can enable Basic authentication as well. There are other authentication options you can select based on your network requirements. We discuss them in detail in Configuring ISA Server 2004. You do not need to enable the Require all users to authenticate option, because you will create Access Rules that require authentication. In this example we'll enable Integrated and Basic. Click OK.
   
5. Figure 5
   
6. Click Apply and then click OK in the Internal Properties dialog box.
   

There's a lot more you can do to customize the Firewall client and Web proxy client configuration settings at the ISA firewall. Check out the Chapter on ISA client types in the book for all the details.

The next step is to install the ISA Firewall client software on the client machines. In order to install the Firewall client on the client machines, you must install the Firewall client software installation file either on the ISA firewall itself, or on a file server somewhere on the corporate network. I highly recommend that you install them on a file server on the network, as you really don't want to allow SMB/CIFS connections to the ISA firewall itself. You can install the Firewall client share from the autorun menu that appears when you put the ISA Server 2004 CD into the CD-ROM drive.

If you choose to not install the Firewall client, you can still get user names and URLs in the log files and reports by configuring the Web browsers as Web proxy clients. There are many ways to configure the browsers as Web proxy clients:

• Automatically configure the browsers during installation of the Firewall client software
  
• Manually configure browsers as Web proxy clients in the Internet Options dialog box
  
• Configure Group Policy to configure clients as Web proxy clients
  
• Use IEAK to configure browsers as Web proxy clients
  
• Use log on scripts to configure clients as Web proxy clients
  

The key issue is that the clients are configured as Web proxy clients. Web proxy clients will send user credentials when challenged by the ISA firewall's Web proxy filter for authentication. SecureNAT clients will not be challenged for authentication and cannot provide user credentials. In addition, only the Web proxy client's connections will register a URL in the log files and reports. Firewall and SecureNAT client connections will never register URL information in the logs and all you'll see are IP addresses. The only exception to this is if you have installed the LogHostname software on the ISA firewall.

Enable Anonymous Access Only for Sites not Requiring User Authentication

In general, ISA firewall policy should be configured using the following principles:

• Anonymous deny rules should be placed on top
  
• Anonymous allow rules should be placed below anonymous deny rule
  
• Authenticated deny rules should be placed below anonymous deny and anonymous allow rules
  
• Authenticated allow rules should be placed below anonymous deny, anonymous allow, and authenticated deny rules
  

In reality, configuration of a well crafted firewall policy that does exactly what your corporate security design requires is a bit more complex. Check out my article Optimizing ISA Server 2004 Firewall Policies at http://techrepublic.com.com/5100-6345_11-5579216.html?tag=search for the best way to configure ISA firewall polices and rule sets for optimal security and performance. Also, check out Stefaan Pouseele's excellent article on ISA firewall access policies, Understanding ISA 2004 Rule Processing at http://www.isaserver.org/articles/ISA2004_AccessRules.html.

The key to getting user names and URLs in log files is to make sure that you do not create any anonymous allow or deny rules pertaining to sites that you want user information for. While the Firewall client will always send user information to the firewall, the Web proxy client will not send user information if authentication is not required.

For example, the only anonymous access rules should be those you require for server access to the Internet. Since servers typically (and should not) have logged on users, you need to create anonymous access rules to enable or deny connections from these servers (access control for servers is usually done by using source IP address). All other rules should require user authentication from either the Firewall client configuration, Web proxy client configuration, or both.

Note that another reason why your anonymous access rules are configured for servers only is that servers should not have the Firewall client software installed. Although they can be configured as Web proxy clients, if the servers require Web access when there is no logged on user, an authenticated access rule will cause the server's attempt to connect to the Internet will fail.

Install Collective Software's Loghostname Web Filter

Throughout this article I've made it clear that in order to get user names in the log files you need to configure the clients as Firewall clients, Web proxy clients, or both. There is no way around this issue. There is no magic field in the TCP, UDP, IP or any other "packet level" header that will enable user information to be placed in the logs for SecureNAT clients. Firewall clients will provide user information in the logs for all Winsock applications used to connect to the Internet (essentially all TCP and UDP applications) and the Web proxy client configuration will enable user information to appear in the logs for Web connections (HTTP/HTTPS/HTTP-tunneled FTP).

In order to get URL information into the logs and reports, you need to configure the clients as Web proxy clients. Firewall client and SecureNAT client connections do not provide URL information in the logs and only IP addresses for sites visited by Firewall and SecureNAT clients appear in the logs and reports. There nothing you can do about it.

Well, there was nothing you could do about it, until now. Now we don't have to suffer from the absence of URLs in the log files for Firewall and SecureNAT clients. Why? Because Collective Software (www.collectivesoftware.com) has released its Loghostname Web Filter!

The Loghostname Web filter magically transforms the log files in on the ISA firewall to provide URL information for Web sites visited by SecureNAT and Firewall clients. If for some reason you won't or can't make your clients Web proxy clients (although you should always use the Web proxy client configuration - but admins don't always do what's best for them), then Loghostname by Collective Software (www.collectivesoftware.com) is a dream come true.

Its important to note that the Loghostname Web filter is a Web filter. Because its a Web filter, it ties into the ISA firewall's Web proxy filter. All communications mediated by the ISA firewall's Web proxy filter are logged in the Web proxy log. Note that SecureNAT and Firewall client Web requests are automatically redirected to the ISA firewall's Web proxy filter when the Web proxy filter is bound to the HTTP protocol. If you unbind the Web proxy filter from the HTTP protocol, then the SecureNAT and Firewall client connections will not be automatically redirected to the Web proxy filter, and Loghostname will not be able to log host names to the Web proxy log. The Web proxy filter must be enabled on the HTTP protocol (which is the default setting on all ISA firewalls) in order to for Internet host names to appear in the ISA firewall's Web proxy log.

Loghostname is a simple Web filter that uses an .msi install file. Just download it from http://www.collectivesoftware.com/Products/ and double click the installation file. This will automatically install the Web filter. After installing the Web filter, restart the Firewall service.

This is what the log file looks like for a SecureNAT client connection before installing Loghostname:

Figure 6

But with Loghostname, the SecureNAT client's log entries look like this:

Figure 7

Nice, eh?

Summary

The ISA firewall makes it easy to get detailed user information in the Web Proxy and Firewall service log files. You just need to force all clients to use the Firewall and Web Proxy client configuration, configure the logs to record user information, and remove anonymous access rules except for those required for servers that need to access the Internet without logged on users. You can also get URL information in the ISA firewall's logs and reports by configuring the clients as Web proxy client, or by installing the Loghostname Web filter. I routinely create this configuration because a major factor in security is accountability. The only way to get accountability is to require user authentication and identification. This level of logging may also be required in environments subject to regulatory requirements.

Pasted from <http://www.redline-software.com/eng/support/articles/isaserver/config/how_to_record_url_and_user_information.php>

Create an Option 252 Entry in DHCP

To create an Option 252 entry in DHCP, do the following:

1. Click Start, point to Programs, point to Administrative Tools, and then click DHCP.
   
2. In the console tree, right-click the applicable DHCP server, click Set Predefined Options, and then click Add.
   
3. In Name, type WPAD.
   

• In Code, type 252.
  
• In Data type, select String, and then click OK.
  
• In String, type http://Computer_Name:Port/wpad.dat where:
  
  • Computer_Name is the fully qualified domain name of the ISA Server computer.
    
  • Port is the port number on which automatic discovery
    
  • \information is published. You can specify any port number. By default ISA Server publishes automatic discovery information on port 8080.
    

1. Right-click Server options, and then click Configure options.
   
2. Confirm that the Option 252 check box is selected.
   

Pasted from <http://technet.microsoft.com/en-us/library/cc713344.aspx>

Create a WPAD Entry in DNS

To create a WPAD entry in DNS, do the following:

1. Click Start, point to Programs, point to Administrative Tools, and then click DNS.
   
2. In the console tree, right-click the applicable forward lookup zone and click New Alias.
   
3. In Alias name, type WPAD.
   
4. In Fully qualified name for target host, type the fully qualified domain name (FQDN) of the WPAD server.
   

Note:

The ISA Server computer or array needs a host (A) record defined before you can create an Alias entry. If a host (A) record is defined, you can click Browse to search the DNS namespace for the ISA Server computer.

Pasted from <http://technet.microsoft.com/en-us/library/cc713344.aspx>