S u p p o r t
for ntop

ntopsupport.com
US Based

FAQ Content Last updated: 22May2006

ntop FAQ...

This is an unsophisticated, automated conversion of the source file, docs/FAQ into html. Please report problems to the ntop-dev mailing list. But remember, it's not about making it look good, it's about making the content available.

Features

Q. What determines the features of ntop?

A. Whatever Luca wants

Q. Why did you do this "x" instead of feature "y"?

A. Don't know. I could guess...

Imagine you are the network manager for a large University network and have to crack down on users who are illegally exchanging copyrighted files or using University resources to run a business without paying for the resources being consumed.

You are a major vendor of infrastructure, whose customers are using networks for new features such as storage area networks and you want to give them the ability to monitor these.

You have a really cool technology that you've just donated to the community via an RFC and wish to jump-start adoption of it.

You are moving heavily into IPv6 and need to be able to monitor your 'new' network just like you monitored the IPv4 one.

or (my favorite)

You really, really, really hate that ntop generates such lousy html code and you decide to scratch that itch.

or (what should happen)

A major corporation, with out resources, time, skills and/or inclination to do it themselves sponsors the development of a feature that's critical to him/her.

Then again, it could just be because it's cool...

Q. Could ntop do "x"

A. Probably - as long as it doesn't move the tool away from it's purpose and it's strengths, almost anything is possible - especially as a plugin.

Q. Will you do "x"

A. Maybe - if it's of interest to a developer, or you provide the code such that it can be merged in, or if you're willing to sponsor the development effort (contact us through http://www.ntop.org/consultancy.html).

Documentation

Q. Why isn't there (any)(more)(better) documentation.

A. (A personal peeve from Burton...)

I get real tired of people complaining that there isn't any documentation and then being unwilling to contribute even the simplest stuff. I've said I'll edit and assemble whatever people send me... and since I started working with ntop in November 2001, I've received maybe six pages of stuff.

I'm trying to get people - who aren't coders - to contribute to ntop the project. The contribution that ANYONE can make is "documentation". A task- specific HOWTO... some sample screen shots... An FAQ entry...

I've tried being nice. I've tried asking. I've tried shaming people into it.

What have I gotten? Zip.

Nasty is all that's left... This is your fair warning. If you show up on the ntop mailing lists and complain about documentation, you will get blasted.

-----Burton

Q. Ok, where can I find what does exist.

A. http://www.ntop.org has pointers and some (very out-dated) documents.

The documentation in the docs/ directory, the Documentation files at SourceForge and some at http://www.ntopsupport.com are basically all that there is.

Search the ntop mailing lists at gmane, http://search.gmane.org. The lists are called gmane.linux.ntop.general and gmane.linux.ntop.devel.

Please contribute to the ntop community by writing things up for inclusion in this FAQ or other documents!

Q. I can't find a file at SourceForge!

A. You can reach the archives through any SourceForge mirror, or the main site:

      http://prdownloads.sourceforge.net/n/nt/ntop

It has ALL the files unless we explicitly delete them...

Problems

Q. I have a problem...

A. Make sure it's a supported release!

We support only the current versions of ntop. This is either:

       * the last release, e.g. v3.1.
       * the current cvs (cvs.ntop.org)
       * the latest development version posted at SourceForge (if one)

If you use a port/package and the latest version available for your OS is some release candidate from a year ago, sorry. Contact the packager and ask them to get current.

Luca usually asks people to try the latest cvs (development) version because problems are frequently already fixed in there.

Q. I'm running something supported and I've tried the latest & greatest. Still, I have a problem.

A. Read the ntop mailing lists.

The ntop mailing lists are at

      http://listgateway.unipi.it/mailman/listinfo/ntop
      http://listgateway.unipi.it/mailman/listinfo/ntop-dev
   and
      http://listgateway.unipi.it/mailman/listinfo/ntop-misc

If you're having non-user problems OR you are using the cvs, you should be reading and posting ntop-dev (for example, the cvs commit messages are posted there). Stuff unrelated to baseline ntop (PF_RING) belongs in ntop-misc.

You can read the lists through gmane (or other gateways) if you don't want to subscribe, but only subscribers can post.

You can download the older messages in large chunks from the mailing list subscription pages. Look for "To see the collection of prior postings to the list".

Q. I looked and I didn't find my problem.

A. Join the mailing list(s) and ask for help.

Before you post, check the "HowTo Ask for Help" at the end of this FAQ.

Please, if at all possible, use the built in PR_ form (the little 'bug' icon on the 'About' tab).

Guidelines for asking questions:

ONE and only ONE problem / issue / question per message.

With a meaningful subject.

         The goal is that if you're asking a common question, the
         subject would have allowed you to find it in the back
         traffic for the mailing list.

Post the information about your environment we ask for.

         We STRONGLY suggest you use the automatically generated "Problem
         Report" form that since it contains much of the necessary information.

Make sure you're in a supported environment (./configure --showoses).

         If it's an unsupported environment, we're interested in your efforts to
         make ntop work, but we don't have the time, resources, knowledge and/or
         interest to do it ourselves.

For software 'crashes', please run ntop under the gdb debugger and capture the full failure information.

         Brief instructions on using gdb are at the end of this file (docs/FAQ).

Q. I posted to the list and nobody answered me.

A. ntop is open source, and the lists are a community resource. If nobody answered your question, then nobody knew the answers off-hand and nobody wanted to spend THEIR time solving YOUR problem.

Q. Do you offer paid support?

A. Yes - contact us through http://www.ntop.org/consultancy.html

Configuring ntop

Q. What does ./configure do?

A. ./configure checks for the tools installed on your system - configuring ntop to compile with the ones you have and skip the ones you don't (or to tell you if you're missing something critical).

Q. Why bother - just compile the code.

A. Then you would have to have a machine configured EXACTLY like Luca's. Nothing else would work. Various OSes and Linux distributions package the files in different ways and put them in different places. Plus some packages put files into directories with release information in them, etc.

Q. OK, so ./configure

A. Is how you tell ntop where to find things. A lot of stuff it can figure out on it's own, but if things get put in 'strange' places, ntop's ./configure has switches you use to tell where to find things.

Q. And the list is?

A. ./configure --help shows the whole list. It's a bit confusing because there are standard options and ntop options mixed in there.

A. So, first let's look at the 'where are things' options. There are two types of files ntop is looking for, '.h' files and libxxxx files.

.h files are also called 'includes' and libxxxx files are called libraries or lib files.

.h files are the C source for functions provided by the OS or by libraries. They are typically in a directory named /usr/include, but they can be placed ANYWHERE.

lib files are compiled libraries of these functions (the .h tells ntop how to call something, the lib file is the actual code). Their names usually begin libxxxx (so the library gd is named libgd).

By convention, libraries end in .so or .a. A .so library is a shared library (Windows DLL), where one copy of the library is used by all programs that want it's functions. A .a library is a non-shared or static library, which must be merged (the technical term is linked) with the code.

ntop uses both - the myrrd library is a static, .a library. When it comes to things like libpcap or libgd, we use shared (.so) libraries.

Library files are typically placed in /usr/lib, where the gnu linker (ld), 'knows' automatically how to find them. However, from OS to OS and distribution to distribution, there are many other common places. Some OSes even have a file telling ld all the places to look.

Q. So ntop looks for these .h and library thingies in a couple of places. What if it doesn't find them?

A. If a basic ./configure can't find something, you'll have to tell ntop where to look.

It's complex and OS/distro dependent. For example, if you install libgd from the Sun Freeware site on to a Solaris machine, the files get put into /usr/local/include and /usr/local/lib, which are not on the lists of 'standard' places for Solaris' versions of gcc (the Gnu c compiler) or ld. So to compile ntop, you have to tell gcc to look in these additional locations.

The things ntop might be looking for are in this part of the ./configure --help output:

       +-External-source-locations:-------------------------------------------------+
       --with-pcap-root=DIR        LBNL pcap located in DIR
       --with-pcap-lib=DIR          or libpcap located in DIR
       --with-pcap-include=DIR      or pcap.h located in DIR
       --with-gdbm-root=DIR        gdbm located in DIR
       --with-gdbm-lib=DIR          or libgdbm located in DIR
       --with-gdbm-include=DIR      or gdbm.h located in DIR
       --with-zlib-root=DIR        zlib located in DIR
       --with-zlib-lib=DIR          or libz located in DIR
       --with-zlib-include=DIR      or zlib.h located in DIR
       --with-gd-root=DIR          gd located in DIR
       --with-gd-lib=DIR            or libgd located in DIR
       --with-gd-include=DIR        or gd.h located in DIR
       --with-libpng-root=DIR      libpng located in DIR
       --with-libpng-lib=DIR        or libpng located in DIR
       --with-libpng-include=DIR    or png.h located in DIR
       --with-ossl-root=DIR        openSSL located in DIR
       --with-ossl-lib=DIR          or libssl located in DIR
       --with-ossl-include=DIR      or ssl.h located in DIR
       --with-localedir=DIR        LOCALE files located in DIR (i18n)
       +-----------------------------------------------------------------------+

You can see that there is a pattern, pretty much every --with-xxxxx-root has a --with-xxxxx-include and --with-xxxxx-lib option.

So, if ntop tells you it can't find something, do this - first look for the File on your system:

     $ locate pcap.h
     /usr/include/pcap/pcap.h

(If you don't have locate, this works too:

     $ find / -type f -name "pcap.h"
   )

And then tell ./configure via --with-pcap-include=/usr/include/pcap

(Some OSes are smart enough to look in a subdirectory of the standard location, but others aren't).

Q. Ok, but why three options?

A. You use either the --with-xxxxx-root option OR either/both of the others at a time. But ntop really only looks at the --with-xxxxx-include and --with-xxxxx-lib options.

Internally, --with-xxxxx-root=/a/b/c is translated into --with-xxxxx-lib=/a/b/c/lib and --with-xxxxx-include=/a/b/c/include (that's the usual pattern).

Now sometimes libraries are installed logically - if the pcap.h file is in /usr/local/pcap/include, the library (libpcap.so) is probably in /usr/local/pcap/lib. Sometimes they are not logical and you will have to use the split options.

The --with-pcap-root=/usr/local/pcap is shorthand for the two options, --with-pcap-include=/usr/local/pcap/include and --with-pcap-lib=/usr/local/pcap/lib.

Q. Oh Ghu - aren't there any short cuts.

A. For the first time you try ./configure, there's a script on SourceForge in The user contributed area that will try to build the ./configure line for you.

Q. And the OTHER options

A. There is a set that tells ntop where to install stuff. For simplicity, the two you might want to change are:

       --prefix=PREFIX         install architecture-independent files in PREFIX
                               [/usr/local]
       --datadir=DIR           read-only architecture-independent data
                               [PREFIX/share]

--prefix tells ntop where to install the various files. The default value is /usr/local, which is where most non-OS software normal goes.

A common choice for libraries (such as pcap) is --prefix=/usr, which puts things like .h files in places easier to automatically find (/usr/include). --prefix=/usr certainly works for ntop. --prefix=/opt is another choice.

The 'proper' choice turns on which model of file organization the OS and sysadmin prefer. That's a fight I'm staying out of.

--datadir tells ntop where to put its databases and output files. The default is /usr/share/ntop, but that will give some sysadmin's agita. Another popular choice is --datadir=/var, which puts all the files in /var/ntop. That may be attractive especially if you make /var/ntop a separate partition, so the rrd files don't eat all your disk space.

Q. What's --enable-iknowbetter Override WILLFAIL

A. There are some error messages where it's possible that thing work (now) that didn't used to, or you're doing development and don't want ntop to stop you from doing something, or there's an error message that you have skipped before without getting bitten.

--enable-iknowbetter will print the message but not stop ntop from finishing ./configure.

Use it at your own risk.

Q. What are the --enable and --disable options?

A. These (and the with/without options) pretty much do what you think - they enable or disable large chunks of ntop functionality:

       +--ntop-specific:------------------------------------------------------------+
       --enable-sslv3              enable ssl v3 support [default=disabled]
       --enable-sslwatchdog        enable Watchdog for ssl hangups [default=disabled]
       --disable-plugins           disable compilation of plugins [default=enabled]
       --enable-static-plugins     Enable static linked plugins sntop, default=dynamic]
       --enable-ignoresigpipe      Ignore SIGPIPE errors [default=do not ignore]
       --disable-snmp              Disable SNMP support [default=disable]
       --enable-i18n               Enable (limited) internationalization [default=disabled]
       --enable-jumbo-frames       Enable Jumbo (9K) Ethernet frames [default=disabled]
       --disable-ipv6              use IPv6 [default=enabled]
       +-----------------------------------------------------------------------+

       +--external-packages:---------------------------------------------------+
       --without-ssl               disable HTPPS support [default=enabled]
       --without-zlib              disable zlib [default=enabled]
       --with-tcpwrap              enable use of TCP Wrapper [default=disabled]
       +-----------------------------------------------------------------------+

Q. What ELSE?

A. There are some so called 'environment variables' you can set that change things too. The common ones are:

        CC          C compiler command
        CFLAGS      C compiler flags
        LDFLAGS     linker flags, e.g. -L if you have libraries in a
                    nonstandard directory 
        CPPFLAGS    C/C++ preprocessor flags, e.g. -I if you have
                    headers in a nonstandard directory 
        CPP         C preprocessor

You would use these variables to override the choices made by ./configure or to help it to find libraries and programs with truly nonstandard names/locations.

The best place to look for examples of these environment variables are in the OS/distribution files in the configureextra directory. For example, (again picking on Solaris), we use LDFLAGS to tell ld to look in some common Solaris locations for libraries via this:

     LDFLAGS="-L/usr/local/lib -R/usr/local/lib ${LDFLAGS}"

What that does is add /usr/local/lib to the locations that ld will check.

Q. ./configure dies in some strange horrible way complaining about version conflicts, lunar phase or whatever.

A. The process of creating portable cross-platform scripts for building software is ugly and hard and prone to failure. There are tools (released by gnu) to help, named automake, autoconf and libtool. (Collectively the auto* tools).

These tools take basic files and generate more complex files based on a series of rules, conventions and macros (much of which is poorly or undocumented). Other processes (e.g. ./configure) take these files and generate more complex files based on a different series of rules, conventions and macros. This ultimately produces the 'Makefile', which a program called make uses - based on (...wait for it...) yet another series of rules, conventions and macros - to actually compile the ntop source.

There are interdependencies among the tools, partial support for older versions in some releases, but not in other releases and many more problems.

Different OSes and different Linux distros ship with wildly different versions. Some even have scripts that attempt to analyze the file and pick the correct version (which just means that trying to code a file with multiple version support confuses the tool).

So if you have a basic, total failure of ./configure, it's usually the auto* tools.

ntop used to ship with a manual script that rebuilt the generated files according to the version(s) of the tools installed on the system you were using to build ntop. Thus the standard answer was 'run ./autogen.sh -1'.

But, this meant that you had to have these 'developer' tools installed and caused much problems and gnashing of teeth.

So we spent 100s of hours rebuilding the scripts to be totally independent of having the tools installed on your system - only to run into problems because the xyz 1.6.1 tool installed by default on OS A isn't quite compatible with the 1.6.3 version on OS B.

So we put technology in place to automatically detect tool versions and rebuild the generated files if necessary. That meant you had to have these 'developer' tools installed and caused much problems.

So we rebuilt the scripts AGAIN and AGAIN, dropped support for old versions of the tools and finally reached a point where it works for most reasonably current platforms. This is a compromise:

       Systems that don't have the tools installed usually work.
       Systems that have bleeding edge versions of these tools may break.
       Systems with very old versions also may break.

The technology to detect versions and rebuild the script files if appropriate is still in there, but it's disabled from normal use.

If you have the auto* tools installed and have ./configure problems, you can activate the automatic rebuild feature via:

      $ export NTOPAUTOREBUILD=yes
      $ ./configure ...

If the rebuild fixes it, that's great. Regardless, please report the problem to the ntop mailing list. Please don't paraphrase the messages cut & paste the ACTUAL MESSAGES into your report. Also, please let us know the version(s) of the tools installed on your system:

      $ aclocal --version
      $ autoheader --version
      $ autoconf --version
      $ automake --version
      $ libtool --version

Q. I can't build ntop... (auto* tools)

A. ntop has been tested with and developed for various versions of the "GNU auto* tools" (automake, autoconf and libtool).

The basic requirements are

automake 1.6.1 or higher (automake 1.4 and 1.5 may or may not work) autoconf 2.51 or higher (autoconf 2.13 will NOT work) libtool 1.4 or higher

But that simple statement hides a lot of complexity and many potential problems. Surprisingly, this isn't solely ntop's fault, it's the combination of GNU auto* tools version to version compatiblity and because we're trying to distribute configure and make files that work on all versions. If ntop were less complex or less clever about trying to build a working program even with many components missing, things would work a lot better.

The big change between autoconf 2.13 and 2.5x generated scripts is that the ./configure and make steps are supposed to run the auto* commands automatically if they're required. As we've seen, this doesn't always work!

If you run into problems, you can ALWAYS recreate the generated files via this procedure:

      rm -f acinclude.m4 aclocal.m4 Makefile.in config.h.in configure Makefile

      find current versions of libtool, config.guess and config.sub and cp
      them into your working directory.

      cat acinclude.m4.ntop libtool.m4.in > acinclude.m4
      aclocal
      autoheader
      autoconf
      automake --gnu --copy --add-missing

and then:

      ./configure ...
      make
      make install

as usual.

Q. But how does it all hang together?

A. There's a vsd (Visio)/pdf in the docs directory - ntop-autotools.pdf.

Compiling ntop

Q. Which packages/libraries do I need to compile ntop:

    Note: In some cases the minimal header files for a tool will be in
          one "package" and the execution library in another. ntop needs
          both so that the ./configure test finds the tool. It's usually
          safest to install both the tool and development packages!

         (Some packages will have additional packages as pre-requisites)

       gdbm (We have people using 1.7.3, 1.8.0, 1.8.2 and 1.8.3)

       libpcap
          We recommend 0.8.3, but ntop works with most 0.7.2 and later versions.

          STAY AWAY from older versions, especially under Linux.

          Note: Building libpcap requires: bison/flex

       gd 
          http://www.boutell.com/gd/gd.html - note that many distros have both
          the 1.8.x and 2.x series installed for differnt packages that require
          them.  This occasionally causes problems.

       libpng (1.2.x - although ntop works with 1.0.x versions, the two libpng
              versions are not compatible and will often cause version conflicts
              and crashes).

       glibc

cpp

       gcc most versions of gcc should work, but there are no promisses.

       libtool  1.4+ (distribution is built with 1.4.2)
                     (there are successes reported with 1.3.4 or 1.3.5 -
                      use --enable-iknowbetter)

       gawk/mawk
          Note: If all you have is original awk, then ./configure will not work.

Q. Will the "xyz" compiler work instead of gcc?

A. We explicitly REQUIRE gcc.

We know that under many systems, a compiler called cc is available. On some, it's a symbolic link to gcc, BUT, when invoked as cc, it often triggers 'old' behaviors for cc compatibility. On others, cc is a retarded compiler just good enough for compiling the kernel.

The one exception is Sun's cc, which - since Luca used to do a lot of development on Solaris - was pretty well supported.

Q. The auto* tools?

A. Only if you are going to rebuild the distributed script files:

           autoconf 2.5+ (distribution with 2.57)
           automake 1.6+ (distribution with 1.8.3)

Q. Optional libraries?

A. Lots. The most common is:

       openssl (for https:// support)
          Note: For security reasons you should only have the most current
                Version of openssl installed.

Q. ntop doesn't compile.

A. Sure, it does - for me :-)

Q. Right smarty-pants...

A. First, check if you're using a supported OS.

The long and short of it is that ntop works under Linux, Mac OS X and FreeBSD.

Luca distributes a Win32 version through http://shop.ntop.org but charges a convenience fee. Or you can fire up Microsoft's Visual C++ or Visual C++ .Net and compile the Win32 version that way.

It should work under MinGW - see the BUILD file in docs.

Anything else, you are in untested waters. Some cases we know there are problems, others we just haven't tested.

Q. Ok, it's a supported environment and it still won't compile.

A. Did you run ./configure? And did it complete successfully?

Usually 'compile' problems for supported platforms are a missing (critical) library or header file, but the user ignored (didn't see) the error/warning message and tried running make anyway.

./configure checks a lot of things. When it's looking for headers and libraries, ntop will report KEY information and PROBLEMS in large, set-off, lines:

     *******************************************************************
     *
     * NOTE: Building ntop for a supported platform
     *       This means we expect ntop to work without major issues
     *
     *            'i686-pc-linux-gnu'
     *
     *    Please keep the ntop-dev mailing list updated with any
     *    successes you have or problems you encounter...
     *
     *   Support for this platform was most recently verified for
     *
     *     RedHat7.2 w/ updates           ntop 2.1.51   on 2002-10-21
     *     Suse i686, 2.4.18-4GB-SMP      ntop 2.1.51   on 2002-10-24
     *
     *******************************************************************

READ THESE BOXES. Even if you don't read the rest of the output, read the boxes. ntop can work around a lot of problems (missing libraries) by disabling features that need them. If, for example, you don't have zlib, ntop will compile a version that doesn't output compressed html pages. If you don't read the boxes, you will never know.

READ THE MESSAGE BOXES!

     *******************************************************************
     *                                                                 *
     * NOTICE:  I know you're used to ignoring output from ./configure *
     *                                                                 *
     *          ntop has a lot of complexity and interdependences.     *
     *                                                                 *
     *          Please, please AT LEAST read the stuff in these        *
     *          boxes!                                                 *
     *                                                                 *
     *>>> The ACTION taken is shown prefixed '>>>'                     *
     *                                                                 *
     *    If the ACTION is unacceptable,                               *
     *??? The REMEDIATION plan is shown prefixed with '???'            *
     *                                                                 *
     *******************************************************************

The box will tell you what's wrong, what ntop did and often how to fix it if you don't like ntop's fix.

READ THE MESSAGE BOXES!

Hint: It may sometimes be that you're missing the header files (often those are in a -devel rpm if you're running RedHat)

If you see a message box and don't understand why ("I'm sure that header file is present"), then look at a file called config.log. Search for the specific header or library reported in the message box and you will see the detailed compiler/linker error messages.

For example, ./configure reports:

        checking for linux/if_pppox.h... no

The first thing to so is check if it's on your system:

     $ locate linux/if_pppox.h
     /usr/include/linux/if_pppox.h

(If you don't have locate, this works too:

     $ find / -type f -name "ethertype.h"
   )

Open up config.log and look for if_pppox.h:

     configure:13086: checking for linux/if_pppox.h
     configure:13103: gcc -c -g -O2 -Wshadow -Wpointer-arith
      -Wmissing-prototypes -Wmissing-declarations -Wnested-externs  -fPIC
      -DLINUX conftest.c >&5
     configure:13158: parse error before '/' token
     In file included from configure:13160:
     /usr/include/linux/if_pppox.h:38: `ETH_ALEN' undeclared here (not in a
         function)
     /usr/include/linux/if_pppox.h:39: `IFNAMSIZ' undeclared here (not in a
         function)
     /usr/include/linux/if_pppox.h:40: confused by earlier errors, bailing out
     configure:13106: $? = 1
     configure: failed program was:
     | #line 13091 "configure"
     | /* confdefs.h.  */
     |
     | #define PACKAGE_NAME "ntop"
     | #define PACKAGE_TARNAME "ntop"
     | #define PACKAGE_VERSION "2.2.91"
     | #define PACKAGE_STRING "ntop 2.2.91"
     | #define PACKAGE_BUGREPORT ""
     | #define PACKAGE "ntop"
     | #define VERSION "2.2.91"
     | #define STDC_HEADERS 1
     ...
     | #define HAVE_NETINET_UDP_H 1
     | /* end confdefs.h.  */
     | net/if.h netinet/if_ether.h
     |
     | #include 
     configure:13123: result: no

You see the actual test program, the actual compile line and the error messages.

Before reporting it to us, chase down where those missing items are declared:

     $ grep -R 'ETH_ALEN' /usr/include/* | grep '#define'
     /usr/include/linux/if_ether.h:#define ETH_ALEN  6
     /usr/include/net/ethernet.h:#define     ETHER_ADDR_LEN  ETH_ALEN

And post that information with your error report. The reason is that these field definitions are often placed in very different places in different OSes and even in different distributions.

FWIW, if you look in the older configure.in:

      AC_CHECK_HEADERS([linux/if_pppox.h], [], [], [net/if.h
         netinet/if_ether.h])

should have been

      AC_CHECK_HEADERS([linux/if_pppox.h], [], [], [#include 
      #include ])

because the macro doesn't do the #include automatically.

Q. Nope, it's not ./configure...

A. If it's not the configuration, then it's usually a problem with your specific system, either:

        - A new release of a supported OS.
        - An uncommon option/configuration of a supported OS.

In other words, something is different from what we've seen or expected.

Review the output from make. The error message will usually give you a Somewhat cryptic description of what's wrong and where. Look for messages about missing files. Post as much information as you can - do locates for the missing files, etc. The more you give us the less we will have to ask you to provide.

Remember, we can't see your box - all that the people on the list see is the information you give in your message.

Q. Compile dies because it's missing depcomp

A. automake/autoconf issue. The problem should have been fixed. If not, just Copy the missing file (or make a symbolic link) into the ntop source directory.

It's in /usr/share/automake on my Linux boxes. Another user reports it is in /usr/local/share/automake in sun8.

If you have automake installed, this will do it automatically:

       $ automake --add-missing --gnu -c

Q. Make fails with a message about being unable to create a .deps file.

A. Check the permissions on the (hidden) .deps (and .libs) directories - if root owns them your non-root userid may not be able to create files in there.

Q. Make fails with a message like this:

     /bin/ld: Warning: size of symbol `pcap_open_dead' changed from 100 to 67
         in pcap.o
     collect2: ld returned 1 exit status
     make[2]: *** [libntop.la] Error 1
     make[2]: Leaving directory `/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3'
     make[1]: *** [all-recursive] Error 1
     make[1]: Leaving directory `/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3'
     make: *** [all] Error 2

A. Ah yes, size of symbol...changed. It means you have a conflict between the version of the shared library (be it libpcap, libgd, whatever) that was used to compile ntop with the version that was used to link ntop.

Q. 'splain please...

A. If you break down a typical link line:

      gcc -shared  address.lo ... ntop_darwin.lo
          -L/usr/local/include
          -L/usr/local/lib
          -L/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd
          -lpthread -lresolv -lnsl -lz -lc -lssl
          -lcrypto -lgd -lpng -lpcap /usr/lib/libgdbm.so
          -lmyrrd
          -Wl,-soname -Wl,libntop-2.2.3.so
          -o .libs/libntop-2.2.3.so

You see the mix of -L and -l parameters. The -L parameters ADD additional places to look for the shared libraries, which are in addition to the 'standard locations' for the system. The -l parameters tell which libraries to include.

Read the man pages (man ld, man ld.so, etc.)

The 'standard locations' are very system dependent, but usually include /usr/lib and /lib. PLUS whatever is (under Linux) in the ld.so.conf file, for example

      $ cat /etc/ld.so.conf
      /usr/kerberos/lib
      /usr/X11R6/lib
      /usr/lib/qt-3.0.5/lib
      /usr/local/lib

So, on this system, to resolve a library, ld looks in the -L values:

      1. /usr/local/include
      2. /usr/local/lib
      3. /linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd

      And then the 'standard' places:

      4. /usr/kerberos/lib
      5. /usr/X11R6/lib
      6. /usr/lib/qt-3.0.5/lib
      7. /usr/local/lib
      8. /lib
      9. /usr/lib

Similarly, if you break apart the gcc COMPILE line and scrap the dups, you'll have a different set of places where gcc looks for the .h files. For those, it's the -I parameters plus whatever is 'standard' on your system, which is dependent on the specific gcc port:

/bin/sh ./libtool --mode=compile gcc -DHAVE_CONFIG_H -DLINUX

       -I. -I/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd 
       -I/usr/local/include
      -g -O2 -fPIC -g -O2 -c
      -Wshadow ... -Wnested-externs
       -o ntop_darwin.lo

Again, read the gcc man page.

A big problem is that - unlike .so files, it's not real clear what directories ARE searched for .h files, only that there is a set.

With me so far? Go back to the message... what it means is that:

      * From one set of locations, at compile time, the size of 
        the parameter list for pcap_open_dead was 100 bytes.
      * From the other set of locations, at link (ld) time, the
        library expects the parameter list to be of size 67.

Danger, Will Robinson...

The most likely cause of this problem is when you tell ntop to look at one location to resolve either the .h or .so and it finds the other, 'automatically' in a 'standard' location, but the two are actually from incompatible versions.

Say you have libpcap 0.6.2 installed by the os in /usr/lib and /usr/include, but you also install libpcap 0.7.2 in /usr/local/lib and /usr/local/include.

      $ locate pcap.h
      /usr/include/pcap.h
      /usr/local/include/pcap.h

and

      $ locate libpcap.so
      /usr/lib/libpcap.so.0
      /usr/lib/libpcap.so
      /usr/lib/libpcap.so.0.6.2
      /usr/lib/libpcap.so.0.6
      /usr/local/lib/libpcap.so.0
      /usr/local/lib/libpcap.so
      /usr/local/lib/libpcap.so.0.7.2
      /usr/local/lib/libpcap.so.0.7

If all you give ./configure is --with-pcap-header=/usr/local/lib

      * At compile time, it finds the pcap.h in /usr/local/lib (0.7.2)
      * At link time, it finds the libpcap.so in the 'default', 
        /usr/lib/libpcap.so which is actually libpcap.so.0.6.2

ntop did EXACTLY what you told it to do. The fact that it makes no sense is a problem, so you get the error message.

Q. So the real solution is?

A. Give ntop pointers to consistent sets of header and library files, and maybe don't have multiple versions of the same library installed at once.

Q. I'm done compiling and it works/doesn't work, what do I do?

A. make install (You'll typically need to be root for this to work)

Q. I'm seeing a lot of "changing search order for system directory" messages?

A. It's not a problem, just confusing. This warning, which appears if you redefine the system include directory, causes lots of confusion. Supposedly, according to this http://gcc.gnu.org/ml/gcc-bugs/2002-10/msg01080.html, it wasn't fixed in gcc 3.2.0, but should be fixed in gcc 3.2.1.

According to this: http://gcc.gnu.org/onlinedocs/cpp/Invocation.html, it's truly harmless on systems which already have /usr/local/include as a standard system library:

      -I dir 
      Add the directory dir to the list of directories to be searched for
      header files. See Search Path. Directories named by -I are searched
      before the standard system include directories. If the directory dir
      is a standard system include directory, the option is ignored to ensure
      that the default search order for system directories and the special
      treatment of system headers are not defeated (see System Headers) .

Other ./configure features

Q. How do I update the Vendor Table (MAC address prefixes)?

A. ntop has (in Makefile), a rule to automatically download the latest vendor information table from the IEEE, the oui.txt file ntop reads.

If you are seeing unknown MAC address prefixes (the 1st three units), try the full IEEE table. To rebuild it:

     # make dnvt

and then copy the new oui.txt over the one installed by ntop originally.

Also note that the table changes over time - there are almost 600 Modifications and/or new assignments between the version shipped with ntop 2.0 and the version on the IEEE site in February 2002.

Q. How do I update the passive ethernet fingerprint database?

A. ntop has (in Makefile), a rule to automatically download the latest ettercap file from SourceForge:

     # make dnetter

It will also compress the file and tell you how big the old and new files were.

Q. How do I update the IP to Country Code table?

A. Again, ntop has (in Makefile), a rule to automatically download the latest data and build the file.

     # make p2c

Note that this is actually a fairly complex script (utils/p2c) and is dependent upon data posted to supposedly well known locations by the various internet number authorities (APNIC, RIPE, etc.)

It's a LOT of data to download so don't do this on a site with a slow link to the internet. Also, read the output carefully and DO NOT apply a file if there are messages you don't understand.

Q. But the IP -> CC data is wrong.

A. Yeah. Most people don't care :-(... it's enough to see various pretty flags.

Sadly, the registries often post files with errors in them. Some Tier 1 ISPs post their own data to elaborate on the registry data. Most ISPs don't post data. Some entries in the files make 'logical' sense, but not physical sense.

If you find another set of data, let us know - the shell script is pretty easy to follow to add another data source.

The user who used to care about this (Mr. Anon E. Mouse) used to send me a file of corrections to apply to the file we posted with ntop.

If such a file exists, it would be posted @ SourceForge in the user contributed area.

Q. I have installed ntop for OSX but I cannot start it as I have installed some libraries using fink and ntop cannot find them.

A. The easiest thing to do is to do: ln -s /sw/lib/myfinklibrary /usr/local/lib Courtesy of Matthew Scholz

Running - Startup

Q. ntop won't start - I get this message:

  ** FATAL ERROR** ... open of /var/ntop/addressQueue.db failed:
       File open error

A. It's either permissions (the ntop -u userid doesn't have read/write access to that file - common if you're upgrading from 2.2 to 3.0).

The other cause is that there's still an instance of ntop running. Make sure you shutdown ntop before starting it.

Q. What is the function of the 'ntop' script in the build directory - should I call it or /usr/local/bin/ntop ?

A. (from the comments in the script):

    # ntop - temporary wrapper script for .libs/ntop
    # Generated by ltmain.sh - GNU libtool 1.4 (1.920 2001/04/24 23:26:18)
    #
    # The ntop program cannot be directly executed until all the libtool
    # libraries that it depends on are installed.
    #
    # This wrapper script should never be moved out of the build directory.
    # If it is, it will not operate correctly.

It allows you to run ntop out of the build directory before doing a "make install" by doing all the necessary linkage magic - such as forcing a relink if it didn't succeed originally - to the files in .libs.

Think of it as simulating make install, but not moving stuff to /usr/local or wherever.

Don't use it - it just causes problems...

Q. Which libraries do I need?

A. To run ntop: glibc, gdbm, libpcap

      For https://, add openssl.
      For other tools and compile options, add the appropriate libraries.

Q. ntop seems to run, but the web server isn't up.

A. Set the password - see docs/1STRUN.TXT

Q. How do you reset Admin password if we lost it?

A. Delete ntop_pw.db and follow the procedure in docs/1STRUN.txt

Q. I'm being prompted to set a password, what do I enter.

A. Anything you want, of 5 or more characters. If ntop can't find the value in it's database, it will prompt you to set on. If this happens every time you start ntop, check the permissions on the ntop_pw.db file.

Q. Characters after the 8th are being ignored in my password.

A. Probably. ntop uses the standard crypt() function provided by the OS.

While crypt() is required part of the single unix specification, the details are implementation dependent. Whatever the limitations of crypt() are, that what ntop's limits are.

Unfortunately, there's no way to tell programatically what these limits are, you have to refer to the man page for crypt(3).

There is a "GNU EXTENSION" to crypt() which implies that changing the 'salt' ntop uses would allow longer passwords. That's set via CONST_CRYPT_SALT in globals-defines.h, but AFAIK this hasn't been tested.

Q. How does the @filename option work e.g. /usr/bin/ntop ... @filename ...

A. The text of 'filename', is copied - ignoring line breaks and comments (anything following a #) - into the command line.

ntop behaves as if all of the text had simply been typed directly on the command line. Multiple @s are permitted in the command line, nesting is not. @s in the file will cause an error.

Both are displayed on the info.html report, the "Started as" shows the actual command line ntop was given and the "Resolved to" shows what ntop processed.

Started as /usr/bin/ntop -i eth0,eth1 @/root/ntop_parms -d -L

with /root/ntop_parms containing:

      -p /usr/share/ntop/protocol.list
      -P /usr/share/ntop
      -w 192.168.42.38:3000
      # -W 192.168.42.38:3001
      -u ntop
      --trace-level 3
      -m 12.239.0.0/16,10.113.0.0/16
      -E
      -K

becomes:

Resolved to /usr/bin/ntop -i eth0,eth1 -p /usr/share/ntop/protocol.list

               -P /usr/share/ntop -w 192.168.42.38 -u ntop --trace-level 3
               -m 12.239.0.0/16,10.113.0.0/16 -E -K -d -L

Remember, most ntop options are "sticky", that is they just set an internal flag. Invoking them multiple times doesn't change ntop's behavior. However, options that set a value, such as --trace-level, will use the LAST value given: --trace-level 2 --trace-level 3 will run as --trace-level 3.

It is recommended that you use FULL pathnames for @filename, since ntop may have different effective directories when run in different ways. However, you may wish to use relative pathnames to take advantage of the different effective directories (say cron vs. command line). Just know where you're starting.

Q. ntop seems to run but I don't see any traffic.

A. Make sure you aren't running against the loopback (127.0.0.1) interface. lo shouldn't see much traffic, only that originating on the host destined for it (e.g. ping 127.0.0.1).

Q. ntop is unable to open its database file. Specifically: I have following messages while running ntop

     wait please: ntop is coming up...
     24/Jul/2003 15:15:23 Initializing IP services...
     
     24/Jul/2003 15:15:23 Initializing GDBM...
     24/Jul/2003 15:15:23  ***FATAL_ERROR*** open of ...name... failed: can't be writer
     24/Jul/2003 15:15:23 Possible solution: please use '-P '

A. Multiple possible choices...

* Another ntop already running

(Most common) You forgot the -P parameter, and ntop is using a default value that doesn't exist. Just as the message says, use the -P parameter to point ntop at the directory you want it to use.

* Some sort of file system problem (non-existent directory, permissions, etc.)

1. The directory shown doesn't exist. Create it.

2. You many not have read/write rights in that directory. This can occur if you run ntop both as root and (as recommended) as a non-root user.

3. Another instance of ntop may already be running, so it has the file open and locked.

Q. What's inside the .db files and what's their format?

A. The files are stored in GDBM format. You can dump their content using tools like this: http://tboudet.free.fr/dumpgdbm/. In principle you should not be interested in the file content as they are temporary caches and contain the configuration as well as the MD5 hash of the web passwords.

I posted a trivial dumpgdbm.c to gmane - search for it.

Q. ntop starts up with this: WARNING: Discarded network 172.20.0.0/16: this is the local network.

A. No worries. The message means exactly what it says - it's a warning that you gave the local network as one of the parameter(s) to -m. Since the local networks are always local, ntop doesn't need to make them pseudo-local.

Q. What are ntop's options?

A. There are a couple of options that appear only if they're not compiled in, and a few that depend on various external libraries, e.g. openSSL.

The best way to see what is actually available is to run ntop with the -h or --help options and see. Also read man ntop.

Q. -4 and -6?

A. The default for the ntop web server is to connect to any address and any family of addresses, so if the NIC has both IPv4 and IPv6 addresses it should respond to both. Use these to restrict the web server.

Q. --trace-level?

A. The settings 0..4 control the amount of information logged or displayed. DETAIL (5) adds a [MSGIDnnnn] field and (VERYNOISY) 6 or greater adds the file:line where the message originated.

-t 5 can be useful with a log watch type package, -t 6 is mostly for debugging.

-t 5:

      [MSGID8757584] OSFP: scanFingerprintLoop() checked 1, resolved 1

-t 6:

      [MSGID8757584] [ntop:698] OSFP: scanFingerprintLoop() checked 1, resolved 1

With 3.2 we've added VERYNOISY and (7) BEYONDNOISY ... they do pretty much what they say. Have LOTS of log space available.

Q. ntop starts up find and then seems to die.

A. Are the ntop threads still there? Use ps to check, for example: # ps -C ntop -m -o "uid,pid,ppid,stat,time,command"

     UID   PID  PPID STAT     TIME COMMAND
     501 18327     1 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18330 18327 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18331 18330 S    00:00:05 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18332 18330 S    00:00:45 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18333 18330 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18334 18330 S    00:00:06 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18335 18330 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18336 18330 R    00:00:32 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18337 18330 S    00:00:41 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...

The 'PID' numbers match the THREADMGMT: numbers in the log:

      ntop[18326]:   INIT: Parent process is exiting (this is normal)
           ^^^^^ is the thread that issued the message

      ntop[18327]:   INIT: Bye bye: I'm becoming a daemon...
               (18327 is the 'master' thread created as ntop becomes a daemon)

      ntop[18327]:   THREADMGMT[t105061296]: SFP: Started thread for fingerprinting 
               A message is issued for each thread started.
                                t^^^^^^^^^ is the 'internal' thread #. Ignore it
      ntop[18331]: THREADMGMT[t105061296]: SFP: Fingerprint scan thread starting [18331]
                                                                                  p^^^^^ 
               is the child pid.  Each one issues a message as it starts up

So:

     UID   PID  PPID STAT     TIME is
     501 18327     1 S    00:00:00 Master
     501 18330 18327 S    00:00:00 (internal used by POSIX)
     501 18331 18330 S    00:00:05 Packet processor
     501 18332 18330 S    00:00:45 Idle Host Scan
     501 18333 18330 S    00:00:00 Address Resolution
     501 18334 18330 S    00:00:06 rrd
     501 18335 18330 S    00:00:00 Web Server
     501 18336 18330 R    00:00:32 eth1
     501 18337 18330 S    00:00:41 eth2

If you connect to the running ntop via gdb, you can see this:

$ gdb /usr/bin/ntop 18327 (gdb) info threads

         ^^^^^^^^^^^^ Shows all the threads and their current state

     9 Thread 114696 (LWP 18337)  0x40255c68 in recvfrom () from
            /lib/i686/libpthread.so.0
     8 Thread 98311 (LWP 18336)  0x40255c68 in recvfrom () from
            /lib/i686/libpthread.so.0
     7 Thread 81926 (LWP 18335)  0x420dcc31 in select () from
            /lib/i686/libc.so.6
     6 Thread 65541 (LWP 18334)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6
     5 Thread 49156 (LWP 18333)  0x40252a35 in __pthread_sigsuspend ()
            from /lib/i686/libpthread.so.0
     4 Thread 32771 (LWP 18332)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6
     3 Thread 16386 (LWP 18331)  0x40252a35 in __pthread_sigsuspend ()
            from /lib/i686/libpthread.so.0
     2 Thread 32769 (LWP 18330)  0x420db1a7 in poll () from
            /lib/i686/libc.so.6
     1 Thread 16384 (LWP 18327)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6

(gdb) thread 2

         ^^^^^^^^ switch to thread 2 and see it's state

[Switching to thread 2 (Thread 32769 (LWP 18330))]#0 0x420db1a7 in

            poll () from /lib/i686/libc.so.6

(gdb) info stack

         ^^^^^^^^^^ check the call stack

#0 0x420db1a7 in poll () from /lib/i686/libc.so.6 #1 0x4024f9de in __pthread_manager () from /lib/i686/libpthread.so.0

                     ^^^^^^^^^^^^^^^^^ running __pthread_manager

Q. All the threads seem to be running ok.

A. Check if there's one that looks like this:

     4 Thread 32771 (LWP 27295)  0x420cd207 in sched_yield () from
            /lib/i686/libc.so.6

Let ntop run a few more seconds (cont command), then check again.

If it's frozen in the sched_yield, you're probably tripping a deadlock situation that we've seen but don't understand - There's a lot of stuff on the 'net, including what seems to be the same problem with other resource intensive tools but no clear answers.

This happens most often on RedHat Linux 8.0 (9 uses a different threading library and the issue hasn't been reported there).

Regardless, if you see the deadlock or just have a hung ntop, there is a run time option to try: --disable-schedyield. This option disables the calls at a slight penalty to ntop's interactivness. If you're experiencing the deadlocks, try it.

And, especially if you're running something other than RedHat 8, please let the ntop-dev list know about it.

Q. Can I set the admin password from a script?

A. Yes, you can call ntop with the option:

      ntop --set-admin-password=password

Q. What's this warning about ownership?

A. The first time you run make install and ntop creates a new directory for ntop's files, there's a problem. So you'll see this warning box:

      ************************************************************
      ************************************************************

        WARNING: This install created a directory for the ntop
                 files and databases:

                   some directory...

                 This directory MUST be owned by the user
                 which you are going to use to run ntop.

                 The command you must issue is something like:

                 chown -R ntop.ntop 
           or    chown -R ntop:users

                 man chown to check the syntax for YOUR system

      ************************************************************
      ************************************************************

Since root created the directory, root is the owner. And since we don't know - yet - what user ntop is going to be run as, we can't issue the chown (CHange OWNership) command for you.

Q. And if I don't do this?

A. ntop won't run. Typically you get a message about a bad -P file or endless prompts for the admin password.

Q. I can't merge interfaces (-M option)?

A. Check your plugins and see if either netFlow or sFlow is active. Regardless of whether you're using them, if they're active, they (silently) force the -M switch on.

If you have used the web configuration to set this, you may be unintentionally overriding your command line.

With v3.2 there are log messages telling you what sets interface merge.

        NOTE: Interface merge disabled by default
        NOTE: Interface merge enabled by default
        NOTE: Interface merge disabled from prefs file
        NOTE: Interface merge enabled from prefs file
        NOTE: Interface merge disabled due to command line switch

Q. Explain -u.

A. -u root means that you are running ntop as root. It means you don't drop root's superuser privledges, so pretty much you can read/write any file. It's not a good idea. While we're not aware of any security problems with ntop, programs that run as root are targets.

-u ntop (or -u whatever) means you run as a normal user and can be assigned only the privledges necessary to run ntop. It also means you can only read/ write files as ntop. So less is exposed.

For even better security, the -u userid should not have a valid shell, so people can't use it to login.

Q. What are the options that reduce ntop's workload?

A. These options turn off certain processing (and thus limit ntop's functionality), and may be appropriate for high volume installations.

-b | --disable-decoders

     This flag disables protocol decoders. Use it for better performance
     or if you feel ntop has problems handling these protocols in your
     environment.

     This switch disables code in a number of places throughout ntop, code
     which analyzes specific protocols, but can place additional load on the
     host.  This switch could be used to run ntop on low-end CPUs or where
     ntop is acting as a collector (netFlow or sFlow) and the GUI is not
     required.

     Disabled is the analysis of:

        DNS Sniffing - where ntop captures DNS information from other hosts'
                       requests to reduce the # of DNS requests ntop must -
                       itself - make.

        NetBIOS   \
        NetWare    \
        AppleTalk   -- resource intensive protocol analysis of less
        bootp/dhcp /   common protocols.
        OSI       /

        http (80) - Request success/failure counting on port 80 and other
                    analysis, including "Virtual Host".

        ftp passive session tracking.

        "Wrong Port" monitoring for: http, ftp and smtp (used with the
              -q | --create-suspicious-packets option to dump "suspicious"
              packets to an analysis file)  With this option, ntop checks
              the payload for each new connection, looking for text usually
              present in http, ftp or smtp requests.  If these are not on the
              "normal" ports (http's 80, ntop's 3000 or squid's 3128, ftp's
              21 or smtp's 25) (or there is a non-ftp or smtp request on the
              standard ports), the packet is logged.

-g | --track-local-hosts

     Use this flag to tell ntop that you care only about local hosts (use
     -m | -- local-subnets to specify local nets).  This flag is useful when
     ntop sees many hosts (e.g. border gateway) but only the local ones need
     to be tracked.

     This switch disables code in a number of places throughout ntop, code
     which allows ntop to track "foreign" hosts (that is ones not local
     according to the IP address(es) of ntop's interfaces or set pseudo-local
     by -m | -- local-subnets).

     Basically, ntop doesn't bother to do DNS resolution on these addresses
     and, for purposes of various counts, uses the "other" bucket instead of
     creating a unique hash table bucket for the specific host.

     This switch could be used to run ntop on low-end CPUs or where ntop is
     acting as a collector (netFlow or sFlow) and the GUI is not required.

-o | --no-mac

     Specifies that ntop should not trust MAC addresses but just IP addresses.
     This option is useful whenever ntop is started on an interface where MAC
     (Media Access Controller - the low-level Ethernet address) addresses can
     not really be trusted (e.g. port/VLAN mirror in Switched Ethernet
     environments).

     Certain processing is performed differently:

          Hash search is via IP not MAC

     Certain capabilities are disabled:

          Analysis of bootp/dhcp requests
          localRoutersList.html report
          Wrong net mask log message and flag
          Analysis of non-tcp/udp protocols like NetWare and Spanning Tree
          Router listing on Host Detailed report.
          Traffic Matrix report

     (Note that this list is subject to change as we learn more about protocols
      that do/do not depend on the MAC address)

     See also -z | --disable-sessions

-z | --disable-sessions

     This flag disables tcp session tracking. Use it for better performance or
     when you don't really need the tracking of sessions.

     Also, in situations where the MAC addresses cannot be trusted, ntop may
     - or may not - be able to accurately track tcp sessions.  There is no easy
     way to tell, so this switch puts control back into the users' hands.

     In versions after 2.0 up to & including 2.1.2, the -j | 
     --border-sniffer-mode flag (predecessor of -o | --no-mac) always turned
     this off.  Many users wanted to try turning session tracking back on, and
     did via code patches with mixed results.

     Suggested usage:  If you enable -o | --no-mac, try running ntop with
     sessions enabled.  If the data looks reasonable, congratulations - your
     network allows session tracking.  If the data does not look reasonable,
     then you will also need to disable session tracking with this switch.

Q. -s | --no-promiscuous doesn't work

A. It should work - it's passed to pcap_open_live.

But, whether the flag is supported by the OS and whether it is actually respected by the interface depends on libpcap. If it fails, you'll see a message and ntop will refuse to startup.

Understand that if it actually does work, it means that ntop sees a lot less comprehensive view of the traffic.

But from the ntop web server, you won't see anything different.

You can check (*nix) by executing ifconfig on the interface.

Note that the parameter specifies if the interface is to be put into promiscuous mode because of ntop.

Even if this parameter is false, the interface could well be in promiscuous mode for some other reason. And in some situations, the interface may be forced by the low level driver into promiscuous mode, without reporting this to the kernel!

Q. How does -m | --local-subnets work?

A. This flag allows users to specify the subnets whose traffic is considered local (called "pseudoLocal" internally).

The format is

        /<# subnet mask bits>[,/<#  subnet 
         mask bits>

For instance "131.114.21.0/24,10.0.0.0/255.0.0.0".

Q. (followup) but what does it MEAN?

A. Surprisingly, it means EXACTLY what it says. Treat traffic on the listed subnet(s) as local.

-m relates to the traffic you see on the wire at the TCP/IP level. -m tells ntop something it can't determine by itself. And that is to treat a range of addresses EXACTLY like it was local.

For example, on my Cable Modem, I see broadcasts for a number of subnets that AT&T (now Comcast) has assigned to this area (I don't see the traffic just the broadcasts).

If you have VLANs or simply network overlays (two or more networks on the same wire, but with separate address spaces). etc.

Those are the cases where you use -m. To tell ntop to treat that traffic as local.

Q. Why

A  ntop differentiates between local traffic and remote traffic.  There are
   actually four classes (although only three are routinely reported) L->L L->R
   R->L and R->R.  Some additional processing is done and some additional 
   reporting is available for L traffic.

Q. I'm confused. Explain, please!

A. Suppose your IP is 1.2.3.4 with a 255.255.255.0 netmask (a/k/a 1.2.3.4/24)

Under the TCP/IP protocol, traffic with any address 1.2.3.1 -> 1.2.3.254 does not get routed. It's "local".

Your buddy is at 1.2.3.9 and the router is 1.2.3.1, so your network looks like this:

the +--------+ world-----+ Router +--1.2.3.1-------------------------------------- +dog +--------+ | 1.2.3.4 | 1.2.3.9

                                  +--------+            +--------+
                                  |  You   |            |  Buddy |
                                  +--------+            +--------+

Say you send a packet to your buddy at 1.2.3.9. You build a packet with SRC=1.2.3.4 DST=1.2.3.9 and your data and cast it out the wire. (For purposes of this illustration, ignore the fact the your TCP stack would recognize the "local" nature of the packet and actually use another, lower level protocol, called Ethernet to deliver it.)

The router (1.2.3.1) looks at it, does the math and ignores it - it's local Your buddy (1.2.3.9) looks at it, says - gee, that's me and reads it

This is L->L traffic.

Now you send a packet to ntop.org at 131.114.21.9. Again, SRC=1.2.3.4 and Now DST=131.114.21.9.

The router (1.2.3.1) looks at it, does the math and says - oops, I have to send it out to the world. Your buddy (1.2.3.9) looks at it, says - gee that's NOT me and ignores it

This is L->R traffic.

Now it's perfectly possible to have multiple (physical) networks on the same physical wire. Say that your ISP chooses to put 1.2.4.1-1.2.4.254 (1.2.4.0/24) on the same wire. (Why would they do this? - maybe it's a big pipe and only a few users or whatever).

A packet from 1.2.4.4 -> 1.2.4.9 is seen by

The router - no, that too is local, ignore it You (1.2.3.4): (1.2.4.9) - not me - ignore it Buddy (1.2.3.9) - um... 1.2.4.9 - not me - ignore it

And that's perfectly legal.

But what if you are the ISP and you want ntop to see ALL the traffic on that wire? ntop will figure out from it's own IP address that the 1.2.3.0/24 traffic is local, but it will classify the 1.2.4.0/24 as REMOTE.

And that is what the --local-subnets switch does. It tells ntop to treat that 1.2.4.0/24 traffic as local.

If there isn't any other traffic on the wire, then telling ntop to treat it as local won't change a thing.

You can always use a packet sniffer, such as tcpdump to scan the traffic on the wire and see what's really there...

Q. And internally?

A. ntop is designed as a hybrid packet analyzer, not a pure Ethernet analyzer (layer 2) nor a pure TCP/IP analyzer (layer 3). Most of ntop's displayed counts are at the TCP/IP level, and that's what confuses people. Internally, ntop works both at the level of the Ethernet frame and the TCP/IP packet.

A single MAC address can be associated with multiple TCP/IP addresses. The MAC address -- unless something is horribly wrong on the network or with the hardware or somebody is deliberately spoofing it -- is guaranteed to be unique and refers to a physical host or network interface. For many reports, ntop displays the information using the MAC address to separate physical devices. Other data is accumulated and displayed at the TCP/IP (level 3) layer.

-m relates to the traffic you see on the wire at the TCP/IP level. -m tells ntop something it can't determine by itself. And that is to treat that range of addresses EXACTLY like it was local.

For example, on my Cable Modem, I see broadcasts for a number of subnets that AT&T has assigned to this area (I don't see the traffic, but you get the picture) in an overlay structure (two or more networks on the same wire, but with separate address spaces).

Q. What about multicast traffic?

A. Multicast traffic uses the 224.0.0.0/4 address range (that is all addresses between 224.0.0.0 and 239.255.255.255 - see RFC 1112). By default then, all multicast traffic is treated as 'Remote' by ntop.

Your actual network topology may be such that multicast traffic could all be local, all be remote or a mixture. You can use the -m | --local-subnets parameter to force some (or all) multicast groups to be counted as 'Local' traffic.

Q. What about VLANs?

A. VLAN traffic is also a special case. ntop treats all traffic within your machine's netmask definition as 'Local'. For instance, you may have a /16 network (netmask of 255.255.0.0), and you are using VLANs to distribute the traffic. In this case ntop will see the a.b.c.d/16 address and classify all traffic within this /16 network as 'Local'.

Don't design your network that way.

Q. But they did!

A. This MIGHT work:

Use an un-numbered interface, so that ntop doesn't assign any implicit 'Local' addresses. Then use the -m | --local-subnets parameter to define what is truly 'Local'. As long as there aren't too many 'Local' networks, this should work. I run something sort of like this (but using /24s not subdividing a /16) for my development network.

Q. What are the default protocols ntop monitors?

A. (These are the ones ntop monitors if the user does not supply a -p parameter) Check addDefaultProtocols() in ntop.c around line 525. The current list (December 2004) is

     Protocol   Ports
     --------   -----

     FTP        ftp ftp-data
     HTTP       http www https 3128      /* 3128 is HTTP cache */
     DNS        name domain
     Telnet     telnet login
     NBios-IP   netbios-ns netbios-dgm netbios-ssn
     Mail       pop-2 pop-3 pop3 kpop smtp imap imap2
     DHCP/BOOTP 67-68
     SNMP       snmp snmp-trap
     NNTP       nntp
     NFS/AFS    mount pcnfs bwnfs nfsd nfsd-status 7000-7009
     X11        6000-6010
     SSH        22
     Gnutella   6346 6347 6348
     Morpheus   1214
     WinMX      6699 7730
     DirectConnect 
     eDonkey    4661-4665
     BitTorrent 6881-6999 6969
     Messenger  1863 5000 5001 5190-5193

Note that the names come from /etc/services (or your system's equivalent). If you add protocols to /etc/services, you can refer to them by name on the -p parameter.

REMEMBER: You must define the list using the format illustrated in the ntop man page. Don't try to read /etc/services. It will fail.

The list changes over time as P2P protocols appear and disappear. Check the cvs and diff ntop.c (around line 550 in void addDefaultProtocols() if you want the history.

Q. What about protocol XYZZY?

A. The analysis of protocols is very limited and unsophisticated. But, theoretically, if it's there in plain text, we could report on it. The more work you can do up front in identifying the protocol (e.g. port #s, header structure, etc.), the easier it would be to add.

Q. I am using a /16 (/25 or whatever) mask and I get this message:

      Truncated network size to 1024 hosts (real netmask 255.255.255.0)

A. Yes. ntop limits each network to 1024 hosts (a /24). If you need more, alter the #define for MAX_SUBNET_HOSTS in globals-defines.h and recompile. Space has to be reserved for this many hosts for each network, so the limit exists to keep memory usage from growing to absurd levels on people with "class A" (/8) interfaces (e.g. 10. or Cable Modems, etc.).

Q. What's with the version check?

Q. Why is ntop connecting to jake.ntop.org?

A. jake.ntop.org is the cannonical name for version.ntop.org, which hosts the xml file used for the version check.

See the man page, 1st run log messages and/or privacyNotice.html page (Click on the "this" in this line: "For information on ntop and information privacy, see this page." on the 1st page ntop presents).

Running - On-going

Q. Explain L and R - why doesn't ntop double count?

A. Classification is all based on what ntop SEES in the packets. ntop sees packets and ONLY packets. Packets have a FROM and a TO address. Which packets ntop sees is determined by the interfaces it is monitoring.

Traffic (packets) is classified based on the joint classification of the FROM address (L or R) and the TO address (L or R).

Only in L->L traffic will ntop see sent=rcvd and 'double count'.

                            Host: 192.168.1.x  www.yahoo.com
                                  L->R  R->L   L->R  R->L
                                  S  R  S  R   S  R  S  R
      192.168.1.x>www.yahoo.com
        HTTP GET ...             30  .  .  .   .  .  . 30

      www.yahoo.com>192.168.1.x
        HTTP 200                  .  .  .  8   8  .  .  .

      www.yahoo.com>192.168.1.x   .  .  .200 200  .  .  .
        ...

      etc.

Q. Why does ntop use so much memory ?

A. ntop holds a lot of information about each host it has seen in an in-memory table. Periodically, it looks at all the entries in the table and flushes any which have been idle for a period of time.

You can change the sizing of the table and the flushing interval via #define statements in globals-defines.h.

But realistically, ntop needs enough memory to hold information about what's active on YOUR network.

To reduce memory, monitor fewer protocols or use the filter (-B "bpf filter") option to monitor only parts of the network.

There have been a couple of discussions on the ntop mailing list about ntop's memory usage - you might read them (search on gmane).

Q. Database.

A. No way. This data just can't be stored in a database - while each record (row) is small, we're talking 1000s of updates/second. As in Enterprise Oracle class speeds...

A. ntop's memory usage for host tables depends on the # of hosts it sees in packet traffic. This is NOT, repeat NOT controllable by ntop in ANY way. If a user kicks off a port scan, 100s of hosts appear. If somebody does a DOS attack against you, 1000s of hosts 'appear'. If a user searches Kazza for an obscure song, it can probe 4K hosts. etc.

Lots of those hosts appear, have a few bytes of traffic and then disappear. Each host has a variable amount of memory - there's a base structure, some optional counter structures and a large # of pointer fields, which may or may not be valued for any given host. It depends on the # of active sessions and a lot of other things, but 8-20K is a good guess - I usually use 12K as an guestimating size. Similarly, sessions may appear and disappear (http: opens a lot, does a small retrieval and closes them), ssh may last for days. etc. Memory is consumed tracking them too.

It's not just one table entry per host - there are a lot of ancillary tables which only get allocated if you have data for them.

Q. Got a guess?

A. Sure. Look in textinfo.html and you will see the line:

     (very) Approximate memory per host.....4.4KB

I'm not sure how much I trust this and I wrote it. This calculation just computes the (current usage - baseline) / # of hosts.

Q. So ntop's memory usage is dependent upon?

Q. So what does the purge do?

A. ntop purges idle hosts. Period.

Idle being defined as having not had packet traffic for a #define able period, 5 minutes by default.

Because of the amount of linked data, these purges take time (lots of free() calls on all those char* values), so we don't purge in 'real time'. First we build a list of idle hosts, then we purge from that list. Any idle host is eligible for purge (unless you tell ntop not to purge idle hosts, which is usable only on small networks).

Over time, all idle hosts are purged. Only to be perturbed by the next burst of activity - say a port scan or everybody logging in back in after lunch. Eventually, there's some form of steady state, but it's HIGHLY dependent upon network activity. Which, remember, is external to ntop and can't be predicted.

Q. Why?

A. Think about the overhead of sorting this huge structure by 'last packet seen time'. 1GB of ram is something like 80K+ hosts and takes a long time to sort (let alone free). During which time, on that busy network, a couple of dozen packets are processed... Meaning maybe your list of idle hosts is wrong.

Q. So how about a hard memory limit?

A. There's no way to make a hard limit without purging ACTIVE hosts (or non-IDLE given ntop's definition of idle).

Think about it ... you're at that 1GB limit and you find a new host. Do you kick off an interim purge (with it's huge overhead?) And wait 2 or 3 s for an available slot?? While 1000s of other packets appear??

Q. Soft limit?

A. It might, might, be possible as a soft limit - but it's got a lot of issues.

First off, tracking memory usage of the hosts tables is itself a huge job. There are multiple places were stuff is purged, added to the structures as pointers, etc. and there's a queue of purged entries for reuse to cut down on the malloc() calls.

Secondly, the purge is resource intensive, and has been the cause of deadlocks before - you don't dare lock the structures for too long - packets keep arriving, and FAST on the busy network that has the memory issues in the first place. Since you can't lock for long, you can only purge a small # of entries.

Q. But what about -x and -X.

A. Ah, yes, grasshopper - you have been reading the man page. Good user. The -x and -X options are pretty crude. They say "if you have allocated n (hosts|sessions), fail the allocation of (host|session) n+1".

It works in the context of preventing memory exhaustion - if you can calculate the right values based on your available memory (and remember that 4K or 12K or whatever per host value is an AVERAGE).

And, no, you can't make them memory dependent - it's all ultimately virtual and ntop doesn't get indication of whether a malloc() call causes swapping. It either works because the allocation (total) is under your limit (or the hard limit of real memory + swap space - OS usage), or it fails.

This type of limit can't make sure you save only the RIGHT hosts.

Q. These options aren't very smart, are they?

A. No. They don't know anything about the hosts. Just the # of them. So failing an allocation might hurt in that the n+1th host might be an important one and yet host #2 is basically trivial junk (the printer that sends and "I'm here" message every three minutes).

It also causes overhead, because the next packet for host n+1 will attempt to allocate a record and fail. Again and again.

But these options may keep ntop from crashing and that's a good thing.

If you use them, I would recommend setting them very high - just at the limit of what your memory permits. Use this as a last resort - say you get hit by a worm or virus.

I recommend that you use other options such as filter and track-local-hosts for the day-to-day controls on what ntop tracks.

Q. Why does ntop drop packets?

A. We used to have an extensive discussion here. But with Luca's merge of ntop-ht into baseline, we shouldn't be dropping packets beyond whatever the OS is dropping.

Until we've learned enought about ntop 3.2 and following, the whole discussion has been moved to FAQarchive.

Q. ntop stops capturing packets, except ARP and other broadcasts. Why?

A. Check if you have a daemon running that periodically checks for and resets interfaces in promiscuous mode? If that happens, all you would see were broadcast packets like ARPs...

Check back in the log and see if there is a message about the interface changing status. Determine why.

Q. How much horsepower do I need to run ntop on a network of size x?

A. Nobody really knows. ntop needs enough memory to store the active hosts and enough cpu to keep up with the average packet flow. The buffer will handle the occasional peak, but if you see frequent lost packets, you're in trouble.

Note that a few packets occasionally lost isn't a big deal for most users. After all, the network itself has losses - I've seen my AT&T Broadband connection have spurts of 30% packet loss. Ideally in a LAN environment, the packet loss should be down in the small #s... the Ethernet standard allows 1 error in 100,000,000(10^8), but most vendors beat that by a long margin (even as high as 1 in 10^12).

Of course, those are lab measurements. In the real world? Not that good. Electrical noise can be a real bugaboo. Remember, at a certain point, if the NIC doesn't understand what it's seeing, it throws it away and declares an error. The key is to keep up with the traffic.

Similarly, the OS kernel does the same thing in it's interrupt handling (throw away packets). Last resort, but better than hanging up the whole machine.

ntop drops packets when the queue gets longer than the permitted length. You can see this in the configuration page as # Queued Pkts to Process and # Max Queued Pkts.

One or two or a small number (you pick your tolerance) is ok, but constant losses isn't. What I'm saying is that as long as ntop can keep up with the NIC, then the data is as good as it gets... if ntop can't keep up, then the data isn't very good.

If you have measurements - network size, traffic flow and %CPU used (with the hardware info, of course), shoot them over to us on ntop and someday maybe we'll be able to give better #s.

Q. What about my Frobozz Model xx Magic Network card? Is it good enough?

A. Probably. Well, a lot of the cheapos just don't have the buffering and cpu offload of a 3c905c or such. If the network isn't that busy, anything will do. For a busy network, buy a decent PCI NIC.

Q. Gigabit Ethernet?

A. No clue. Remember that you can suck a lot more traffic over that network than an old PC can handle (i.e. the bandwidth limitations of 32bit PCI and PC100/133 RAM, heck even PC2700 DDR).

Q. Do "zero copy" drivers help?

A. Yeah. Maybe. Once upon a time, I read about "zero copy" - look here http://people.freebsd.org/~ken/zero_copy/ for the FreeBSD stuff. Quoting:

      "What is "zero copy"?

      Zero copy is a misnomer, or an accurate description, depending on how
      you look at things.

      In the normal case, with network I/O, buffers are copied from the user
      process into the kernel on the send side, and from the kernel into 
      the user process on the receiving side.

      That is the copy that is being eliminated in this case. The DMA or 
      copy from the kernel into the NIC, or from the NIC into the kernel is 
      not the copy that is being eliminated. In fact you can't eliminate 
      that copy without taking packet processing out of the kernel altogether.
      (i.e. the kernel has to see the packet headers in order to determine 
      what to do with the payload)

      Memory copies from userland into the kernel are one of the 
      largest bottlenecks in network performance on a BSD system, so 
      eliminating them can greatly increase network throughput, and 
      decrease system load when CPU or memory bandwidth isn't the 
      limiting factor."

What's that mean? It means that BEST CASE, you have to copy the data NIC->Kernel and without zero copy, it happens TWICE. Then, once you're in the kernel, it has to hand the data off to libpcap (another copy) and from libpcap to ntop. So we're moving the data 3 or 4 times... best case!

Let's do some off the cuff math. Looking here

      http://www6.tomshardware.com/mainboard/20020501/ddr400vsrambus-05.html
   shows a table of memory types and max bandwidth (picking a few):

      Label  Name  Effective Clock  Data Bus Bandwidth
      PC100  SDRAM       100 MHz     64 Bit  0,8 GB/s
      PC133  SDRAM       133 MHz     64 Bit  1,06 GBb/s
      PC2700 DDR333      166 MHz     64 Bit  2,7 GB/s

See the problem? A fully loaded GigE link is 1 Gb/s. 4x that is 0.5 GB/s - so if ALL that's going on is the capture of packets from the network, the CPU can keep up. Maybe. Those bandwidth numbers are theoretical, best case (nice sequential access). Throw in some other system activity, cache misses and CAS/RAS issues... and um...

The moral is that if you're going to use ntop to monitor big fat links, you need screaming fast iron.

Q. Can I disable logging? Totally?

A. No.

Q. I'm seeing weird "hosts" on my network with names like "Bridge Sp. Tree/OSI Route". What are they?

A. There is a list of "special" MAC address prefixes in vendor.c, specialMacInfo[]. There are blocks of MAC addresses reserved (sometimes not formally) for special uses, such as sharing information about Spanning Tree for bridges. These do not have an IP address - they operate at a lower level - so nothing gets displayed in some of ntop's fields.

A reference about protocols at the wire level is here:

      http://www.oreillynet.com/pub/a/network/2001/03/02/net_2nd_lang.html

If you only want to see TCP/IP, then I suggest you use -B "ip" to filter only TCP/IP protocol on your ntop line...

Q. How do I see fully qualified names for all my hosts? Some are netbios names!

A. ntop doesn't SEND NetBIOS queries, it sniffs them off the traffic already on the network.

There is only ONE case where ntop uses the NetBIOS names, which is if it can't resolve them via DNS (both it's own queries and from sniffing responses to other's queries off the network).

So, if you have a properly functioning DNS, you'll see DNS names. If these are (for example) internal names, unknown to the DNS server, you'll see NetBIOS names if they are available. Lastly, you'll get IP addresses...

If you do have a DNS, and the name is resolved as part of the default domain, you won't see a fully qualified name back from the DNS, so ntop won't have that information.

So, on a real network you'll often get a mix of name resolution types:

    Host                            IP Address      MAC Address
                                                                   Other Name(s)
    netnews.attbi.com               63.240.76.16
    tigger.homeportal.2wire.net     192.168.0.xx   00:D0:09:xx:xx:xx
    homeportal.homeportal.2wire.net 192.168.0.1    00:D0:9E:xx:xx:xx
    swallowtail                     192.168.0.XX   00:A0:CC:xx:xx:xx SWTL [DMN]
    12-xxx-xxx-xxx.client.attbi.com 12.xxx.xxx.xxx 00:D0:9E:xx:xx:xx
    12-xxx-xxx-yyy.client.attbi.com 12.xxx.xxx.yyy

Q. What does this log message (and others like it) mean? **WARNING** releaseMutex() call with an UN-LOCKED mutex [hash.c:559] last

               unlock [pid 22176, pbuf.c:2598]

A. Those messages are part of an error check in our mutex handling routines.

If things are working properly, then these messages tell us things are really, really, really wrong.

releaseMutex() call with an UN-LOCKED mutex means that the corresponding accessMutex() call never occured.

releaseMutex() call with an UN-INITIALIZED mutex means that the access/release occured before the createMutex().

The file/line comments, e.g. [hash:559], tell you where the access/release was called from (e.g. hash.c @ 559) and where the last recorded release was.

There are others, look in leaks.c for the entire set.

Q. How serious are they?

A. None of these stop ntop from processing, but they're indications of unprotected accesses to shared data areas, which could lead to lost counts.

In the development cycle after 3.1 was released, I was able to track down and fix the remaining cause of spurious messages (but I've thought this was fixed before).

For most users, these warning messages shouldn't be ignored.

The root of the problem is that POSIX mutexes are stupid and don't record a lot of state information. To debug things like deadlock, it's really, really, nice to know where the mutex was locked from.

So ntop has added 'extra' information to the mutex data for recording this. This added data is not protected by the mutex and could get out of sync. So we lock a state change mutex whenever we are changing the state data.

BUT: You can't hold the state change mutex while waiting for the main mutex or you deadlock.

So the 'access' call becomes this pattern (see utils.c):

lock(statechangemutex)

     set state data

     tryLock(mutex)
     if(fails) {
       unlock(statechangemutex)
       lock(mutex)  <-- which may block, but does not hold the statechangemutex!
       lock(statechangemutex)
     }
     set state data
     unlock(statechangemutex)

Q. Threads. ugh...

A. There's a good set of articles on POSIX threads at

      http://www-106.ibm.com/developerworks/library/l-posix1/,
      http://www-106.ibm.com/developerworks/library/l-posix2/ and
      http://www-106.ibm.com/developerworks/library/l-posix3/.

Q. What does the message "URL security(1): ERROR: Found percent in URL...DANGER... rejecting request" mean?

A. It means that ntop received a request with a percent sign (%) in it, often used as part of Unicode exploits against various web servers. Since there is no situation where ntop should process this, we reject it. URLsecurity in http.c is the place where these tests occur.

Q. What does the message "Rejected request from address x.y.z.t (it previously sent ntop a bad request)" mean?

A. Once you send ntop a request that URLsecurity rejects, the sending address goes into a ring buffer on a 5 minute timeout where we simply drop subsequent requests... rather than waste cycles ignoring an attack...

Q. What are the other URL security(#) codes?

A. 1. Found a % in the request (Unicode problems) 2. Found a parameter type code (//, &&, ??) 3. Found a directory transversal code (..) 4. Found a prohibited (RFC1945) character 5. Found a bad extension

Q. ntop doesn't report any traffic at all.

A. Understand how ntop works: It simply listens on the interface(s) for packets, then counts and interprets them. If there aren't any packets, ntop doesn't count things.

ntop does not sample. It processes every packet it sees and counts them. Only if there is more traffic than ntop can handle for a long period of time will the packet queue hit it's limit and packets be lost. But this is still not sampling.

Make sure that there's traffic on the interface(s) you are using. You can use tcpdump or a similar network sniffer tool to check.

If you are on a segmented network (i.e. switched), you may not see traffic that isn't destined for the ntop machine unless you configure the switch to set the port for the ntop host into "mirror" or "management" mode (different vendors call it different things, but it's a mode where ALL traffic is copied to a specific port, regardless of which port the destination host is on).

If there is more than one interface in the ntop host, perhaps you aren't listening on the one that has traffic? Check using ifconfig:

eth0 Link encap:Ethernet HWaddr 00:D0:09:77:85:B9

             inet addr:192.168.0.34  Bcast:192.168.0.255  Mask:255.255.255.0
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:1105906 errors:0 dropped:0 overruns:0 frame:0
             TX packets:601935 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:100
             RX bytes:119869887 (114.3 Mb)  TX bytes:112203781 (107.0 Mb)
             Interrupt:11 Base address:0xc000

If the RX and TX numbers are increasing, this shows that traffic IS flowing...

If you have an unnumbered interface (listening only), remember you need to use -m to tell ntop what is local and what isn't:

eth1 Link encap:Ethernet HWaddr 00:30:F1:54:55:00

             UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
             RX packets:1596612 errors:0 dropped:0 overruns:0 frame:0
             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:100
             RX bytes:566953031 (540.6 Mb)  TX bytes:0 (0.0 b)

You can select an interface using the '-i' flag, e.g. -i eth1 or -i eth0,eth1.

Q. ntop doesn't understand xxxxx?

A. True. IP Packets have a source address & port and a destination address & port... you MUST get your head out of the application layers and revert to that simple concept.

How does Apache handle virtual hosts? It analyzes the flow at the application level (layer 4) not the wire/packet/protocol (layers 1, 2 and 3). It does this by re-assembling packets into a layer 4 message (e.g. GET http://virtual.host.name.com/page.html)...

Now there are some layer 4 analysis routines - virtual hosts was added in 2.2 (and the folks who have virtual hosts have been pretty pleased), ftp, http, and some others - mostly looking for traffic on non-standard ports, etc.

So, since ntop works at the packet level, it doesn't understand virtual hosts. Unless it's SPECIFICALLY coded for. ntop is a NETWORK analyzer, not an application level one.

Q. tcpwrappers doesn't work

A. Oh yes it does... for http: connections

1) You have to configure it this way before compiling ntop:

        ./configure --enable-tcpwrap

2) You must have the headers and libraries installed on the build machine

      (and on the execution machine if they aren't the same).

Remember to make the appropriate entries in hosts.allow (e.g. ntop:192.168.0.) and hosts.deny (e.g. ntop:ALL)

However, tcpwrappers and https:// is known not to work - see docs/KNOWN_BUGS

Q. My filter doesn't work! I'm running ntop like this:

      /usr/local/bin/ntop -u nobody -L -d -E -w 3000 \
                      -m 192.168.10.0/24,xxx.xxx.xxx.xxx/32 \
                      -M -i eth0,eth1 \
                      (src net 192.168.10.0/24 or src host xxx.xxx.xxx.xxx ) \
                       and not dst net 192.168.10.0/24

A. Yup, it doesn't work. Use the -B option and put the filter in quotes:

      -B "(src net 192.168.10.0/24 or src host xxx.xxx.xxx.xxx ) and 
           not dst net 192.168.10.0/24"

ntop used to assume anything it didn't recognize was a filter. But not since 2.1.3. If you try this now, you should see a log warning that says maybe you forgot the quotes

Q. I have experienced problems defining multiple filters: ntop reports 'syntax error'

A. If you believe the filter is syntactically correct then it's likely that the Libpcap you have used has been compiled using an old non-reentrant version of flex. Please make sure you're using version 2.5.4 or above.

Q. Can you give some additional examples of filters?

A. man tcpdump -- see "expression"

A couple of simple examples, courtesy of B. Loic:

       -B "host not 192.168.1.100 and not 192.168.1.101"

to exclude hosts 192.168.1.100 and 192.168.1.101 from tracking (FQDN such as www.yahoo.com will work too).

If you need to exclude a full IP range, you will want to use something like

        -B "net not 192.168.1.0/24"

Q. What about the backtrace?

A. Sadly, probably useless. Why? Too much information isn't available:

  Here's one (and a pretty good, obvious one at that):

      ntop[23720]:   **FATAL_ERROR** RRD: caught signal 11 SIGSEGV
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE: backtrace is:
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  1. /usr/bin/ntop [0x42028c48]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  2. /usr/bin/ntop(getopt_long+0x43) [0x420c4e83]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  3. /usr/lib/ntop/plugins/rrdPlugin.so(rrd_update+0x61) [0x44157f7d]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  4. /usr/lib/ntop/plugins/rrdPlugin.so [0x441481a2]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  5. /usr/lib/ntop/plugins/rrdPlugin.so(updateCounter+0x19) [0x44148a15]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  6. /usr/lib/ntop/plugins/rrdPlugin.so [0x4414ba49]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  7. /lib/i686/libpthread.so.0 [0x41138941]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  8. /usr/bin/ntop(__clone+0x3a) [0x420da1ca]

    See: It doesn't give the routines, just the hex offsets from some points in the executable.
    It seems to say that rrdPlugin's updateCounter called rrd_update which called getopt_long. But
    then getopt_long called something else and that's what died...  plus we don't see the parameters.

    Sometimes it works great for me on my box, because I can use the tools to figure out precisely
    where it means.  But often I use gdb too.  Move off my box and the odds of your compile and my 
    compile (and our ./configure and all the tool versions) matching so that the offsets mean something
    are pretty small.

Q. What do these log messages mean?

     **ERROR** mVLAN: Host (identical IP/MAC) found on multiple VLANs
     mVLAN: ntop continues but will consolidate and thus probably overcount this traffic
     mVLAN: Up to 10 examples will be printed
     mVLAN: Host 192.168.aaa.bbb (00:12:17:xx:yy:zz) VLANs 3 and 4

A. Once again, they mean what they say. ntop found something pretty weird. It found packets for the same 'host' (that is the same IP and MAC address) on two different VLANs. In this case, IP 192.168.aaa.bbb, MAC 00:12:17:xx:yy:zz on VLAN 3 and VLAN 4.

It's not WRONG, but it's definitely odd. If we see a packet twice, we count it twice, so - via this message - we're warning you that we might be double counting some traffic. We warn you ONCE per host (per ntop run) and for only up to 10 hosts, so as not to innundate you with meaningless warnings.

On the about page is a count (if you've even seen this junk), in the "Host/Session counts - global" section. If it's just a couple, ignore it.

If you see a lot of this stuff, you may want to rethink your ntop sensor placement or at least understand what you are doing. On my home LAN, I use a 802.1q VLAN trunk to move traffic between two switches and have a passive tap on the trunk. Traffic to/from the firewall moves over this trunk, so I see the same packet on the YELLOW(DMZ) VLAN and the GREEN(TRUSTED) VLAN all the time.

Why even bother trapping and warning? Well, there is one bad thing from ntop's perspective with duplicated hosts, which would occur if we didn't trap this, and that is that you would also see a lot of RRD errors when ntop tries to update the same host twice:

**WARNING** RRD: rrd_update(/usr/.../ipBytesSent.rrd) error: illegal attempt to update using time

                    1109523297 when last update time is 1109523297 (minimum one second step)

Running - Web Server

Q. Why does ntop display bits of my web site, instead of its own pages?

A. ntop is designed to search the current working directory for data files, such as the html subdirectory, before it searches the default directories.

This is a feature.

You are one of those rare souls who happen to have had an unrelated subdirectory 'html' with a file named 'index.html' as a subdirectory of the current working directory at the time that you launched ntop. cd to an acceptable directory, such as /usr/share/ntop, before launching ntop.

Q. What are High/Medium/Low risk flags

A. They are set in reportUtils.c based on fairly self-obvious functions Well documented in the help.html page, reachable by clicking on the problem descriptions or via http://127.0.0.1:3000/help.html

Often seen if you are monitoring a backbone or common network (high) or if you have cloned MAC addresses for, say, a home Firewall box.

Q. What does the "Users" flag mean on a host?

A. If you go to the "Info about host xxxx" page, there will be data in the "Known Users" section, if it's acting as a server for certain protocols.

In sessions.c, the function updateHostUsers() is used to maintain the list of "users" of a host. In handleSession(), as part of the protocol level analysis, the "user" information for various protocols is pulled out of the packets. Stuff like the "X-Kazaa-Username" header, the "MAIL FROM:" header, etc.

We tag users as one or more of the following types:

        P2P_USER, SMTP_USER, FTP_USER, POP_USER, IMAP_USER

Note that for P2P, we also record - where possible - whether this user is in P2P_UPLOAD_MODE and/or P2P_DOWNLOAD_MODE.

Q. Why are some of the host names in different colors?

A. Colors are used on several of the ntop pages to convey extra information to the user. (in particular the ACTIVE TCP SESSIONS and the LOCAL HOST STATS pages). There are five colors used to depict how long ago the host was first seen by ntop.

The pages which display these colors use a html style sheet called style.css located in the normal html subdirectory (where ntop is installed). This happens by setting the 'class=' parameter of the html 'A' (Anchor or hyper-link) tag. The style sheet defines the following:

      Age of host  'class' name  Color code         Color description
      (minutes)
      -------------------------------------------------------------------
      0-5          A.age0min    { color:#FF0000 }   Red
      5-15         A.age5min    { color:#FF00FF }   Fuchsia/Magenta
      15-30        A.age15min   { color:#FF7F00 }   Coral (lt orange)
      30-60        A.age30min   { color:#007FFF }   Slate blue
      60+          A.age60min   { color:#0000FF }   Blue

The color legend is displayed on the About | Configuration page (info.html).

Q. What does the P2P flag mean on a host?

A. If ntop knows enough to tag you as a P2P user, it's also looking at the other headers to see if it can track what files you're exchanging. If a host (i.e. a workstation) downloads a file from another host ("server"), the file name is recorded in the list ntop maintains for both of them.

If a host has at least one file name recorded, it's tagged with the "P2P" flag.

Q. What does a "Virtual Host" mean.

A. If a single instance of a web server handles many web sites, all of the references resolve to the same name. The web server uses the "Host:" header to determine which "index.html" page to serve up.

ntop monitors port 80 (http:) exchanges and looks for the Host: which allows it to build a list of virtual hosts being handled by the web server.

Running - Web Server (https:)

Q. SSL is not working! I have the following error in the log/terminal:

     10/Jun/2002 22:58:17 Started thread (6151) for network packet sniffing on
          eth0.1700:error:140EC0AF:SSL routines:SSL2_READ_INTERNAL:non sslv2
          initial packet:s2_pkt.c:187:

A. You forgot to put https:// instead o