From Helpful

Sort of a set:

• Linux notes
• Linux admin notes
• Specific or lowish level linux notes,

Loop devices

Traffic shaping notes

Introduction

Linux traffic shaping throttles network data. In home use, its main use is probably keeping your latency low (general networking snappy) by avoiding saturating your upload bandwidth.

The model that linux traffic shaping uses is a tree of individual throttlers (one tree per interface), which allows you to not only limit speed but also give preferential treatment based on some criteria.

All traffic enters this tree at the root, and is routed according to rules (think iptables and tcpdump type rules) until it ends up at the leaf of the tree, which sends it out onto the network.

Each of tree elements has:

• a capacity, which acts as a hard limit for the element
• a user-set bandwidth that this element should target,
• a particular resolution of control - long-term allows bursting but will make longer transfers oscillate between fast and slow, while short-term is a harder, more immediate limit
• options that control e.g. whether elements may borrow any unallocated leftovers from sibling nodes

Consider, for example, a case in which you have a LAN in which you want no throttling, but which has a gateway on it; everything going to that needs to be limited, and probably to different degrees. Let's say you are hosting an (DNATted) internet server on this LAN. You could, for example, think up the following:

                   capacity    target speed
+ root             
|--+ local:         100Mbit    no limiting
|  `--- nfs          80Mbit    no bursting  (NFS is known for congesting networks, so avoid that)
`--+ inet :           1Mbit    
   |--- voip:         1Mbit    no limiting
   |--- http:       320Kbit    30KByte/s 
   |--- ftp:        320Kbit    20KByte/s
   `--- fileshare:  120Kbit    15KByte/s, no bursting

Note the mix of Kbit(/s) and KByte(/s). It's unnecessary of me and entirely confusing to mix them. It is intentional - I'm pointing out you should be aware of this to avoid making your own mistakes.

Note also that the limited inet serves have a target speed lower than the capacity.

Clarifying some terms

Traffic policing and traffic shaping refer to limiting incoming and outgoing data, though 'shaping' can and sometimes is used to describe both.

Less ambiguous terms are egress (outgoing) and ingress (incoming).

Ingress control is less useful, as all you can do is drop packets. If the sender doesn't slow down, that just means data loss, or eventual retransmission (e.g. in TCP) which would slow the delivery rate, but mean unnecessary overhead, and increased latency.

A qdisc is a queueing discipline, a name for a particular type of scheduler algorithm that handles a given queue of packets in a particular way. The choice of algorithm is important if you want some specific behaviour.

Classless qdiscs can be described as those whose behaviour is hardcoded and not configurable beyond basic rate figures. They may well have multiple flows and may be somewhat smart, but since you cannot add filters, classes, or attach child qdiscs, they are leaves in the tree and tend to serve a fairly simple final purpose.

Classful qdiscs are used to create the major logic in the tree. Filters appear only inside classful qdiscs and handle the classification of packets into one of the qdisc's classes.

Classes are the configurable inner workings that handle different flows of data. Each such class may have some properties according to the qdisc it is in (e.g., individually throttlable flows), and also contains a child qdisc.

Each class may be isolated, meaning it will not borrow from or give away bandwidth to sibling classes, or bounded, meaning it will not borrow.

bursting means you may exceed the limit on the short term. Usually indicates that a qdisc observes the target rate in the long rather than the short term.

Common qdisc types

Qdiscs serve multiple purposes. Most are structured as classifier → fifos → scheduler, some may just limit, some equalize for fairness, and other such purposes.

Classless qdiscs

pfifo_fast

Used for TOS purposes: the (sixteen) TOS values are mapped to three bands (FIFOs), namely interactive (0), bulk (2) and inbetween (1). When there are packets waiting on more than one bands, packets from lower-numbered bands are always emitted first.

I believe SSH sets TOS, meaning it gets this preferential treatment. Many programs don't set this, though. You can change ToS with firewall rules, though.

When you have traffic support compiled in but have not looked at configuring it, each interface has a single pfifo_fast as the root (and only) qdisc. (so TOS is observed)

bfifo, pfifo

Byte based and packet based fifos, respectively. Usually the default qdiscs in classful qdiscs(verify).

Mentioning 'fifo' in a configuration seems to imply bfifo(verify).

sfq, esfq

'Stochastic Fair Queuing' is not a limiter, but a classless qdisc that attempts to equalize individual flows (mostly TCP connections and UDP streams), meaning flows will not as easily throttle each other when the qdisc is saturated. It is useful as a leaf when you anticipate many concurrent connections. Note that this process balances flows, not programs.

Flows are directed into the qdisc's bands, and the qdisc emits packets from these bands in a round-robin way. The way it maps flows to bands is a based on a process which is simple and cheap but sometimes doesn't balance very well. The 'perturb' option lets you specify how often it should change the details of the hash, which makes it more likely that distribution works well over the long run.

ESFQ is a variation that allows more control over the band decision process, and some other details.

tbf

Token Bucket Filter is useful when you need a simple hard bandwidth cutoff, and allow no bursting either.

The mentioned tokens are virtual 'you may send out a packet now' tokens that are replenished at a configured rate (with a maximum amount).

Apparently uses one token (i.e. sends one packet) per jiffy, where that hiffy depends on the kernel configuration, often 1/100, 1/1000 or 1/125 of a second.

Note that that kernel rate both controls the actual meaning of the outgoing rate, and implies a maximum speed (independent of and possibly much lower than the interface speed).

gred

(verify) Apparently not very interesting in home networks

Classful

prio

Same idea as pfifo_fast, but you can configure the bands and TOS mapping yourself.

cbq

'Class Based Queuing' is a featured averaging shaper that keeps to the limit set over a longer period of time.

Each class may be isolated, meaning it will not borrow from or give away bandwidth to sibling classes, or bounded, meaning it will not borrow.

HTB may be more interesting in various setups.

htb

Hierarchical Token Bucket is one of the more controllable methods, resembling tbf but in a simple hierarchy in which the parent can give tokens to the children.

Roughly:

• rate roughly means the targeted rate. Under differently taxed conditions and depending on your design, this can act as a cap and a rough guarantee.

• ceil caps the rate under borrowing conditions (value seems to default to parent ceil)
• burst caps the amount of bytes bursted at 'ceil' speed (per jiffy - note the tbf-like limit this imposes; you may have to raise this under some conditions)
• cburst caps the amount of bytes bursted as fast as the interface can handle it. Setting this high lowers equalizing properties.

hfsc

(verify)

dsmark

(verify)

Setup

monitoring

To watch traffic in detail, look at bmon.

A quick and dirty solution:

#to watch the status of the classes:
watch -n 1 -d tc -s class show dev eth0

#and to see the qdisc structure:
watch -n 1 -d tc -s qdisc show dev eth0

Notes:

• the dev eth0 shows one device instead of all.
• ls is equivalent to and less typing than show
• a tc -s is also a way to see if you've got tc support. You'll likely see a pfifo_fast (per device) when you have it.

tc

You can write rules for the tc command, which comes from iproute2 and is little more than a hook to kernel calls.

tcng

There is now also tcng (traffic control next generation), which translates a more readable and maintainable 'tcng' language to tc commands. See the configuration example below, and run it through tcc. Aren't you happy you didn't have to write what comes out of that yourself?

I wrote a simple bash script to be able to quickly replace the current configuration: this clears the current one, parses the tcc-style config file, and runs the resultant tc commands. (it does not, however, check whether the new configuration parsed properly before clearing the current one)

#!/bin/bash
### The first argument should be a tcc-style configuration file
if [ ! -r "$1" ]; then
  echo "You did not specify a (readable) configuration file"
  exit
fi;
 
### The second, optional argument is the device. It defaults to eth0
# To use this, use DEVICE in your config instead of a particular device name
DEVICE=${2:-eth0}
 
#clear current configuration, if any
tc qdisc del dev $DEVICE root 2>/dev/null
# bash' -x option shows each tc command as it's run.  
#  Remove it if that is too verbose for you
tcc $1 | egrep -v '(^#|^[\s]*$)' | sed "s/DEVICE/$DEVICE/" | \
  xargs -n 1 --replace=@ bash -x -c '@'

units

Example

Leaves the local net unthrottled, limit incoming and outgoing SSH only somewhat (because of SCP and SFTP) (and assuming your broadband upstream is a little more, e.g. 100KByte/s), does web serving at a decent but nondisruptive rate, and throttles everything else down pretty hard.

Note that this setup will affect every service you don't list here, so in practice you may want to write it the other way around: specifically slow down anything you don't want to be disruptive, and having the catch-all rule be barely throttled or unthrottled.

dev "eth0" {
    egress {
        class ( <$local>  )  if ip_dst:16 == 192.168.0.0;
        class ( <$fast>   )  if tcp_sport == 22 || tcp_dport == 22;
        class ( <$med>    )  if tcp_sport == 80;
        class ( <$other>  )  if 1;
        htb ( class ( rate 100Mbps, ceil 100Mbps ) {
            $local  = class                              {sfq;}
            $fast   = class ( rate 80kBps, ceil 80kBps ) {sfq;}
            $med    = class ( rate 40kBps, ceil 40kBps ) {sfq;}
            $other  = class ( rate 15kBps, ceil 20kBps ) {sfq;}
        }
    }
}

Options (like rate, ceil) inherit (though in this case they are eplicitly overridden in all but the $local class).

Warning: Case matters(verify) - those are megabits (b) per second and kilobytes (B) per second. Note that since 'bit' also exists, you probably want to avoid abbreviations like '8Mb', since it is more confusable than the equivalent '8Mbit.'

Note that this is NOT the same as the units used in tc(verify)

See also [1]

Semi-sorted

MARK

You can use netfilter MARK values for shaping, but only indirectly, in that you can't use them in shaping rule tests.

Instead, you have to see what qdisc handle you something sent to, and use that number as the mark in your firewall. I am not yet sure how handle and flowid relate to this, and how to decide the mark on column'd handles.

Yes, this means that if you use this, you have to rewrite the mangle table every time you structurally change your traffic shaping setup.

Since tcng allows U32 type checking, you can usually avoid it. Cases in which you can't includes traffic that is NATted by the same machine, since it arrives at shaping after it has been translated.

To check whether the mangle rule seems to be working, or at least getting data, try:

watch -n 1 -d iptables -t mangle -vL

See also

• http://tldp.org/HOWTO/Traffic-Control-HOWTO/

• http://www.opalsoft.net/qos/DS.htm

• http://www.mastershaper.org/index.php/Main_Page

Bridging notes

The effects of using linux as a bridge (rather than a hardware brige):

• you can use firewalling, monitoring, traffic shaping ((verify)),etc.
• the bridge can participate in the network (however...)
• the bridge can eat CPU on busy networks, because it handles pretty much all traffic it sees (NICs have to be in promiscuous mode)

Errors

Add bridge failed: Package not installed

You don't have kernel support for bridging.

Compile it in. It should be in 'Networking' → 'Networking options'.

See also

• http://www.linuxfoundation.org/en/Net:Bridge

Multiple X screens, xrandr

Multiple monitors - different methods

Clone

The same on all outputs. Generally useless, other than in specific situations (e.g. monitor+projector combinations).

Separate X instances

Separate in terms of input, rendering surfaces, and whatnot.

Now regularly used for multi-user login.

Multiple X screens

X itself supports running multiple separate displays in the same X instance. Separate in terms of rendering targets. For example, windows cannot appear half on one, half on another.

Window managers usually treat these as completely separate too. You can move the mouse back and forth, so if you don't mind the separation, this may serve your needs.

Not so commonly used.

(Merged rendering)

There are a few different implementations of the idea of gluing different configured physical screens into one logical X screen. The fact that it will end up rendered on multiple physical things may be transparent to most of X.

Implementations:

• Xinerama is the open version
• MergedFB is technically better but not as supported
• TwinView is nVidia's implementation
• Big Desktop is ATI's name

Xinerama

An extension of X, and transparent to drivers, so this option may work the most setups. It is in some ways a simple-and-stupid hack, however (it e.g. duplicates resources to get around the fact that X Core has resources per screen).

Usual implications:

• hardware 3D and/or video overlays may work only on one screen (at all, or at a time)

• You cannot rotate parts of a xinerama setup - XRanrR is disabled when xinerama is enabled. May be supported in future versions. (written when?(verify))

• You cannot change the configuration on the fly, or even easily base it on detection, which matters in some situations, such as when you use an external screen on a laptop.

MergedFB

Similar idea to Xinerama, better implementation: supports 3D on all heads, supports XRandR(verify).

However, the driver has to specifically support it. A number of open drivers have relatively experimental support, some usable. Commercial nVidia and ATI drivers do not ((verify)).

TwinView

Needs binary nVidia driver.

Supports XRandR (verify)

Big Desktop

Needs binary ATI driver(verify).

Supports XRandR (verify)

fglrxconfig

Other

XRandR, a.k.a RandR

The X Resize and Rotate Extension (a.k.a. X RandR, XRandR, RandR) allows resolution changes on-the-fly, and also rotation assuming your setup supports it.

Check

To double-check whether X is loading it, check the X log:

grep -i randr /var/log/X*.log

You'ld see something like:

(==) RandR enabled
(II) Initializing built-in extension RANDR

or even

(II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message.
(--) RandR disabled
(II) Initializing built-in extension RANDR

xrandr

xrandr is the command line interface to the RandR extension.

capability query

Without any options it will report what it can do. To query separate displays (e.g. on a dual-monitor setup):

xrandr -q

...and for specific displays, which is useful on multi-head monitors set up with different dispalys:

xrandr -q -d :0.0
xrandr -q -d :0.1

resolution change

You can change the resolution on-the-fly by specifying one of the numbers (if you do this automatically from some script, you could specify a specific resolution if you want that script to be portable between setups)

xrandr -q 1 

rotation

When you do a query, you will see the relevant options below the list of resolutions, something like:

Rotations possible - normal
Reflections possible - none

This particular output mens that reflection is not supported, and that there is only one rotation, 'normal' - in other words, that rotation is not supported. There seem to be various reasons, including:

• the card does not support it
• the driver does not support it
• the current X config does not support it:
  • the combination of X extensions does not allow it (verify) (Which? XV? Xcomposite?)

TODO: Look at the effects of TwinView and xinerama. Do both imply you can't?

gnome-randr-applet

nvidia-settings

Unsorted

MergeFB Clone

http://www.ogmaciel.com/?p=272

Wacom tablet notes

Your distro probably has a package that has the drivers. If not, you want http://linuxwacom.sourceforge.net/

X configuration

You still have to configure X yourself. See:

• http://linuxwacom.sourceforge.net/index.php/howto/x11
• the man page (man wacom)

Note that aside from the global config, you can also use wacomcpl or xsetwacom to set behaviour details on general startup, and more interestingly, each user's login.

Sections

cursor: Mouse

To use it as a mouse, use the following:

Add to ServerLayout (regardless of input type):

InputDevice    "cursor1"    "SendCoreEvents"

Now, for USB devices add:

Section "InputDevice"
  Driver        "wacom"
  Identifier    "cursor1"
  Option        "Type"       "cursor"
  Option        "USB"        "on"                  
  Option        "Device"     "/dev/input/event0"  
                   # possibly /dev/input/wacom
EndSection

For the older serial devices, add:

Section "InputDevice"
   Driver        "wacom"
   Identifier    "cursor1"
   Option        "Type"       "cursor"
   Option        "Device"     "/dev/ttyS0"          
                              # (...you probably only have one serial port) 
 EndSection

You also need to tell X about using this mouse in the ServerLayout section. Usually you will have

• a regular mouse as a "CorePointer"
• a (few) wacom section(s) configured to "SendCoreEvents" so that it will also act like a mouse.

Even if you don't have another mouse configured, don't set the wacom as a CorePointer - you'll use pressure sensitivity and all other wacom-specific features (e.g. xidump cursor1, xsetwacom set cursor1 ...). You'll instead get the error:

X Error: 170 BadDevice, invalid or uninitialized input device

To use as more than a mouse, add more sections:

• with identical device settings
• with another
  Type
  (stylus, eraser, and possibly pad),
• further behaviour options (most may differ per Type), and
• mention it as a SendCoreEvents type InputDevice

Note that programs must know about and claim devices to react to pressure and such. Gimp is one of the relatively few that does. xidump is another, which is a test app for the X configuration.

stylus: pen

Option "Type" "stylus"
InputDevice    "stylus"    "SendCoreEvents"

Eraser: back-side eraser

Option "Type" "eraser"
InputDevice    "eraser"    "SendCoreEvents"

pad: Expresskeys and strip

To get events for the keys and strip on the Intuos3, Cintiq 21UX, Graphire4, and Bamboo, add:

Option  "Type" "pad"
InputDevice    "pad"

No SendCoreEvents, even for the strip, apparently(verify).

Behaviour

Rotate

Option "Rotate" "NONE"       # default
Option "Rotate" "CW"         # 90 degrees right
Option "Rotate" "CCW"        # 90 degress left
Option "Rotate" "HALF"       # 180 degrees

To keep movement in X and Y direction porportional in Absolute, you need to set the active area to the screen shape:

Option "KeepShape" "On"

Relative or absolute mode

Option "Mode" "Absolute"     #or "Relative"

Defaults:

• cursor is Relative
• stylus and eraser are Absolute

Relative mode speed can be multiplied (default 1.0(verify))

Option "Speed" "0.7"

Pressure

Button response threshold:

Option "Threshold" "number"

Default is MaxPressure*3/50.

Pen response curve:

Option "PressCurve" "0,0,100,100"   # linear, default

Buttons

To report presses as different buttons:

Option "Buttonn" "action" # where action is 0 (disabled), a button number, or a keyboard event

For example, my mouse's scroll is inverted, so:

Report buttons only when tip is pressed:

Option "TPCButton" "on"

Pad and Screen mapping

Active area:

Option "TopX" "number"
Option "TopY" "number"
Option "BottomX" "number"
Option "BottomY" "number"

Multiple pen associations

Compiling a kernel

Manual work

You usually have one or more kernels in /usr/src/linux-someversiondetail, and /usr/src/linux is a symlink to one of them.

The one it links to is the one considered current (usually managed by you, sometimes by a package manager or assisted by one). This is significant only in that some third party kernel-module compilers may assume this link contains the headers/source they should use.

Configuring and compiling a new kernel

Hint: it's probably sensible to compile everything you may ever use as modules, that way you don't have to install-and-reboot as much when things change; modules take only filesystem space, no memory. (I wonder why this isn't common as a default)

Moving to a newer (or significantly different) kernel:

• Copy out .config from the current one to the new one,
• link /usr/src/linux to the new source
• go to this source, run
  make oldconfig
  , to see everything that's new or has changed explicitly.
• Then configure and as usual (people tend to use make menuconfig, make xconfig for nice option navigation)
• and compile/install as usual

Compiling a kernel

After configuration:

• compile the kernel:
  make bzImage
  makes a fairly well compressed image (note: not related to bzip2), or just make to generate a slightly less compressed file called vmlinuz.

• compile kernel modules:
  make modules
  to build the modules and
  make modules_install
  to copy them to directories the system actualy uses.

• install the kernel - using your preferred method.

Installing a compiled kernel

Note there are sometimes ways to make installation easier. Some distributions allow compilation into a distribution package, and in some the package manager automatically adds a kernel installed that way to the boot menu.

Simple automatic:

The simplest way is probably:

make install

This copies overly everything you could need to /boot. If you always do this, this will replace the previous kernel, so means you won't have to change boot menu settings -- and that it's also sort of risky in case you new kernel fails. I do believe it copies out the old kernel and appends something like .old, but this would mean you can do recover exactly one make install

By package

Various systems allow you to package the compiled kernel for at least organizing your compiled kernels, and sometimes easier installation and removal. For example, debian allows

make deb-pkg

which creates a .deb package. Other options include rpm-pkg, binrpm-pkg, tar-pkg, targz-pkg and tarbz2.png.

By copying the kernel

Manually installing a kernel is actually fairly simple. It consists of copying the kernel image to /boot and telling the boot loader about it.

Copy out vmlinuz / bzImage, from a place depending on architecture (e.g. ./arch/i386/boot/bzImage) to some unique and descriptive name in /boot, then point your bootloader towards it.

Manually updating the bootloader

These days grub is the boot loader of choice since it's more flexible and forgiving than the now fairly oldschool lilo

You can edit the boot list manually, but grub makes it a little easier: [update-grub http://www.fifi.org/cgi-bin/man2html/usr/share/man/man8/update-grub.8.gz update-grub man page] looks for all files starting with vmlinuz- and adds menu entries for them.

There are good reasons to do this manually, though, since editing the list yourself means you can ensure the same order and the same default/fallback options, which can be handy on headless servers.

Notes:

• /boot is mostly there for organisation. Historically it's also there so that you can make sure the partition lies at the start of the hard drive so that very simple boot loaders can access it (aren't required to support LBA addressing)

Boot parameters

• General: http://www.fifi.org/cgi-bin/man2html/usr/share/man/man7/bootparam.7.gz
• Kernel drivers can also accept these, though don't often need to.

Structure of linux in a wide sense

This is quite rough. It will be revised.

Bare booting

In terms of the basic system necessities, we have:

• The BIOS, which looks for a (storage) device to boot from and hands control to it
• A bootloader, which gives you the option of choosing one of several kernels (GRUB is now a common bootloader. It is more flexible than the oldschool LILO) (See also bootability)
• The kernel with its built-in drivers, stored on the filesystem, loaded by the bootloader. Provides interfaces to hardware, handles things like threads, swapping and such. Tends to starts a few processes to be able to all it's configured to do.

Actually, there're more details. See e.g. this page at IBM illustrating the lower levels of the boot process.

Also perhaps more general bootability details.

The system's framework

The parts always provided and/or necessary, started automatically:

• init, the parent of all processes, started by the kernel. Runs necessities, and things like the shells to log into (e.g. the fairly six standard consoles under Alt-F1 to -F6), and kicks off the rest of the bootup, usually using runlevels as a model.
• Core executables and libraries that are required by init or services.

• Some that deals with loading services in some order and considering dependencies between them. Runlevels are still fairly common, adapted from SysV init runlevels. This is what the /etc/init.d directory comes from, and rc0.d to rc6.d/, if you have them.

• Services themselves. Usually, few or none of these are strictly necessary for any of the system's core to work.

Things that run

Many things work not in an always-present way like the kernel does, but more on demand. This includes:

• Programs, of course. They may hook into libraries, the kernel and (when allowed by the kernel) even hardware fairly directly.

• Libraries loaded by the system on-use. This refers to most anything you cause to tun as a user. Not privileged like the kernel and kernel modules.

• And, in fact, kernel modules. These are files loaded by the kernel on demand, or possibly on use or by anticipated request ('coldplug'), and means additional drivers. They act like kernel code, but can be loaded and unloaded at will (whereas the kernel is a monolithic unit). Almost all kernel code can be both internal to the kernel and compiled as a module.

The graphical interface is actually a program, usually started from the service management. This may strike Windows users as odd, as they are used to thinking the graphical interface is the system - unless they're old enough to remember Windows 3.1.

Some window managers put a whole another behavioural and functional layer on top of the system. For example, Gnome's filsystem-like configuration manager is a neat idea, and KDE's kioslaves are sometimes really useful, but both of these are essentially just two of the more directly visible libraries.