View source for Virtualization, emulation, simulation

{{stub}}
{{virtualization_emulation_simulation}}

=Hardware level - emulation, simulation, and hardware virtualization=

Generally, 
: '''simulation''' means '''imitating the resulting behaviour'''
: sometimes just the surface level is enough (but that's often so specific-purpose that it's too much work)
: so you often go deeper, stop at the highest level you can get away with and that does not actually make things harder


: '''emulation''' means '''replicating how the internals work'''
: as precisely as sensible or required, often at a much lower level 
: either out of necessity (e.g. emulating how an old CPU works so you can run anything on it)
: or out of preference (e.g. emulating imperfections as well as the specs)



Theory often presents this as two alternatives, but in practice is more of a sliding scale - which is [[bimodal]], sure.

<!--
varying with 
* how different the hardware is

* how different the platform is in other ways

* how changeable the code you're running is
:: for games


Games are an interesting example of many. 

For example, games initially made to run under of windows (and graphics drivers) 
may need some invasive change in the graphics subsystem (and often almost nothing in their game logic).

That change in the graphics system can be considered simulating the older graphics stack.



Games on older game consoles are often so hardware-specific that
the only approach to getting anything working is to emulate that hardware,
and run the (often-original) code on there.

Simulation in this context could only really mean writing your own imitation of the game your own way,
and trying to match it as close as possible.

-->


<!--
Say you want to play pacman. If you've done a bit of programming, then maybe 
you take a modern programming language, and a few weeks, and make something that looks convincingly like 1980s pacman.



Say, if you have a pacman game that ran on WinXP, but does not run now. 
Most of the incompatibilities might be in details like how you asked the system to play an audio file, 
and you can fix that by intercepting that call and doing it another way.
: now you're emulating an older version of windows, but otherwise it's running as-is


Now say you have a version of a cartridge or disk or tape for pacman on a 1980s era game console.

There are no libraries saying 'now play this audio file' - this is directly talking to all hardware,
that this isn't just about "intercept the way it plays a file and do it another way"
and that hardware (CPU and peripherals) will be ''so'' unlike any PC (modern or from the same eighties)
It is unlikely to function at all unless you know end emulate the specs ''and'' idiosyncracies of that hardware.

The thing is that old hardware has peculiarities that are not documented well,
but are sometimes specifically abused.

Say, if you implement what the parts of an Playstation do on paper, you will probably not get Crash Bandicoot to run - there's a whole interesting story behind that one




Now consider both the [https://www.c64-wiki.com/wiki/SD2IEC SD2IEC] and the [https://www.c64-wiki.com/wiki/1541_Ultimate 1541 Ultimate].
: Both take floppy images on an SD card, both present themselves as a [https://www.c64-wiki.com/wiki/Commodore_1541 1541 floppy drive] to a C64 or similar. 
: The SD2IEC is more of an simulation in that it only really cares to speak the image's data on the bus.
: The 1541 Ultimate emulates so many parts of the drive to a low enough level that it supports strange uses like that by fastloaders.
-->



<!--
But you can also get fuzzier about what you even wanted, e.g. whether this is about hardware, or results.

For example, [https://www.bryanbraun.com/after-dark-css/ after dark in CSS] shows you toasters, but the code has absolutely no relation to After Dark's, or the windows 3.1 it was often run on.
It ''only'' cares to show you images of moving toasters.
I guess you could call this ''imitation'', because presumably they don't even act quite the same.

-->


==When hardware is compatible, and software environment is quite similar==
<!--

In some cases, the change in environment over time can be fixed with a little translation.

Say, Windows's compatibility mode basically means windows presents itself differently, 
doing some translation, but ''relatively'' small amounts, 
so that programs expecting an older version of windows can still be run.

Microsoft actually puts a lot of work into little program-specific tweaks so that things stay working for many years{{verify}})

You could say it's simulating an older version of windows.
Or perhaps call it none of these terms.
-->

==When software environment is quite different==
<!--

Take [https://www.winehq.org/ Wine], which letting you run windows binaries on linux
: the code is for the same CPU architecture (x86), so at opcode level basically just runs
: however, every call towards the windows kernel runs into the issue that there ''is'' no windows kernel in linux, so instead all such calls are intercepted and handled somehow (libraries, and often some linux kernel calls)
: also deals with typically-installed windows libraries


[https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux#WSL_1 WSL1] is the same idea, but in the opposite direction, letting you run linux binaries on windows.  

It's not a linux kernel, but a translation layer to windows libraries (and kernel)


Both are essentially simulating an operating system

{{comment|(side note: WSL2 is completely different to WSL1: it's a VM running a modified linux kernel)}}
-->


==When hardware, or software environment, is completely different==
<!--

Take DOS-era games    (similar story for most game consoles, actually)

* You cannot run such a program directly on a CPU either, because it does not understand that other things might be running on the same CPU - it assumes full control.

* programs would directly talk to video cards and sound cards.
: Video cards and sound cards that also differs so significantly from modern hardware that there is no simple translation even if the previous point were no issue


As a result, running DOS/console games on a modern PC is different enough that you ''have'' to go to simulating/emulating all the hardware the code needs to talk to. 

Exactly where that lies ''between'' simulation/emulation depends more on the specific piece of hardware.


Xbox was a weird case.  Yes, it's a Pentium III with PCI bus connecting most other hardware, Nvidia GPU, IDE based hard and DVD Drive, and USB, but enough of the core hardware (particularly the MCPX southbridge) is custom enough that no, you can't "just run" xbox on PC.

It needs to be emulated too - but yeah, various fragments of code may be ''more easily'' transated to modern x86 than some completely different architectures.



sits ''largely'' on the emulation end, mainly because there is little to no hardware overlap between them and the PC you probably want it on.

To go into how this often a gliding scale, consider:
* classical game consoles tended to use CPUs we no longer use, and custom hardware setups.
: Emulating down to instructions, with all effects perfect to the cycle, is possible but almost always a few factors slower, so game emulators will often blur the lines a bit, with clever tricks like optimizing JIT-like cross-compilation

* consoles had very specific hardware, and games typically talked to the hardware directly
: these accesses have to intercepted, and redirected to simulations/emulations of that hardware


The last is a point in itself.
: while you could emulate each piece of hardware down to the silicon, that's often slow to do (easily one or two orders of magnitude slower)
:: ...so it can often be a lot more efficient to simulate what it does.
::: e.g. memory management is just bookkeeping
::: e.g. a sound chip that is mostly makes squarewaves can be simulated. It takes proper care, though, because people used these chips in ''interesting'' ways, (so you may have to implement imperfections and less-well-documented things)
:: in general, a particular game not running on an emulator (yet) is because it does something that simulation code did not consider.


Say, a sound chip or a floppy drive is often easier and faster to simulate than emulate - dealing only with the bytes it stores and reports, completely ignoring the nature of the medium.

Still, if someone uses it in an unusual or undocumented way (e.g. clever copy protection), then maybe it doesn't work for those few cases. (it's often quite doable, and potentially easier overall, to patch those oddities in)


Similarly, audio chips like PSGs had some specific (ab)use that let you make more interesting sounds with them, and the most basic simulation may not let you do that well.


For another sound related example, the C64's [[SID chip]] is a decent example where the implementation was a little worse than its designer intended.  
: In particular, there wasn't enough silicon for a clean filter, meaning it was a dirty filter (and varied, due to production process). Yet the grittiness that it produced became a characteristic sound that people love, and these imperfections can be simulated (at significant CPU cost).
: Also, the ability to produce samples relies on poor DC isolating creating a tick (which was mostly gone in the second revision of the chip - so many soldered the mistake back in). To-spec implementation wouldn't allow this, so you need to simulate this imperfection.
-->

==Same-architecture emulation, virtualization - virtual machines==

<!--

VM, Virtual Machine, us used in more than one sense, 
but in this context it means "we can run a guest OS, 
because as far as it's concerned, it looks enough like base hardware, 
when in fact we're cheating and it can't see the system that is actually hosting it"


When that's on basically the same hardware and same instruction set,
the work in hosting that guest OS ''mostly'' lies in intercepting hardware access,
and pure number crunching might actually happen at basically the same speed as a program on the host OS.

Hardware access ''has'' to be intercepted
: ...to be able to present what ''looks'' like low level hardware to the guest OS,
: also to not immediately conflict with the host OS and crash.
: (it's also good for security, which is often part of the point)


After a while, some care was taken in CPUs design to help such hardware virtualization, meaning distinct OSes can run on the same CPU without being aware of each other, or able to affect each other.

Roughly speaking, this means that some of the mentioned tasks, that were previously had to be handled via interception in software can now often be left up to hardware.


Arguably, none of this is truly simulation or emulation, because the architecture not only ''looks'' like what you want, it ''is'' what you want.

But if you need to call it one of the two, it'd be emulation, in that the little extra that is there, be it implemented in software or hardware, 

...at the same time, VM software will often want to present a unique network card, video card, sound card, mouse, etc., which are all software simulations in that they only need to act right.

-->

<!--

If your application's machine code instruction set matches the machine it's on,
in theory there's little you need to get it working,
and there are performance benefits from having as little code inbetween.

Ideally, that just means launching a program directly under your OS.

If it's too many versions ago (say, windows 3 apps)
 


What if you want to run completely isolated operating systems?

Historically that meant the OS ran another OS in a process,
because while the machine code is the same, ''something'' needs to pay attention that
the guest OS doesn't clash with the host OS's management of memory and IO.
Which in practice they would do ''immediately''. 

So most cases of Virtual Machines were for many years meant "a program that
translates all the memory and IO of the guest, to get this working". 

They could still mostly let through just-doing-calculations instructions straight to the CPU.


VMs are so useful that they affected CPU design.
-->



<!--

See [http://en.wikipedia.org/wiki/Comparison_of_virtual_machines Comparison of virtual machines (wikipedia)]. 



Some of the more common virtual machine software:
* VirtualBox
** Free
** Runs on Windows, OSX, linux, Solaris
** Has 3D support, but no lowish shader mode version, so it can be best to keep unnecessary 3D off
** http://www.virtualbox.org/

* VMWare 
** Personal installations, PC: Free (basically)
** VMWare Fusion (Mac version): Not free. Version 2.0 has Pixel Shader 2 support, so Vista Aero and games run fairly well.
** VMWare for server virtualisation: Not free
** http://www.vmware.com/

* Qemu
** http://www.qemu.org/
** Runs on x86, x86_64, PowerPC, can emulate about a dozen processors and host a whole bunch of guest OSes

* MS Virtual PC
** Available for PC, Mac
** http://www.microsoft.com/windows/virtualpc/default.mspx 

* Parallels
** Parallels Desktop for Mac (OSX)
** Parallels Workstation for Windows, Linux
** http://www.parallels.com/


Virtual server setups (not necessarilty graphical)
* [http://en.wikipedia.org/wiki/Xen Xen]
* [http://www.linuxvirtualserver.org/ LVS (Linux Virtual Server)]
* [http://linux-vserver.org/Welcome_to_Linux-VServer.org vserver]

-->

==Tangent: protection==

Protection is not virtualization, but plays a useful part of it - and arguably a pretty necessary one in practice.

====Memory protection====


Early CPUs (and various microcontrollers and some comparably simple processors today) ran with all code having access to everything.
Why not, you know? 

Also, early computers tended to run exactly one thing at a time.
We do not have any reason to distrust what else might be happening when there ''is'' nothing else happening.


When you want to run distinct processes, 
this is awkward for many reasons, 
such as having full rights to trample over each other (meaning stability and security is impossible to guarantee), 
they can't easily know where the other is in memory, 
so can't easily avoid each other even if you wanted to.

So e.g. Intel thought up [https://en.wikipedia.org/wiki/Protected_mode protected mode] {{comment|(the thing before then was called [https://en.wikipedia.org/wiki/Real_mode real mode], apparently because the addresses were real, and later were always virtual)}}. Protected mode allowed things like
: virtual memory - letting a process deal with their own private memory space, and having the OS map those to actual memory in a way where they are kept
: safe multi-tasking - as a result of the above, and some other details
: paging - put specific parts of memory on secondary storage (this made that so much simpler to do at OS level)


Protected memory was introduced with the 286 but was awkward, so only really started seeing adoption around the 386.


For efficiency reasons, the mapping (and usually protection) step is assisted by a hardware element called MPU.


See also [[On computer memory#Intro]]




On most computers, memory is both [[On_computer_memory#On_virtual_memory|virtualized]], and protected.

It is the [[MMU]] that handles both. 

On some platforms memory is not virtualised, but still protected, and there is an MPU doing the latter.

-->

====Protection rings====
{{stub}}

Basically all platforms designed for memory isolation will have some sort of privilege system - that will have to go beyond just memory (for, at lower level, mildly complex reasons).



For example, On x86, there are multiple privilege levels, called '''protection rings''',
the still-fairly-general idea of "you must be this privileged (or better) to do the thing."


It is effectively the CPU being told (by the kernel) when to say no to IO accesses.
Which is complementary to the kernel (VMM)telling the MPU (and then CPU) when to say no to memory accesses.


x86 processors have four such levels, and diagrams always make a point of this, but as far as I can tell, most OSes only actually use two of them:
: "stuff the the kernel can do", i.e. everything - often in ring 0
: programs, which have to go via the OS to speak to most hardware anyway - often in ring 3


A few OSes have experimented with using more, and e.g. put drivers in a third ring (less privileged than the kernel, more than programs), but most have apparently decided this is not worth the extra bother, for both technical and practical reasons.

OpenVMS apparently uses four.


For more concrete implementation details, look for [https://en.wikipedia.org/wiki/Protection_ring#Privilege_level IOPL].



ARM processors have no rings like that, but have a fairly regular user mode, and a supervisor mode that gives more control,
that can be used in much the same way{{verify}}.
<!--
[https://heyrick.eu/assembler/regs.html]
-->

====not-really-rings====

AMD-V and Intel VT-x and other '''hardware-''assisted'' virtualization''' are regularly referred to as '''Ring -1'''.

Which they are not, because it's an entirely different mechanism from IOPL.

Yet it gets the point across that it allows multiple OSes to run, unmodified, 
meaning they will make privileged instructions directly as if they are Ring 0 on the CPU,
and thsee hardware features ensures that functions at all, doesn't cause conflicts, or big security holes.
{{comment|(the alternative is paravirtualization, where OSes ''are'' aware they are running in a hypervisor, and call into that hypervisor instead of issuing privileged instructions)}}



There is also a System Management Mode ([https://en.wikipedia.org/wiki/System_Management_Mode SMM]),
referring to a CPU mode where regular OS is suspended and external firmware is running instead.

This was initially (386 era) meant for hardware debugging, later also used to implement some BIOS-level/assisted features like Advanced Power Management, USB legacy support, and a handful of other relatively specific things.

SMM is sometimes referred to as '''Ring -2'''.
Also not accurate, though it ''could'' potentially be used for something resembling that.



There was a specific exploit of a specific chipset[https://en.wikipedia.org/wiki/Intel_Management_Engine#Ring_-3] that did something similar to SMM, and is referred to '''Ring -3'''.
This is even less accurate, and instead seems to be a reference to the CPU sleep state this could be exploited in{{verify}} and not related to IO protection.



Some footnotes:
* VT-x and such are often disabled by default on e.g. most home PCs
: partly because people theorize about Ring -1 malware, "...so if you don't need it, why risk it?"
: on certain server hardware it's enabled by default because of how likely it is you will need it

* there is no point for software to use rings that the OS doesn't, because you would be using security model that the OS doesn't enforce.

==Hardware virtualization==

===On hypervisors===
{{stub}}

Without any of this virtual stuff, an OS's kernel effectively runs in [https://en.wikipedia.org/wiki/Protection_ring#Supervisor_mode supervisor mode], which is a name for a flag that allows (only) such system tasks to do everything to the hardware, and we trust you to never do a thing wrong. 
In x86/x64 this roughly means [[protection rings|protection ring 0]] but there are other platforms's answers to the same idea.


So when you want to run multiple OSes in parallel -- natively on the same actual hardware rather than fully simulated hardware as in VMs -- then you have to deal with OSes assuming they have supervisor mode.
So we need something managing the supervisor mode stuff of multiple OSes.

And since hyper- is the superlative of super-, we call that a hypervisor.

The job of a hypervisor is mainly to make sure that IO and memory access makes sense for all of the things it's taking care of.
This is ''largely'' isolation, and mainly things related to the [[VMM]] (memory and a bit of IO) and [[protection rings]] (IO).


====Hosted and native hypervisors====
{{stub}}

There is useful distinction between 
* '''hosted hypervisors''' (a.k.a. type 2), where there is a main OS, and each guest OS is effectively a process within the host OS (and inherits its protections)

* '''native hypervisors''' (a.k.a. type 1), where the hypervisor is part of the hardware, so that its protections apply even when the OS basically runs at what it considers to be Ring 0, and 
: for most intents and purposes this means, running ''alongside'' the host OS, and only guests insofar that the host probably manages when they start and stop, and probably stayed in control of the monitor you poke at
: the hypervisor also controls resources, e.g. hyper-V's vCPUs are an abstraction around timeslicing your physical cores{{verify}}[https://social.technet.microsoft.com/wiki/contents/articles/1234.hyper-v-concepts-vcpu-virtual-processor-q-a.aspx] 

The distinction is sometimes fuzzy. 
For example, Linux's KVM is a kernel module that basically makes the kernel a nearly-native hypervisor -- while being part of an OS, suggesting a hosted hypervisor.


<!--

If the above is a bit dry theory, consider the same from a different angle:

Emulation of one OS in another is nice for functionality, but initially not speed. This is largely because of safety: you have to check that everything host and guests do is isolated. In straight emulation, that means software.

When running virtual machines became widespread, it begged the question whether you could run embedded OSes to, basically, have each run at Ring 0, getting hardware to do the checks for you.


The answer is "sure, by adding another level below".[https://en.wikipedia.org/wiki/Protection_ring#Hypervisor_mode]
That is what AMD-V and Intel VT-x are and do: '''hardware-''assisted'' virtualization'''. 

It means that the bytecode (that happens to be an OS accessing hardware assuming it's in Ring 0) can run unmodified - while actually being completely isolated from other things that think they're running in Ring 0 this way, or are - in particular the host system.



People refer to this as '''Ring -1''' (in reference to [[protection rings]]). Which it isn't because it's a completely different mechanism, but intuitively gets the point across: as far as the multiple operating systems are concerned, each of them are running Ring 0 on the CPU.

In practical terms, this allowed VMs to work withour the latency incurred from a hosting hypervisor.


There is still a minor, uniform speed hit, because the hardware-assisted virtualisation is an extra non-instant layer - that you only neeed if you want to do the above, and can be turned off if you only ever run one OS.




There is also an inbetween, '''para-virtualization''', 
where the guest is aware it's a guest to a host, and chooses to talks directly to the virtualized hardware,
rather than relying on it being intercepted.

e.g. Xen can run both para-virtualized and hardware-virtualized.



http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/

https://en.wikipedia.org/wiki/Hypervisor#Classification

https://en.wikipedia.org/wiki/Hardware-assisted_virtualization





'''Virtualization''' refers to the mechanics of doing this -- and to in practice also means you abstact computer ''resources''. 

Sometimes for clever provisioning and competitive pricing, sometimes for security, sometimes for marketing.

In recent years, it refers primarily to abstracting a whole computer (platform virtualization), 
used e.g. for using a single hardware computer to host multiple VMs.



It means you can effectively sell a part of a computer's power for a part of that computer's price,


Most flexible, and arguably more secure than OS containers in that there are more steps before you can roam freely in the wider system actually running the bit you got -- but it in all cases it requires some serious exploits.



===IO in virtualization===

Intel's VT-x and AMD's AMD-V  are the wider standards dealing with hardware virtualization.



Aside from sharing the CPU, another thing you need for your guest OSes to (think they are) running in ring 0 is a way for each to talk to hardware peripherals independently.

This means DMA and interrupts must be handled, and more specifically, remapped. 

This is called I/O MMU virtualization (or sometimes 'PCIe Passthrough' because that's what it mostly amounts to).
: this is covered by 
:: Intel VT-d, short for "Virtualization Technology for Directed I/O" (''part'' of VT-x),
:: AMD-Vi information (previously just IOMMU)


for this to actually function, we also need
* motherboard chipset support
* motherboard firmware support (BIOS or UEFI) 
* PCIe device support (specifically function level reset) 
...there are some footnotes, particularly for older buses than PCIe.




https://en.wikipedia.org/wiki/X86_virtualization#I/O_MMU_virtualization_(AMD-Vi_and_Intel_VT-d)


Also relevant acronyms include are 
* HAP

Nested paging (a.k.a. SLAT, Second Level Address Translation) [https://en.wikipedia.org/wiki/Second_Level_Address_Translation]
* Intel's SLAT implementatin is called EPT (Extended Page Table)
* AMD's SLAT implementation is called RVI (Rapid Virtualization Indexing )
* some ARM also supports SLAT



===KVM===

===Xen===





-->

==Paravirtualization==
<!--
https://wiki.xenproject.org/wiki/Paravirtualization_(PV)

-->

==Nested virtualization==
<!--

https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen#Booting_Dom0:

-->

=OS level - Virtualization, jails, application containers, OS containers=

<!--
http://phk.freebsd.dk/sagas/jails.html
-->

===chroot jails===

chroot() changes a process's apparent filesystem-root directory to a given path.

So only really affects how path lookups work.

This is useful for programs that ''want'' to run in an isolated environment of files. 
Such as a clean build system, which seems to be its original use.


At the same time, if program ''wants'' to break out of these, it will manage.
It was never designed as a security feature, so it ''isn't'' a security feature.
See [[chroot]] for more words about that.



===FreeBSD Jails===

Inspired by early chroot() and the need to compartimentalize,
BSD jails actually ''did'' have some security in mind.

It only lets processes talk to others in the same jail,
and considers syscalls, sockets/networking and some other things.

It's been around for a while, is mature, and allows decently fine-grained control.


===Solaris/illumos Zones===

If you come from linux angle: these are much like LXC {{verify}}, and were mature before it.

Due to the Solaris pedigree, this combines well with things like ZFS's snapshots and cloning.

<!--

https://wiki.smartos.org/display/DOC/Zones

https://illumos.org/man/5/zones

https://illumos.org/man/5/brands


KVM zones

LX zones - emulate linux within the confines of a zone. Basically a bunch of specific hackery to be interoprable



https://www.youtube.com/watch?v=TrfD3pC0VSs

https://www.youtube.com/watch?v=hgN8pCMLI2U
-->

===Linux containers===


There are a few kernel features that isolate, monitor, and limit processes. 


So when we say that processes running in containers are really just processes on the host, that's meant literally.
The main way in which they differ is that by default they can share/communicate nothing with processes that are not part of the same container.


While these isolations are a "pick which you want" gliding scale, 'Linux containers' basically refers to using ''all'' of them to isolate things in ''all'' the ways that matter to security. ...and often with a toolset to keep your admin person sane, and usually further convenience tools. [[Docker]] is one such toolset.


The main two building blocks that make this ''possible'' are [[#namespaces|namespaces]] and [[#cgroups|cgroups]].

Just those two would probably still be easier to break out of than a classical VM, so when you care about security you typically supplement that with 
: [[capabilities]], intuitively understood as fine-grained superuser rights
: [[seccomp]], which filters allowed syscalls,
: [[SELinux]] / AppArmor / other [[mandatory access control]]. Often not necessary, but still good for peace of mind.

{{comment|(Note that a good number of these fundaments resemble BSD jails and illumos zones)}}

<!--
Docker's focus on copy-on-write storage to build images ''mostly'' isn't necessary, but quite useful because fast and manageable.
-->


'''Uses''' 

Various other names around this area - [[docker]], LXC, [[kubernetes]], rkt, runC, systemd-nspawn, OpenVZ - are tools around the above - runtimes, management, provisioning, configuring, etc.

Some of them the gears and duct tape, some of them the nice clicky interfaces around ''that'', some of them aimed at different scales, or different purposes.


rkt and runC just on running (lightweight container runtime, fairly minimal wrappers around libcontainer)

containerd - manages image transfer, lifecycle, storage, networking, execution via runc

[[docker]] is built on top of containerd (and indirectly runc). And is aimed at making single-purpose containers, on building them, on portability.

'''[[LXC]]''' shares history, libraries, and other code with docker, but is aimed more at being a multitenant machine virtualisation thing.

While you could e.g. do [[microservices]] with LXC, and a fleshed-out OS in docker, both are more work and bother and tweaking to get them quite right {{comment|(e.g. docker avoids /sbin/init not only because it's not necessary, but also because standard init does some stuff, like setting the default gateway route, that the inside of a container would have to work around)}}.

[[kubernetes]] focuses almost purely on ''orchestrating'' (autimated deployment, scaling, and other management) systems within one or more hosts.

<!--
openVZ more about running and migrating
-->

See also:
* https://en.wikipedia.org/wiki/LXC
* https://en.wikipedia.org/wiki/Docker_(software)
* https://en.wikipedia.org/wiki/Kubernetes

===Xen===

<!--

Xen is a [[hypervisor]], that runs VMs for you. 

You can use it to run multiple OSes basically-natively.




https://en.wikipedia.org/wiki/Xen

-->

===Linux KVM===
<!--

Linux's KVM is a kernel module that basically makes the kernel a nearly-native hypervisor -- while being part of an OS (this is somewhat unusual in that native hypervisor versus hosted hypervisor is usually a clearer distinction)


-->

=From a hosting perspective=
<!--
From this perspective, there were a few rough eras:

* Classical:  App installed directly on OS
: no resource boundaries, meaning apps (and users, if there were multiple) could trample on each other with resources 
: maybe with chrooted shells, but that only does so much
: any security flaws could affect all apps (and all users, if there were multiple) on the same host 
: could be solved by putting people on separate servers, but this would significantly under-utilize servers, and be pricier

* Virtualization with VMs
: fairly strict resource boundaries 
:: each has its own filesystem, OS and so process space, a pre-set amount of memory
:: CPU sharing is a little flexible
: good security isolation between VMs
: VMs are disposable over a long enough term

* Virtualization with containers
: relaxed isolation of 
:: system - they share the Operating System (OS) (but cannot view resources from the host)
:: filesystem - has its own, but also more easily shared from host
:: memory - shared from host (with optional limits, which have footnotes)


Whether VMs or containers are better depends on 'what for?', 
but containers have become popular for more practical reasons:
it's ''much'' easier to deploy one, making it a prime choice for [[CI/CD]] style development

This quickly became its own thing. See [[Cloudy notes]].

-->

=A few notes on...=
===namespaces===
'''Namespaces''' limits what you can see of a specific type of resource, by implementing a mapping between within-a-container resources to the host's.


This allows a cheap way to have a container see their own - and have their host manage these as distict subsets. <!-- {{comment|(Note this also helps allowing instances to be moved. Because e.g. consider that e.g. a the guest's running process's PID should not change, but the host PID would if you moved it)}} -->

Linux has grown approximately six of these so far:
* PID - allows containers to have distinct process trees
* user - user and group IDs. {{comment|(E.g. allows UID 0 (root) inside a container to be non-root on the host)}}
* mount - can have its own filesystem root {{comment|(chroot-alike)}}, can have own mounts (e.g. useful for /tmp)
* network - network devices, addresses and routing, sockets, ports, etc.
: some interesting variations/uses
* UTS - mostly for nodename and domainname
* IPC - for SysV IPC and POSIX message queues


For example, you can
 sudo unshare --fork --pid --mount-proc bash
which separates ''only'' the processes' PID namespace.
You can run top (because your filesystem is still there as before), you can talk to the network as before, etc. -- but no matter what you do you'll only see the processes you started under this bash.


See also [https://en.wikipedia.org/wiki/Linux_namespaces namespaces] [https://lwn.net/Articles/531114/]

===cgroups===

'''Control groups''' concept is about balancing and metering resources.

While they came into knowledge with containers, cgroups apply to ''all'' of the host and not just containers, they just effectively default to no limits.

<!--
Very roughly, you can throw a bunch of processes into a cgroup and you get reporting and control over the resources that group uses.

This is e.g. used in 
* docker

* optional in systemd
: useful e.g. to get statistics per service
-->


They're a useful tool to limit how crazy a known-to-misbehave app can go,
without going near anything resembling namespaces or containers.

And/or just to report how much a process or group of them has used<!-- - see e.g. its use in systemd-->.


Resources types (which cgroups calls subsystems) include:
* Memory - heap and stack and more (interacts with page cache{{verify}})
: allows OOM killer to work per group
: this makes sense for containers, but also for isolating likely/known miscreants and not have OOM killer accidentally shoot someone else in the foot
* cpu - sets weights, not limits
* cpuset - pin to a CPU
* blockIO - limits and/or weights
: also network, but does little more than tagging for egress 
* devices
* hugeTLB
* freezer - allows pausing in a cgroup


Cgroups set up a hierarchy for each of these.
Nodes in each hierarchy refers to (a group of) processes.

The API for cgroups is a filesystem, /sys/fs/cgroups.
Which, being a verbose/finicky, there are now nicer tools for.


See also
: https://en.wikipedia.org/wiki/Cgroups cgroups
: https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
: https://www.youtube.com/watch?v=sK5i-N34im8

===LXC===

<!--

LXC containers are created from templates, 
in part to inherit a bunch of tweaking 
that you would otherwise have to do manually.


By example:
 lxc-create -t ubuntu -n tomo
 lxc-list


LXD is a container manager.
It builds on top of LXC, basically providing alternative to various LXC

See also: 
* https://en.wikipedia.org/wiki/LXD



Privileged versus unprivileged containers:

Unprivileged containers means that even root in the container maps to non-root outside.
Good for security.





 lxc_container: utils.c: get_template_path: 1135 No such file or directory - bad template: ubuntu
 lxc_container: lxccontainer.c: lxcapi_create: 1223 bad template: ubuntu
 lxc_container: lxc_create.c: main: 274 Error creating container tomo
You don't have templates installed, probably because it's in a separate package.

 'debootstrap' command is missing
Install the package.

 lxc-ls: command not found
It's probably in a package called lxc-extra


'''Networking'''
Default is virtual Ethernet pairs{{verify}}

 failed to attach 'veth54PCQG' to the bridge 'virbr0': No such device
You need a bridge device on the host to be able to attach containers on it.
virbr0 is usually created by libvirt's service - do you have that installed and started? (try sudo systemctl start libvirtd)

If you want more control over the networking, dive into config files

https://unix.stackexchange.com/questions/179353/failed-to-attach-veth87vsij-to-the-bridge-virbr0-no-such-device

http://blog.scottlowe.org/2013/11/25/a-brief-introduction-to-linux-containers-with-lxc/


-->

=More specific software=

(...rather than things that mostly just name thing building blocks)


===Virtualbox===

===VMware===

===Parallels===


===Linux OpenVZ===

See also:
* https://en.wikipedia.org/wiki/OpenVZ

===Qemu notes===

{{stub}}


Usage notes:

* VNC is handy (while installing and in general), since you get a view on the entire machine, including BIOS and bootup.

* If the mouse in your VNC window seems to be wrong (offset from the real mouse position), this is likely because of motion acceleration applied by the guest OS. An easy way to fix this is to get absolute positioning by making the mouse device report itself tablet-style: use '''<tt>-usbdevice tablet</tt>''' when running qemu.

* The easiest VM-to-host networking option is probably '''<tt>-net nic,vlan=1 -net user,vlan=1</tt>''' (see networking notes below on what that actually means)

* '''<tt>-no-acpi</tt>''' may be useful for XP or anything else that tries to use ACPI a lot if prsent (but will run fine without it); it seems that emulating ACPI for the way XP and some other things use it uses unnecessarily much host CPU.


====On networking====

Qemu can emulate network cards in the guest (with NIC model and mac address configurable) and connects it its own VLAN - which is basically like a virtual router, giving you flexibility when interconnecting. You can, for example, connect each guest to other guests and/or to the host networking. 


Of course, many of us just have a single guest and want to give it internet access. This means the default, which is equivalent to <tt>-net nic,vlan=1 -net user,vlan=1</tt>, is good. That means means "emulate a network card in the guest, connect it to VLAN1; connect qemu's usermode networking {{comment|(to the host)}} to VLAN1 as well". 
In other words, this setup mkaes VLAN1 is a simple interconnect between the VM and the host.


You can get various possible interconnections. Some of the options:
* Qemu VLAN to host via usermode network stack 
** easy gateway to the outside (includes things like a basic in-qemu DHCP server), without needing any interface bother on the host side, nor much bother in the client
** allows port redirects to the inside, which can be handy

* Qemu VLAN to host via tun/tap

* Qemu VLAN to host via VDE (usermode tool that manages tun/tap)

* Qemu VLAN to other Qemu VLAN


See also:
* http://en.wikibooks.org/wiki/QEMU/Networking
* http://en.opensuse.org/Qemu_networking

====On images====

http://en.wikibooks.org/wiki/QEMU/Images


Some older versions of Qemu have a few problems with qcow type images, and will report that they can't open the disk image (which would normally point to permission problems). 
Use another type, or use a different version of qemu.
If you have such a problematic image, you can fix it by converting it to another type{{verify}} (using <tt>qemu-img</tt>).



Note that formats like qcow and qcow2 are sparse, but things like defragmenting and wiping in the guest OS will cause space not actually used by the guest filesystem data to be used on the host size. If you want to crimp it again, you can often shrink the image file by doing something like:
 qemu-img convert -f qcow2 windows.img -O qcow2 windows-shrunken.img
(...and then swap them)


=====Growing or shrinking guest disk size=====
{{stub}}

Possible, but some work as you're both resizing the filesystem and the disk that the guest OS sees.

Also, while most systems beside windows seem to deal with this well enough, windows copies part of the disk geometry into its boot code to identify the boot disk later, which causes problems in some cases, (possibly when you resize across certain sizes, seemingly two-power sizes?{{verify}}).
The error you'll get is the rather uninformative '''"A disk read error occurred"'''. You can fix this (by editing the MBR (probably easiest while it's still raw format, depending a little on how you do it), and there are probably some ways tools like fdisk might fix this too<!--(perhaps fdisk /mbr fixes this too?{{verify}})-->.


As to the actual resizing, the easiest option is probably to go via raw format.
For example:
* (if shrinking:) shrink the partition within the VM so you know for sure that only the first so-many bytes of the (virtual) disk is being used. You'll need something which actually moves the data stored on the partition (gparted is an easy enough and free option).

* convert the image to a raw-type image

* truncate or extend the raw file (representing-a-disk) as you want. It's probably easiest to use <tt>dd</tt>, truncating and/or adding as necessary (look at dd's <tt>bs</tt>, <tt>count</tt>, and such)

* convert this new raw image to your favorite image format

* use gparted within the VM to resize the filesystem to the new disk size




For example, I started with a 6-gig image {{comment|(initially created using <tt>qemu-img create windows.qcow2 -f qcow2 6G</tt>)}}.

After installing windows, and stripping Windows down to take ~1.6GB I decided that a 3GB virtual disk would be enough, so:
* used gparted (within qemu, starting its liveCD using <tt>-boot d -cdrom gparted.iso</tt>) to resize the partition to ~2.7GB (under 3GB to avoid problems from rounded numbers - I could've worked exactly instead))}}.

* converted the image to raw:
 qemu-img convert -f qcow2 windows.qcow2 -O raw windows.raw

* Copied the first ~3G to a new raw image. {{comment|(Note that the following actually specifies 3000MB, ~2.9G, but this is still very comfortably more than what is actually used, since the gparted we did means that only the first ~2.7GB of that 6G raw image is used by the partition)}}
 dd if=windows.raw of=win3g.raw bs=10M count=300

* converted this truncated raw to a qcow2 image again using qemu-img
 qemu-img convert -f raw win3g.raw -O qcow2 win3g.qcow2

* started the VM with this new disk image, and booted the gparted liveCD again to grow the partition to the actual new disk size.

<!--
http://fugitivethought.com/blog.php?action=view&blog_id=77
-->

====Experiment with XP====
{{stub}}
Choose the memory you'll need ahead of time. Windows can be picky about hardware changes, deciding it's now in a different computer. The below uses 384MB, which is comfortable enough for basic use.

* create drive image (max 5GB), here of type qcow2
 qemu-img create windows.img -f qcow2 5G

* While installing windows:
 qemu -no-acpi -localtime -usbdevice tablet -net nic,vlan=1 -net user,vlan=1 windows.img -m 384 -vnc :2 -boot d -cdrom winxpsetup.iso


Now you can connect a VNC client to :2 and watch it boot, install windows, and such.

Once windows is installed, the CDROM isn't really necessary {{comment|(arguably it can't hurt as windows can always find any files needed for extra installation)}}.

You may wish to configure XP to listen for Remote Desktop and use that instead of (or in addition to) VNC (note: you may wish to add a password to VNC access). To use remote desktop you'll also need to forward a port to the inside.

The command I use for regular runs of this VM:
 qemu -no-acpi -localtime -usbdevice tablet -net nic,vlan=1 -net user,vlan=1 windows.img -m 384 -redir tcp:3389::3389


<!--
====See also====
http://www.tuxradar.com/content/howto-linux-and-windows-virtualization-kvm-and-qemu
-->

===Multiple===

====bhyve====

For BSD


https://bhyve.org/

====Proxmox====
{{stub}}

Proxmox Virtual Environment, often just proxmox, sometimes PVE, is a [[hosted hypervisor]] (running on a slightly modified Debian (or Ubuntu) that puts a management web interface in front of allowing both:
* VMs ([[Linux KVM]] style)
* containers (LXC style)



If you heard container and wonder if that can run docker ''directly'' -- you probably don't want to; 
you can do this either in a VM (arguably better isolation) or in LXC instance (needs two config changes to nest like that but works).

====Kubernetes====


====Qubes OS====
<!--


Qubes is a security oriented desktop OS (with Xen at its core),
focused on compartementalizing everything, including different functional parts of the OS itself, 
using separate VMs, e.g. separating the usb, the network stack, the firewall, the display manager, system administration, and user applications.

It allows granular management of the trust involved.


This is at the same time sort of "though experiment" levels of theoretic overkill, 
and potentially a very pragmatic answer nothing else gives:
if you ever want to run any software you do not fully trust, but ''do'' actually need to use,
and want some real guarantees about what that will ''not'' leave behind,
how do you do it?


"Use a VM"? Yes, but unless you already have an image representing
"fully installed and usable VM ready to install that one app, to be reverted to later",
that is an hour of work; Qubes is that, made practical.



https://www.qubes-os.org/



'''Qubes on servers'''

You can.

Most Qubes features are really intended for GUI client stuff, 
and if you ignore those parts, you are mostly left with [[Xen]].


And you might like more management, like that of proxmox VE.


That said, there ''are'' cases where qubes would be 
* more robust against specific attacks
* a little more featured than that, e.g. considering [https://www.qubes-os.org/doc/qrexec/ qeexec].




-->