Elasticsearch notes: Difference between revisions
mNo edit summary |
|||
(31 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{#addbodyclass:tag_tech}} | |||
{{search stuff}} | {{search stuff}} | ||
==Choice side== | |||
===Broad intro=== | ===Broad intro=== | ||
{{stub}} | {{stub}} | ||
Line 28: | Line 30: | ||
It's not very hard to interface your own form with the API, though. | It's not very hard to interface your own form with the API, though. | ||
=== | ===Subscription model=== | ||
{{stub}} | {{stub}} | ||
From an "I don't care about jargon and legalese" perspective, | |||
* you can get a managed cloudy thing - split into a few variants | |||
* or you can self-host - ''also'' split into a few variants (not quite the same?) | |||
{{comment|(Before 6.3 the free license needed to be refreshed every year, now basic/free does not need an explicit license anymore?{{verify}})}} | |||
Around self-hosting there are two or three variants: Basic, platinum, and enterprise (Gold is gone?) | |||
: | : Basic is perfectly good for smaller scale | ||
:: | : the differences between platinum and enterprise mostly matter once you do this on large scale [https://www.elastic.co/subscriptions] | ||
: | |||
There is a perfectly free subset of ES (ES configured with a Basic license). | |||
' | ...but the default is bit a Basic license, it's a 30-day trial license in which you get more features, to entice you to buy the fuller product. | ||
If you know you want to stick with basic features, then after this 30-day period (or earlier, if you know what you want) | |||
you would | |||
: not only need to switch to Basic license, | |||
: but also disable some features (apparently all X-pack things{{verify}}) | |||
...but if you didn't know that, this is annoying in the form of "automated install configured a trial license for me, but things just stopped working ''completely'' after 30 days and now I'm panicking", because it takes more than a little reading what you now need to disable and why. {{comment|(My logs showed "Elasticsearch Search and Analytics" -- which seemed to just be a confusing name for the machine learning and alerting stuff)}} | |||
<!-- | |||
For example, | |||
: | * "Current license is non-compliant for search application and behavioral analytics. Current license is active basic license" | ||
: | : is a feature it (confusingly) calls "Elasticsearch Search and Analytics", which is the machine learning and alerting stuff. | ||
: either buy a license, or disabling those features: | |||
xpack.ml.enabled: false | |||
xpack.graph.enabled: false | |||
xpack.watcher.enabled: false | |||
: (note that if this is a "basic license or not?" situation there is more than this to disable) | |||
--> | |||
<!-- | |||
What is X-pack[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-xpack.html]? | |||
An extension that groups | |||
* "[https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html advanced security]" (meaning? TLS for whatever connects directly to the service?) | |||
: | * [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/monitoring-overview.html monitoring], | ||
: | * [https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-settings.html machine learning] | ||
: | ...and possibly a little more? | ||
It ''used'' to be a closed-source set of extended features, | |||
and is now open-sourced but still very still paid-for? | |||
trial for that[https://www.elastic.co/guide/en/kibana/7.17/managing-licenses.html]) | |||
:: is it optional? | |||
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-xpack.html | |||
xpack.security.enabled (in elasticsearch.yml) | |||
: defaults to true | |||
: note that if you enable this, kibana needs some extra config to be able to connect | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html | |||
This gets more complex when considering more varied ES features, and more of the ELK stack, because | |||
: some components are dual-licensed public + proprietary | |||
: | : some components are proprietary only, most of which are paid-only ES features[https://www.elastic.co/subscriptions] | ||
:: though many of them are advanced / scale features that a lot of setups won't need. | |||
Apparently: | |||
* Before 6.3, | |||
: you will need a license - you can get a free basic license at https://register.elastic.co/ | |||
:: note this is is a 1-year license and needs to be refreshed | |||
* Since 6.3, Basic (free) does not need an explicit license | |||
* | |||
xpack.monitoring.collection.enabled | |||
--> | --> | ||
===License details=== | |||
{{stub}} | |||
<!-- | <!-- | ||
There used to be entirely proprietary parts.{{verify}} | |||
Many of those are now source-available ''but'' cannot be used without a subscription, | |||
or taken elsewhere, without some legal implications. | |||
So this is all more source-available rather than open source. | |||
'''Core functionality versus lots of automatic niceness''' | |||
There are a lot of features that you will care about running a large cluster, | |||
and a business off it, and would ''absolutely'' want to pay for. | |||
...they you may not care about if you just want to add search to a hobby project. | |||
This is, understandably, tied into the licensing, and part of the proprietary/open mix. | |||
Broadly, a choice between | |||
* SSPL (a more-viral, less-open variant of the AGPL) | |||
* Elastic License - ''not'' open | |||
So the free version cannot be used/sold as a services with the ES branding removed. | |||
This is related to.... | |||
'''Amazon kerfuffle''' | |||
Around 2021, there was a lisence change from [[Apache2]] to [[SSPL]] - similar to [[AGPL]] but even more aggressive, so people don't quite considered it a free/open license anymore), seemingly aimed specifically at limiting Amazon selling it {{comment|(which just prompted{{verify}} Amazon to fork it into its own OpenSearch[https://en.wikipedia.org/wiki/OpenSearch_(software)] - the SSPL seems made for another such case, MongoDB, and similarly Amazon just forked to DocumentDB{{verify}})}} | |||
--> | --> | ||
==Implementation details you'd want to know to use it== | |||
===The major moving parts=== | |||
=== | ====Some terminology==== | ||
{{stub}} | {{stub}} | ||
{{comment|(note that the core of this comes from the underlying lucene)}} | |||
'''mapping''' - basically a schema, telling ES of how to interpret each field | |||
: e.g. mentions types | |||
: if a field is not already defined in the mapping by you, it will adds fields as you mention them, with types guessed based on the first value it sees | |||
'''field''' - a distinct thing in the mapping | |||
: (usually) indexed to be searchable | |||
: (usually) fetchable | |||
: (there are some field type variants - see below) | |||
'''document''' - individual search result. | |||
: you can see it as the JSON you sent in, and/or as the part of that that got put into fields (and usually indexed) as specified by the mapping | |||
:: side note: ''most'' fields can have zero or more values in a document, though in a lot of practical cases it's practically just one, and in a few cases it's restricted to just one | |||
: original JSON is also stored in _source | |||
'''index''' | |||
: if you squint, a grouping of documents - a collection you search as a whole | |||
: you can have multiple indexes a per cluster (and may well want to) | |||
: an index is divided into one or more shards, which is about replication in general and distribution in a cluster | |||
:: (and shards are divided into segment files, which is more about how updating documents works) | |||
{{comment|...and this is the point at which you can stop reading this section if you're just doing some experiments on one host. You ''will'' get to them once you if and when you need to scale to multiple hosts.}} | |||
'''shard''' | |||
: you typically split an index into a number of shards, e.g. to be able to do [[horizontal scaling]] onto nodes | |||
: internally, each shard is a self-contained searchable thing. In the sense of the complete set of documents you fed in, this is just a ''portion'' of the overall thing we we call index here | |||
: Shards come in two types: primary (the basic type), and replica. | |||
'''segment''' - a shard consists of a number of segments (segments are individual files). | |||
: Each segment file is immutable (which eases a lot of management, means no blocking, eases parallelism, eases cacheing). | |||
'''replica''' is an exact copy of a shard | |||
: the point of replicas is robustness against node failure: | |||
:: if you have two copies of every shard (and those copies are never on the same hardware), then one node can always drop out and search won't be missing anything | |||
:: without duplication, node failure would mean missing a part of every index distributed onto it | |||
: you could run without replicas to save some memory | |||
: memory is unlikely to be much of your cloudy bill (something like ES is CPU and transfer hungry), so using replicas is typically worth it for long term stability | |||
'''node''' - distinct ES server instance | |||
: in larger clusters, it makes sense to have some nodes take on specific roles/jobs - [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/modules-node.html#node-roles this] discusses some of that | |||
'''cluster''' - a group of nodes, serving a set of indices created in it | |||
: | : nodes must mention a shared ID to join a cluster | ||
====Combined with==== | |||
In practice, people often pair it with... {{comment|(see also "{{search|ELK stack}}")}} | |||
* [https://www.elastic.co/kibana Kibana] | |||
: Web UI, makes a bunch of inspection and monitoring easier, including a dashboard interface to ES | |||
: also seen in tutorials, there largely for its '''console''' interactively poking ES without writing code | |||
: itself pluggable, with ''bunch'' of optional things | |||
And, '''if you're not coding your own ingest''', | |||
* [https://www.elastic.co/logstash Logstash] is log processing/aggregator and [https://github.com/logstash-plugins plugins] let it be flexible | |||
: you ''can do'' all of that yourself, but logstash can do a lot of work for you, or ''at least'' be a quick start | |||
* [https://www.elastic.co/beats/ Beats] - where logstash is a more generic, configurable thing, beats is a set of specific-purpose ingest scripts, e.g. for availability, log files, network traffic, linux system metrics, windows event log, cloud service metrics, [https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html] - and a lot more contributed ones[https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html] | |||
: e.g. [https://www.elastic.co/guide/en/beats/metricbeat/7.16/index.html metricbeat], which is stores ES metrics in ES | |||
===Indices=== | |||
====Fields, the mapping, field types==== | |||
{{stub}} | |||
By default, ES picks up every part of the document (remember it's JSON), but you can configure it | |||
:: | : to have parts go towards multiple fields | ||
: to have parts go to no field at all | |||
:: for ''all'' documents based on your mapping having a field but setting "enabled": false, | |||
:: from other behaviour, such as "values should never be bigger than X bytes" config on a field | |||
{{comment|Note that in terms of the index, a document can have multiple values for a field. So fields can be considered to store ''arrays'' of values, and most searches will match if any of those match.}} | |||
Each '''regular field''' (there are others) gets its own behavior in terms of | |||
: whether it is stored and/or | |||
: | : whether it is indexed | ||
: | :: if indexed, ''how'' it is transformed before going into the index | ||
:: | |||
{{comment|(For completeness, a '''runtime field'', is evaluated at runtime, and not indexed or in the _source. | |||
[https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html]. This can be handy for certain calculated data that you won't search on, or to experiment with fields you'll later make regular fields{{verify}})}} | |||
The '''mapping'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html][https://www.elastic.co/blog/found-elasticsearch-mapping-introduction] is the thing that lists all fields in an index, mostly: | |||
** | * name | ||
* | * data type(s) | ||
* configured processing when new data comes in | |||
* and, | '''Explicit mapping'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html], | ||
amounts to specifying a mapping ''before'' you add documents. | |||
This can be preferred when | |||
* you want specific interpretations of some fields (e.g. identifier that shouldn't be parsed as text, an IP address, a date), and/or | |||
* you want specific features (e.g. {{inlinecode|flattened}}, {{inlinecode|join}}, {{inlinecode|search_as_you_type}}) | |||
'''Dynamic mapping'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html] is (like '[[schemaless]]') a nice name for | |||
: "I didn't set a mapping for a field, so am counting on ES guessing right based on the first value it sees" | |||
: if you're mainly handling text, this usually does the right or at least a very sane thing. | |||
{{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic-parameters dynamic-parameters]}} allows a little more variation of that: | |||
:: | : '''true''' - unknown fields are automatically added to the mapping as regular fields (indexed) -- the default just described | ||
:: it'll mostly end up with boolean, float, long, or text/keyword | |||
: '''runtime''' - unknown fields are automatically added to the mapping as ''runtime'' fields (not indexed) | |||
: '''false''' - unknown fields are ignored | |||
: '''strict''' - unknown fields mean the document is rejected | |||
Note that | |||
* '' | * with {{inlinecode|true}} and {{inlinecode|runtime}}, the created field's type is based on the communicated-JSON's type - see [https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html this table] | ||
* {{inlinecode|runtime}} and {{inlinecode|dynamic}} will | |||
** try to detect whether text seems to be a date[https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection] (apparently if it follows a specific configured date template{{verify}}) | |||
** optionally detect numbers[https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#numeric-detection] (and whether they are reals or integers), but this is disabled by default - probably because it's much less error-prone to control this yourself. | |||
* You can also alter mappings later - see [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html] | |||
* | :: but this comes with some restrictions/requirements/footnotes | ||
:: | |||
* ''' | * dynamic-parameters is normally set per index{{verify}} and inherited, but you ''can'' be more precise about it | ||
'''Text-like field types''' | |||
The documentation makes a point of a split into | |||
* '''''text family'''''[https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html] - text, match_only_text | |||
* '''''keyword family'''''[https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type] - keyword, constant_keyword, wildcard | |||
* ...and it seems that e.g. search-as-you-type are considered miscellaneous | |||
* '''{{inlinecode|text}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html] {{comment|(previously known as {{inlinecode|string analysed}})}} | |||
: flexible search of free-form text, analysis can will transform it before indexing (you probably want to know how - see e.g. [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/analysis.html analysis]) | |||
: no aggregations | |||
: no sorting | |||
''' | * '''{{inlinecode|keyword}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html] {{comment|(previously known as {{inlinecode|string not_analysed}})}} | ||
: if | : structured content (exact search only?), e.g. identifiers, a small/known set of tags, also things like emails, hostnames, zip code, etc. | ||
: should be a little faster to match (if only because of smaller index size){{verify}} | |||
: allows sorting | |||
: allows aggregation | |||
: can make makes sense for serial numbers/IDs, tags - even if they ''look'' like numbers, you will probably only ever search them up as text equality (compared to storing those as a number, keyword may be a little larger yet also saves some index complexity necessary for numeric-range search) | |||
* {{inlinecode|constant_keyword}} [https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#constant-keyword-field-type] - all documents in the index have the same value | |||
:: e.g. when you send each log file to its own index, this might assist some combining queries {{verify}} | |||
* {{inlinecode|wildcard}} - [https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type] | |||
:: assists [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html wildcard] and [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html regexp] queries | |||
: | |||
: | |||
''' | * '''search_as_you_type'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-as-you-type.html] | ||
: | : fast for prefix (match at start) and infix (terms within) matches | ||
: mainly for autocompletion of queries, but could work for other shortish things (if not short, the index for this will be large) | |||
: n-gram style, can have larger n (but costs in terms of index size, so only useful if you usually want many-token matches?) | |||
: kept in memory, costly to build | |||
' | Further notes: | ||
: | * it seems that fields that are dynamically added to the mapping and detected as text will get two indexed fields: a free-form text one, and keyword with ignore_above of 256 | ||
: not | :: this is useful if you don't know what it will be used for | ||
:: but for e.g. identifiers it's pointless to also tokenize it | |||
:: and for free-form text it will probably do very little -- that is, for all but short documents it ends up _ignored. {{comment|(It's a clever edge-caes trick to deal with cases where the only value is actually something other than text, and is ''otherwise'' almost free)}} | |||
:: separately, some things you may wish to not index/serch on, but still store it so you can report it as part of a hit | |||
''' | '''Data-like field types (primitives)''': | ||
* '''boolean'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/boolean.html] | |||
''' | * '''numbers''': [https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html] | ||
: | ** '''byte''', '''short''', '''integer''', '''long''', '''unsigned_long''' | ||
: | ** '''float''', '''double''', '''half_float''', '''scaled_float''' | ||
:: allows range queries | |||
* and, arguably, keyword (see above) | |||
'''Data-like fields (specifics but text-like)''': | |||
* '''date''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html] | |||
:: (internally milliseconds since epoch (UTC) | |||
''' | * '''date_nanos''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/date_nanos.html] | ||
: | |||
''' | * '''version''' - text that represents semantic versioning | ||
: | :: mostly for sortability? | ||
* '''ip''' - IPv4 and IPv6 addresses | |||
:: allows things like subnet searches ([[CIDR]] style) | |||
* '''geospatial''' - points and shapes [https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html] | |||
:: including distance and overlap | |||
Different way of treating JSON as transferred | |||
* | * '''object'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html] - | ||
:: each subfield is separately mapped and indexed, with names based on the nesting dot notation, see e.g. the linked example | |||
* '''flattened'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html] | |||
:: takes the JSON object and indexes it as one single thing (basically an array of its values combined) | |||
* '''nested'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html] basically a variant of object that allows some field indexability | |||
- | |||
Other stuff | |||
* '''binary'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html] | |||
:: not searchable | |||
:: not stored (by default) - which makes sense. A search index is not a blob store | |||
:: seems to be there for the possibility of some plugin extracting text to index?{{verify}} or fishing it out of _source?{{verify}} | |||
''' | * '''join'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html] | ||
:: relations to other documents, by id{{verify}} | |||
:: seems to mean query-time lookups, so | |||
::: you could get some basic lookups for much cheaper than separate requests | |||
::: at the same time, you can make things ''slower'' by doing unnecessary and/or multiple levels of lookups (if you need relational, a relational database is better at that) | |||
'''token_count'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/token-count.html] | |||
: stored as an integer, but takes text, analyses it into tokens, then counts the number of tokens | |||
: seems intended to be used via multi-field, to also get the length of text | |||
'''dense_vector'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html] (of floats by defaults, byte also possible) | |||
: if indexes, you can use these for knn searches[https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html] | |||
<!-- | |||
'''rank_feature'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/rank-feature.html] | |||
: basically a number usable to boosts a document in ''any'' search{{verify}} | |||
: used with [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-rank-feature-query.html rank_features] queries | |||
'''rank_features'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/rank-features.html] | |||
: | |||
'''histogram'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/histogram.html] | |||
: | : a length-matched pair of {{inlinecode|values}} (double) and counts (integer) | ||
: | : not indexed | ||
: (so) seems mostly useful as pre-aggregated data, and usable by only some aggregation functions | |||
''' | '''percolator'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/percolator.html] | ||
: | |||
'''sparse_vector'''[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/sparse-vector.html] | |||
: usable for scoring | |||
: deprecated | |||
--> | |||
====Managing indices (and thinking about shards and segments)==== | |||
{{stub}} | |||
Remember that shards are made of segments, and segments are immutable. | |||
ES collects changes in memory, and only occasionally dumps that that into a new segment file. | |||
: (this is why documents are not ''immediately'' searchable and we call it [https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html ''near''-realtime]) | |||
'''Refresh''' - is that act of writing to a new segment | |||
: also helps updates not be blocking - existing segments are searched, new one once they're done | |||
'''Merge''' refers to how smaller segments are periodically consolidated into fewer files, [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules-merge.html] | |||
: also the only way that delete operations actually flush (remember, segments are immutable). | |||
Refresh interval is 1s by default, but various setups may want that larger - it's a set of tradeoffs | |||
* Refresh can be too quick in that the overhead of refresh would dominate, and less work is spent actually doing useful things (and be a little pricier in pay-per-CPU-use cloudiness) | |||
* Refresh can also be too slow both in that | |||
** new results take a ''long'' time to show up | |||
** the heavier load, and large new segments, could make fore more irregular response times | |||
The default is 1s. Large setups might increase that up to the order of 30s. | |||
(more in the tradeoffs below) | |||
---- | |||
'''index lifecycle management''' (ILM) lets you do slightly higher-level management like | |||
* create a new index when one grows too large | |||
* do things like creating an index per time interval | |||
Both can help get the granularity you want when it comes to | |||
backing them up, | |||
duplicating them, retiring them to adhere to data retention standards. | |||
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/ | https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-lifecycle-management.html | ||
Also on the topic: a '''reindex''' basically means "read data, delete data in ES, ingest again". | |||
: You would not generally want this over a merge. | |||
: It's mainly useful when you make structural schema changes, and you want to ensure the data uniformly conforms to that. | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html | |||
====other field types==== | |||
'''multi-fields'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html] | |||
let you specify that one thing in your JSON should be picked up towards multiple fields. | |||
* e.g. a city name | |||
** as a keyword, for exact match, sorting and aggregation | |||
** and towards the overall text search, via an analyser | |||
* text as a searchable thing, and also store its length via token_count | |||
'''runtime fields''' [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/runtime.html] | |||
: mostly about query time - not part of the index (so won’t display in _source) | |||
: just calculated as part of the response | |||
: can be defined in the mapping, or in an individual query | |||
: (seem to be a refinement of script fields?) | |||
: easy to add and remove to a mapping (because there is no backing storage for them) | |||
<!-- | |||
: when to use them? | |||
:: on-the-fly aggregations, filtering, and sorting | |||
::: ...on fields you rarely do that on. Things you search and filter all the time should probably be indexed | |||
--> | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html | |||
: https://www.elastic.co/blog/getting-started-with-elasticsearch-runtime-fields | |||
: you can search specific indices (comma separated names, or _all), or just one - sometimes useful | |||
'''Metadata fields''' (that is, those other than e.g. the fields from your document, and runtime/script fields) | |||
* '''{{inlinecode|_id}}''' - document's ID | |||
* '''{{inlinecode|_type}}''' - document's mapping type | |||
: deprecated | |||
* '''{{inlinecode|_index}}''' - index that the document is in | |||
* '''{{inlinecode|_source}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/7.16/mapping-source-field.html] | |||
: is the original JSON that indexing got | |||
:: note that everything needs to be indexed | |||
:: this 'original document' is also used in update and reindex operations | |||
::: so while you can ask it to not store _source at all, that disables such operations | |||
:: that original submission can be handy for debug | |||
:: that original submission can be core functionality for your app in a "we found it by the index, now we give you the actual document" way | |||
: you can filter what parts of _source are actually sent in search results - see [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/search-fields.html#source-filtering source filtering] | |||
* '''{{inlinecode|_size}}''' - byte size of _source | |||
* '''{{inlinecode|_ignored}}''' - fields in a document that indexing ignored for any reason - see | |||
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/mapping-fields.html others | |||
====More on indexing==== | |||
{{stub}} | |||
The [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html bulk API] ''could'' be used for all CRUD operations, | |||
but in practice is probably ''mostly'' used for indexing. | |||
=====Do bulk adds where possible===== | |||
While you can do individual adds, doing bunches together reduces overheads, | |||
and can make refreshes more efficient as well, | |||
largely relating to the refresh interval. | |||
However, keep in mind that if some part of a bulk operations fails, you need to notice, and deal with that correctly. | |||
=====Missing data===== | |||
Setting a field to null (or an array of nulls, or an empty array) means it is not added to the index. | |||
Sounds obvious, but do a mental check that "that field can not be matched in a lot of searches" | |||
(except for missing, must_not exists, and such{{verify}}) is the behaviour you intend. | |||
Note that it also affects aggregation on the relevant fields, and some other details. | |||
=====Ignoring data and fields===== | |||
Field values may be ignored for varied reasons, including: | |||
* use of [https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html ignore_malformed] | |||
:: by default, using a type that can't be converted means the entire '''rejects the document''' add/update operation | |||
::: though note that in bulk updates it only considers that one operation failed, and will report a partial success | |||
:: if you set {{inlinecode|ignore_malformed}}[https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html], it will instead reject only bad values, and process the rest normally | |||
* [https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html ignore_above] (on keyword fields) | |||
You can search for documents where this happened ''at all'' {{comment|(you'd probably do that for debug reasons)}} with a query like | |||
"query":{ "exists":{"field":"_ignored"} } | |||
<!-- | |||
Values like NaN are not supported (by the JSON parser?) | |||
If you're working from e.g. pandas, you have some extra RTFMing to do for processing on ''that'' side, because it's e.g. likely to add NaN in text (object) fields to have NA, and you'll have to filter those out before feeding it to ES. | |||
res = df.apply(lambda x: x.fillna(0) if x.dtype.kind in 'biufc' else x.fillna('.')) | |||
--> | |||
===== | =====Text processing===== | ||
You want curacao to match Curaçao? Want fishes to match fish? | |||
To computers those are different characters and different strings, | |||
whether such variations are semantically equivalent, or close enough to be fuzzy with, and ''how'', might vary per language. | |||
Analysers represent relatively simple processing that helps, among other things, normalize data (and implicity the later query) for such fuzziness. | |||
<!-- | |||
"Can't I do this myself?" | |||
Sure. Note that the the _source will now contain your own normalization, so if you also report data (e.g. fulltext) out of ES, that will now not be original. | |||
--> | |||
An ''analyzer'' is a combination of | |||
* a ''character filter'' | |||
* a ''tokenizer'' | |||
* a ''token filter'' | |||
(a normalizer seems to be an analyzer without a tokenizer - so practically mainly a character filter (can also have a token filter but this is probably not so useful) | |||
'''Analyzers''' takes text, and applies a combination of character filters, tokenizer, and token filters, usually to the end of | |||
* | * split it into tokens (e.g. words) | ||
* stripping out things (e.g symbols and punctuation) | |||
* normalize (e.g. lowercasing) | |||
There are '''built-in analyzers''' to cover a lot of basic text search needs | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html standard]}} (the default if you don't specify one) | |||
** lowercases | |||
** splits using [[Unicode Text Segmentation]] (for english is mostly splitting on spaces and punctuation, but is better behaved default for some other languages) and lowercases | |||
** removes most punctuation, | |||
** can remove stop words (by default does not) | |||
* [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html language-specific analysers] - currently arabic, armenian, basque, bengali, brazilian, bulgarian, catalan, cjk, czech, danish, dutch, english, estonian, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, and thai. | |||
:: details vary | |||
:: can remove stopwords | |||
:: can apply stemming (and take a list of exception cases to those stemming rules) | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simple-analyzer.html simple]}} | |||
:: no lowercasing | |||
:: splits on non-letters | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stop-analyzer.html stop]}} - simple plus stopword removal, so | |||
:: no lowercasing | |||
:: splits on non-letters | |||
:: can remove stopwords | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html pattern]}} - splits by regexp | |||
:: can lowercase | |||
:: can remove stopwords | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-whitespace-analyzer.html whitespace]}} | |||
:: no lowercasing | |||
:: splits on whitespace | |||
* {{inlinecode|[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-fingerprint-analyzer.html keyword]}} - does nothing, outputs what it was given | |||
* [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-fingerprint-analyzer.html fingerprint] | |||
:: reduces text in a way that helps detect duplicates, see something like [https://openrefine.org/docs/technical-reference/clustering-in-depth] | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html | |||
===Data streams=== | |||
<!-- | |||
When you start to do metrics, and daily analytics, you append all the time. | |||
Your data may ''amounts'' to a timeseries, | |||
and storing it in a classical index is ''possible'', | |||
but awkward and its size would add up slowly but surely. | |||
If you were to make that work without data streams, that might amount to | |||
: use ILM to start a new index daily (doing what amounts to [[log rotation]]) | |||
: use index aliases help keep your query's reference to an index the same over time, rather than changing them daily too | |||
: use ILM to move old indices to cheaper archival hardware | |||
A data stream amounts to a different sort of storage, | |||
an append-only time series thing (that still seems backed by indices internally) | |||
that requires [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html index templates]. | |||
append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are .. | |||
Data streams are sort of that but more automated (though you still configure rollover with ILM) and controlled (append-only, and you can only append to the last), which makes it a lot easier to deal with continuously-generated data streams. | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html | |||
https://aravind.dev/elastic-data-stream/ | |||
--> | |||
===Snapshots, Searchable snapshots=== | |||
<!-- | |||
Snapshots let you | |||
* back up a cluster live | |||
* | |||
* | * recover after losing files | ||
* | * searchable snapshots[https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html] - low search volume on things basically archived to cheaper and slower-to-access storage [https://www.elastic.co/guide/en/elasticsearch/reference/current/put-snapshot-repo-api.html] | ||
: | |||
Snapshots can go to e.g. | |||
: | : e.g. shared filesystem. | ||
: Plugins provide [https://www.elastic.co/guide/en/elasticsearch/plugins/7.16/repository-s3.html S3], [https://www.elastic.co/guide/en/elasticsearch/plugins/7.16/repository-gcs.html GCS], [https://www.elastic.co/guide/en/elasticsearch/plugins/7.16/repository-azure.html Azure]), [https://www.elastic.co/guide/en/elasticsearch/plugins/7.16/repository-hdfs.html HDFS] | |||
A snapshots contains | |||
: cluster state (settings, templates, pipelines) | |||
: all data streams | |||
: all open indices | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html | |||
--> | |||
===Install=== | |||
<!-- | |||
You might want to consider going the docker way - it makes cluster setup easier. | |||
"max virtual memory areas vm.max_map_count is too low" | |||
See <tt>kernel-doc/Documentation/sysctl/vm.txt</tt>, but | |||
these relates to the amount of allocations (also mmap, mprotect) done. | |||
The default is 65530, most processes use fewer than a thousand, | |||
but e.g. malloc debuggers need more, and so does ElasticSeaarch for some reason. | |||
--> | --> | ||
=== | ====Things you may want to think about somewhere before you have a big index==== | ||
<!-- | <!-- | ||
Note that for a bunch of of things - not least importantly indexes - there are static settings (only settable at creation time) and dynamic settings (changeable later) | |||
* | * '''index template''' | ||
: basically the settings that new indexes shouldbe created with | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html | |||
Index settings [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules.html] | |||
* Refresh | |||
: background: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/near-real-time.html | |||
: interval setting: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-refresh-interval-setting | |||
* Merge scheduling | |||
: | : [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules-merge.html] | ||
: | |||
* Analysis of text fields | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/analysis.html | |||
Mapping - in particular whether you want to control it rather than use dynamic mapping [https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules-mapper.html] | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/ | More advanced: | ||
* Sharding details in clusters: | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-modules-allocation.html | |||
: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/allocation-total-shards.html | |||
Consider whether you need security | |||
ES has | |||
* IP filtering | |||
* its own user auth | |||
:: role-based access control | |||
* auditing | |||
* TLS (towards clients) | |||
These things are mostly useful around multitenancy, | |||
or when your ES deployment is on a different provider (e.g. public cloud deployment), | |||
and may not be necessary within your own cloudiness. | |||
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/configuring-security.html | |||
ES/Kibana security | |||
* | Minimal | ||
* user passwords for ES | |||
* optional passwords for Kibana | |||
* | Basic Security | ||
* TLS between nodes, isolates clusters | |||
* No TLS for Kibana | |||
* | Basic + HTTPs external | ||
* TLS (HTTPS) on all ES and Kibana, Kibana | |||
--> | |||
===APIs=== | |||
{{stub}} | |||
ES grew a lot of APIs over time. | |||
Perhaps the most central are | |||
* the document CRUD part (Index, Get, Update, Delete, respectively), | |||
* and the searching. | |||
'''Document CRUD''' | |||
* '''Index''' - add a document (and implicitly get it indexed soon) | |||
:: e.g. {{inlinecode|PUT /test/_doc/1}} with the JSON doc in the body | |||
* '''Get''' - get a specific document by ID | |||
: e.g. GET test/_doc/1 {{comment|(HEAD if you only want to check that it exists)}} | |||
: this basically just gives you {{inlinecode|_source}}, so you can filter what part of that gets sent[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html#get-source-filtering] | |||
* '''Update'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html] | |||
:: e.g. {{inlinecode|POST /test/_update/1}} | |||
: allows things like <tt>"script" : "ctx._source.votes += 1"</tt> | |||
* | * '''Delete'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html]) | ||
:: | :: e.g. {{inlinecode|DELETE /test/_doc/1}} | ||
Note that there is also a '''bulk API''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html] | |||
: that lets you do many of the above at once | |||
: POST _bulk | |||
: where the body can contain many index, delete, update | |||
: libraries often support the bulk API in a way that makes sense for how the rest of that library | |||
"Is update really update?" | |||
It seems that update is essentially | |||
* construct a a new document based on the current version's _source | |||
* add 1 to the version | |||
* save that to a new segment | |||
:: ...even if it turns out there is no difference (...by default - you can specify that it should check and not do that - see detect_noop) | |||
It seems that | |||
the old version will be ''considered'' deleted but still be on disk in that immutable segment it was originally entered in (until the next merge), | |||
and searches just happen to report only the latest (using _version){{verify}} | |||
So if you do continuous updates on all of your documents, you ''can'' actually get the update process to fall behind, which can mean you temporarily use a lot more space (and search may be slightly slower too). | |||
"Is there a difference between indexing a new version and updating the document?" | |||
Very little in terms of what gets done at low level (both will create a new document, mark old as deleted) | |||
It's most that ''for you'' there is a question of what you do is easier to express as | |||
* a new version of the data | |||
* the changes you want to see, or a script that makes them | |||
Also, keep in mind ''how'' you are updating. If it's fetch, change, send, that's more steps than 'here is how to change it' in a script. | |||
So if you do not keep track of changes to send to elasticsearch, | |||
you could update by throwing everything at at with detect_noop - which only applies to update. | |||
Search[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html] | |||
: | : GET /indexname/_search | ||
Multi Search[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html] | |||
:: | : multiple searches with a single API call | ||
: GET /indexname/_msearch | |||
:: ...though the request can switch index | |||
Async search | |||
Multi Get[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-multi-get.html] | |||
: Retrieves multiple JSON documents by ID. | |||
: GET /_mget | |||
: | |||
<!-- | |||
helpers.bulk | |||
helpers.streaming_bulk | |||
helpers.parallel_bulk | |||
python wrapper | |||
Index creation | |||
* https://elasticsearch-py.readthedocs.io/en/v7.16.3/api.html#indices | |||
--> | |||
===Search=== | |||
====Shapes of a query==== | |||
{{stub}} | |||
Most APIs take a query that looks something like: | |||
<syntaxhighlight lang="javascript"> | |||
{ | |||
"query": { | |||
"match_phrase": { | |||
"fieldname": "word word" | |||
} | |||
} | |||
} | |||
</syntaxhighlight > | |||
That is, | |||
there is a specific [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html Query DSL], | |||
where you expressing a structured search as an abstract syntax tree, which is communicated as JSON. | |||
Notes: | |||
* the field names will vary with specific types of search | |||
:: In fact even the nesting will | |||
* You might like [https://www.elastic.co/guide/en/kibana/7.17/console-kibana.html Kibana's console] to play with these, as it has some auto-completion of the search syntax. | |||
* a quick "am I even indexing" test would probably use {{inlinecode|match_all}}[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html] | |||
'''Exceptions / other options''' | |||
The only exception to "AST as JSON" seems to be '''URI search'''[https://www.elastic.co/guide/en/elasticsearch/reference/master/search-uri-request.html] which exposes basic searches in a URL (without needing to send a JSON style query in the body), but unless this covers ''all'' your needs, you will eventually want to abandon this. Might as well do it properly up front)}} | |||
The only exceptions to "you must create query ASTs yourself" seem to be: | |||
* '''<tt>query_string</tt>'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html] - | |||
: still uses that JSON, but feeds in a single query string in that specific lucene syntax, to be parsed internally | |||
:: exposes a bunch of the search features in a shorter-to-write syntax | |||
::: ...that you have to learn | |||
::: ...that is fairly breakable | |||
::: ...that allows frankenqueries that unduly load your system | |||
::: ...so you probably don't want to expose this to users unless all of them are power users / already know lucene | |||
* '''<tt>simple_query_string</tt>'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html] | |||
: same idea as the previous, but using the somewhat more simplified syntax | |||
:: less breaky, still for experts | |||
...but again, until this covers all your needs ''and'' doesn't ope | |||
These exceptions may looks impler but in the long run you will probably need to abandon them. | |||
If you want to learn it once, learn the AST way. | |||
(that sais, a hacky translation to the latter syntax is sometimes an easier shortcut for the short term). | |||
====API-wise==== | |||
The '''search API'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html] ({{inlinecode|/''indexname''/_search}}) should cover many basic needs. | |||
: The '''search template API'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template-api.html] does the same sort of search, but you can avoid doing some complex middleware/client-side query construction by asking ES to slot some values into an existing saved template. | |||
'''Multi search'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html] ({{inlinecode|/''indexname''/_msearch}}) is sort of like bulk-CRUD but for search: you send in multiple search commands in one request. Responds with the same number of result sets. | |||
: '''Multi search template API'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-search-template.html] - seems to just be the combination of multi search and use of templates. | |||
=== | =====On result state, and getting more consistency between search interactions===== | ||
In search systems, it can be a good tradeoff to | |||
: get a good indication of how many there roughly, and stop counting when the number amounts to "well, um, ''many''" -- rather than get a precise count | |||
: only fetch-and-show the complete data for only the first few -- rather than everything | |||
: forget the search immediately after serving it -- rather than keeping state around in RAM and/or on disk, for who knows how long exactly | |||
...because it turns out we can do that faster. | |||
This is ES stops counting after 10000 (max_result_window), only return the first 10 (size=10, from=0) | |||
Also, if search-and-fetch-a-few turns out to be cheap in terms of IO and CPU, we can consider doing them without storing state. | |||
This also because we know that in interactive browser use, most people will never check more than a few. | |||
If in fact someone ''does'' the somewhat unusual thing of browing to page 2 or 3 or 4, | |||
you can redo the same search and fetch some more (use {{inlinecode|from}} to fetch what ''amounts'' to the next bunch). | |||
'''However''', if the index was refreshed between those actions, this new search will shift items around, | |||
so might get the same item again, or never show you are didn't see one one. | |||
: | Whenever the consistency or completeness really matters, you are probably looking for async or PIT: | ||
Also, if you use this to back an API that allows "fetching everyting", you won't have won much. | |||
'''Async search'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html] - lets you start searches in the background. | |||
Search functionality is mostly equivalent, there's some minor parameter differences (think cache) | |||
: the searches are store din their own index{{verify}} so you may want a mechanism to delete the searches by ID if you need to, and/or lower the time these are kept (default is 5 days, see keep_alive) | |||
:: note that you {{inlinecode|wait_for_completion_timeout}} lets you ask for "return regular search if you finish quickly, make it async if not" | |||
'''Point in time API'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html] - consider that document updates and refreshes means you will generally get the ''latest'' results. | |||
If instead it is more important to get a consisten set, you can use PIT (or presumably async search?) | |||
<!-- | |||
Consider efficiency | |||
It helps speed/load to try to express queries in simpler queries, like | |||
term and match (and most match_ variants), | |||
( | and compounds of such. | ||
In general, you may wish to stick to the more-efficient-to-evaluate where possibly; | |||
somewhat more costly (up-front), | |||
and require search.allow_expensive_queries to be true | |||
--> | |||
=====Searching multiple indices===== | |||
...mostly points to Multi search (_msearch), which looks something like: | |||
<syntaxhighlight lang="javascript"> | |||
GET /index1/_msearch | |||
{ } | |||
{ "query": {"multi_match":{ "query":"fork", "fields":["plaintext"] } } } | |||
{"index": "index2"} | |||
{ "query": {"multi_match":{ "query":"fork", "fields":["title","plaintext"] } } } | |||
{"index": "index3"} | |||
{ "query": {"multi_match":{ "query":"fork", "fields":["plaintext"] } } } | |||
</syntaxhighlight > | |||
Note that you can also do: | |||
<syntaxhighlight lang="javascript"> | |||
GET /index1,index2,index3/_search | |||
{ "query": { "multi_match":{"query":"fork", "fields":["title","plaintext"] } } } | |||
</syntaxhighlight > | |||
On the upside, you don't have to creatively merge the separate result sets (because multi search ''will not do that for you''). | |||
On the downside, you don't get to control that merge, e.g. scoring can be difficult to control, you don't really get to specify search fields per index anymore, or source filtering fields - though some forethought (e.g. to field naming between indices) can help some of those. | |||
=====Other, mostly more specific search-like APIs, and some debug stuff===== | |||
'''knn'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search-api.html] search searches a dense_vector close to the query vector. | |||
: separate API is depracated, being moved to [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-api-knn an option] | |||
'''suggester'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html] - suggests similar search terms based on [[edit distance]] | |||
'''terms_enum'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-terms-enum.html] - | |||
''' | '''count'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-count.html] | ||
''' | '''explain'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-explain.html] | ||
'''profile'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html] | |||
'''validate'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-validate.html] | |||
'''shards'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-shards.html] - report which shards would be accessed | |||
====composing queries==== | |||
=====Compound queries===== | |||
{{stub}} | |||
Compound queries[https://www.elastic.co/guide/en/elasticsearch/reference/current/compound-queries.html] wrap others (leaf queries, or other compound queries), for | |||
One reason is logical combinations of other queries {{comment|(note that when you do this for filtering only, there are sometimes other ways to filter that might be more efficient in specific situations)}} | |||
'''{{inlinecode|bool}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html] | |||
:: lets you combine requirements with requirements like | |||
::: {{inlinecode|must}} | |||
::: {{inlinecode|filter}} is <tt>must</tt> without contributing to scoring | |||
::: {{inlinecode|must_not}} | |||
::: {{inlinecode|should}} ('portion of leafs must', and more matches scores higher) | |||
The rest is mostly about more controlled scoring (particularly when you make such combinations) | |||
'''{{inlinecode|dismax}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-dis-max-query.html] | |||
:: if multiple subqueries match the same document, the highest score gets used | |||
'''{{inlinecode|boosting}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-boosting-query.html] | |||
:: have subqueries weigh in positively and negatively | |||
'''{{inlinecode|constant_score}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-constant-score-query.html] | |||
:: results from the search this wraps all get a fixed constant score | |||
'''{{inlinecode|function_score}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html] | |||
:: results form the search this wraps with scripting that can consider values from the document and query | |||
=====Term-level and full-text queries===== | |||
{{stub}} | |||
'''Term queries''' are''not'' put through an analyser, so stay one string, so are mostly used to match exact strings/terms. | |||
'''Full-text queries''' are put through an [[#More_on_indexing|analyzer]] (the same way as the text fields it searches), which does some alterations, and means the search then deals with multiple tokens. | |||
''' | '''{{inlinecode|term}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html] | ||
: exact value | |||
: works in text fields but is only useful for fairly controlled things (e.g. username) | |||
'''{{inlinecode|terms}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html] | |||
: like term, matches ''any'' from a provided list | |||
'''{{inlinecode|terms_set}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-set-query.html] | |||
: like terms, but with a minimum amount to match from a list | |||
''' | '''{{inlinecode|exists}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html] | ||
: Returns documents that contain any indexed value for a field. | |||
''' | '''{{inlinecode|fuzzy}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html] | ||
: Returns documents that contain terms similar to the search term. Elasticsearch measures similarity, or fuzziness, using a Levenshtein edit distance. | |||
''' | '''{{inlinecode|ids}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html] | ||
: Returns documents based on their document IDs. | |||
''' | '''{{inlinecode|prefix}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html] | ||
: Returns documents that contain a specific prefix in a provided field. | |||
'''{{inlinecode|range}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html] | |||
: Returns documents that contain terms within a provided range. | |||
'''{{inlinecode|regexp}}'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html] | |||
{{ | : Returns documents that contain terms matching a regular expression. | ||
'''{{inlinecode|wildcard}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html] | |||
: Returns documents that contain terms matching a wildcard pattern. | |||
{{inlinecode|match}} [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html] | |||
:: | : matches text, number, date, or boolean data | ||
::: {{ | : in a single field | ||
:: | : optionally does things like phrase or proximity queries, and things like edit-distance fuzziness | ||
: text is analysed; all tokens are searched - effectively defaults to OR of all terms (and/so apparently minimum_should_match=1) | |||
{ | |||
"query": { | |||
"match": { | |||
"fieldname":{ | |||
"query": "scary matrix snowman monster", | |||
"minimum_should_match": 2 //optional, but | |||
} | |||
} | |||
} | |||
} | |||
'''{{inlinecode|multi_match}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html] | |||
: match, but searches one query in multiple fields | |||
{ | |||
"query": { | |||
"multi_match" : { | |||
"query": "this is a test", | |||
"fields": [ "subject", "message" ] | |||
} | |||
} | |||
} | |||
'''{{inlinecode|combined_fields}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-combined-fields-query.html] | |||
: somewhat like multi_match, but instead of doing the query to each field separately, it acts as if it matched on a single field (that consists of the mentioned fields combined), useful when the match you want could span multiple fields | |||
{ | |||
"query": { | |||
"combined_fields" : { | |||
"query": "database systems", | |||
"fields": [ "title", "subject", "message"], | |||
"operator": "and" | |||
} | |||
} | |||
} | |||
'''{{inlinecode| | '''{{inlinecode|match_phrase}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html] | ||
: | : analyses into tokens, then matches only if ''all'' are in a field, in the same sequence, and by default with a slop of 0 meaning they must be consecutive | ||
{ | |||
"query": { | |||
"match_phrase": { | |||
"message": "this is a test" | |||
} | |||
} | |||
} | |||
'''{{inlinecode| | '''{{inlinecode|match_phrase_prefix}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase-prefix.html] | ||
: | : like match_phrase, but the ''last'' token is a prefix match | ||
'''{{inlinecode| | '''{{inlinecode|match_bool_prefix}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-bool-prefix-query.html] | ||
: | : constructs a {{inlinecode|bool}} of term-{{inlinecode|should}}s, but the last last is a prefix match | ||
''' | '''{{inlinecode|intervals}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-intervals-query.html] | ||
: imposes rules based on order and proximity of matching terms | |||
'''{{inlinecode|query_string}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html] | |||
: lucene-style query string which lets you specify fields, and AND,OR,NOT, in the query itself [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax] | |||
: maybe don't expose to users, because it's quick to error out | |||
'''{{inlinecode|simple_query_string}}''' [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html] | |||
=====Span queries===== | |||
''' | <!-- | ||
'''Span queries'''[https://www.elastic.co/guide/en/elasticsearch/reference/current/span-queries.html] | |||
use token-position information to be able to express terms's order, proximity , | |||
This seems mostly useful technical documents, | |||
and other things that seem to conform to templates enough that this specificity is meaningful (rather than just rejects a lot) | |||
* span_term[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-term-query.html] - like regular term, but works when combining span queries with some of the below | |||
'' | * span_containing[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-containing-query.html] | ||
: | : takes list of span queries, returns only those spans which also match a second span query | ||
: similar to near but more controlled? | |||
* span_near[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-near-query.html] - multiple span queries must match within distance, and with optional order requirement | |||
* span_or[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-or-query.html] - match by ''any'' span query | |||
* span_not[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-not-query.html] - excludes based on a span query | |||
* span_first[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html] - appear within the first N positions of field | |||
* span_field_masking[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-field-masking-query.html] - lets you do span-near or span-or across fields | |||
: | * span_multi[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-multi-term-query.html] - Wraps a term, range, prefix, wildcard, regexp, or fuzzy query. | ||
* span_within[https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-within-query.html] - one span must appear within another | |||
--> | |||
====Result set==== | |||
<!-- | |||
* size - how many hits to actually return | |||
:: together with from, you can imitate pages | |||
:: from & size don't allow fetching more than 10000 documents. If you need to, look at [https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after] | |||
:: ...also, these seem to be distinct searches{{verify}}, so you're assuming index doesn't change; if you care about that, consider async searches | |||
* sort | |||
* search_type - how distribution is handled | |||
:: query_then_fetch - scoring per shard | |||
:: dfs_query_then_fetch - overall scoring. More precise but slower. | |||
'''Source filtering, stored fields, runtime fields''' | |||
Values present in the | |||
fields is a little more controlled/flexible | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html | |||
In queries: | |||
* fields | |||
_source is the complete document as ingested. | |||
Source filtering refers to asking for fewer fields from that. | |||
The upside is that it means less transfer. | |||
The downside is that _source needs to be parsed before it can be filtered, which for larger/more complex documents can add up. | |||
In queries: | |||
* _source - what fields of the _source you want returned in the results. can be true (all, default), false (none), or a comma separated list of field names | |||
: (see also _source_includes, _source_excludes) | |||
Stored fields[https://www.elastic.co/guide/en/elasticsearch/reference/7.15/search-fields.html#stored-fields] are stored in the index, and let you say "don't send _source at all, just these specific stored fields are enough for me" | |||
If _source is large, this can save parsing time (and traffic for anyone who forgets to do source filtering). | |||
The downside is that it duplicates data, so if there is little improvement, then it's just bloat. | |||
(stored fields are also something you want when you don't save _source at all - see the indexing section) | |||
In queries: | |||
* stored_fields - what stored-only fields to also return? | |||
: interacts with the source_ | |||
There are also doc value fields, [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#docvalue-fields] | |||
In queries: | |||
* docvalue_fields | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html | |||
https://www.elastic.co/guide/en/elasticsearch/reference/7.15/search-fields.html#stored-fields | |||
--> | |||
====Aggregation==== | |||
<!-- | |||
Aggregation is any calculation from multiple documents. | |||
Which can be particularly useful when you store structured information - like metrics. | |||
It is not as commonly used in full-text search. | |||
Grouped into | |||
* Metric aggregations[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html] - whole-set things like | |||
: min, max, avg, percentile, boxplot, ardinality, stats, extended_stats, geo_bounds, geo_centroid, matrix_stats | |||
: t_test | |||
: rate, | |||
: text: count, min_length, max_length, avg_length, entropy | |||
: ...and more | |||
* Bucket[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html] - per-group-of-documents things like | |||
: missing | |||
: sampler | |||
: adjacency_matrix, histogram, date_histogram, auto_date_histogram, | |||
: categorize_text, significant_terms, significant_text, terms (...counter) | |||
: range | |||
: filter | |||
: geo_distance (and other geo things) | |||
: children, nested | |||
* Pipeline[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html] - take input from aggregations | |||
: bucket_selector | |||
: bucket_sort | |||
: avg_bucket, sum_bucket | |||
: stats_bucket, extended_stats_bucket | |||
: cumulative_sum | |||
: cumulative_cardinality | |||
: max_bucket, min_bucket | |||
: moving_avg, moving_percentiles, moving_fn, | |||
: serial_diff | |||
: percentiles_bucket | |||
: bucket_correlation | |||
: bucket_count_ks_test | |||
: normalize | |||
: inference | |||
: derivative | |||
: bucket_script | |||
: ...and more | |||
https://logz.io/blog/elasticsearch-aggregations/ | |||
--> | |||
==further notes== | |||
===Performance considerations=== | |||
<!-- | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html | |||
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster | |||
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster | |||
--> | |||
====query considerations==== | |||
<!-- | |||
'''Search only the fields you need'''. | |||
Searching more fields (via query_string or multi_match) ''roughly'' means | |||
a number of separate searches | |||
that are then merged. | |||
If you ''quite typically'' want to search the same set of multiple fields (that you also want to be able to search separately), | |||
it may be worth it to have a single field that contains the combination. | |||
: You can do that without duplicating things in _source by using copy-to in the mapping (probably ''only'' gets values from being copied into) | |||
'''Filtering''' search, without scoring (and sorting by that score) will tend to be faster. | |||
: If you're mostly just aggregating data, avoid scoring. | |||
: ...yet in a text search engine, scoring tends to be fairly central | |||
'''Consider converting a subset of search specifications to queries yourself, instead of exposing arbitrary searching to users''' | |||
They ''will'' at some point manage a franken-query that takes CPU-minutes. | |||
In more expert cases, you probably can't do much about this - more flexibility is a feature. | |||
But consider whether "present nice form, the have code decide query that isn't going to spend CPU-minutes" is a thing you can do. | |||
--> | |||
====More general settings==== | |||
<!-- | |||
'''Lower refresh interval if a minute later is also acceptable''' | |||
: | Increasing from the default 1sec ("I want it now") to something on the order of 10 to 30 sec reduces the overhead that comes from this work, | ||
(a little more so when not using bulk import). | |||
Exactly how many documents to add at once before things level off varies, so some trial and error can be good | |||
[https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_use_bulk_requests] | |||
(also bulk inserts may wish to temporarily increase it) | |||
Where: refresh_interval in each index's _settings | |||
Other aspects to the '''indexing buffer''' are sometimes interesting[https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html]. | |||
Mostly, if the indexing buffer is too small, it'll still flush a lot. | |||
You shouldn't need more than a a few hundred MB{{verify}}, and since it defaults to 10% of Java's heap size, that's usually on the right order. | |||
--> | --> | ||
=== | ====indexing considerations==== | ||
<!-- | |||
'''Bulk alterations''' | |||
Any CRUD you can do in bulk will lower latencies of things done in serial. | |||
'''low/no refresh during bulk impoty''' | |||
Index adding in bulk also makes refresh more efficient and merge necessary less often | |||
You can set index.refresh_interval higher for just that import | |||
''' | Or even disable this, and do a manual refresh request at the end. | ||
This can make sense on some things that ''aren't'' continuously updated, though in most cases it doesn't save much. | |||
'''Bulk import''' | |||
Setting index.number_of_replicas to 0 during that import and setting it back later (so that the replication happens later - when?{{verify}}) | |||
Indexing is a moderately heavy operation. | |||
You can consider having one cluster do the indexing, | |||
then [https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html replicating] its results onto a distinct search-only cluster that is exposed to users. | |||
'''Parallel import''' | |||
Once you built a cluster, a single process feeding in documents is unlikely to max out the ingest capacity. | |||
However, if you do and ES gets behind, it will throw TOO_MANY_REQUESTS (429) so watch for that. | |||
'''Consider auto-IDs''' | |||
If continuously index a lot of documents, | |||
and use IDs that are ''only'' really there to guarantee uniqueness (and not for their actual value), | |||
then consider using auto-generated IDs rather than your own. | |||
Checking whether an ID is in the store is basically free with auto-ids (it's basically a counter), whereas with your own it's an index check that grows with index size. | |||
'''use keyword for identifiers, not text''' | |||
Identifiers may not be used like numbers, even when they are. | |||
That is, they're often there for known-item search. | |||
If you don't (or only exceptionally) need to do range searches on them, | |||
you can keep the indexing slimmer by making them a keyword instead. | |||
- | You can even consider a multi-field if you want faster exact search, and somewhat slower range search | ||
'''Avoid <tt>nested</tt> if you don't need it''' | |||
as it can make things a factor slower. | |||
'''Avoid <tt>join</tt> if you don't need it''' | |||
Parent-child relationships can make even more difference. | |||
Consider that a little duplication (as long as it doesn't become the bulk) may be well worth it in terms of search speed. | |||
'''If you want relational/join stuff, then considering the split/distribution details can help''' | |||
For example, if you can direct the joined documents to end up in the same shared, | |||
operations involving joins and such ([https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-request-inner-hits.html], ) should end up faster. | |||
You have some control over that in that documents are assigned to shards something like | |||
(hash(_routing) % num_routing_shards) / routing_factor | |||
where | |||
: _routing defaults to the document _id, but can be set | |||
'''If searches are typically narrow, then split indices the same way you query it''' | |||
E.g. if you parse logs, consider grouping into a daily index | |||
(see ILM, and for this case probably specifically Data Streams instead), | |||
which could really localize certain queries. | |||
If you have distinct users who typically search their own data, | |||
consider giving each their own index. | |||
'''Add fields to dumb down common searches''' | |||
Don't overdo it (particularly when potentially bloating the index), but... | |||
If | If you know searches always look in a specific year, consider a year (probably a keyword field), so that you don't have to do a range search on a date field. | ||
In general, rounded dates are likelier to get searches cached in the [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-cache.html query cache] | |||
Aggregations are basically query-specific | |||
If you do a lot of range aggregations - see e.g. [https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html#_pre_index_data] | |||
([https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html aggregations] are sort of indices) | |||
--> | |||
====Routing==== | |||
<!-- | |||
' | For a given document, routing determines which of an index's shards it should go in, | ||
(based mostly on the document ID and the cluster size). | |||
The default tends to spread documents quite evenly (particularly with auto IDs), | |||
and you don't need to touch this. | |||
This can be useful for a few reasons. | |||
One example is when there are subsets of sorts in your data. | |||
For example, you could group data per user without having to create one index per user. | |||
It is arguably more of a trick than structure, | |||
and it's up to ''you'' to be consistent about specifying the routing in all operations (index, search, more) | |||
because if you e.g. send search to look in only the wrong shard, it will just miss things. | |||
Note also that it may be less load-balanced in a large cluster when it e.g. turns out you have one user with a lot of data. | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html | |||
--> | |||
====Clusters==== | |||
<!-- | |||
https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html#scalability | |||
e.g. an ingest node intercepts index requests (bulk or not), applies transformations, | |||
and it then passes the documents back to the index or bulk APIs. | |||
by default, all nodes can ingest, so it's more whether you've configured any processing. | |||
In some cases you may want to dedicate one or a few nodes to ingest, for offload. | |||
Though note that doing the processing in code before handing it to be indexed is, essentially, the same, and this can be more about how centralized, vesionable, or accidentally-skippable that processing may be. | |||
--> | |||
===Security notes=== | |||
<!-- | |||
You have probably run into the fact that out of the box, ES may be configured to use SSL, | |||
with a self-signed certificate, meaning that all interaction that is not explicitly told | |||
about the public key that was generated at install time (or told to not veryify the license) will fail. | |||
xpack.security.http.ssl.enabled | |||
--> | |||
===ES errors and warnings=== | |||
<!-- | <!-- | ||
'''Failed to get search application count to include in Enterprise Search usage''' / '''Current license is non-compliant for search application and behavioral analytics. Current license is active basic license. Search Applications and behavioral analytics require an active trial, platinum or enterprise license.''' | |||
which, if you're watching your logs, is in a quite-spammy amount of lines (java stack traces). | |||
( | |||
https://github.com/elastic/elasticsearch/issues/96365 | |||
If you didn't mean to use monitoring and an upgrade seems to have enabled it, edit kibana.yml and disable xpack.monitoring.collection.enable | |||
Probably an update enabled xpath monitoring? | |||
Not that that should be | |||
'''License information could not be obtained from Elasticsearch due to ConnectionError''' | |||
'''Monitoring execution failed''' | |||
...seems to come from pretty catch-all code. | |||
Somewhere buried in that stack trace might be some 'Caused by' lines, one of which should be a little more helpful. | |||
Look for those instead. | |||
'''primary shard is not active''' | |||
There are multiple reasons it isn't active. | |||
* you manually stopped a node that contained a primary shard | |||
* a node's disk is almost full, and ES refuses to allocate new shards to it (default at 85%) {{comment|or even try to reallocate (default at 90%)}} | |||
: if you have a large disk, 15% might be 100+ Gbyte, and you might want to set these settings (cluster.routing.allocation.disk.watermark.low {{comment|, cluster.routing.allocation.disk.watermark.high}}) to absolute values instead [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-settings] | |||
: https://stackoverflow.com/questions/27547091/primary-shard-is-not-active-or-isnt-assigned-is-a-known-node | |||
PUT /_cluster/settings | |||
{ | |||
"persistent" : { | |||
"cluster.routing.allocation.disk.watermark.flood_stage":"98%", | |||
"cluster.routing.allocation.disk.watermark.high" : "97%", | |||
"cluster.routing.allocation.disk.watermark.low" : "95%" | |||
} | |||
} | |||
--> | --> | ||
Line 1,634: | Line 1,748: | ||
<!-- | <!-- | ||
Revision as of 00:42, 21 April 2024
Some practicalities to search systems
Lucene and things that wrap it: Lucene · Solr · ElasticSearch Search-related processing: tf-idf |
Choice side
Broad intro
ElasticSearch is largely seen as a document store that is searchable.
You could see it as a document store with CRUD HTTP API which happens to do indexing.
Because of its heritage (Lucene at its core) it is is largely seen as text search, but it is also reasonably suited for logging and certain metrics.
(You could use it like a database engine, though in that niche-purpose NoSQL-ey way where it doesn't do strong consistency where you may not really want it for a primary store).
Compared to some other text indexing, it
- wraps a few more features, e.g. replication and distribution
- does more to make more data types easily searchable, part of why it is reasonable for data logging and metrics (in fact the company behind it is currently leaning on the monitoring/metrics angle),
- eases management and is a little more automatic at that (e.g. index stuff that gives more consistent latency over time).
ES libraries are relatively thin wrappers around communicating with the HTTP API, and more about convenience.
There is no user interface included, perhaps in part because ES is fairly flexible. It's not very hard to interface your own form with the API, though.
Subscription model
From an "I don't care about jargon and legalese" perspective,
- you can get a managed cloudy thing - split into a few variants
- or you can self-host - also split into a few variants (not quite the same?)
(Before 6.3 the free license needed to be refreshed every year, now basic/free does not need an explicit license anymore?(verify))
Around self-hosting there are two or three variants: Basic, platinum, and enterprise (Gold is gone?)
- Basic is perfectly good for smaller scale
- the differences between platinum and enterprise mostly matter once you do this on large scale [1]
There is a perfectly free subset of ES (ES configured with a Basic license).
...but the default is bit a Basic license, it's a 30-day trial license in which you get more features, to entice you to buy the fuller product.
If you know you want to stick with basic features, then after this 30-day period (or earlier, if you know what you want) you would
- not only need to switch to Basic license,
- but also disable some features (apparently all X-pack things(verify))
...but if you didn't know that, this is annoying in the form of "automated install configured a trial license for me, but things just stopped working completely after 30 days and now I'm panicking", because it takes more than a little reading what you now need to disable and why. (My logs showed "Elasticsearch Search and Analytics" -- which seemed to just be a confusing name for the machine learning and alerting stuff)
License details
Implementation details you'd want to know to use it
The major moving parts
Some terminology
(note that the core of this comes from the underlying lucene)
mapping - basically a schema, telling ES of how to interpret each field
- e.g. mentions types
- if a field is not already defined in the mapping by you, it will adds fields as you mention them, with types guessed based on the first value it sees
field - a distinct thing in the mapping
- (usually) indexed to be searchable
- (usually) fetchable
- (there are some field type variants - see below)
document - individual search result.
- you can see it as the JSON you sent in, and/or as the part of that that got put into fields (and usually indexed) as specified by the mapping
- side note: most fields can have zero or more values in a document, though in a lot of practical cases it's practically just one, and in a few cases it's restricted to just one
- original JSON is also stored in _source
index
- if you squint, a grouping of documents - a collection you search as a whole
- you can have multiple indexes a per cluster (and may well want to)
- an index is divided into one or more shards, which is about replication in general and distribution in a cluster
- (and shards are divided into segment files, which is more about how updating documents works)
...and this is the point at which you can stop reading this section if you're just doing some experiments on one host. You will get to them once you if and when you need to scale to multiple hosts.
shard
- you typically split an index into a number of shards, e.g. to be able to do horizontal scaling onto nodes
- internally, each shard is a self-contained searchable thing. In the sense of the complete set of documents you fed in, this is just a portion of the overall thing we we call index here
- Shards come in two types: primary (the basic type), and replica.
segment - a shard consists of a number of segments (segments are individual files).
- Each segment file is immutable (which eases a lot of management, means no blocking, eases parallelism, eases cacheing).
replica is an exact copy of a shard
- the point of replicas is robustness against node failure:
- if you have two copies of every shard (and those copies are never on the same hardware), then one node can always drop out and search won't be missing anything
- without duplication, node failure would mean missing a part of every index distributed onto it
- you could run without replicas to save some memory
- memory is unlikely to be much of your cloudy bill (something like ES is CPU and transfer hungry), so using replicas is typically worth it for long term stability
node - distinct ES server instance
- in larger clusters, it makes sense to have some nodes take on specific roles/jobs - this discusses some of that
cluster - a group of nodes, serving a set of indices created in it
- nodes must mention a shared ID to join a cluster
Combined with
In practice, people often pair it with... (see also "ELK stack")
- Web UI, makes a bunch of inspection and monitoring easier, including a dashboard interface to ES
- also seen in tutorials, there largely for its console interactively poking ES without writing code
- itself pluggable, with bunch of optional things
And, if you're not coding your own ingest,
- you can do all of that yourself, but logstash can do a lot of work for you, or at least be a quick start
- Beats - where logstash is a more generic, configurable thing, beats is a set of specific-purpose ingest scripts, e.g. for availability, log files, network traffic, linux system metrics, windows event log, cloud service metrics, [2] - and a lot more contributed ones[3]
- e.g. metricbeat, which is stores ES metrics in ES
Indices
Fields, the mapping, field types
By default, ES picks up every part of the document (remember it's JSON), but you can configure it
- to have parts go towards multiple fields
- to have parts go to no field at all
- for all documents based on your mapping having a field but setting "enabled": false,
- from other behaviour, such as "values should never be bigger than X bytes" config on a field
Note that in terms of the index, a document can have multiple values for a field. So fields can be considered to store arrays of values, and most searches will match if any of those match.
Each regular field (there are others) gets its own behavior in terms of
- whether it is stored and/or
- whether it is indexed
- if indexed, how it is transformed before going into the index
(For completeness, a 'runtime field, is evaluated at runtime, and not indexed or in the _source. [4]. This can be handy for certain calculated data that you won't search on, or to experiment with fields you'll later make regular fields(verify))
The mapping[5][6] is the thing that lists all fields in an index, mostly:
- name
- data type(s)
- configured processing when new data comes in
Explicit mapping[7], amounts to specifying a mapping before you add documents. This can be preferred when
- you want specific interpretations of some fields (e.g. identifier that shouldn't be parsed as text, an IP address, a date), and/or
- you want specific features (e.g. flattened, join, search_as_you_type)
Dynamic mapping[8] is (like 'schemaless') a nice name for
- "I didn't set a mapping for a field, so am counting on ES guessing right based on the first value it sees"
- if you're mainly handling text, this usually does the right or at least a very sane thing.
dynamic-parameters allows a little more variation of that:
- true - unknown fields are automatically added to the mapping as regular fields (indexed) -- the default just described
- it'll mostly end up with boolean, float, long, or text/keyword
- runtime - unknown fields are automatically added to the mapping as runtime fields (not indexed)
- false - unknown fields are ignored
- strict - unknown fields mean the document is rejected
Note that
- with true and runtime, the created field's type is based on the communicated-JSON's type - see this table
- runtime and dynamic will
- You can also alter mappings later - see [11]
- but this comes with some restrictions/requirements/footnotes
- dynamic-parameters is normally set per index(verify) and inherited, but you can be more precise about it
Text-like field types
The documentation makes a point of a split into
- text family[12] - text, match_only_text
- keyword family[13] - keyword, constant_keyword, wildcard
- ...and it seems that e.g. search-as-you-type are considered miscellaneous
- text[14] (previously known as string analysed)
- flexible search of free-form text, analysis can will transform it before indexing (you probably want to know how - see e.g. analysis)
- no aggregations
- no sorting
- keyword[15] (previously known as string not_analysed)
- structured content (exact search only?), e.g. identifiers, a small/known set of tags, also things like emails, hostnames, zip code, etc.
- should be a little faster to match (if only because of smaller index size)(verify)
- allows sorting
- allows aggregation
- can make makes sense for serial numbers/IDs, tags - even if they look like numbers, you will probably only ever search them up as text equality (compared to storing those as a number, keyword may be a little larger yet also saves some index complexity necessary for numeric-range search)
- constant_keyword [16] - all documents in the index have the same value
- e.g. when you send each log file to its own index, this might assist some combining queries (verify)
- wildcard - [17]
- search_as_you_type[18]
- fast for prefix (match at start) and infix (terms within) matches
- mainly for autocompletion of queries, but could work for other shortish things (if not short, the index for this will be large)
- n-gram style, can have larger n (but costs in terms of index size, so only useful if you usually want many-token matches?)
- kept in memory, costly to build
Further notes:
- it seems that fields that are dynamically added to the mapping and detected as text will get two indexed fields: a free-form text one, and keyword with ignore_above of 256
- this is useful if you don't know what it will be used for
- but for e.g. identifiers it's pointless to also tokenize it
- and for free-form text it will probably do very little -- that is, for all but short documents it ends up _ignored. (It's a clever edge-caes trick to deal with cases where the only value is actually something other than text, and is otherwise almost free)
- separately, some things you may wish to not index/serch on, but still store it so you can report it as part of a hit
Data-like field types (primitives):
- boolean[19]
- numbers: [20]
- byte, short, integer, long, unsigned_long
- float, double, half_float, scaled_float
- allows range queries
- and, arguably, keyword (see above)
Data-like fields (specifics but text-like):
- date [21]
- (internally milliseconds since epoch (UTC)
- date_nanos [22]
- version - text that represents semantic versioning
- mostly for sortability?
- ip - IPv4 and IPv6 addresses
- allows things like subnet searches (CIDR style)
- geospatial - points and shapes [23]
- including distance and overlap
Different way of treating JSON as transferred
- object[24] -
- each subfield is separately mapped and indexed, with names based on the nesting dot notation, see e.g. the linked example
- flattened[25]
- takes the JSON object and indexes it as one single thing (basically an array of its values combined)
- nested[26] basically a variant of object that allows some field indexability
Other stuff
- binary[27]
- join[28]
- relations to other documents, by id(verify)
- seems to mean query-time lookups, so
- you could get some basic lookups for much cheaper than separate requests
- at the same time, you can make things slower by doing unnecessary and/or multiple levels of lookups (if you need relational, a relational database is better at that)
token_count[29]
- stored as an integer, but takes text, analyses it into tokens, then counts the number of tokens
- seems intended to be used via multi-field, to also get the length of text
dense_vector[30] (of floats by defaults, byte also possible)
- if indexes, you can use these for knn searches[31]
Managing indices (and thinking about shards and segments)
Remember that shards are made of segments, and segments are immutable.
ES collects changes in memory, and only occasionally dumps that that into a new segment file.
- (this is why documents are not immediately searchable and we call it near-realtime)
Refresh - is that act of writing to a new segment
- also helps updates not be blocking - existing segments are searched, new one once they're done
Merge refers to how smaller segments are periodically consolidated into fewer files, [32]
- also the only way that delete operations actually flush (remember, segments are immutable).
Refresh interval is 1s by default, but various setups may want that larger - it's a set of tradeoffs
- Refresh can be too quick in that the overhead of refresh would dominate, and less work is spent actually doing useful things (and be a little pricier in pay-per-CPU-use cloudiness)
- Refresh can also be too slow both in that
- new results take a long time to show up
- the heavier load, and large new segments, could make fore more irregular response times
The default is 1s. Large setups might increase that up to the order of 30s.
(more in the tradeoffs below)
index lifecycle management (ILM) lets you do slightly higher-level management like
- create a new index when one grows too large
- do things like creating an index per time interval
Both can help get the granularity you want when it comes to backing them up, duplicating them, retiring them to adhere to data retention standards.
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index-lifecycle-management.html
Also on the topic: a reindex basically means "read data, delete data in ES, ingest again".
- You would not generally want this over a merge.
- It's mainly useful when you make structural schema changes, and you want to ensure the data uniformly conforms to that.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
other field types
multi-fields[33] let you specify that one thing in your JSON should be picked up towards multiple fields.
- e.g. a city name
- as a keyword, for exact match, sorting and aggregation
- and towards the overall text search, via an analyser
- text as a searchable thing, and also store its length via token_count
runtime fields [34]
- mostly about query time - not part of the index (so won’t display in _source)
- just calculated as part of the response
- can be defined in the mapping, or in an individual query
- (seem to be a refinement of script fields?)
- easy to add and remove to a mapping (because there is no backing storage for them)
- https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html
- https://www.elastic.co/blog/getting-started-with-elasticsearch-runtime-fields
- you can search specific indices (comma separated names, or _all), or just one - sometimes useful
Metadata fields (that is, those other than e.g. the fields from your document, and runtime/script fields)
- _id - document's ID
- _type - document's mapping type
- deprecated
- _index - index that the document is in
- _source[35]
- is the original JSON that indexing got
- note that everything needs to be indexed
- this 'original document' is also used in update and reindex operations
- so while you can ask it to not store _source at all, that disables such operations
- that original submission can be handy for debug
- that original submission can be core functionality for your app in a "we found it by the index, now we give you the actual document" way
- you can filter what parts of _source are actually sent in search results - see source filtering
- _size - byte size of _source
- _ignored - fields in a document that indexing ignored for any reason - see
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/mapping-fields.html others
More on indexing
The bulk API could be used for all CRUD operations, but in practice is probably mostly used for indexing.
Do bulk adds where possible
While you can do individual adds, doing bunches together reduces overheads, and can make refreshes more efficient as well, largely relating to the refresh interval.
However, keep in mind that if some part of a bulk operations fails, you need to notice, and deal with that correctly.
Missing data
Setting a field to null (or an array of nulls, or an empty array) means it is not added to the index.
Sounds obvious, but do a mental check that "that field can not be matched in a lot of searches" (except for missing, must_not exists, and such(verify)) is the behaviour you intend.
Note that it also affects aggregation on the relevant fields, and some other details.
Ignoring data and fields
Field values may be ignored for varied reasons, including:
- use of ignore_malformed
- by default, using a type that can't be converted means the entire rejects the document add/update operation
- though note that in bulk updates it only considers that one operation failed, and will report a partial success
- if you set ignore_malformed[36], it will instead reject only bad values, and process the rest normally
- by default, using a type that can't be converted means the entire rejects the document add/update operation
- ignore_above (on keyword fields)
You can search for documents where this happened at all (you'd probably do that for debug reasons) with a query like
"query":{ "exists":{"field":"_ignored"} }
Text processing
You want curacao to match Curaçao? Want fishes to match fish?
To computers those are different characters and different strings, whether such variations are semantically equivalent, or close enough to be fuzzy with, and how, might vary per language.
Analysers represent relatively simple processing that helps, among other things, normalize data (and implicity the later query) for such fuzziness.
An analyzer is a combination of
- a character filter
- a tokenizer
- a token filter
(a normalizer seems to be an analyzer without a tokenizer - so practically mainly a character filter (can also have a token filter but this is probably not so useful)
Analyzers takes text, and applies a combination of character filters, tokenizer, and token filters, usually to the end of
- split it into tokens (e.g. words)
- stripping out things (e.g symbols and punctuation)
- normalize (e.g. lowercasing)
There are built-in analyzers to cover a lot of basic text search needs
- standard (the default if you don't specify one)
- lowercases
- splits using Unicode Text Segmentation (for english is mostly splitting on spaces and punctuation, but is better behaved default for some other languages) and lowercases
- removes most punctuation,
- can remove stop words (by default does not)
- language-specific analysers - currently arabic, armenian, basque, bengali, brazilian, bulgarian, catalan, cjk, czech, danish, dutch, english, estonian, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, and thai.
- details vary
- can remove stopwords
- can apply stemming (and take a list of exception cases to those stemming rules)
- no lowercasing
- splits on non-letters
- stop - simple plus stopword removal, so
- no lowercasing
- splits on non-letters
- can remove stopwords
- pattern - splits by regexp
- can lowercase
- can remove stopwords
- no lowercasing
- splits on whitespace
- keyword - does nothing, outputs what it was given
- reduces text in a way that helps detect duplicates, see something like [37]
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html
Data streams
Snapshots, Searchable snapshots
Install
Things you may want to think about somewhere before you have a big index
APIs
ES grew a lot of APIs over time.
Perhaps the most central are
- the document CRUD part (Index, Get, Update, Delete, respectively),
- and the searching.
Document CRUD
- Index - add a document (and implicitly get it indexed soon)
- e.g. PUT /test/_doc/1 with the JSON doc in the body
- Get - get a specific document by ID
- e.g. GET test/_doc/1 (HEAD if you only want to check that it exists)
- this basically just gives you _source, so you can filter what part of that gets sent[38]
- Update[39]
- e.g. POST /test/_update/1
- allows things like "script" : "ctx._source.votes += 1"
- Delete[40])
- e.g. DELETE /test/_doc/1
Note that there is also a bulk API [41]
- that lets you do many of the above at once
- POST _bulk
- where the body can contain many index, delete, update
- libraries often support the bulk API in a way that makes sense for how the rest of that library
"Is update really update?"
It seems that update is essentially
- construct a a new document based on the current version's _source
- add 1 to the version
- save that to a new segment
- ...even if it turns out there is no difference (...by default - you can specify that it should check and not do that - see detect_noop)
It seems that the old version will be considered deleted but still be on disk in that immutable segment it was originally entered in (until the next merge), and searches just happen to report only the latest (using _version)(verify) So if you do continuous updates on all of your documents, you can actually get the update process to fall behind, which can mean you temporarily use a lot more space (and search may be slightly slower too).
"Is there a difference between indexing a new version and updating the document?"
Very little in terms of what gets done at low level (both will create a new document, mark old as deleted)
It's most that for you there is a question of what you do is easier to express as
- a new version of the data
- the changes you want to see, or a script that makes them
Also, keep in mind how you are updating. If it's fetch, change, send, that's more steps than 'here is how to change it' in a script.
So if you do not keep track of changes to send to elasticsearch, you could update by throwing everything at at with detect_noop - which only applies to update.
Search[42]
- GET /indexname/_search
Multi Search[43]
- multiple searches with a single API call
- GET /indexname/_msearch
- ...though the request can switch index
Async search
Multi Get[44]
- Retrieves multiple JSON documents by ID.
- GET /_mget
Search
Shapes of a query
Most APIs take a query that looks something like:
{
"query": {
"match_phrase": {
"fieldname": "word word"
}
}
}
That is,
there is a specific Query DSL,
where you expressing a structured search as an abstract syntax tree, which is communicated as JSON.
Notes:
- the field names will vary with specific types of search
- In fact even the nesting will
- You might like Kibana's console to play with these, as it has some auto-completion of the search syntax.
- a quick "am I even indexing" test would probably use match_all[45]
Exceptions / other options
The only exception to "AST as JSON" seems to be URI search[46] which exposes basic searches in a URL (without needing to send a JSON style query in the body), but unless this covers all your needs, you will eventually want to abandon this. Might as well do it properly up front)}}
The only exceptions to "you must create query ASTs yourself" seem to be:
- query_string[47] -
- still uses that JSON, but feeds in a single query string in that specific lucene syntax, to be parsed internally
- exposes a bunch of the search features in a shorter-to-write syntax
- ...that you have to learn
- ...that is fairly breakable
- ...that allows frankenqueries that unduly load your system
- ...so you probably don't want to expose this to users unless all of them are power users / already know lucene
- exposes a bunch of the search features in a shorter-to-write syntax
- simple_query_string[48]
- same idea as the previous, but using the somewhat more simplified syntax
- less breaky, still for experts
...but again, until this covers all your needs and doesn't ope
These exceptions may looks impler but in the long run you will probably need to abandon them.
If you want to learn it once, learn the AST way.
(that sais, a hacky translation to the latter syntax is sometimes an easier shortcut for the short term).
API-wise
The search API[49] (/indexname/_search) should cover many basic needs.
- The search template API[50] does the same sort of search, but you can avoid doing some complex middleware/client-side query construction by asking ES to slot some values into an existing saved template.
Multi search[51] (/indexname/_msearch) is sort of like bulk-CRUD but for search: you send in multiple search commands in one request. Responds with the same number of result sets.
- Multi search template API[52] - seems to just be the combination of multi search and use of templates.
On result state, and getting more consistency between search interactions
In search systems, it can be a good tradeoff to
- get a good indication of how many there roughly, and stop counting when the number amounts to "well, um, many" -- rather than get a precise count
- only fetch-and-show the complete data for only the first few -- rather than everything
- forget the search immediately after serving it -- rather than keeping state around in RAM and/or on disk, for who knows how long exactly
...because it turns out we can do that faster.
This is ES stops counting after 10000 (max_result_window), only return the first 10 (size=10, from=0)
Also, if search-and-fetch-a-few turns out to be cheap in terms of IO and CPU, we can consider doing them without storing state.
This also because we know that in interactive browser use, most people will never check more than a few.
If in fact someone does the somewhat unusual thing of browing to page 2 or 3 or 4, you can redo the same search and fetch some more (use from to fetch what amounts to the next bunch).
However, if the index was refreshed between those actions, this new search will shift items around, so might get the same item again, or never show you are didn't see one one. Whenever the consistency or completeness really matters, you are probably looking for async or PIT:
Also, if you use this to back an API that allows "fetching everyting", you won't have won much.
Async search[53] - lets you start searches in the background. Search functionality is mostly equivalent, there's some minor parameter differences (think cache)
- the searches are store din their own index(verify) so you may want a mechanism to delete the searches by ID if you need to, and/or lower the time these are kept (default is 5 days, see keep_alive)
- note that you wait_for_completion_timeout lets you ask for "return regular search if you finish quickly, make it async if not"
Point in time API[54] - consider that document updates and refreshes means you will generally get the latest results.
If instead it is more important to get a consisten set, you can use PIT (or presumably async search?)
Searching multiple indices
...mostly points to Multi search (_msearch), which looks something like:
GET /index1/_msearch
{ }
{ "query": {"multi_match":{ "query":"fork", "fields":["plaintext"] } } }
{"index": "index2"}
{ "query": {"multi_match":{ "query":"fork", "fields":["title","plaintext"] } } }
{"index": "index3"}
{ "query": {"multi_match":{ "query":"fork", "fields":["plaintext"] } } }
Note that you can also do:
GET /index1,index2,index3/_search
{ "query": { "multi_match":{"query":"fork", "fields":["title","plaintext"] } } }
On the upside, you don't have to creatively merge the separate result sets (because multi search will not do that for you).
On the downside, you don't get to control that merge, e.g. scoring can be difficult to control, you don't really get to specify search fields per index anymore, or source filtering fields - though some forethought (e.g. to field naming between indices) can help some of those.
Other, mostly more specific search-like APIs, and some debug stuff
knn[55] search searches a dense_vector close to the query vector.
- separate API is depracated, being moved to an option
suggester[56] - suggests similar search terms based on edit distance
terms_enum[57] -
count[58]
explain[59]
profile[60]
validate[61]
shards[62] - report which shards would be accessed
composing queries
Compound queries
Compound queries[63] wrap others (leaf queries, or other compound queries), for
One reason is logical combinations of other queries (note that when you do this for filtering only, there are sometimes other ways to filter that might be more efficient in specific situations)
bool [64]
- lets you combine requirements with requirements like
- must
- filter is must without contributing to scoring
- must_not
- should ('portion of leafs must', and more matches scores higher)
- lets you combine requirements with requirements like
The rest is mostly about more controlled scoring (particularly when you make such combinations)
dismax [65]
- if multiple subqueries match the same document, the highest score gets used
boosting [66]
- have subqueries weigh in positively and negatively
constant_score [67]
- results from the search this wraps all get a fixed constant score
function_score [68]
- results form the search this wraps with scripting that can consider values from the document and query
Term-level and full-text queries
Term queries arenot put through an analyser, so stay one string, so are mostly used to match exact strings/terms.
Full-text queries are put through an analyzer (the same way as the text fields it searches), which does some alterations, and means the search then deals with multiple tokens.
term [69]
- exact value
- works in text fields but is only useful for fairly controlled things (e.g. username)
terms [70]
- like term, matches any from a provided list
terms_set [71]
- like terms, but with a minimum amount to match from a list
exists[72]
- Returns documents that contain any indexed value for a field.
fuzzy[73]
- Returns documents that contain terms similar to the search term. Elasticsearch measures similarity, or fuzziness, using a Levenshtein edit distance.
ids[74]
- Returns documents based on their document IDs.
prefix[75]
- Returns documents that contain a specific prefix in a provided field.
range [76]
- Returns documents that contain terms within a provided range.
regexp[77]
- Returns documents that contain terms matching a regular expression.
wildcard [78]
- Returns documents that contain terms matching a wildcard pattern.
match [79]
- matches text, number, date, or boolean data
- in a single field
- optionally does things like phrase or proximity queries, and things like edit-distance fuzziness
- text is analysed; all tokens are searched - effectively defaults to OR of all terms (and/so apparently minimum_should_match=1)
{ "query": { "match": { "fieldname":{ "query": "scary matrix snowman monster", "minimum_should_match": 2 //optional, but } } } }
multi_match [80]
- match, but searches one query in multiple fields
{ "query": { "multi_match" : { "query": "this is a test", "fields": [ "subject", "message" ] } } }
combined_fields [81]
- somewhat like multi_match, but instead of doing the query to each field separately, it acts as if it matched on a single field (that consists of the mentioned fields combined), useful when the match you want could span multiple fields
{ "query": { "combined_fields" : { "query": "database systems", "fields": [ "title", "subject", "message"], "operator": "and" } } }
match_phrase [82]
- analyses into tokens, then matches only if all are in a field, in the same sequence, and by default with a slop of 0 meaning they must be consecutive
{ "query": { "match_phrase": { "message": "this is a test" } } }
match_phrase_prefix [83]
- like match_phrase, but the last token is a prefix match
match_bool_prefix [84]
- constructs a bool of term-shoulds, but the last last is a prefix match
intervals [85]
- imposes rules based on order and proximity of matching terms
query_string [86]
- lucene-style query string which lets you specify fields, and AND,OR,NOT, in the query itself [87]
- maybe don't expose to users, because it's quick to error out
simple_query_string [88]