🐇 Tsubasa is a tiny, and simple Elasticsearch microservice to abstract searching objects!

  • By Noel
  • Last update: Jul 4, 2022
  • Comments: 14

🐇 翼 Tsubasa • Lint on Commit Docker Pulls GitHub Sponsors

Tiny, and simple Elasticsearch microservice to abstract searching objects!

Why did you build this?

Tsubasa was built to be a simple abstraction to not use the official SDKs to search objects within an Elasticsearch index, so this is just a simple way to retrieve data from it.

Installation

⚠️ Tsubasa is alpha software, it is NOT production ready! Beware~ >!<

Sweet, you want to use Tsubasa for your own use cases! You can install the Tsubasa server:

System Requirements

This is the minimum system requirements to bootstrap Tsubasa. You don't need to worry about this if you're running on the Helm Chart since that handles it for you.

  • 2GB or higher of system RAM
  • 2 CPU Cores or higher
  • Go 1.17 or higher
  • An instance of Elasticsearch running. This supports single-node and multi-node instances and multiple authentication methods.

Helm Chart

You can install Tsubasa on your Kubernetes cluster with a single command! You will need to index the Noelware Helm Charts under the "noel" user (which is me! :D)

⚠️ You are required to be using Kubernetes >=1.22 and Helm 3!

$ helm repo add noel https://charts.noelware.org/~/noel

You should be able to search the tsubasa repository when using the helm search command.

Now, you should be able to just run tsubasa on a single command:

$ helm install <my-release> noel/tsubasa

...and the server should be running now. The helm chart assumes you installed an Elasticsearch cluster installed.

Docker Image

You can use the official Docker images on ghcr.io or on Docker Hub!

~ ; ... coming soon >o< ... ; ~

Locally with Git

~ ; ... coming soon >o< ... ; ~

Configuraton

Tsubasa is configured using a TOML file which must be in the following locations:

  • /app/noel/tsubasa/config.toml if using the Docker Image
  • $ROOT/config.toml if running locally or wanting to contribute to Tsubasa
  • TSUBASA_CONFIG_FILE environment variable, which will override both clauses above.

You can find an example in the documentation!

Contributing

~ ; ... coming soon >o< ... ; ~

License

Tsubasa is released under the Apache 2.0 License by Noel!

Download

tsubasa.zip

Comments(14)

  • 1

    同学，您这个项目引入了81个开源组件，存在2个漏洞，辛苦升级一下

    检测到 auguwu/tsubasa 一共引入了81个开源组件，存在2个漏洞

    漏洞标题：David Kitchen bluemonday 安全漏洞
缺陷组件：github.com/microcosm-cc/[email protected]
漏洞编号：CVE-2021-42576
漏洞描述：David Kitchen bluemonday是  （David Kitchen）开源的一个应用程序。用于在Go中实现的HTML清理程序。
bluemonday sanitizer 存在安全漏洞，该漏洞源于Go中1.0.16之前的bluemonday和Python中 0.0.8之前的bluemonday(在pybluemonday中)，不能正确地强制与SELECT、STYLE和OPTION元素关联的策略。
影响范围：(∞, 1.0.16)
最小修复版本：1.0.16
缺陷组件引入路径：floofy.dev/tsubasa@->github.com/microcosm-cc/[email protected]

    另外还有2个漏洞，详细报告：https://mofeisec.com/jr?p=a966e0

  • 2

    fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.3.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/elastic/go-elasticsearch/v8 | require | minor | v8.2.0 -> v8.3.0 |

    Release Notes

    elastic/go-elasticsearch

    v8.3.0

    Compare Source

    API

    • ML.InferTrainedModelDeployment renamed to InferTrainedModel
    • ML.PreviewDatafeed has two new parameters, start and end. Documentation
    • ML.StartTrainedModelDeployment has three new parameters, number_of_allocations, threads_per_allocation and queue_capacity. Documentation
    • Cluster.DeleteVotingConfigExclusions has a new master_timeout parameter.
    • Cluster.PostVotingConfigExclusions has a new master_timeout parameter.
    • Snapshot.Get has a new index_names parameters (boolean). Whether to include the name of each index in the snapshot. Defaults to true.

    New APIs

    • Security.HasPrivilegesUserProfile (Experimental API) Documentation

    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by Mend Renovate. View repository job log here.

  • 3

    fix(deps): update module github.com/spf13/cobra to v1.5.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/spf13/cobra | require | minor | v1.4.0 -> v1.5.0 |

    Release Notes

    spf13/cobra

    v1.5.0

    Compare Source

    Spring 2022 Release 🌥️

    Hello everyone! Welcome to another release of cobra. Completions continue to get better and better. This release adds a few really cool new features. We also continue to patch versions of our dependencies as they become available via dependabot. Happy coding!

    Active help 👐🏼

    Shout out to @​marckhouzam for a big value add: Active Help https://github.com/spf13/cobra/pull/1482. With active help, a program can provide some inline warnings or hints for users as they hit tab. Now, your CLIs can be even more intuitive to use!

    Currently active help is only supported for bash V2 and zsh. Marc wrote a whole guide on how to do this, so make sure to give it a good read to learn how you can add this to your cobra code! https://github.com/spf13/cobra/blob/master/active_help.md

    Group flags 🧑🏼‍🤝‍🧑🏼

    Cobra now has the ability to mark flags as required or exclusive as a group. Shout out to our newest maintainer @​johnSchnake for this! https://github.com/spf13/cobra/pull/1654 Let's say you have a username flag that MUST be partnered with a password flag. Well, now, you can enforce those as being required together:

    rootCmd.Flags().StringVarP(&u, "username", "u", "", "Username (required if password is set)")
rootCmd.Flags().StringVarP(&pw, "password", "p", "", "Password (required if username is set)")
rootCmd.MarkFlagsRequiredTogether("username", "password")

    Flags may also be marked as "mutally exclusive" with the MarkFlagsMutuallyExclusive(string, string ... ) command API. Refer to our user guide documentation for further info!

    Completions 👀

    Documentation 📝

    Testing & CI ⚙️

    Beep boop, bot commits 🤖

    Misc 💭

    Shoutout to ALL our contributors (and all the new first time contributors!!) - great work everyone!! Cobra and it's huge impact wouldn't be possible without you 👏🏼 🚀 🐍

    Full Changelog: https://github.com/spf13/cobra/compare/v1.4.0...v1.5.0

    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by Mend Renovate. View repository job log here.

  • 4

    fix(deps): update module github.com/pelletier/go-toml/v2 to v2.0.2

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/pelletier/go-toml/v2 | require | patch | v2.0.1 -> v2.0.2 |

    Release Notes

    pelletier/go-toml

    v2.0.2

    Compare Source

    What's Changed

    Fixed bugs
    Other changes

    Full Changelog: https://github.com/pelletier/go-toml/compare/v2.0.1...v2.0.2

    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by Mend Renovate. View repository job log here.

  • 5

    chore(deps): update dependency alpine to v3.16

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | alpine | final | minor | 3.15 -> 3.16 |

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 6

    chore(deps): update goreleaser/goreleaser-action action to v3

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | goreleaser/goreleaser-action | action | major | v2 -> v3 |

    Release Notes

    goreleaser/goreleaser-action

    v3

    Compare Source

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 7

    fix(deps): update module github.com/pelletier/go-toml/v2 to v2.0.1

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/pelletier/go-toml/v2 | require | patch | v2.0.0 -> v2.0.1 |

    Release Notes

    pelletier/go-toml

    v2.0.1

    Compare Source

    What's Changed

    Fixed bugs
    Documentation

    New Contributors

    Full Changelog: https://github.com/pelletier/go-toml/compare/v2.0.0...v2.0.1

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 8

    chore(deps): update docker/setup-qemu-action action to v2

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | docker/setup-qemu-action | action | major | v1 -> v2 |

    Release Notes

    docker/setup-qemu-action

    v2

    Compare Source

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 9

    chore(deps): update docker/setup-buildx-action action to v2

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | docker/setup-buildx-action | action | major | v1 -> v2 |

    Release Notes

    docker/setup-buildx-action

    v2

    Compare Source

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 10

    chore(deps): update docker/login-action action to v2

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | docker/login-action | action | major | v1 -> v2 |

    Release Notes

    docker/login-action

    v2

    Compare Source

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 11

    fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.2.0

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/elastic/go-elasticsearch/v8 | require | minor | v8.1.0 -> v8.2.0 |

    Release Notes

    elastic/go-elasticsearch

    v8.2.0

    Compare Source

    Client

    • Fixed a serialisation error for retry_on_conflict in the BulkIndexer. Thanks to @​lpflpf for the help!
    • Fixed a concurrent map error in the BulkIndexer when custom headers are applied. Thanks to @​chzhuo for the contribution!

    API

    New APIs

    Configuration

    📅 Schedule: At any time (no schedule defined).

    🚦 Automerge: Enabled.

    ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

  • 12

    route: add GET/PUT /elastic/{index}/mappings

    This is an abstraction to create, update, and retrieve index mappings on a specific index.

    GET /elastic/{index}/mappings

    Returns the index mappings if any, or null if not found.

    200 OK

    {
   "shards": Int, # Returns how many shards
   "replicas": Int, # Returns how many replicas
   "data": { # The raw mapping from Elastic
       [... index mapping ...]
    }
}

    PUT /elastic/{index}/mappings

    Create or updates the index mapping of the kind.

    200 OK

    {
    "updated": bool # Returns a boolean if it was updated or created
}
  • 13

    feat: update/create index mappings on objects

    This will be tagged as a 0.4 release but I wanted to do some sort of query DSL:

    I proposed this idea internally (in my brain, I have no friends except my boyfriend 😢) that I wanted to create a query DSL to make searching objects more easily than what the current Elasticsearch query is.

    It will be called the Tsubasa Query Language (short-hand: TQL) that is similarly to GraphQL but it's stored on the server of all the data types available (under the filesystem called datatypes.tql or in a key-store database (Redis, etcd)).

    To define a data type, you can do:

    define(DataType, "<DATA_TYPE_NAME>", "<elastic data type>")

    This will create a new entry in etcd, Redis, or in the filesystem as datatypes.json as so:

    [
   { "data_type": "<DATA_TYPE_NAME>", "redirect_to": "<elastic data type>" }
]

    Tsubasa will cache the datatypes.json object (or under Redis as the hash: tsubasa:datatypes; etcd: tsubasa/datatypes) in the container and now you can create a query:

    search("index", FUZZY) { # Search on index "index" using the fuzzy query match type
    pretty() # Append ?pretty=true
    trackTotalHits() # Appends ?track_total_hits=true
}

    (this issue is unfinished)

  • 14

    Dependency Dashboard

    This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

    This repository currently has no open or pending branches.

    Detected dependencies

    dockerfile
    Dockerfile
    • golang 1.18-alpine
    • alpine 3.16
    release.Dockerfile
    • alpine 3.16
    github-actions
    .github/workflows/codeql.yml
    • actions/checkout v3
    • actions/setup-go v3
    • github/codeql-action v2
    • github/codeql-action v2
    .github/workflows/lint.yml
    • actions/checkout v3
    • actions/setup-go v3
    • actions/cache v3
    • golangci/golangci-lint-action v3
    .github/workflows/release-snapshot.yml
    • actions/checkout v3
    • actions/setup-go v3
    • docker/login-action v2
    • docker/login-action v2
    • docker/setup-qemu-action v2
    • docker/setup-buildx-action v2
    • goreleaser/goreleaser-action v3
    .github/workflows/release-stable.yml
    • actions/checkout v3
    • actions/setup-go v3
    • docker/login-action v2
    • docker/login-action v2
    • docker/setup-qemu-action v2
    • docker/setup-buildx-action v2
    • goreleaser/goreleaser-action v3
    • dawidd6/action-get-tag v1
    gomod
    go.mod
    • github.com/elastic/go-elasticsearch/v8 v8.3.0
    • github.com/getsentry/sentry-go v0.13.0
    • github.com/go-chi/chi/v5 v5.0.7
    • github.com/pelletier/go-toml/v2 v2.0.2
    • github.com/sirupsen/logrus v1.8.1
    • github.com/spf13/cobra v1.5.0
    • [ ] Check this box to trigger a request for Renovate to run again on this repository

Popular Tag

Related