Right now, sometimes sending unexpected payloads can cause 500 responses with panics:

import requests
payload = {
  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
  "data": {
    "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",
    "data": {
      "product_id": "ASO01043",
      "price": 49.95
    }
  }
}
requests.post('http://localhost:8080/com.snowplowanalytics.snowplow/tp2', json=payload)

2022/09/06 23:08:07 [Recovery] 2022/09/06 - 23:08:07 panic recovered:
POST /com.snowplowanalytics.snowplow/tp2 HTTP/1.1
Host: localhost:8080
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 208
Content-Type: application/json
User-Agent: python-requests/2.27.1


runtime error: invalid memory address or nil pointer dereference
/usr/local/Cellar/go/1.19/libexec/src/runtime/panic.go:260 (0x103b9e4)
        panicmem: panic(memoryError)
/usr/local/Cellar/go/1.19/libexec/src/runtime/signal_unix.go:835 (0x1053a1c)
        sigpanic: panicmem()
/Users/jpcassil/buz/pkg/snowplow/eventBuilder.go:169 (0x1bf16f9)
        setTstamps: e.DvceCreatedTstamp = *getTimeParam(params, "dtm")
/Users/jpcassil/buz/pkg/snowplow/eventBuilder.go:355 (0x1bf3ee7)
        BuildEventFromMappedParams: setTstamps(&event, params)
/Users/jpcassil/buz/pkg/envelope/builderSnowplow.go:106 (0x1bf70fe)
        BuildSnowplowEnvelopesFromRequest: spEvent := snowplow.BuildEventFromMappedParams(c, event.Value().(map[string]interface{}), *conf)
/Users/jpcassil/buz/pkg/handler/snowplow.go:21 (0x2d9da2e)
        SnowplowHandler.func1: envelopes := envelope.BuildSnowplowEnvelopesFromRequest(c, h.Config, h.CollectorMeta)
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
        (*Context).Next: c.handlers[c.index](c)
/Users/jpcassil/buz/pkg/middleware/identity.go:50 (0x2da8e5c)
        Identity.func1: c.Next()
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
        (*Context).Next: c.handlers[c.index](c)
/Users/jpcassil/buz/pkg/middleware/cors.go:28 (0x2da899a)
        CORS.func1: c.Next()
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
        (*Context).Next: c.handlers[c.index](c)
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x18220d7)
        CustomRecoveryWithWriter.func1: c.Next()
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
        (*Context).Next: c.handlers[c.index](c)
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:555 (0x181fe93)
        (*Engine).handleHTTPRequest: c.Next()
/Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:511 (0x181f964)
        (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/local/Cellar/go/1.19/libexec/src/net/http/server.go:2947 (0x14ad873)
        serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/Cellar/go/1.19/libexec/src/net/http/server.go:1991 (0x14a73db)
        (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/Cellar/go/1.19/libexec/src/runtime/asm_amd64.s:1594 (0x10705a0)
        goexit: BYTE    $0x90   // NOP

We should consider building in some nice mechanisms for graceful failures and helpful responses, where applicable.

I'm not sure what about this payload in particular caused a panic.

Perhaps I misunderstood this section: https://github.com/silverton-io/honeypot/blob/main/website/docs/examples/quickstart.md#4-send-events-to-honeypot

but it indicates that the page it serves up at 8080 has snowplow integrated for easy testing. I don't think that's the case... I just get 404 page not found and I also get 404s when trying to POST or GET anything from localhost or where I have this served up in GCP right now. Not sure if I'm doing something wrong...

Right now sinks do not retry if they fail. Having that level of guarantee is important.

Sinks should be independently configurable to retry yes/no, and it would be cool to configure the retry strategy. Exponential decay with min/max? Something else?

Currently there's one way of providing secrets to honeypot - a file titled config.yml in the same directory as the binary. This is obviously non-ideal.

Options (probably all of them are ideal)

1. Add env var specifying config file location override
2. Add env-var-based config overrides: LEVEL1_LEVEL2_VARNAME
3. Add default config.yml to the docker build

This PR:

1. Shuffles the terraform directory under deploy
2. Adds system and env vars to multiple environments can be rapidly provisioned using the same code
3. Shuffles variable building from main to locals
4. Autogenerate cloud run service revision names
5. Adds outputs
6. Make bucket naming be user-flexible (admittedly this might be a step backward, since the using project id's in bucket names to keep them globally unique is 🔥 )

Issue 391

Terraform for the GCP deploy as outlined by the GCP documentation steps. First ever Terraform for GCP so may not be following all the best standards. Main hurdle is that it has to be applied in two steps because of the manual process of uploading the image. Otherwise seems to work.

It would be cool if this thing just split events into tables without being configured to do so. But could be configured in reverse to say "no don't do that".

buz endpoints like /stats, /schemas etc right now are kind of 'need to know', but ideally we have an auto generated documentation for them with descriptions of what they do and what we can expect back when we hit them.